Pre/post scripts not executing

Support
#1

Duplicacy in docker environment.

I have written each a pre-check and post-prune script that execute well on their own. They are in the correct temporary folder within the appdata structure and duplicacy sees them, recognises them and according to the log executes them, however nothing happens. The logs do not show a completion and the backup is stuck for ever.

Running check command from /cache/localhost/all Options: [-log check -storage Backup_Tower -threads 4 -a -tabular]
2022-03-07 11:15:13.625 INFO STORAGE_SET Storage set to sftp://XXXX@XXX.XXX.XXX.XXX//mnt/user/Backups_Tower/
2022-03-07 11:15:13.625 INFO SCRIPT_RUN Running script /cache/localhost/all/.duplicacy/scripts/pre-check

Permissions of the scripts are 777. I have seen this mentioned before on the forum, but no conclusion was reached (Pre Command and Post Command Scripts - #13 by sss).

is there a particular language needed to make the bash file executable?

Any help will be greatly appreciated.

#2
  • Is the script executable? (chmod +x post-prune)
  • does the script specify the interpreter? (E.g. #!/usr/bin/zsh in the header)
#3

Is the script executable? (chmod +x post-prune) <

I use chmod -R 777 $LOCATION to make the files executable

does the script specify the interpreter? (E.g. #!/usr/bin/zsh in the header)<

Files start with #!/bin/bash

#4

Is there perhaps a bare bones bash script available which is verified to work? That way I would be able to check whether it is an issue with my duplicacy install or my scripts.

I also had another thought/question: as part of running the script, does duplicacy run it in the same environment/way that the OS does? Or does it perhaps run the scripts in a sandbox-ish environment where it perhaps does not have access to some of the OS resources? (I would be very surprised at this, but it does not hurt to ask).

#5

What does your script do? Can you make sure that it is not getting stuck? From the log it looked like Duplicacy was correctly running the script and waiting it to finish.

#6

It uses os/exec:

To confirm that it runs — add touch /tmp/hello to your script and check if that file gets created. If it does — issue is with the content of your script.

#7

The script turns on the backup server and turns it off at the end (two separate scripts). Both scripts are tested and have been used many times with the previous backup solution.

I guess you mean whether it is getting stuck? I can tell it isn’t running/getting stuck as the backup server does not turn on, and when I check the OS events log I can see that the wake command was not sent. When I run the script on its own from within the OS, it works fine and I get the notifications that I have set.

That is what I thought too, however even when I leave it ‘running’ nothing happens. No error occurs, the duplicacy interface simply shows the progress bar.

#8

I will try that now, and report on the result

#9

Interesting, two observations:

  1. when adding to the original script, nothing happens, nothing changes.

  2. I make an empty new script
    #!/bin/bash
    touch /tmp/hello
    exit
    to see if that would just make the file in /tmp/ and it does not, however the script does complete and the backup attempts to run.

I also tried mkdir /tmp/hello, no result. Tried it in command line and it creates the folder no problem.

#10

@saspus I just realised that I am using your docker version of duplicacy on unraid. Could it be that local permissions are potentially not set properly?

The docker has USR_ID=0 and GRP_ID=0 set.

#11

0 is root, so it will have access to everything that container has access to.

Are you looking in the /tmp/ in the container or on the host? That /tmp/ refers to folder inside the container. If you want it to see folder on the host you would need to map it into the container. I’m not even sure if Alpine has the /tmp/ folder there to begin with.

Return value from the script is used to decide whether the operation may proceed. I would explicitly write exit 0 there.

#12

I was checking the OS /tmp/ as well as .duplicacy/.

thanks for the tip

#13

/tmp/ won’t be there. You need to open shell in the container and check there. see docker exec -it <container name or id> /bin/sh.

Do that, and run your script there standalone – make sure it does what you expect it to do. Then have duplicacy execute it. Generally, if duplicacy logged that it was about tor unit – it was going to run it, there is nothing else but call to exec.

#14

I thought of the same thing :slight_smile:

unfortunately when I try running simple commands in the container bash window, it can not execute some of the functions that the OS can. It seems that access to some of the needed functions is missing:

On duplicacy bash command window:
bash-5.1# ssh
bash: ssh: command not found

On OS bash command window:
root@Tower:~# ssh
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b bind_address] [-c cipher_spec] [-D [bind_address:]port] [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file] [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address] [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]] destination [command]

In summary, to run my two scripts there are two functions that are provided by the OS that the duplicacy bash windows does not have access to: etherwake and ssh.

The way the frist script is set up is, to (1) check the IP of backup server, if online, continue, if offline, then (2) etherwake Backup Server and (3) wait in 30s increments for response from server.
It is at step 3 where the script never completed because it is waiting forever.

bash-5.1# bash pre-check
pre-check: line 55: etherwake: command not found
Attempting to wake server now
Backup server is offline....
.............................Checking backup server attempt...1
Backup server not started yet
Waiting 30 seconds to check again

I will add a max count clause. That will avoid the infinite loop.

So now that I am very close to the error at hand, how am I able to make sure that functions are passed through from the OS(unraid) to the container?

#15

Of course! That’s the whole point of containers :slight_smile: – to create isolated minimum environment to run the specific software. In that container I’m using Alpine linux with bare minimum functionality (see FROM alpine:latest in the Dockerfile). Note however that since duplicacy does not have any dependencies (it’s a monolithic self-contained executable) it does not benefit from isolation, therefore containerizing it is of questionable utility. Maybe docker hub provides a handy distribution channel (the only reason I made that container initially was to get duplicacy on synology docker without messing with copying files), but that’s about it.

You have two options:

  1. Run duplicacy-web on unraid directly. I’ve done it on Synology: Duplicacy Web on Synology Diskstation without Docker | Trinkets, Odds, and Ends. With some tweaks (mainly around sheduler – synology 6.xx used upstart, I don’t know what Unraid is using) it can be made run on Unraid as well. Literally the only thing that needs to be done is to ensure it starts on boot at a right time. Kind reader left the configuration for systemd in the comment section.
  2. Install tools you need into container. They will be wiped out on the container update, but if you use “:mini” tag you don’t ever need to update it (unless I fix a bug there; but given how little code there is it’s unlikely that any issues will pop up). Or just add installation of those tools in your pre- script :slight_smile: – then you don’t need to worry about container updates.

For example, if you need SSH – open shell in container and do apk --update add --no-cache openssh. Example is in docker file here. Bitbucket.

And list of available packages: Alpine Linux packages

#16

Oh I see! I guess I actually never thought about this. Up until now I considered the whole docker environment more of a software distribution system.

Thank you for the background. That makes total sense now.

I will try the web version.

One additional way that would solve my problem, would be the ability to execute a file on the host from the docker. I.e. my script just tells the host OS to execute the file. Would something like this be possible?

#17

I guess, nothing is impossible…

Think of docker as chroot+networking with extra steps. It’s one step removed from a VM, where kernel is virtualized too, not just userspace. In all those cases the goal is for the contained application to be unable to reach or even be aware of a host.

Hence, you have options:

  1. You can ssh from a container to the host and run any software you want. Setup passwordless auth, and your pre- file will simply contain ssh hostname.local '/path/to/stuff/to/run/on/a/host.sh'. But this is a bit roundabout.
  2. Another popular approach is to have host share a socket file with the guest, and can communicate with the app running on a host. This is how containers like Watchtower work – via the socket file passed to it it can control host docker instance.

But in case of duplicacy – just run it on unraid as-is. You only get drawbacks, extra work and no benefits in trying to run it in a container.

Why do you want to run stuff on the host though? As far as I understand you need to send WoL to a server and wait for it to be up and running, and then perhaps SSH to it to shutdown. Can you not do it inside a container?

#18

Forgive my confusion on this topic, but didn’t we just establish that the docker does not support the dependencies that I want to use (ssh & etherwake) to run the scripts inside the container?

#19

No, why? The container, being based on a minimalistic linux, does not have those utilities installed, but absolutely nothing prevents you from installing them, right from your pre- and post- scripts if they are missing. The openssh can be installed directly with apk, and etherwake (or wakeonlan) by simply copying the executable to the container filesystem (or simply dropping them to /config/bin folder on the host, for persistence)

#20

oh wow! this has been an extremely educational day for me! Thank you for your guidance @saspus.

Final solution for me:
add an extra folder add-ons in the duplicacy appdata folder which holds a copy of the etherwake executable file.
At start of script run apk add openssh and cp add-ons/etherwake sbin. Now dependencies are available to run the script. Works like a charm.