Run web-ui in a docker container?

[gkoerk]

I am sure I have the files in the right place, but I still get this error:

/root/.duplicacy-web/bin/duplicacy_web
Created a new configuration.
Failed to locate the CLI executable

This is despite the fact that upon container startup, I can do the following inside the container:

~ # cd /root/.duplicacy-web/bin
~/.duplicacy-web/bin # ls -lart
total 47624
-rwxrwxr--    1 root     root      26327201 Nov 15 13:54 duplicacy
-rwxrwxr--    1 root     root      22428329 Nov 15 13:54 duplicacy_web
drwxrwxrwx    1 root     root          4096 Nov 15 13:54 .
drwxr-xr-x    1 root     root          4096 Nov 15 13:56 ..
~/.duplicacy-web/bin #

Is it looking for a specific filename or format? If it looks in ~/.duplicacy-web/bin, does it not like my filename?Here is my dockerfile now (it’s gotten really ugly as I hack it together with Duct Tape

FROM alpine:latest
RUN mkdir -p /app
RUN mkdir -p /root/.duplicacy-web/bin
RUN chmod 777 /root/.duplicacy-web/bin
COPY duplicacy /app/duplicacy
COPY duplicacy_web /app/duplicacy_web
RUN cp /app/duplicacy /root/.duplicacy-web/bin/duplicacy
RUN cp /app/duplicacy_web /root/.duplicacy-web/bin/duplicacy_web
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy_web
EXPOSE 8080
#ENTRYPOINT ["/root/.duplicacy-web/bin/duplicacy_web"]

The last line is commented out so the completed image may be run interactively to get a prompt. Trying to run the coomand in ENTRYPOINT manually starts to create a config, but is still giving an error that it cannot find the CLI. Any ideas?

[gchen]

The CLI has to be named duplicacy_linux_x64_2.1.2. The version number may change later when there are newer releases on github, but you can ‘freeze’ the version by adding the line to duplicacy.json:

    "cli_version": "2.1.2"

[gkoerk]

Renaming the file allowed me to clean things up. This is now working as a draft we can refine if it’s of any value to you. One incredibly important thing that’s missing is: where is duplicacy_web storing its data? I want to put it in a docker named volume or (more likely) - bind that volume to a host volume. Right now, no settings are persistent beyond restarts of the container.

FROM alpine:latest
COPY duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2
COPY duplicacy_web /root/.duplicacy-web/bin/duplicacy_web
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_web
EXPOSE 8080
ENTRYPOINT ["/root/.duplicacy-web/bin/duplicacy_web"]

And my duplicacy.yml (docker-compose) stack:

version: '3'
services:

  duplicacy:
    image: gkoerk/duplicacy:latest
    volumes:
      # Expose backup folders to the container:
      - /share/appdata:/backup/appdata
      - /share/Photos:/backup/photos
      - /share/downloads:/backup/downloads
      - /share/media/movies:/backup/movies
      # Mount external storage as one backup destination.
      - /share/USBDisk1:/destination/external
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.port=8080
        - traefik.network=traefik_public
        - 'traefik.frontend.rule=Host:backup.domain.com'          

networks:
  traefik_public:
    external: true

[gkoerk]

Where does duplicacy_web store persisted data?

[gchen]

All files are under ~/.duplicacy-web. The most important one is duplicacy.json which stores almost all config information.

The folder filters has all include/exclude patterns so this folder should be backed up too.

The folders stats and logs are less important; if you don’t want to keep the history you don’t need to back up these two.

And definitely no need to back up the repositories folder, as all files there are temporary (most are cache files).

[gkoerk]

Okay - here is a better docker-compose.yml. Note that it offers persistence, but you will need to copy out the duplicacy.json file. Also - I can’t get Traefik to reverse-proxy the web app. Any idea what might be the issue? Right now I just get a “waiting for backup..com” and eventually a gateway timeout.

Dockerfile:

FROM alpine:latest
COPY duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2
COPY duplicacy_web /root/.duplicacy-web/bin/duplicacy_web
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_web
EXPOSE 8080
ENTRYPOINT ["/root/.duplicacy-web/bin/duplicacy_web"]

To build your image you run the following from within the same folder as the Dockerfile, and reference it as the image name in your docker-compose.yml (yes, there are several other ways to do this, please feel free to improve upon this however you see fit):

docker build -t <imagename>:<tag> .

docker-compose.yml (works via docker-compose up -d or in Swarm mode via docker stack deploy:

version: '3'
services:

  duplicacy:
    image: <imagename>:<tag>
    ports:
      - 8090:8080
    volumes:
      - /share/appdata/duplicacy/duplicacy.json:/root/.duplicacy-web/duplicacy.json
      - /share/appdata/duplicacy/stats:/root/.duplicacy-web/stats
      - /share/appdata/duplicacy/log:/root/.duplicacy-web/logs
      - /share/appdata:/backup/appdata
      - /share/Photos:/backup/photos
      - /share/downloads:/backup/downloads
      - /share/media/movies:/backup/movies
      - /share/media/music:/backup/music
      - /share/media/tv:/backup/tv
      - /share/USBDisk1:/destination/external
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.port=8080
        - traefik.network=traefik_public
        - 'traefik.frontend.rule=Host:backup.gkoerk.com'          

networks:
  traefik_public:
    external: true

And the command to setup the traefik_public network:

• docker-compose:

docker network create --driver=bridge --subnet=172.1.1.0/21 traefik_public

• docker stack deploy:

docker network create --driver=overlay --subnet=172.1.1.0/21 --attachable traefik_public

`

[k2eric]

I’ve also been toying around with running the web UI in a Docker container and though I’d share my working Dockerfile:

FROM alpine:latest
    
RUN ARCHITECTURE=linux_x64                                                                             && \
    SHA256_DUPLICACY=034720abb90702cffc4f59ff8c29cda61f14d9065e6ca0e4017ba144372f95d7                  && \
    SHA256_DUPLICACY_WEB=b0a9fc35f249ee8fa5d7da0fe0f8487041661ed19b24c115fd01aee37049ccfe              
    VERSION_DUPLICACY=2.1.2                                                                            && \
    VERSION_DUPLICACY_WEB=0.2.8                                                                        && \
                                                                                                          \
    # add Bash for our entrypoint.sh, and ca-certificates so Duplicacy doesn't complain about certs
    apk update                                                                                         && \
    apk add --no-cache bash ca-certificates                                                            && \
                                                                                                          \
    # download, check, and install duplicacy
    wget --quiet -O /usr/local/bin/duplicacy                                                              \ 
 https://github.com/gilbertchen/duplicacy/releases/download/v${VERSION_DUPLICACY}/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
    echo "${SHA256_DUPLICACY}  /usr/local/bin/duplicacy" | sha256sum -s -c -                           && \
    chmod +x /usr/local/bin/duplicacy                                                                  && \
                                                                                                          \
    # downlooad, check, and install the web UI
    wget --quiet -O /usr/local/bin/duplicacy_web                                                          \
        https://acrosync.com/duplicacy-web/duplicacy_web_${ARCHITECTURE}_${VERSION_DUPLICACY_WEB}      && \
    echo "${SHA256_DUPLICACY_WEB}  /usr/local/bin/duplicacy_web" | sha256sum -s -c -                   && \
    chmod +x /usr/local/bin/duplicacy_web                                                              && \
                                                                                                          \
    # duplicacy_web expects to find the CLI binary in a certain location
    # https://forum.duplicacy.com/t/run-web-ui-in-a-docker-container/1505/2
    mkdir -p ~/.duplicacy-web/bin                                                                      && \
    ln -s /usr/local/bin/duplicacy ~/.duplicacy-web/bin/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
                                                                                                              \
    # create the logs directory and touch a log so that we can tail it before we start the server
    mkdir ~/.duplicacy-web/logs                                                                        && \
    touch ~/.duplicacy-web/logs/duplicacy_web.log                                                      && \
                                                                                                          \
    # listen on all interfaces
    echo '{"listening_address":"0.0.0.0:3875"}' > ~/.duplicacy-web/settings.json

COPY ./entrypoint.sh /usr/local/bin/entrypoint.sh

ENTRYPOINT [ "/usr/local/bin/entrypoint.sh" ]

entrypoint.sh looks something like this:

#!/usr/bin/env bash
tail -f ~/.duplicacy-web/logs/duplicacy_web.log &
duplicacy_web &

It also includes a Bash trap to catch Ctrl-C to shut down the server, but I’ve left that out for clarity.

I can’t get Traefik to reverse-proxy the web app. Any idea what might be the issue?

The trick for me was to get the web UI to listen on 0.0.0.0 by writing an initial duplicacy.json, otherwise the server listens only on localhost and is therefore inaccessible outside of the container.

[gkoerk]

The trick for me was to get the web UI to listen on 0.0.0.0 by writing an initial duplicacy.json

Could you share the additions/edits or an example of your duplicacy.json (redacted of course)? I don’t see how to add the listening address.

[k2eric]

Could you share the additions/edits or an example of your duplicacy.json (redacted of course)? I don’t see how to add the listening address.

From the Dockerfile I posted above, near the bottom:

echo '{"listening_address":"0.0.0.0:3875"}' > ~/.duplicacy-web/settings.json

Note that this line sets the contents of settings.json, not duplicacy.json. I made no other edits.

Here’s an updated Dockerfile that you can build and test:

FROM alpine:latest

RUN ARCHITECTURE=linux_x64                                                                             && \
    SHA256_DUPLICACY=034720abb90702cffc4f59ff8c29cda61f14d9065e6ca0e4017ba144372f95d7                  && \
    SHA256_DUPLICACY_WEB=322e8865fa5f480952938be018725bf02bd0023a26512eb67216a9f0cb721726              && \
    VERSION_DUPLICACY=2.1.2                                                                            && \
    VERSION_DUPLICACY_WEB=0.2.10                                                                       && \
                                                                                                          \
    # add Bash for our entrypoint.sh, and ca-certificates so Duplicacy doesn't complain about certs
    apk update                                                                                         && \
    apk add --no-cache bash ca-certificates                                                            && \
                                                                                                          \
    # download, check, and install duplicacy
    wget --quiet -O /usr/local/bin/duplicacy                                                              \
        https://github.com/gilbertchen/duplicacy/releases/download/v${VERSION_DUPLICACY}/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
    echo "${SHA256_DUPLICACY}  /usr/local/bin/duplicacy" | sha256sum -s -c -                           && \
    chmod +x /usr/local/bin/duplicacy                                                                  && \
                                                                                                          \
    # downlooad, check, and install the web UI
    wget --quiet -O /usr/local/bin/duplicacy_web                                                          \
        https://acrosync.com/duplicacy-web/duplicacy_web_${ARCHITECTURE}_${VERSION_DUPLICACY_WEB}      && \
    echo "${SHA256_DUPLICACY_WEB}  /usr/local/bin/duplicacy_web" | sha256sum -s -c -                   && \
    chmod +x /usr/local/bin/duplicacy_web                                                              && \
                                                                                                          \
    # duplicacy_web expects to find the CLI binary in a certain location
    # https://forum.duplicacy.com/t/run-web-ui-in-a-docker-container/1505/2
    mkdir -p ~/.duplicacy-web/bin                                                                      && \
    ln -s /usr/local/bin/duplicacy ~/.duplicacy-web/bin/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
                                                                                                          \
    # create the logs directory and touch a log so that we can tail it before we start the server
    mkdir ~/.duplicacy-web/logs                                                                        && \
    touch ~/.duplicacy-web/logs/duplicacy_web.log                                                      && \
                                                                                                          \
    # listen on all interfaces
    echo '{"listening_address":"0.0.0.0:3875"}' > ~/.duplicacy-web/settings.json

ENTRYPOINT [ "/usr/local/bin/duplicacy_web" ]

Run it with docker run --rm -p 9991:3875 046107c386da, the visit http://localhost:9991 and you should see the login screen.

Does that help?

[saspus]

Here’s an updated Dockerfile that you can build and test:

I’ve tweaked (and simplified) your docker file and added explicit declarations to move cache, logs, and config location into sane places (they would reside on different volumes with different backup and retention policies and to declare such possibilities, including required ports, to be user-friendly.

I really don’t like the way Duplicacy stuffs transient, logs, and config data into the same folder.

Dockerfile:

FROM alpine:latest

ENV VERSION=0.2.10

RUN  apk --update add --no-cache bash ca-certificates && \
    wget -nv -O /usr/local/bin/duplicacy_web                      \
        https://acrosync.com/duplicacy-web/duplicacy_web_linux_x64_${VERSION} && \
    chmod +x /usr/local/bin/duplicacy_web  && \
    rm -rf /tmp/* && rm -rf /var/cache/apk/*                                                             

COPY ./entrypoint.sh /usr/local/bin/entrypoint.sh                          
                            
VOLUME /config
VOLUME /logs
VOLUME /cache

EXPOSE 3875/tcp

ENTRYPOINT [ "/usr/local/bin/entrypoint.sh" ]

Entrypoint:

#!/usr/bin/env bash  

# trap ^C
trap 'kill ${!}; exit' SIGHUP SIGINT SIGQUIT SIGTERM
         
# create duplicacy folders
                                     
mkdir -p    ~/.duplicacy-web/logs \
            ~/.duplicacy-web/filters \
            ~/.duplicacy-web/repositories 
            
            
echo Creating sane exports and files
mkdir -p    /config/filters \
            /logs \
            /cache 

touch       /logs/duplicacy_web.log
            
            
echo '{"listening_address":"0.0.0.0:3875"}' > /config/settings.json
echo '{}'                                   > /config/duplicacy.json

echo Link data to where duplicacy expects it

ln -s /config/settings.json     ~/.duplicacy-web/settings.json                  
ln -s /config/duplicacy.json    ~/.duplicacy-web/duplicacy.json
ln -s /config/filters           ~/.duplicacy-web/filters
ln -s /logs                     ~/.duplicacy-web/logs
ln -s /cache                    ~/.duplicacy-web/repositories

echo Logging tail of the log from this moment on
tail -0 -f /logs/duplicacy_web.log & 

echo Starting duplicacy
duplicacy_web &

# wait for events.
wait

To run:

docker build --tag=saspus/duplicacy-web .
docker run  -p 3875:3875/tcp \
        -v ~/Library/Duplicacy:/config  \
        -v ~/Library/Logs/Duplicacy/:/logs \
        -v ~/Library/Caches/Duplicacy:/cache \
        -v ~:/MyHomeToBackup:ro \
        saspus/duplicacy-web

[k2eric]

Great ideas! I’ve incorporated some of your tricks into my image.

For fun, I’ve published a new GitHub repo and would be happy to accept feedback or contributions:

https://github.com/ehough/docker-duplicacy

To use it:

1. docker run -p 3875:3875 erichough/duplicacy
2. Visit http://localhost:3875

The GitHub readme has a few other tips on how to use the image, but the above instructions should work well for testing.

[saspus]

Any reason why are you manually downloading the CLI engine? Duplicacy web will download the correct one on the first run itself.

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

# redirect the log to stdout
ln -s /dev/stdout /var/log/duplicacy_web.log                                              && \

I’m stealing it! How I did not think of this.

[k2eric]

Thanks for the feedback!

any reason why are you manually downloading the CLI engine? Duplicacy web will download the correct one on the first run itself.

I didn’t know duplicacy_web would do that - clever!

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Have you pulled the latest? I think I fixed that exact bug last night in this commit but will do some more testing. Thanks for the heads up.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

I considered just using wget, but paranoia dies hard The recent hack of PEAR (a fairly critical component of PHP’s ecosystem) was a good reminder for me that the bad guys will eventually find their way into anything.

Let us know if you end up sharing or publishing your image, and thanks again for the feedback.

[saspus]

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Have you pulled the latest? I think I fixed that exact bug last night in this commit but will do some more testing. Thanks for the heads up.

Ah, yes, indeed, I looked at it yesterday, but posted comment today.
I also did not realize you can create symlinks to non-existent files. That is pretty useful.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

I considered just using wget , but paranoia dies hard The recent hack of PEAR (a fairly critical component of PHP’s ecosystem) was a good reminder for me that the bad guys will eventually find their way into anything.

I guess it’s a good point. And it’s probably not too much hassle to keep updating the Dockerfile when new version is released.

Let us know if you end up sharing or publishing your image, and thanks again for the feedback.

Thank you too for the ideas - it’s my first docker container, have never done it before.
I posted link to it in the other message, copying here: Docker Hub

That said I really don’t like the idea of linking to individual config files; what if next version introduces another new file?

I thing duplicacy_web shall support getting path to configuration folder from environment. If it is not set, then by all means let is use ~/.duplicacy-web. @gchen, do you think this would be useful/feasible?

Another question for @gchen, is there an exhaustive list of stuff that can be configured in settings.json?

[gkoerk]

I got the following errors in the docker logs when running your docker image. It looks like it is running, but it isn’t accessible via browser. Any ideas what I’m doing wrong?

Log directory set to /var/log,
2019/01/27 15:03:33 Failed to marshal the configuration: json: error calling MarshalJSON for type main.StorageCredentials: No master password provided,
2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,
2019/01/27 15:03:33 A new license has been downloaded for 9c9bde4d2bf9,
2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,
2019/01/27 15:03:33 Failed to get the value from the keyring: keyring/dbus: Error connecting to dbus session, not registering SecretService provider: dbus: DBUS_SESSION_BUS_ADDRESS not set,
2019/01/27 15:03:33 Failed to decrypt the testing data using the password from KeyChain/Keyring: crypto/aes: invalid key size 0,
2019/01/27 15:03:34 Duplicacy CLI 2.1.2, Duplicacy CLI 2.1.2,
2019/01/27 15:03:34 Temporary directory set to /var/cache/duplicacy/repositories,
2019/01/27 15:03:34 Schedule  next run time: 2019-0128 05:00,
2019/01/27 15:03:34 Duplicacy Web Edition Beta 0.2.10 (5771BE), 
Duplicacy Web Edition Beta 0.2.10 (5771BE),
Starting the web server at http://[::]:3875,

[k2eric]

Those logs messages look normal to me, except for this one:

Failed to marshal the configuration: json: error calling MarshalJSON for type main.StorageCredentials: No master password provided,

But since the rest of the server starts up, it doesn’t seem like it’d be interfering.

What’s your full docker-run command? Are you using traefik?

[saspus]

2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,

2019/01/27 15:03:33 A new license has been downloaded for 9c9bde4d2bf9,

Please see this (my) comment. You need persistent machine-id (via dbus, or manually) and persistent hostname (via --hostname flag for docker). Otherwise you won’t be able to stabilize the license.

Duplicacy Web Edition 0.2.10 Beta is now available

Ah! That totally makes sense. Docker creates hostnames that look like ids, that confused me. So, this is what I’ve learned needs to be done to stabilize the license when changing the container (which is based on alpine linux): install dbus: apk add dbus. link ln -s /var/lib/dbus/machine-id /etc/machine-id. pass --hostname argument when launching container. This result in correct behavior, docker seems to ensure that machine-id is generated consistently on the same host, no need to save/rest…

[saspus]

Update. Further testing revealed that the machine-id is regenerated if the container layer caches are purged completely. And also that there is no need to link it to /etc/machine-id; duplicacy finds it at original location just fine.

So I added back code to persist the machine-id in the config directory.

I’ve also added option to run duplicacy under specific user and group in the container as opposed to root.

Updated container to the docker hub. Feel free to re-use or just use as-is. Source is here

I was testing it on Synology, seems to work fine so far.

[k2eric]

Thank you for your research regarding the machine-id!

machine-id is regenerated if the container layer caches are purged completely.

/var/lib/dbus/machine-id will be identical for everyone running your image since it’s “baked in”. So if two different users happen to use the same hostname (e.g. duplicacy), Duplicacy would detect a license violation, no? Having a static machine-id also seems, to me at least, to be contrary to the semantics of what the value is supposed to represent.

I’ve updated my image to generate a temporary machine-id when the container launches, but also allow a few different ways for users to supply their own static values (i.e. bind-mount or environment variable).

[saspus]

/var/lib/dbus/machine-id will be identical for everyone running your image since it’s “baked in”

Ooooops. You are right. I did not think that through. I assumed that the image will be rebuilt every time from the source, which is not the case.

I think the solution would be to install dbus in init script, as opposed to backed into the image; as the machine-id seems to be generated by dbus-uuidgen in the dbuses post-install script.

Proof of concept:

aleximac:~ alex$ docker run -it alpine /bin/sh
/ # apk add dbus
...
(3/3) Installing dbus (1.10.24-r1)
Executing dbus-1.10.24-r1.pre-install
Executing dbus-1.10.24-r1.post-install
Executing busybox-1.28.4-r2.trigger
OK: 5 MiB in 16 packages
/ # cat /var/lib/dbus/machine-id
50ad3d4b81530f1ee29a3f7c5c50ea85
/ # exit
aleximac:~ alex$ docker run -it alpine /bin/sh
/ # apk add dbus
...
(3/3) Installing dbus (1.10.24-r1)
Executing dbus-1.10.24-r1.pre-install
Executing dbus-1.10.24-r1.post-install
Executing busybox-1.28.4-r2.trigger
OK: 5 MiB in 16 packages
/ # cat /var/lib/dbus/machine-id
076c0a40baf41f59667dc0595c50ea98
/ #

Thank you for catching that!

I’ll fix that and check in the fix in the evening.

Edit. Actually, I think it would be even better to just leave dbus backed in the image, but run the dbus-uuidgen script in the init script unless the saved machine-id already exists in the config folder, in which case just copy it.

Edit. This is what I ended up doing; this takes care of new machine-id generation, verification and correction.

# Preparing persistent unique machine ID
if ! dbus-uuidgen --ensure=/config/machine-id; then 
	echo machine-id contains invalid data. Regenerating.
	dbus-uuidgen > /config/machine-id
fi

# Copying machine-id to container
cp /config/machine-id /var/lib/dbus/machine-id
chmod o+r,g+r /var/lib/dbus/machine-id