Run web-ui in a docker container?


#1

First of all - nice work! Soon you’ll be able to differentiate Duplicacy from other backup tools on the basis of performance AND functionality/ease of use.

Note: I am running duplicacy on a Linux x64 server.

Is there any way we can get this into a docker image? I’m fairly religious about running my web apps in docker images (reverse-proxied by Traefik). Benefits?

  1. Minimize the attack vector (if you don’t run docker as root).

  2. Allows you to mount just the storage you want backed up to the image.

  3. Eliminates the need to start the application before modifying the port it listens on (as has been mentioned, port 8080 will commonly be in use on an existing (shared) server. In a docker container, you can choose to:

    1. Leave it at 8080 (since it’s only on the container) and reverse proxy to that port, without needing to bind it to the host at all (very common with several web apps I run).

    2. Bind port 8080 internally to whatever port you want on the host without touching the code.

Here’s the Dockerfile I’ve created. The only error I’m getting is that it cannot find the CLI executable. I tried ensuring the CLI version of Duplicacy was available inside the image (at /bin, in the path) but no matter how I executed to GUI runtime, it gave the CLI missing error inside the container upon startup. That’s the one piece I need to figure out.

FROM alpine:latest
ADD duplicacy_web_linux_x64_0.1.0 /bin/duplicacy_web
ADD duplicacy_linux_x64_2.1.2 /bin/duplicacy
RUN chmod +x /bin/duplicacy_web /bin/duplicacy
EXPOSE 8080

I’ve tried each of the following as the last line in the Dockerfile:
ENTRYPOINT ["/bin/duplicacy_web"]
CMD ["/bin/duplicacy_web"] and other variations. I just need to get the CLI found inside the image. I’m not exactly a developer though. Ideas?

Oh - here’s how a very simple docker run would work:

docker run -d -p <any_port>:8080 -v /volume/to/backup:/backup/name_of_volume -v /second/volume/to/backup:/backup/name_of_second_volume <repo>/duplicacy_web

And a simple docker-compose.yml:

image: <repo>/duplicacy_web
container_name: duplicacy_web
volumes:
  - /volume/to/backup:/backup/name_of_volume
  - /second/volume/to/backup:/backup/name_of_second_volume
ports:
  - <any_port>:8080

Then all your backup repositories are nicely mounted inside the container. (By the way, I presume the web GUI doesn’t change the underlying architecture in which .duplicacy folder is present in each repo. If the repo information is kept elsewhere, then you could mount the backup repositories as “read only”).


Screenshots of the new web-based GUI
How to install and setup Duplicacy CLI on Synology?
#2

The web GUI always looks for the CLI under ~/.duplicacy-web/bin. If it can’t find it there, it will download the latest release from github.com. So I think if you can copy the CLI under that directory it should work.


#3

I am sure I have the files in the right place, but I still get this error:

/root/.duplicacy-web/bin/duplicacy_web
Created a new configuration.
Failed to locate the CLI executable

This is despite the fact that upon container startup, I can do the following inside the container:

~ # cd /root/.duplicacy-web/bin
~/.duplicacy-web/bin # ls -lart
total 47624
-rwxrwxr--    1 root     root      26327201 Nov 15 13:54 duplicacy
-rwxrwxr--    1 root     root      22428329 Nov 15 13:54 duplicacy_web
drwxrwxrwx    1 root     root          4096 Nov 15 13:54 .
drwxr-xr-x    1 root     root          4096 Nov 15 13:56 ..
~/.duplicacy-web/bin #

Is it looking for a specific filename or format? If it looks in ~/.duplicacy-web/bin, does it not like my filename?Here is my dockerfile now (it’s gotten really ugly as I hack it together with Duct Tape :slight_smile:

FROM alpine:latest
RUN mkdir -p /app
RUN mkdir -p /root/.duplicacy-web/bin
RUN chmod 777 /root/.duplicacy-web/bin
COPY duplicacy /app/duplicacy
COPY duplicacy_web /app/duplicacy_web
RUN cp /app/duplicacy /root/.duplicacy-web/bin/duplicacy
RUN cp /app/duplicacy_web /root/.duplicacy-web/bin/duplicacy_web
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy_web
EXPOSE 8080
#ENTRYPOINT ["/root/.duplicacy-web/bin/duplicacy_web"]

The last line is commented out so the completed image may be run interactively to get a prompt. Trying to run the coomand in ENTRYPOINT manually starts to create a config, but is still giving an error that it cannot find the CLI. Any ideas?


#4

The CLI has to be named duplicacy_linux_x64_2.1.2. The version number may change later when there are newer releases on github, but you can ‘freeze’ the version by adding the line to duplicacy.json:

    "cli_version": "2.1.2"

#5

Renaming the file allowed me to clean things up. This is now working as a draft we can refine if it’s of any value to you. One incredibly important thing that’s missing is: where is duplicacy_web storing its data? I want to put it in a docker named volume or (more likely) - bind that volume to a host volume. Right now, no settings are persistent beyond restarts of the container.

FROM alpine:latest
COPY duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2
COPY duplicacy_web /root/.duplicacy-web/bin/duplicacy_web
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_web
EXPOSE 8080
ENTRYPOINT ["/root/.duplicacy-web/bin/duplicacy_web"]

And my duplicacy.yml (docker-compose) stack:

version: '3'
services:

  duplicacy:
    image: gkoerk/duplicacy:latest
    volumes:
      # Expose backup folders to the container:
      - /share/appdata:/backup/appdata
      - /share/Photos:/backup/photos
      - /share/downloads:/backup/downloads
      - /share/media/movies:/backup/movies
      # Mount external storage as one backup destination.
      - /share/USBDisk1:/destination/external
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.port=8080
        - traefik.network=traefik_public
        - 'traefik.frontend.rule=Host:backup.domain.com'          

networks:
  traefik_public:
    external: true

#6

Where does duplicacy_web store persisted data?


#7

All files are under ~/.duplicacy-web. The most important one is duplicacy.json which stores almost all config information.

The folder filters has all include/exclude patterns so this folder should be backed up too.

The folders stats and logs are less important; if you don’t want to keep the history you don’t need to back up these two.

And definitely no need to back up the repositories folder, as all files there are temporary (most are cache files).


#8

Okay - here is a better docker-compose.yml. Note that it offers persistence, but you will need to copy out the duplicacy.json file. Also - I can’t get Traefik to reverse-proxy the web app. Any idea what might be the issue? Right now I just get a “waiting for backup..com” and eventually a gateway timeout.

Dockerfile:

FROM alpine:latest
COPY duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2
COPY duplicacy_web /root/.duplicacy-web/bin/duplicacy_web
RUN chmod 774 /root/.duplicacy-web/bin/duplicacy_linux_x64_2.1.2 /root/.duplicacy-web/bin/duplicacy_web
EXPOSE 8080
ENTRYPOINT ["/root/.duplicacy-web/bin/duplicacy_web"]

To build your image you run the following from within the same folder as the Dockerfile, and reference it as the image name in your docker-compose.yml (yes, there are several other ways to do this, please feel free to improve upon this however you see fit):

docker build -t <imagename>:<tag> .

docker-compose.yml (works via docker-compose up -d or in Swarm mode via docker stack deploy:

version: '3'
services:

  duplicacy:
    image: <imagename>:<tag>
    ports:
      - 8090:8080
    volumes:
      - /share/appdata/duplicacy/duplicacy.json:/root/.duplicacy-web/duplicacy.json
      - /share/appdata/duplicacy/stats:/root/.duplicacy-web/stats
      - /share/appdata/duplicacy/log:/root/.duplicacy-web/logs
      - /share/appdata:/backup/appdata
      - /share/Photos:/backup/photos
      - /share/downloads:/backup/downloads
      - /share/media/movies:/backup/movies
      - /share/media/music:/backup/music
      - /share/media/tv:/backup/tv
      - /share/USBDisk1:/destination/external
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.port=8080
        - traefik.network=traefik_public
        - 'traefik.frontend.rule=Host:backup.gkoerk.com'          

networks:
  traefik_public:
    external: true

And the command to setup the traefik_public network:

  • docker-compose:

docker network create --driver=bridge --subnet=172.1.1.0/21 traefik_public

  • docker stack deploy:

docker network create --driver=overlay --subnet=172.1.1.0/21 --attachable traefik_public

`


Duplicacy Web Edition 0.1 Beta
#9

I’ve also been toying around with running the web UI in a Docker container and though I’d share my working Dockerfile:

FROM alpine:latest
    
RUN ARCHITECTURE=linux_x64                                                                             && \
    SHA256_DUPLICACY=034720abb90702cffc4f59ff8c29cda61f14d9065e6ca0e4017ba144372f95d7                  && \
    SHA256_DUPLICACY_WEB=b0a9fc35f249ee8fa5d7da0fe0f8487041661ed19b24c115fd01aee37049ccfe              
    VERSION_DUPLICACY=2.1.2                                                                            && \
    VERSION_DUPLICACY_WEB=0.2.8                                                                        && \
                                                                                                          \
    # add Bash for our entrypoint.sh, and ca-certificates so Duplicacy doesn't complain about certs
    apk update                                                                                         && \
    apk add --no-cache bash ca-certificates                                                            && \
                                                                                                          \
    # download, check, and install duplicacy
    wget --quiet -O /usr/local/bin/duplicacy                                                              \ 
 https://github.com/gilbertchen/duplicacy/releases/download/v${VERSION_DUPLICACY}/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
    echo "${SHA256_DUPLICACY}  /usr/local/bin/duplicacy" | sha256sum -s -c -                           && \
    chmod +x /usr/local/bin/duplicacy                                                                  && \
                                                                                                          \
    # downlooad, check, and install the web UI
    wget --quiet -O /usr/local/bin/duplicacy_web                                                          \
        https://acrosync.com/duplicacy-web/duplicacy_web_${ARCHITECTURE}_${VERSION_DUPLICACY_WEB}      && \
    echo "${SHA256_DUPLICACY_WEB}  /usr/local/bin/duplicacy_web" | sha256sum -s -c -                   && \
    chmod +x /usr/local/bin/duplicacy_web                                                              && \
                                                                                                          \
    # duplicacy_web expects to find the CLI binary in a certain location
    # https://forum.duplicacy.com/t/run-web-ui-in-a-docker-container/1505/2
    mkdir -p ~/.duplicacy-web/bin                                                                      && \
    ln -s /usr/local/bin/duplicacy ~/.duplicacy-web/bin/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
                                                                                                              \
    # create the logs directory and touch a log so that we can tail it before we start the server
    mkdir ~/.duplicacy-web/logs                                                                        && \
    touch ~/.duplicacy-web/logs/duplicacy_web.log                                                      && \
                                                                                                          \
    # listen on all interfaces
    echo '{"listening_address":"0.0.0.0:3875"}' > ~/.duplicacy-web/settings.json

COPY ./entrypoint.sh /usr/local/bin/entrypoint.sh

ENTRYPOINT [ "/usr/local/bin/entrypoint.sh" ]

entrypoint.sh looks something like this:

#!/usr/bin/env bash
tail -f ~/.duplicacy-web/logs/duplicacy_web.log &
duplicacy_web &

It also includes a Bash trap to catch Ctrl-C to shut down the server, but I’ve left that out for clarity.

The trick for me was to get the web UI to listen on 0.0.0.0 by writing an initial duplicacy.json, otherwise the server listens only on localhost and is therefore inaccessible outside of the container.


#10

Could you share the additions/edits or an example of your duplicacy.json (redacted of course)? I don’t see how to add the listening address.


#11

From the Dockerfile I posted above, near the bottom:

echo '{"listening_address":"0.0.0.0:3875"}' > ~/.duplicacy-web/settings.json

Note that this line sets the contents of settings.json, not duplicacy.json. I made no other edits.

Here’s an updated Dockerfile that you can build and test:

FROM alpine:latest

RUN ARCHITECTURE=linux_x64                                                                             && \
    SHA256_DUPLICACY=034720abb90702cffc4f59ff8c29cda61f14d9065e6ca0e4017ba144372f95d7                  && \
    SHA256_DUPLICACY_WEB=322e8865fa5f480952938be018725bf02bd0023a26512eb67216a9f0cb721726              && \
    VERSION_DUPLICACY=2.1.2                                                                            && \
    VERSION_DUPLICACY_WEB=0.2.10                                                                       && \
                                                                                                          \
    # add Bash for our entrypoint.sh, and ca-certificates so Duplicacy doesn't complain about certs
    apk update                                                                                         && \
    apk add --no-cache bash ca-certificates                                                            && \
                                                                                                          \
    # download, check, and install duplicacy
    wget --quiet -O /usr/local/bin/duplicacy                                                              \
        https://github.com/gilbertchen/duplicacy/releases/download/v${VERSION_DUPLICACY}/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
    echo "${SHA256_DUPLICACY}  /usr/local/bin/duplicacy" | sha256sum -s -c -                           && \
    chmod +x /usr/local/bin/duplicacy                                                                  && \
                                                                                                          \
    # downlooad, check, and install the web UI
    wget --quiet -O /usr/local/bin/duplicacy_web                                                          \
        https://acrosync.com/duplicacy-web/duplicacy_web_${ARCHITECTURE}_${VERSION_DUPLICACY_WEB}      && \
    echo "${SHA256_DUPLICACY_WEB}  /usr/local/bin/duplicacy_web" | sha256sum -s -c -                   && \
    chmod +x /usr/local/bin/duplicacy_web                                                              && \
                                                                                                          \
    # duplicacy_web expects to find the CLI binary in a certain location
    # https://forum.duplicacy.com/t/run-web-ui-in-a-docker-container/1505/2
    mkdir -p ~/.duplicacy-web/bin                                                                      && \
    ln -s /usr/local/bin/duplicacy ~/.duplicacy-web/bin/duplicacy_${ARCHITECTURE}_${VERSION_DUPLICACY} && \
                                                                                                          \
    # create the logs directory and touch a log so that we can tail it before we start the server
    mkdir ~/.duplicacy-web/logs                                                                        && \
    touch ~/.duplicacy-web/logs/duplicacy_web.log                                                      && \
                                                                                                          \
    # listen on all interfaces
    echo '{"listening_address":"0.0.0.0:3875"}' > ~/.duplicacy-web/settings.json

ENTRYPOINT [ "/usr/local/bin/duplicacy_web" ]

Run it with docker run --rm -p 9991:3875 046107c386da, the visit http://localhost:9991 and you should see the login screen.

Does that help?


#12

I’ve tweaked (and simplified) your docker file and added explicit declarations to move cache, logs, and config location into sane places (they would reside on different volumes with different backup and retention policies and to declare such possibilities, including required ports, to be user-friendly.

I really don’t like the way Duplicacy stuffs transient, logs, and config data into the same folder.

Dockerfile:

FROM alpine:latest

ENV VERSION=0.2.10

RUN  apk --update add --no-cache bash ca-certificates && \
    wget -nv -O /usr/local/bin/duplicacy_web                      \
        https://acrosync.com/duplicacy-web/duplicacy_web_linux_x64_${VERSION} && \
    chmod +x /usr/local/bin/duplicacy_web  && \
    rm -rf /tmp/* && rm -rf /var/cache/apk/*                                                             

COPY ./entrypoint.sh /usr/local/bin/entrypoint.sh                          
                            
VOLUME /config
VOLUME /logs
VOLUME /cache

EXPOSE 3875/tcp

ENTRYPOINT [ "/usr/local/bin/entrypoint.sh" ]

Entrypoint:

#!/usr/bin/env bash  

# trap ^C
trap 'kill ${!}; exit' SIGHUP SIGINT SIGQUIT SIGTERM
         
# create duplicacy folders
                                     
mkdir -p    ~/.duplicacy-web/logs \
            ~/.duplicacy-web/filters \
            ~/.duplicacy-web/repositories 
            
            
echo Creating sane exports and files
mkdir -p    /config/filters \
            /logs \
            /cache 

touch       /logs/duplicacy_web.log
            
            
echo '{"listening_address":"0.0.0.0:3875"}' > /config/settings.json
echo '{}'                                   > /config/duplicacy.json

echo Link data to where duplicacy expects it

ln -s /config/settings.json     ~/.duplicacy-web/settings.json                  
ln -s /config/duplicacy.json    ~/.duplicacy-web/duplicacy.json
ln -s /config/filters           ~/.duplicacy-web/filters
ln -s /logs                     ~/.duplicacy-web/logs
ln -s /cache                    ~/.duplicacy-web/repositories

echo Logging tail of the log from this moment on
tail -0 -f /logs/duplicacy_web.log & 

echo Starting duplicacy
duplicacy_web &

# wait for events.
wait

To run:

docker build --tag=saspus/duplicacy-web .
docker run  -p 3875:3875/tcp \
        -v ~/Library/Duplicacy:/config  \
        -v ~/Library/Logs/Duplicacy/:/logs \
        -v ~/Library/Caches/Duplicacy:/cache \
        -v ~:/MyHomeToBackup:ro \
        saspus/duplicacy-web

#13

Great ideas! I’ve incorporated some of your tricks into my image.

For fun, I’ve published a new GitHub repo and would be happy to accept feedback or contributions:

https://github.com/ehough/docker-duplicacy

To use it:

  1. docker run -p 3875:3875 erichough/duplicacy
  2. Visit http://localhost:3875

The GitHub readme has a few other tips on how to use the image, but the above instructions should work well for testing.


#14

Any reason why are you manually downloading the CLI engine? Duplicacy web will download the correct one on the first run itself.

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

# redirect the log to stdout
ln -s /dev/stdout /var/log/duplicacy_web.log                                              && \

I’m stealing it! How I did not think of this.


#15

Thanks for the feedback!

I didn’t know duplicacy_web would do that - clever!

Have you pulled the latest? I think I fixed that exact bug last night in this commit but will do some more testing. Thanks for the heads up.

I considered just using wget, but paranoia dies hard :slight_smile: The recent hack of PEAR (a fairly critical component of PHP’s ecosystem) was a good reminder for me that the bad guys will eventually find their way into anything.

Let us know if you end up sharing or publishing your image, and thanks again for the feedback.


#16

Ah, yes, indeed, I looked at it yesterday, but posted comment today.
I also did not realize you can create symlinks to non-existent files. That is pretty useful.

I guess it’s a good point. And it’s probably not too much hassle to keep updating the Dockerfile when new version is released.

Thank you too for the ideas - it’s my first docker container, have never done it before.
I posted link to it in the other message, copying here: Docker Hub

That said I really don’t like the idea of linking to individual config files; what if next version introduces another new file?

I thing duplicacy_web shall support getting path to configuration folder from environment. If it is not set, then by all means let is use ~/.duplicacy-web. @gchen, do you think this would be useful/feasible?

Another question for @gchen, is there an exhaustive list of stuff that can be configured in settings.json?


#17

I got the following errors in the docker logs when running your docker image. It looks like it is running, but it isn’t accessible via browser. Any ideas what I’m doing wrong?

Log directory set to /var/log,
2019/01/27 15:03:33 Failed to marshal the configuration: json: error calling MarshalJSON for type main.StorageCredentials: No master password provided,
2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,
2019/01/27 15:03:33 A new license has been downloaded for 9c9bde4d2bf9,
2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,
2019/01/27 15:03:33 Failed to get the value from the keyring: keyring/dbus: Error connecting to dbus session, not registering SecretService provider: dbus: DBUS_SESSION_BUS_ADDRESS not set,
2019/01/27 15:03:33 Failed to decrypt the testing data using the password from KeyChain/Keyring: crypto/aes: invalid key size 0,
2019/01/27 15:03:34 Duplicacy CLI 2.1.2, Duplicacy CLI 2.1.2,
2019/01/27 15:03:34 Temporary directory set to /var/cache/duplicacy/repositories,
2019/01/27 15:03:34 Schedule  next run time: 2019-0128 05:00,
2019/01/27 15:03:34 Duplicacy Web Edition Beta 0.2.10 (5771BE), 
Duplicacy Web Edition Beta 0.2.10 (5771BE),
Starting the web server at http://[::]:3875,

#18

Those logs messages look normal to me, except for this one:

Failed to marshal the configuration: json: error calling MarshalJSON for type main.StorageCredentials: No master password provided,

But since the rest of the server starts up, it doesn’t seem like it’d be interfering.

What’s your full docker-run command? Are you using traefik?


#19

Please see this (my) comment. You need persistent machine-id (via dbus, or manually) and persistent hostname (via --hostname flag for docker). Otherwise you won’t be able to stabilize the license.


#20

Update. Further testing revealed that the machine-id is regenerated if the container layer caches are purged completely. And also that there is no need to link it to /etc/machine-id; duplicacy finds it at original location just fine.

So I added back code to persist the machine-id in the config directory.

I’ve also added option to run duplicacy under specific user and group in the container as opposed to root.

Updated container to the docker hub. Feel free to re-use or just use as-is. Source is here

I was testing it on Synology, seems to work fine so far.