Run web-ui in a docker container?

[saspus]

Any reason why are you manually downloading the CLI engine? Duplicacy web will download the correct one on the first run itself.

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

# redirect the log to stdout
ln -s /dev/stdout /var/log/duplicacy_web.log                                              && \

I’m stealing it! How I did not think of this.

[k2eric]

Thanks for the feedback!

any reason why are you manually downloading the CLI engine? Duplicacy web will download the correct one on the first run itself.

I didn’t know duplicacy_web would do that - clever!

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Have you pulled the latest? I think I fixed that exact bug last night in this commit but will do some more testing. Thanks for the heads up.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

I considered just using wget, but paranoia dies hard The recent hack of PEAR (a fairly critical component of PHP’s ecosystem) was a good reminder for me that the bad guys will eventually find their way into anything.

Let us know if you end up sharing or publishing your image, and thanks again for the feedback.

[saspus]

There is a bug though in your Dockerfile: if user maps out the /etc/duplicacy volume all the changes you made prior to that in the RUN directive (e.g. creating symlinks) will be lost. You need to set those up in the entry point script instead.

Have you pulled the latest? I think I fixed that exact bug last night in this commit but will do some more testing. Thanks for the heads up.

Ah, yes, indeed, I looked at it yesterday, but posted comment today.
I also did not realize you can create symlinks to non-existent files. That is pretty useful.

Also I don’t think you need to verify SHA checksums; if network error occurs wget will return failure.

I considered just using wget , but paranoia dies hard The recent hack of PEAR (a fairly critical component of PHP’s ecosystem) was a good reminder for me that the bad guys will eventually find their way into anything.

I guess it’s a good point. And it’s probably not too much hassle to keep updating the Dockerfile when new version is released.

Let us know if you end up sharing or publishing your image, and thanks again for the feedback.

Thank you too for the ideas - it’s my first docker container, have never done it before.
I posted link to it in the other message, copying here: Docker Hub

That said I really don’t like the idea of linking to individual config files; what if next version introduces another new file?

I thing duplicacy_web shall support getting path to configuration folder from environment. If it is not set, then by all means let is use ~/.duplicacy-web. @gchen, do you think this would be useful/feasible?

Another question for @gchen, is there an exhaustive list of stuff that can be configured in settings.json?

[gkoerk]

I got the following errors in the docker logs when running your docker image. It looks like it is running, but it isn’t accessible via browser. Any ideas what I’m doing wrong?

Log directory set to /var/log,
2019/01/27 15:03:33 Failed to marshal the configuration: json: error calling MarshalJSON for type main.StorageCredentials: No master password provided,
2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,
2019/01/27 15:03:33 A new license has been downloaded for 9c9bde4d2bf9,
2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,
2019/01/27 15:03:33 Failed to get the value from the keyring: keyring/dbus: Error connecting to dbus session, not registering SecretService provider: dbus: DBUS_SESSION_BUS_ADDRESS not set,
2019/01/27 15:03:33 Failed to decrypt the testing data using the password from KeyChain/Keyring: crypto/aes: invalid key size 0,
2019/01/27 15:03:34 Duplicacy CLI 2.1.2, Duplicacy CLI 2.1.2,
2019/01/27 15:03:34 Temporary directory set to /var/cache/duplicacy/repositories,
2019/01/27 15:03:34 Schedule  next run time: 2019-0128 05:00,
2019/01/27 15:03:34 Duplicacy Web Edition Beta 0.2.10 (5771BE), 
Duplicacy Web Edition Beta 0.2.10 (5771BE),
Starting the web server at http://[::]:3875,

[k2eric]

Those logs messages look normal to me, except for this one:

Failed to marshal the configuration: json: error calling MarshalJSON for type main.StorageCredentials: No master password provided,

But since the rest of the server starts up, it doesn’t seem like it’d be interfering.

What’s your full docker-run command? Are you using traefik?

[saspus]

2019/01/27 15:03:33 Failed to retrieve the machine id: machineid: open /etc/machine-id: no such file or directory,

2019/01/27 15:03:33 A new license has been downloaded for 9c9bde4d2bf9,

Please see this (my) comment. You need persistent machine-id (via dbus, or manually) and persistent hostname (via --hostname flag for docker). Otherwise you won’t be able to stabilize the license.

Duplicacy Web Edition 0.2.10 Beta is now available

Ah! That totally makes sense. Docker creates hostnames that look like ids, that confused me. So, this is what I’ve learned needs to be done to stabilize the license when changing the container (which is based on alpine linux): install dbus: apk add dbus. link ln -s /var/lib/dbus/machine-id /etc/machine-id. pass --hostname argument when launching container. This result in correct behavior, docker seems to ensure that machine-id is generated consistently on the same host, no need to save/rest…

[saspus]

Update. Further testing revealed that the machine-id is regenerated if the container layer caches are purged completely. And also that there is no need to link it to /etc/machine-id; duplicacy finds it at original location just fine.

So I added back code to persist the machine-id in the config directory.

I’ve also added option to run duplicacy under specific user and group in the container as opposed to root.

Updated container to the docker hub. Feel free to re-use or just use as-is. Source is here

I was testing it on Synology, seems to work fine so far.

[k2eric]

Thank you for your research regarding the machine-id!

machine-id is regenerated if the container layer caches are purged completely.

/var/lib/dbus/machine-id will be identical for everyone running your image since it’s “baked in”. So if two different users happen to use the same hostname (e.g. duplicacy), Duplicacy would detect a license violation, no? Having a static machine-id also seems, to me at least, to be contrary to the semantics of what the value is supposed to represent.

I’ve updated my image to generate a temporary machine-id when the container launches, but also allow a few different ways for users to supply their own static values (i.e. bind-mount or environment variable).

[saspus]

/var/lib/dbus/machine-id will be identical for everyone running your image since it’s “baked in”

Ooooops. You are right. I did not think that through. I assumed that the image will be rebuilt every time from the source, which is not the case.

I think the solution would be to install dbus in init script, as opposed to backed into the image; as the machine-id seems to be generated by dbus-uuidgen in the dbuses post-install script.

Proof of concept:

aleximac:~ alex$ docker run -it alpine /bin/sh
/ # apk add dbus
...
(3/3) Installing dbus (1.10.24-r1)
Executing dbus-1.10.24-r1.pre-install
Executing dbus-1.10.24-r1.post-install
Executing busybox-1.28.4-r2.trigger
OK: 5 MiB in 16 packages
/ # cat /var/lib/dbus/machine-id
50ad3d4b81530f1ee29a3f7c5c50ea85
/ # exit
aleximac:~ alex$ docker run -it alpine /bin/sh
/ # apk add dbus
...
(3/3) Installing dbus (1.10.24-r1)
Executing dbus-1.10.24-r1.pre-install
Executing dbus-1.10.24-r1.post-install
Executing busybox-1.28.4-r2.trigger
OK: 5 MiB in 16 packages
/ # cat /var/lib/dbus/machine-id
076c0a40baf41f59667dc0595c50ea98
/ #

Thank you for catching that!

I’ll fix that and check in the fix in the evening.

Edit. Actually, I think it would be even better to just leave dbus backed in the image, but run the dbus-uuidgen script in the init script unless the saved machine-id already exists in the config folder, in which case just copy it.

Edit. This is what I ended up doing; this takes care of new machine-id generation, verification and correction.

# Preparing persistent unique machine ID
if ! dbus-uuidgen --ensure=/config/machine-id; then 
	echo machine-id contains invalid data. Regenerating.
	dbus-uuidgen > /config/machine-id
fi

# Copying machine-id to container
cp /config/machine-id /var/lib/dbus/machine-id
chmod o+r,g+r /var/lib/dbus/machine-id

[saspus]

I’ve updated my image to generate a temporary machine-id when the container launches, but also allow a few different ways for users to supply their own static values (i.e. bind-mount or environment variable).

  # no luck. generate a temporary machine-id
  cat /dev/urandom | tr -dc 'a-f0-9' | fold -w 32 | head -n 1 > $path
  log "using temporary, random machine-id of $(cat $path)"

Any reason why don’t you use dbus_uuidgen which is intended to generate machine-id, and instead simulate its behavior (which may change)? Also I’m not sure urandom is diverse enough, i’d add some sort of crypto hash on top. Or just use dbus tools.

Also, i’d validate user-supplied values (dbus-uuidgen can do that for you) - duplicacy likely expects compliant string.

[k2eric]

Any reason why don’t you use dbus_uuidgen which is intended to generate machine-id, and instead simulate its behavior (which may change)?

Mostly because I don’t want to install dbus into the image just to create a 32 character string, especially since it can be done with a one-liner.

Also I’m not sure urandom is diverse enough, i’d add some sort of crypto hash on top

dbus-uuidgen internally reads from /dev/urandom, too.

Not to get off-topic, but I don’t know of a better source of random data. And I don’t think that hashing a random value will produce a more random value; If anything, hashing could have the opposite effect.

[saspus]

Mostly because I don’t want to install dbus into the image just to create a 32 character string, especially since it can be done with a one-liner.

I’d argue that anything could be written in one line; but it is less readable than using dedicated tool. In addition, duplicacy uses dbus, so it does make sense to use the same environment instead of simulating it; space is cheap.
Also, dbus may be useful later, to communicate with keychain for example to work with encryption keys, etc.

dbus-uuidgen internally reads from /dev/urandom , too.

They sort of make an attempt to add variability by concatenating current time to it, but honestly, I did expect more.

Not to get off-topic, but I don’t know of a better source of random data.

Since you are reading random data almost very first thing after boot, the entropy pool may not be deep enough yet (see /proc/sys/kernel/random/entropy_avail). /dev/urandom will not block and return poor quality random data; and since this is used to derive license keys we really want to minimize collisions so I would at least wait for sufficient entropy to accumulate. Simply using /dev/random should do the trick, as it will block until enough entropy is available.

and I don’t think that hashing a random value will produce a more random value;

The amount of entropy won’t change, but correlation between bits will (when using crypto hash, not just CRC). In fairness, I can’t imagine that somebody will try to exploit that to get free duplicacy license but addressing this does not cost anything and may reduce probability of collisions.

[k2eric]

I’d argue that anything could be written in one line; but it is less readable than using dedicated tool. In addition, duplicacy uses dbus, so it does make sense to use the same environment instead of simulating it; space is cheap.
Also, dbus may be useful later, to communicate with keychain for example to work with encryption keys, etc.

k2eric:

I hear ya and respect your idea, but in this case I just have a different opinion. ¯\_(ツ)_/¯

Since you are reading random data almost very first thing after boot, the entropy pool may not be deep enough yet (see /proc/sys/kernel/random/entropy_avail ).

I disagree here. Remember that a container shares the same kernel with the host, so reading from /dev/{u}random in the container will, behind the scenes, ask the same kernel for random bytes. You can verify this yourself:

cat /proc/sys/kernel/random/entropy_avail && docker run --rm alpine cat /proc/sys/kernel/random/entropy_avail

You’ll see that the available entropy is essentially the same, even in the freshly-started container.

You could make the argument that a physical system or virtual machine will have low entropy on boot, but even then many distros will keep a seed file (/lib/systemd/systemd-random-seed or /var/lib/systemd/random-seed) to help with entropy after booting.

/dev/urandom will not block and return poor quality random data; and since this is used to derive license keys we really want to minimize collisions so I would at least wait for sufficient entropy to accumulate. Simply using /dev/random should do the trick, as it will block until enough entropy is available.

The consensus among the experts, as I interpret it at least, is that urandom is recommended for the vast majority of all cryptographic uses. There are a few rare instances where /dev/random is better like embedded systems after boot, or information-theoretic security.

The amount of entropy won’t change, but correlation between bits will (when using crypto hash, not just CRC) … but addressing this does not cost anything and may reduce probability of collisions.

Again I have to disagree. A hash of any kind (crypto or otherwise) simply maps a set of bits to another set of a fixed number bits; it won’t improve the randomness of the final value. Even if you salt the input or use a randomness extraction function on it, you’re relying on additional random data that needs to come from somewhere.

But I agree with you that we’re dealing with a software license and not protecting someone’s bank account or medical history, so we might be going overboard But it’s a good discussion, and I look forward to continuing to work together to develop images for Duplicacy.

[gotcode]

Does anyone know where the stats for the dashboard are stored in the container? I’ve mounted logs and stats to the host container. I see the last run of the backups and schedules within those tabs. But every time I restart the container I lose the stats/history on the dashboard screen.

Thanks!

[saspus]

Are you asking about saspus/duplicacy-web or erichough/duplicacy ?

In the former stats are kept in the /config/stats folder, in the latter stats go to a separate volume /var/cache/duplicacy.

[gotcode]

I was asking about erichough/duplicacy.

I have that stats directory mounted on the host, but the dashboard stats still don’t persist between docker container restarts. I can see the stats on the schedules page, but not on the dashboard.

Any thoughts? Here’s my docker command:

docker run --name=duplicacy -v /etc/localtime:/etc/localtime:ro -p 3875:3875 -v /media/data/working/duplicacy-logs/:/root/.duplicacy-web/logs -v /media/data/working/duplicacy-stats/:/root/.duplicacy-web/stats -v /media/data/working/duplicacy/:/etc/duplicacy -v /media/:/storage erichough/duplicacy

[saspus]

I’m not the author, but take a look at Dockerfile. You’ll find that the stats go under /var/cache/duplicacy. It needs to be mapped to persist.

[gotcode]

Oh gotcha.

I am actually mapping it. In the container, ~./duplicacy-web/stats is symlinked to /var/cache/duplicacy.

I see the stat files being persisted on the host, but it doesn’t look to me like those are the right files to populate the dashboard.

In the version you created, do the dashboard stats persist from run to run?

[saspus]

In the version you created, do the dashboard stats persist from run to run?

Just checked; seems to be preserving the stats just fine:

This is my test script to entirely wipe the container and image and start anew; the dashboard displays all past activity between invocations:

echo Deleting container
export container=$(docker ps -a |grep duplicacy-web-container | awk '{print $1}');
if [ -n "$container"  ]; then
    docker rm $container || exit 1
fi
echo Deleting Image
export image=$(docker images |grep saspus/duplicacy-web | awk '{print $3}')

if  [ -n "$image" ]; then
    docker rmi $image || exit 2
fi

temp=/tmp/dupl01
mkdir $temp

# build
docker build --tag=saspus/duplicacy-web .   || exit 3

docker run  --name duplicacy-web-container             \
        --hostname duplicacy-web-docker                \
         --publish 3875:3875/tcp                       \
             --env USR_ID=$(id -u)                     \
             --env GRP_ID=$(id -g)                     \
          --volume $temp/config:/config                \
          --volume $temp/logs:/logs                    \
          --volume $temp/cache:/cache                  \
          --volume $temp/backuproot:/backuproot:ro     \
          --volume $temp/storage:/storage              \
                   saspus/duplicacy-web

edit: comment out the build command, you don’t need to build it of course.

[gotcode]

Thank you. I removed the container and image, rebuilt everything, and it seems to be working now. I’m not sure what I did differently, but recreating the volume mappings per your post seems to have done the trick. Thanks for helping!