I recently migrated my system and data (Unraid to Ubuntu/ZFS) and since the move, my nightly backups have gone from taking minutes to around 6 hours. Initially, I thought it might have been due to permissions or ownership changes that were initially made during the transfer, but it has now been a week where there have been no file changes.
Duplicacy tries to process every file in the filesystem, only to find almost all chunks have been cached, uploading only a small number of changes actually uploaded. From an uploading perspective, only file changes/incrementals are sent, but from OS/data processing perspective, it’s always a full run that causes hours of high IO activity.
My experience on the other system is that when indexing starts, only “changed since last backup” files are considered for processing and therefore only takes minutes due to a small number files having actually changed.
I have considered a few things:
- Time difference between the docker instance and the host - they have the same time and timezone
- permissions - the container has been run as root and as a less privileged user who owns the files. no change.
- mount point - initially the mount point was ro, but was changed to rw. no change.
Things to try
- ZFS : for performance reasons, I have set
atime=offon the zpool. I would hope that ctime or mtime being on would be sufficient…but maybe not? I could try to backup a non-zfs point to see if the behaviour changes.
- Docker - perhaps there is some Docker shenanigans going on, but I have always run Duplicacy from Docker before, not sure what has changed.
- I have run “-v -d” on a backup, but didn’t see anything obvious…just millions of lines similar to
- Somehow my include pattern is causing problems?
Are there any settings that would log why a file is considered to be changed and needs further processing? (i.e. does it keep track of file attributes to help manage incrementals?)
Any help with getting to the bottom why Duplicacy thinks every file needs to be processed would be very welcome.
- Duplicacy Web Edition 1.7.2 running in Docker on Ubuntu 22