Trying to (unsuccessfully) copy existing data storage

[david1]

Hi there. Was hoping to get some help on copying an existing storage repository to a new one. I’m running Duplicacy Web Edition 1.8.1. Command line version is 3.2.3.

The existing storage repository is on OneDrive and scheduled backups have been working just fine. I have four backups being backed up to that repository with daily backups, prunes and checks. With the release of 1.8.1, I noticed the addition of zstd compression and noticed someone else mentioning that they were able to save quite a bit of storage by enabling it, so I decided to follow their lead.

In the Web UI, I paused all backups, prunes and checks to the old repository. Then I set up a copy job from the existing repository to a new one I had created, also on OneDrive, with compression turned on, and ran it. It failed a few times, maybe because it was being throttled by OneDrive, but eventually I seem to have gotten quite close to completion. The last time I tried it, these were the results from the log:

Copy storage OneDrive-fava2 to storage OneDrive-fava2-compressed, status: Failed, log file: copy-20240512-082322.log
--------------------------------------------------------------------------------------------------------------------
Running copy command from /root/.duplicacy-web/repositories/localhost/all
Options: [-log copy -from OneDrive-fava2 -to OneDrive-fava2-compressed]
2024-05-12 08:23:22.394 INFO STORAGE_SET Source storage set to one://Backups/fava2
2024-05-12 08:23:30.163 INFO STORAGE_SET Destination storage set to one://Backups/fava2 - Compressed
2024-05-12 08:30:52.333 INFO SNAPSHOT_COPY Chunks to copy: 2, to skip: 133409, total: 133411
2024-05-12 08:30:53.928 WARN DOWNLOAD_RETRY Failed to decrypt the chunk ba969607fd0de3104dea75d7040b6a26f079becfa8c9d204cc57e07d4ce6cc88: cipher: message authentication failed; retrying
2024-05-12 08:30:54.988 WARN DOWNLOAD_RETRY Failed to decrypt the chunk ba969607fd0de3104dea75d7040b6a26f079becfa8c9d204cc57e07d4ce6cc88: cipher: message authentication failed; retrying
2024-05-12 08:30:56.122 WARN DOWNLOAD_RETRY Failed to decrypt the chunk ba969607fd0de3104dea75d7040b6a26f079becfa8c9d204cc57e07d4ce6cc88: cipher: message authentication failed; retrying
2024-05-12 08:30:56.971 ERROR DOWNLOAD_DECRYPT Failed to decrypt the chunk ba969607fd0de3104dea75d7040b6a26f079becfa8c9d204cc57e07d4ce6cc88: cipher: message authentication failed
Failed to decrypt the chunk ba969607fd0de3104dea75d7040b6a26f079becfa8c9d204cc57e07d4ce6cc88: cipher: message authentication failed

When I run a check on the new repository, I get the following:

Running check command from /root/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage OneDrive-fava2-compressed -chunks -tabular -a -tabular]
2024-05-12 10:06:51.641 INFO STORAGE_SET Storage set to one://Backups/fava2 - Compressed
2024-05-12 10:06:58.693 INFO SNAPSHOT_CHECK Listing all chunks
2024-05-12 10:09:56.908 INFO SNAPSHOT_CHECK 0 snapshots and 0 revisions
2024-05-12 10:09:56.911 INFO SNAPSHOT_CHECK Total chunk size is 561,222M in 133409 chunks
2024-05-12 10:09:56.911 INFO SNAPSHOT_CHECK 
2024-05-12 10:09:56.911 INFO SNAPSHOT_VERIFY Verifying 0 chunks
2024-05-12 10:09:56.911 INFO SNAPSHOT_VERIFY All 0 chunks have been successfully verified

I’m running a check on the old repository, but because of its size is taking a while. But so far, everything looks fine. The last check I ran after the last backup to the old repository was also fine.

On the Web UI, the old repository shows as being 761.94G with 4 BackupIDs and 336 revisions. Chunks are 159,205 with 4 revisions. However, for the new repository, it shows 588.48G, 0 Backup IDs with 0 revisions and 133,409 chunks.

At this stage, I’m not quite sure what else to do to successfully copy the old repository to the new so that I can start using the new one. I’m puzzled by the check command shows 0 snapshots and 0 revisions, and why the check command isn’t verifying any chunks. I’m also not sure I know how to address or fix the “Failed to decrypt” messages above. Or why the original repo shows as having 159,205 chunks but the new repo only has a total of 133,411.

Any suggestions or recommendations would be appreciated.

Update: The check on the old repository finished with the following errors at the end:

2024-05-12 21:45:49.684 WARN DOWNLOAD_RETRY Failed to decrypt the chunk acc4fa3ce0605a39bb4bb67bde68c9de23578ccb7de05d818cfc188d4f859282: cipher: message authentication failed; retrying
2024-05-12 21:45:51.083 WARN DOWNLOAD_RETRY Failed to decrypt the chunk acc4fa3ce0605a39bb4bb67bde68c9de23578ccb7de05d818cfc188d4f859282: cipher: message authentication failed; retrying
2024-05-12 21:45:52.492 WARN DOWNLOAD_RETRY Failed to decrypt the chunk acc4fa3ce0605a39bb4bb67bde68c9de23578ccb7de05d818cfc188d4f859282: cipher: message authentication failed; retrying
2024-05-12 21:45:54.204 ERROR DOWNLOAD_DECRYPT Failed to decrypt the chunk acc4fa3ce0605a39bb4bb67bde68c9de23578ccb7de05d818cfc188d4f859282: cipher: message authentication failed
2024-05-12 21:45:54.261 INFO SNAPSHOT_VERIFY Added 38977 chunks to the list of verified chunks
Failed to decrypt the chunk acc4fa3ce0605a39bb4bb67bde68c9de23578ccb7de05d818cfc188d4f859282: cipher: message authentication failed

[saspus]

I would run check -chunks on your original one-drive, to make sure that the chunks are not corrupted. (There was a bug in OneDrive for business where it would save partial file)

You might need to delete the verified_chunks file for duplciacy to recheck all chunks.

If that works, and you’ve added the second storage as copy-compatible, it shall work.

[david1]

Thank you saspus. Most appreciated.

[david1]

In case it might be helpful to others:

• when I ran check -chunks, it detected multiple corrupted chunks in the original storage. Each time I ran it, it stopped as soon as it detected the first corrupt chunk, so had to run it multiple times
• each time a corrupted chunk was detected, I deleted the chunk and the revision in which it was found. Thankfully the deleted revisions were ones I could afford to delete.
• after doing the above 4 or 5 times, finally got a clean check. I then checked the copy as well with check -chunks just to make sure the copy was good

After all the above was done, I ran a backup on the new compressed copy and it all proceeded smoothly. Took quite a bit of time to do all of the above, but I suppose worth it - using -zstd best I was able to reduce storage by just under 22%, or about 154 GB.

[saspus]

Two remarks.

• you can add -persist flag to check to prevent it from stopping in first error. That way you would find all of them in one go.
• after you have manually deleted snapshot revision file, there are likely now some chunks in the storage that don’t belong to any snapshot and can be deleted to free some space. To clear them out you can run prune with -exhaustive flag. It will take a while though, especially on slow backends like OneDrive.

[david1]

Och. Silly me. That -persist flag would have literally saved me days. Oh well. Live and learn. And thank you very much for pointing out the -exhaustive flag. That’s very helpful and I’ll try it on the next run.