Prune stopped with Failed to decrypt the chunk

Bug
#1

Please describe what you are doing to trigger the bug:

Pruning a remote storage (which is a Copy of a local one).
More specifically:

duplicacy_win_x64_2.2.3.exe -log prune -keep 0:30 -threads 4 -all -storage offsite

Belonging to a series of steps. Daily:
machines.BACKUP to local_server
Weekly:
In server.local, weekly:

server.PRUNE local
server.PRUNE offsite
server.COPY local to offsite

Please describe what you expect to happen (but doesn’t):

Old snapshots get deleted.
Some form of automatic recovery of corrupted chuncks.

Please describe what actually happens (the wrong behaviour):

ERROR DOWNLOAD_DECRYPT Failed to decrypt the chunk BIG_HASH: The storage doesn’t seem to be encrypted.

Performed server.COPY to offsite, succeeds, but PRUNE offsite still fails with same error.

Performed offsite.CHECK -a -tabular, same error in the same chunck.

Searched for the file on offsite storage, it existed but was all “00 00 …”.
(There was a *.fsl with same name and non-00 content.)
Deleted it.
Re-ran server.PRUNE offsite, success, with log:

WARN DOWNLOAD_RESURRECT Fossil SAME_BIG_HASH has been resurrected

In the meantime the consumed storage kept increasing because of 3 months without prunes.
If it was this easy for duplicacy to recover a non-existent file, shouldn’t it try to repair the original 00’ed chunck failing to decrypt?

Keep up the good work!!

#2

DOWNLOAD_RESURRECT only means to rename the fossil file (the one with the .fsl suffix) to the original file name, but if the original file exists Duplicacy will never attempt to repair it.

I wonder why the original file has all 0’s. Are you running prune and copy in parallel?

#3

Not in parallel, no, at least not on purposes.
Both are called sequentially in the same task (scheduled) configured to “not start if there’s already one”.

However:

  • This task is run in background and idle CPU+IO priority.
  • The task is configured to be killed if running for 1 day (usually takes 1h though).
  • The machine can be (forced) rebooted at any time. Power loss are not uncommon.
  • The offsite server can also be rebooted at any time.
  • The offsite doesn’t perform any Backup/Copy/Prune over that storage, but sporadically performs Checks (using duplicacy-web).

Even with this conditions it ran (almost) perfectly for more than 1 year with no intervention.
Congrats!

P.S.: I use @TheBestPessimist scripts, and got all the logs of all duplicacy operations stored since forever. If you think it’s worth to further diagnose, let me know and I’ll find the corresponding ones.