Prune never completes

[Charles]

Okay, so this might be an odd case because I didn’t run prune for about a year. When I finally started running it, it would run for a while, eventually giving an error:
“Failed to locate the path for the chunk […] net/http: TLS handshake timeout”

I downloaded the latest version, ran a backup, then a prune.

The latest version 2.1.1 seems to have a less verbose prune, but it decided to resurrect some of the chunks and got as far as snapshot 294 the first time on the new version

I then ran another backup, and another prune and got as far as 295 with no other output than “Deleting snapshot [snapshot] at revision 295”

EDIT:
Correction, there may be a pattern here. The second prune actually didn’t end until after it once again said that “Chunk [blah] does not exist in the storage” so maybe it is erroring out at that point.

A check command reports that there are a lot of chunks that do not exist at revision 1 for this storage.

Additional information: the backup is roughly 6.4 TB to GCP cold storage

Please advise

[Charles]

PS
I did look at this link you sent me, but if the solution is in there, I’m not really sure what it is

Deleted chunk during backup

Right, the prune command erases/fossilizes chunks first and then delete the snapshot files, so if the connection is lost before the snapshot files are deleted then these snapshot files will reference missing chunks. The other way is better – it would just leave some unreferenced chunks in this case.

[Charles]

Okay. One last thing I noticed… Each prune since the day I started running it has gotten 1 snapshot revision further each day. It first failed at revision 280, then 281 the next day and so on. Not entirely sure what this means.

EDIT
An old issue I opened when I first started using Duplicacy comes to mind. I just found it and found the final solution that you came up with. Could it be that the retry algorithm implemented for backups also needs to be applied for prune?

Backup failing for Google Cloud Storage

Although the official GCS client library claims that a retry mechanism is in place for all methods, after some code digging I came to the conclusion that at least the upload method doesn’t support it because of the way the code is written. I’ll add a retry logic to the outer layer in the GCS storage backend. Should be done by the end of tomorrow.

[gchen]

You should run the check command first to find out which revisions have missing chunks. Because the check command bails out on the first missing chunk, you’ll need to run it multiple times and specify different revision ranges each time. Then you’ll need to manually delete affected revision files from the storage. Once the check -a command passes, you can run duplicacy prune -exhaustive -exclusive to remove unreferenced chunks (make sure no backup is running).

[Charles]

This seems impractical for nearly 400 revisions

[Charles]

I suppose at the expense of revision history, I could mass delete all but the last few snapshots in one swoop. How do you “manually” delete them? Is there a command, or do you mean, just delete the snapshot files?

[Charles]

Even deleting these by hand in gcp seems impractical. They are not in numerical order because they aren’t padded with zeroes and you have to click these one at a time. Even if I am able to remove these, I am still going to have a problem with the prune command timing out and not having a proper retry mechanism. Wouldn’t you rather fix the issue and use my data as a test to make sure it works?

[gchen]

Please try the new 2.1.1 release (commit e8b892) available from the releases page. It should work far better than before:

• This simple change might allow you to continue checking other snapshots once a missing chunk is identified. Some chunks are chunks of other chunks and if they are missing then it is hard for Duplicacy to continue. But there aren’t many of these chunks so I think in most cases the change will work for you.
• [This change] will skip (and delete) fossil collections created by aborted prune commands.

If GCP keeps timeout for the prune command, I wonder, for a temporary workaround, if you can turn on the s3 interoperability and switch to the s3 backend instead?

[Charles]

Apparently I’m missing chunks from every backup including the most recent even though prune hasn’t even been running for a while…

EDIT: I also added some debugging lines to the application and rebuilt it. I don’t know why it doesn’t seem to care about most missing chunks but then it does after a while and throws an error instead. It’s been a while since I’ve looked at this so I don’t remember what’s going on…

Please advise
This is a major issue

EDIT2
PS The debugging lines I added for prune are
lines 202 and 204 in duplicacy_gcsstorage.go
and
lines 235 and 238 in duplicacy_gsstorage.go