Prune failure on Google team drives

[miab_03c559c2]

Hi all,

Want to see if anyone else has had problems with Google (Team) Drives and the prune operation.

Background

Have a duplicacy backup going to google team (shared drive). Backup took a while (total 500Gb) but after initial long period; the incremental is quite ok.

Have about 7 revisions and having the prune operation to delete oldest and prune the backup store.

What is happening

The web interface gives me a failed after about 15 minutes of run time. Looking at the logs I see this:

Running prune command from C:\Users\core/.duplicacy-web/repositories/localhost/all
Options: [-log prune -storage Google_Duplicacy -keep 0:1800 -keep 7:30 -keep 1:7 -a]
2020-07-24 09:38:57.111 INFO STORAGE_SET Storage set to gcd://0ALEn0ohi2D08Uk9PVA@Google_Duplicacy
2020-07-24 09:39:01.829 INFO RETENTION_POLICY Keep no snapshots older than 1800 days
2020-07-24 09:39:01.829 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 30 day(s)
2020-07-24 09:39:01.829 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 7 day(s)
2020-07-24 09:39:05.703 INFO SNAPSHOT_DELETE Deleting snapshot Google at revision 2
2020-07-24 09:39:05.935 INFO SNAPSHOT_DELETE Deleting snapshot Google at revision 3
2020-07-24 09:39:06.170 INFO SNAPSHOT_DELETE Deleting snapshot Google at revision 4
2020-07-24 09:39:06.412 INFO SNAPSHOT_DELETE Deleting snapshot Google at revision 5
2020-07-24 09:39:06.649 INFO SNAPSHOT_DELETE Deleting snapshot Google at revision 6
2020-07-24 09:41:10.858 INFO FOSSIL_COLLECT Fossil collection 1 saved
2020-07-24 09:50:12.987 INFO GCD_RETRY [0] Maximum number of retries reached (backoff: 64, attempts: 15)
2020-07-24 09:50:12.988 ERROR SNAPSHOT_DELETE Failed to delete the snapshot Google at revision 2: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions
Failed to delete the snapshot Google at revision 2: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions

So for some reason it is saying I don’t have permissions to delete the file. This is odd as duplicacy was the program that created the file and when I go via G Drive web interface I can delete files just fine.

Retrying leads to same result (the logs are mostly similar but subsequent runs complain of FOSSIL_GHOSTSNAPSHOT that should have been deleted and Chunk {hash} is already a fossil. Duplicacy saves a new fossil collection, but in the end the same failure occurs.

If safe, I suppose I could just delete the fossils by hand but obviously would be nicer to have a automatic handling of the issue.

Any thoughts?

Note

I did search for previously threads on Google issues and did see the below which has the same type of error message but in that thread the problem seemed to be someone uploading a lot (so exceeding daily limits) which is not the case here.

Googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded, Maximum number of retries reached (backoff: 64, attempts: 15)

Hi, I’ve been trying to perform an initial backup of about 4.5TB and 500k files to Google Drive for several days now, starting with 8 threads, then 6, and now even just 1, and every time I inevitably run into this at some point: [2019-08-12 03:22:32 PDT] Uploaded chunk 38492 size 181015734, 20.86MB/s 23:19:18 63.4% [2019-08-12 03:31:54 PDT] [0] Maximum number of retries reached (backoff: 64, attempts: 15) [2019-08-12 03:31:54 PDT] Failed to upload the chunk 9560062f248c9fac8743f024b64b19ea359…

[gchen]

Yes, you can delete the revision file manually in the Google Drive web interface. Do other revisions (3-6) have the same issue? Did you re-generate the token file after these backups were done?

[miab_03c559c2]

Hi Gang,

From the root folder of the backup I see the following:

• chunks
• fossils
• snapshots

When you say delete the revision file manually - do you mean the folders under snapshots ? I see sub folders named ‘1’ ‘2’ ‘3’ etc - so I just delete the whole of folders 3 to 6 then?

In terms of the token - it is working fine; I am doing a nightly backup and the logs report that is running fine without errors.

Given that I have other subsequent backups, is it also safe to take out the entire fossils folder?

[gchen]

1, 2, 3, etc should be files not folders.

I would delete 2 first, and then run the prune command again to see if 3-6 can be deleted.

[miab_03c559c2]

Yep, it is a file. Looks fine to hand delete the files:

• Delete file named ‘2’ by itself and the prune command is much the same but with one less INFO FOSSIL_GHOSTSNAPSHOT complaint.
• Afterward deleted the remaining 3-6 and with prune command doesn’t complain about ghost snapshots anymore and just has the following:

2020-07-27 07:38:53.515 INFO FOSSIL_COLLECT Fossil collection 6 found
2020-07-27 07:38:53.515 INFO FOSSIL_POSTPONE Fossils from collection 6 can't be deleted because deletion criteria aren't met
2020-07-27 07:38:53.515 INFO SNAPSHOT_NONE No snapshot to delete

I am the only one using this backend store and there is no concurrent tasks - so is it ok to delete the contents of the fossil folder too?

[miab_03c559c2]

Even if manually deleting the old snapshots/fossils is ok, is there any insight on why this is happening?

[saspus]

I had similar issue. The workaround is here: Prune Fails: GCD_RETRY: The user does not have sufficient permissions for this file