Google Drive checks constantly saying chunks can't be found

Support
#1

Hey there,

I have been backing up using Duplicacy Personal for about 3 months now, and it mostly works fine.

I have it set up to back up to a local drive, as well as to Google Drive.
I can authenticate and it backs up just fine to Google Drive, but it always says there are missing chunks, which makes me fearful if I ever need to restore.

I have verified that the chunks exist on Google Drive, and I have cleared the local cache to make sure it’s not a stale cache causing the issue.

I have also been through every prune log, and the chunks that it says don’t exist haven’t been listed in any log files saying they were removed or marked as fossils.

I can remove the snapshots and let it rebuild itself, but I really don’t want to upload close to 1TB of data from scratch again. It seems to affect almost every snapshot.

It seems as though Duplicacy just can’t see the files that it has created - again, I have 100% verified that they exist, have not been deleted and are present on Google Drive.

Pruning works as expected, when snapshots are over the age I’ve specified, it removes them, and it isn’t looking for the ones that have been removed - it is explicitly saying that chunks that 100% definitely exist, do not exist. I am willing to provide screenshots and proof of this, but I don’t believe it’s needed at this point.

I have been through the forum topic for fixing missing chunks, and have scoured the internet far and wide for this, and I can’t find anything conclusive. To my somewhat trained eye, it looks like a permissions/timeout issue (Google Drive operations do seem to take a long time to do anything, and I have reasonably fast internet), as in Duplicacy can’t see the files that it has created - it could also have something to do with the way it names files (the first two characters being the folder name, and the rest being the actual filename itself). If I search for the full name of the “missing” chunk, I can’t find it, but if I remove the first two characters, I can.

Happy to hear any advice anyone might have!

Cheers

#2

This is very interesting and puzzling.

Are there any other errors? Maybe if you run with -d flag you will get more clarity: perhaps missing chunk error is caused by google api failure that duplicacy did not handle/did not log and eventually higher level api that checks for chunk reported failure — as if chunk is missing.

How have you setup google drive? Right into folder under My Drive or with some shenanigans involving shared drives, custom google projects, and/or service accounts? There are at least two known issues with using google drive shared folders (user needs manager permissions, content manager is not enough, and there is a limit on 400,000 files in shared drive) — but there could be more.

#3

Hey, thanks for responding.

No other errors. I’ve actually reconfigured everything because I had a drive failure last night (coincidentally) so I am running a full backup to the cloud now anyway.

Google drive is configured normally using the GUI, just a regular Google Drive account with it going directly to a folder. I own the account, it’s not a shared folder or custom Google project.

If it fails again after this upload, I’ll run it with the -d flag and let you know of the output - looking at a 2 day upload currently, however :stuck_out_tongue:

I will note, this only seemed to start happening about a month after I set everything up, so it seems as if something broke and then it started playing up. I’m hoping that perhaps redoing the entire backup scheme will resolve it.

Thanks!

#4

This is one scenario when something happens and then the aftermath keeps flooding logs — it’s when prune operation is interrupted after deleting the chunks but before having a chance to delete snapshots. Then every check will complain about missing chunks — those that are referred by the ghost snapshot.

However in your case the situation is different — the file that does exist cannot be open (or found?) for some reason.

I’m wondering maybe list operation times out and returns partial list?

But we should definitely get to the root cause as re-doing the backup is hardly useful as it may happen again.

#5

Yeah, that is what I was thinking too. I’m running on Unraid on the latest official docker template. (the one you made)

At this point I need to wait for it to finish this upload before it will let me do anything, but I’ll let you know!

#6

The first two characters are stripped off the file name and used as the name of the subdirectory, so the chunk indeed exists in the storage but for some reason Duplicacy can’t see it.

Do you see two subdirectories with the same two-character name? If so then Duplicacy will only looks at one and not be able to find chunks in the other subdirectory. This happened to other users before: Repeatable Chunk check failure with google drive

1 Like
#7

Interesting - I didn’t check every folder, but the ones I searched only had one entry per name. For example, there was a file that was “missing” that started with b9, and there was only one b9 folder.

Unfortunately I had to delete everything and start over again due to a backup drive failure, and I’ve set up Google Drive as a copy of Local storage now, so I’ve completely started uploading everything from scratch. I know that isn’t an ideal solution for everyone, but if it starts happening again, I’ll come back to this forum and let you know so we can gather some diagnostics.

It is entirely possible that there were multiple entries for every failing snapshot, but it was literally every single snapshot/chunk that was failing the check, and I don’t recall seeing any duplicate entries.