Copy to GDrive and missing chunks

Support
#1

Hi Guys,

I’ve got a question about copy command.

I’ve got my local repository that I’m periodically copy-ing to the remote GDrive.

Repository is quite large (~5TB), and it took couple of days to transfer initially. Neverthenless, now, copy takes much less time, since this repositiory doesn’t change very often, and incremental changes are much smaller.

I’ve wanted to check integrity of both of these repositiories, so I’ve ran check command on both, and it looks like my local one is fine.
However, GDrive respository is missing something:

Chunk 08d10c13889120e215dddb4ce47be51384432e13d8be42bd54090190c0d28d53 can't be found

And indeed, I’ve checked GDrive manually, and there isn’t any 08 direcotry there, so this chunk is actually missing.

However, when I’m running copy command, it works flawlessly, and doesn’t throw any error (I guess that it thinks that older snapshots are fine, so it’s processing only new ones).

I’ve got 2 questions:

  • What might have caused this chunk gone missing? I haven’t deleted these chunks on GDrive on my own, so is it possible that duplicacy haven’t copied it in the first place?
  • Can I somehow repair this situation, besides manually checking which chunks are missing from copy command and copying them one by one?
#2

It is impossible for a 5TB storage to not have a 08 directory. There should be around 1.25 million chunks in a 5TB storage and each of the xx subdirectory should contain about 4.8K chunks. A missing subdirectory means something went wrong.

You can fix it by manually removing all files under the snapshots directory in Google Drive and then run a copy job again which will only copy missing chunks.

#3

This is exactly what I’ve tried by myself - but it looks like this new “copy” command has skipped it again - still no “08” folder on my GDrive.

I’ve grepped outpot of this copy command, and it looks like it have skipped all the chunks, and have uploaded anything.

Is it possible that some kind of cache is breaking it?

#4

You really should have a 08 directory in your Google Drive somewhere - otherwise, 1/256th of your chunks are missing, and Duplicacy would report more than just one chunk.

Google Drive won’t necessarily show those directories in alphabetical order, and you may have to scroll down and wait for it to load.

At what stage does a check operation say that the chunk can’t be found? Is there anything else in the output log?

1 Like
#5

oookkay, that’s interesting:

Indeed - 08 probably was there before, but the alphabetical order mislead me here, and I haven’t found in the first place. Now I can agree that 08 is here right now.

Anyways, I’ve deleted besides deleting snapshots directory on GDrive (that haven’t helped much as I’ve noted in the previous post), I’ve deleted cache from the .duplicacy folder stored locally.

Than I’ve re-ran copy command, and this time it have copied ~150 chunks (but not 08d10c13889120e215dddb4ce47be51384432e13d8be42bd54090190c0d28d53 that was mentioned by check as missing last time).

Right now check command ran on the GDrive storage doens’t throw any error.

Can someone explain what exactly might have caused an error here?

And why check is saying that everything is fine even when 08d10c138891... wasn’t transferred (it’s hard to use GDrive’s webui with so many files, but it looks like 08 folder is missing d10c13889120e215dddb4ce47be51... file as well) ?

#6

You probably had a revision on GDrive that was different from the local storage (but with the same revision number).

The copy command didn’t check if the contents of the revisions were identical; when it saw the same revision number it thought the two revisions were the same and would not copy from the source to the destination.

Moreover, the copy command doesn’t update the local cache of the destination storage. So after you removed all revisions from the destination storage, the copy command only copied the revision from the source to the destination storage, and the revision file in the cache for GDrive became stale. The check command read the cache revision which still references the 08d1... chunk.

So deleting the local cache is the key to fixing the issue.