All backups missing chunks

Support
#1

Continuing the discussion from Prune never completes:

My issue seems to be bigger than failure to prune. I also have several terabytes of early deletions on gcp (EDIT: and I am missing chunks from every snapshot). There should be none because I keep all snapshots for at least 90 days to avoid early deletion charges (if I’m going to pay for it, it might as well still be stored).

I am concerned that I don’t have a single valid backup of my data.

Prune command (backup runs daily)
duplicacy.exe -log prune -keep 0:360 -keep 30:120 -keep 7:92 -keep 1:91 >> %logDir%\duplicacy_prune_%filesafetimestamp%.log

#2

The prune logs are all stored under .duplicacy/logs and you may be able to find out why a chunk is missing. If the missing chunk doesn’t show up in any of those logs, then it could have been deleted due to other reasons.

You can try restoring the missing chunk by modifying the repository id and running an initial backup. Just modify the .duplicacy/preferences file to use a new repository id and run duplicacy backup. If this missing chunk is even missing from the latest snapshot then I think there is a large chance that you can recover it using this method.

#3

All of these issues is starting to cost me money on my gcp due to all of the hits. Is it possible not to reinitialize the repo and just re-upload these chunks? Maybe modify the last snapshot to exclude it or something… or make it think they were pruned and need to be reuploaded?

#4

Wait I think I see what you are saying and I guess it doesn’t seem that bad. That will take a long time because it’s a lot of data though… also, that will mess with my pruning since it will see two backups in the storage.

When you say I need to run an initial backup, do you mean something other than my normal backup command?

#5

I would suggest not running prune until the new initial backup is done. No, you don’t need any special command for the initial backup, just the regular duplicacy backup. There is no previous backup for an initial backup, so for every chunk it encounters a lookup is performed to check if the chunk exists in the storage. In contrast, a subsquent backup always assumes all chunks referenced by last backup exit so no lookup is performed on any of these chunks.

#6

So before this post yesterday, I had added a filter for “System Volume Information” and tried the backup again to see if it fixed the missing chunks somehow. When the backup was run, it seemed to try to upload a bunch of files that I had thought were already backed up and took forever by comparison to most days. Afterwards, the verification only showed 2 chunks missing now. There were some more warnings about some files it doesn’t have permissions to. I have fixed these permissions and will run the backup and verify again, but perhaps the issue stemmed from having files it failed to upload this whole time… for prune and everything. We will see. I will report my findings here.

On a side note, I’m not sure why adding a filter made it try to re-upload a bunch of files… Was it skipping these before maybe? I may check previous snapshot listings to see if these exist.

#7

It did indeed upload more files that I thought were already included once I added more filters and fixed permissions to an unrelated folder (not the one I expected to be uploaded).

Reverifying now. I still have one file that is a problem, but I just added it to the filter list, so I may need to do this once more if it does not verify. Stupid file ends with a “.” which Windows does not like. Tired a few things like vb script to change it… nothing has worked so far.

2018-08-23 13:07:00.883 WARN SKIP_FILE File Shares/Access/Media/GoogleDrive/MyFirstGoogleAppsScript. cannot be opened

Anyway… is it normal for the backup to skip files if other files in the backup have issues?

#8

Finally the last 100 snapshots are all clean. The first 299 snapshots are all missing chunks likely from incomplete prunes. What is the recommendation now? Run the prune command normally and hope it completes this time? Run with special options?

PS
Should I just manually delete the folder on gcp for the new repository name after changing it back?

#9

Everything has been fixed. Prune still fails