Check with missing chunks does not update stats?

Support
#1

OK, so I am running a long copy operation (several TBs and several days). So when I run check command on the destination storage it fails with missing chunks errors. This is not surprising, copy is a long way from completed.

What is surprising though is that this check run doesn’t seem to update GUI stats for the destination storage. It still shows data from a few days ago (last successful check?). The thing is, I can see that check correctly counted at least number of chunks and total size (it is in the debug log). Yet .stats file is untouched.

Is this intended behavior?

#2

This is a bug. Check shall not be checking incomplete snapshots. Also copy shall not upload snapshots before all its chunks are uploaded.

This stems from a simple requirement that interrupting any operation must leave the datastore in a consistent state.

#3

OK, there are two things here:

I mean, :d: copy probably doesn’t :wink: But I am using server-side copy which has nothing to do with :d:

Now that statement is a bit strange, what is the purpose of the check command then? However you come up with them, damaged snapshots will happen. It could be file-based copy in my case, it could be damaged chunks on complete snapshots. Regardless of the reason, I don’t see why check command wouldn’t update stats if it can figure out storage size and number of chunks (it certainly can in my case). I’d like to be able to monitor storage size at least - whether the state is consistent or not.

#4

If you’re doing copies outside of Duplicacy, you’re potentially leaving its database in an inconsistent state, so a check might fail and stop, and the GUI doesn’t gather all the proper information.

This probably shouldn’t be all that surprising. Stats should probably be relied on only when there’s a successful check, as the error could be anywhere - you’ve got more important things to deal with. :slight_smile:

You can fix this by copying the \chunks path first, and then \snapshots, or just wait for the copy to finish.

#5

But check does get the relevant information. The very first step it does is enumerating all chunks and fossils. This is sufficient to report storage size and number of chunks in storage. This data would be correct whether or not the storage itself is consistent or not with respect to snapshots.

Waiting is not an amazing option when operation can take more than a week. Also, whatever the reason is for failing check, I’d like to know storage size as right now it is quite possible to run out of quota without any indication from check. E.g. in my case storage is still reported at 15GB/3k chunks, while in reality it is about 2TB/400k and counting. This is not insignificant discrepancy.

Again, the main point is that check already has that information, it’s only a question of passing it up into stats. If someone is concerned that this data could be incorrect for inconsistent storages (though I don’t quite see how that would be the case), it could be protected by an argument a-la -persist, e.g. -update-stats-on-error.

#6

I know this, but it’s only one part of compiling the -stats, which the CLI simply doesn’t get to yet (as it discovered missing chunks), and thus the GUI is unable to interpret this data, as there is none.

Then you should have copied the \chunks first, and later the \snapshots :confused:

Or just relied on Duplicacy copy

You can probably temporarily fix this very easily by removing the affected snapshot files that were already copied, and copy/sync do them again at the end.

But you’re doing a manual file transfer, so you must have a manual method to determine disk quota (df, du etc…).

Point is, you’re creating an inconsistent state by mixing out-of-bound operations on backup storage and expecting Duplicacy to work around that for the sake of… statistics. IMO, really not worth the coding effort.

#7

This turned out to be a rather philosophical discussion. Bottom line is, missing chunks can happen for various reasons, not just doing OOB operations. You may not be able to fix missing chunks right away (if at all). If that is to happen, stats cannot be used, even though :d: actually collects them. The only stats that might not be correct are chunks/volume for individual snapshots, everything else including totals should be fine. Stats are not critical to :d: functionality, but these are useful.

Whether or not this is worth the effort to implement is a separate story, I wouldn’t expect it to be on top of the priority list by any means.