doing a check - files with my storage at B2
There is a 224GB at this storage for thsi backup.
“Listing all chunks”
and now no reply for 10 minutes… how long should I wait? hours? days?
It’s hard to tell if anything is happening…
Well, its been 5 hours with no response, but I’ve been getting usuage alerts from B2… so something is happening…?
There is really something wrong, but there are many variables that can influence.
See the result of a check command of a bucket of mine on B2 with ~ 200GB, lasted only a few minutes:
2019-09-16 22:43:07.348 INFO REPOSITORY_SET Repository set to [redacted]
2019-09-16 22:43:07.362 INFO STORAGE_SET Storage set to b2://[redacted]
2019-09-16 22:43:11.406 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2019-09-16 22:43:11.631 TRACE BACKBLAZE_CALL [0] URL request 'HEAD https://f001.backblazeb2.com/file/[redacted]/nesting' returned status code 404
2019-09-16 22:43:11.632 INFO SNAPSHOT_CHECK Listing all chunks
2019-09-16 22:43:11.632 TRACE LIST_FILES Listing chunks/
2019-09-16 22:45:33.436 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot [redacted]
2019-09-16 22:46:56.057 INFO SNAPSHOT_CHECK 1 snapshots and 83 revisions
2019-09-16 22:46:56.063 INFO SNAPSHOT_CHECK Total chunk size is 192,782M in 157212 chunks
2019-09-16 22:46:56.878 INFO SNAPSHOT_CHECK All chunks referenced by snapshot [redacted] at revision 1 exist
2019-09-16 22:46:58.308 INFO SNAPSHOT_CHECK All chunks referenced by snapshot [redacted] at revision 2 exist
...
2019-09-16 22:48:28.839 INFO SNAPSHOT_CHECK All chunks referenced by snapshot [redacted] at revision 83 exist
2019-09-16 22:48:29.783 INFO SNAPSHOT_CHECK All chunks referenced by snapshot [redacted] at revision 84 exist
...
snap | rev | | files | bytes | chunks | bytes | uniq | bytes | new | bytes |
[redacted] | 1 | @ 2019-06-17 16:38 -hash | 1 | 426,493K | 362 | 427,053K | 5 | 2,644K | 362 | 427,053K |
[redacted] | 2 | @ 2019-06-17 17:38 | 11 | 1,551M | 1283 | 1,554M | 1283 | 1,554M | 1283 | 1,554M |
...
[redacted] | 83 | @ 2019-08-17 11:17 | 4 | 1,926M | 1639 | 1,929M | 1639 | 1,929M | 1639 | 1,929M |
[redacted] | 84 | @ 2019-08-17 14:57 | 6 | 1,999M | 1564 | 2,006M | 1564 | 2,006M | 1564 | 2,006M |
[redacted] | all | | | | 135831 | 166,373M | 135831 | 166,373M | | |
```towerbr, if you just ran check, that is significantly different form check -files
OK, so its been running 8+ hours, with no change:
“listing all chunks”
is all it says. Would be nice to have some further feedback…
This is 2.1.0 GUI on Windows.
I can tell on B2 and my CPU that it is doing something… as I have CPU utilization and duplicacy is downloading lots of packets (about 10Mbp-100Mbps download).
Looks to me like it is downloading everything from B2 to check it. And that makes sense, since it must download everything to check the hash, right?
So if it is downloading 224GB at 50 Mbps… that might take 10+ hours…
Will I get nothing from the console until then?
I’ll check again in the AM… should be done by then… I hope!
Hey, something popped up!
It’s been 10 hours now and it said so far…
C:\Users\Carl>"C:\Program Files (x86)\Duplicacy\duplicacy_win_x64_2.1.0.exe" check -files
Storage set to b2://XXXXXXXXXXXXXXXXXX
Listing all chunks
All files in snapshot MOTOCAT-Carl-Carl at revision 1 have been successfully verified
but it is continuing to churn along.
Strangely, it appears to have downloaded significantly more than 225GB… and it is still going. (The bucket only has 225GB in it.)
Can anyone tell me what is going on? I’m going to bed for at least 8 hours now…
I’m curious what it will say in the morning!
You could be hitting the bug mentioned in this issue:
I suspect that it checks each revision without remembering any of the chunks it checked in previous revisions. I started a “check -files -all” on my 500Gig local storage (external USB drives) two days ago and it’s still going, Each revision seems to take the same time even tho I know that many are very similar.
Ah, didn’t notice the -files parameter, sorry, ignore my post. 
Well, just checked this morning, and I hit the 1TB download cap! and B2 stopped stuff because of my limit.( My limit was set at $10 for this test.)
It does not make sense as there was only 225GB of data in that bucket. Why would it download 1TB?
Here is how the console ended:
C:\Users\Carl>"C:\Program Files (x86)\Duplicacy\duplicacy_win_x64_2.1.0.exe" check -files Storage set to b2://XXXXXXXXXXXXXXX Listing all chunks All files in snapshot MOTOCAT-Carl-Carl at revision 1 have been successfully verified All files in snapshot MOTOCAT-Carl-Carl at revision 39 have been successfully verified Chunk 60dcc64b322bb0e9c16281ec3313db14be4c70a4c0ef6ca567a63e748e443d77 can't be found
I assume the “can’t be found” may have happened when B2 limited out?
So was this working properly, and it’s just a giant project?
or is something broken in my older SW version and I just need to upgrade?
Or is check -files foolish with large repositories?
Tips and ideas? How do I verify my backup?
check without the -files option will make sure that all of the chunks needed are there. I personally don’t see a need to use the -files option on Backblaze B2: I trust that they are storing my files reliably.
One possible thing to do if you are worried about the files not being stored reliably (or weird network errors…) is to switch the backup to another storage (another bucket) and when that one is done nuke the first storage. Then there’s no cost to download and it will refresh your chunks even if they were corrupted. (Tho it would cost up to double for the storage itself.)
A way to check the backup and restore process is to restore just some part of your backup. If you restore less than a Gig it’s free from Backblaze B2. I do a partial restore now and then and locally difference that restore with the original files.
check -files currently will download every file in every revision. It is not smart enough to skip identical files in other revisions that have already verified. Therefore, it may download more data than the total size of the storage.
1 Like
Hmm… so it seems check -files isn’t a great plan with 200 revisions of snapshots.
If I understand correctly, does that mean it is going to download every file for the backup for every revision? So that means 200 times my approx 200GB repository/backup?
If so, can I instead just check the latest revision, and will that check all the files that existed in my repository at the last backup/revision? (Not just the latest file changes, right?)
I mean, I really am just trying to check the integrity of the backup of my machine as it exists now. I don’t need to check the integrity of previous backups / changes/ deletes etc.
check -files for the latest revision should be enough. Every revision is a full snapshot, so yes, it checks all files as a whole, not as incremental changes.
Thank you!
I’ll try this soon…
Ok, well, that didn’t work.
I was attempting to check just the latest revision, hoping that would match the approximate size of my current repository.
I ran this:
C:\Users\Carl>“C:\Program Files (x86)\Duplicacy\duplicacy_win_x64_2.1.0.exe” check -r 3896 -files
Storage set to b2://Mxxxxxxxxxxxxxxxxxxxxxxxx
Listing all chunks
And it ran all night, downloading 750+GB, when the repository is only 225GB. And never finished, as I stopped it at that point.
So… any idea what I am doing wrong?
Perhaps it has something to do with the fact I am using my one PC to backup 3 repositories… the PC itself (225GB), network drive “V” (800GB) and another network drive “W” (500GB).
Could it somehow be doing through all the repositories instead of just the PC repository?
When I do list it only lists the PC repository, “Carl”, not the others, “V” and “W”.
In fact, I can’t get it to list the snapshots for the other repositories… but I know they are being backed up! When I do:
C:\Users\Carl>“C:\Program Files (x86)\Duplicacy\duplicacy_win_x64_2.1.0.exe” list -id MOTOCAT-Carl-V
It replies only with the storage having been set for the PC, “Carl”, not the correct one for “V”, and returns no snapshots:
Storage set to b2://xxxxxxxxxxxxxxxx-CARL
I’m feeling lost…
Very likely, yes.
The documentation for check isn’t entirely clear and I haven’t used it much myself with the -files option (only done it manually a few times), but I believe it’ll check all repositories by default if you don’t specify one via -id, e.g.:
duplicacy -log check -files -id Carl -r 3896
This is because, by default, the list command only shows revisions for the repository from which you run it from. To see the others, use:
duplicacy list -all
Not sure why it didn’t show anything when you specified the -id (maybe a typo?) but it doesn’t take too much extra time to just list everything in there with -all.
duplicacy list -all
This doesn’t show the other repositories for me, only the PC “Carl” one… any other ideas?
I could try the check with the -id as you suggest… but since I can’t even list the other repositories, I’m not convinced that is the issue. Also, when I do the check, it shows the proper storage for PC “Carl”… and never gets out of that…
Odd. Have you other repositories completed at least one backup yet? If so, double-check they’re backing up to the same storage URL (bucket?) as the one on the PC. Ofc this doesn’t explain why it’s downloading a lot of data even when you’ve specified a particular revision of a particular repository.
Either backups for other repositories never completed, or you used the same backup id for all repositories. Check the snapshots directory on your storage to see if there are files named 1, 2, etc under subdirectories.
Replies to the comments below:
All three backups are running, under three tabs in the GUI.
Logs look good for all three backups.
I do not share buckets for the separate backups. Could that be a issue?
preferences for the PC
"id": "MOTOCAT-Carl-Carl", "storage": "b2://MotoCat-Garage-PC-CARL",
preferences for V
"id": "MOTOCAT-Carl-V", "storage": "b2://SynologyMotoNAS-VAULT",
preferences for W
"id": "MOTOCAT-Carl-W", "storage": "b2://SynologyMotoNAS-SERVER",
The snapshots directory in each bucket look good. there is 1, and then higher numbers in the thousands, (as 2, 3 etc. have been pruned).
The list -all command still only shows the result for the one storage:
“C:\Program Files (x86)\Duplicacy\duplicacy_win_x64_2.1.0.exe” list -all
Storage set to b2://MotoCat-Garage-PC-CARL
Snapshot MOTOCAT-Carl-Carl revision 1 created at 2018-04-16 20:29 -hash
Snapshot MOTOCAT-Carl-Carl revision 39 created at 2018-04-23 21:00
…
Snapshot MOTOCAT-Carl-Carl revision 3899 created at 2019-09-19 12:00
Other ideas?