How to interpret the -stats option output?

#1

Hello,

Could you please explain how to interpret the -stats option output for the backup and check commands?

> duplicacy backup -stats
Backup for E:\Data at revision 5 completed
Files: 4108 total, 6,642M bytes; 2460 new, 6,218M bytes
File chunks: 1383 total, 6,649M bytes; 243 new, 1,202M bytes, 1,123M bytes uploaded
Metadata chunks: 3 total, 1,160K bytes; 3 new, 1,160K bytes, 507K bytes uploaded
All chunks: 1386 total, 6,650M bytes; 246 new, 1,203M bytes, 1,123M bytes uploaded
Total running time: 03:52:53

I understand that I have 4108 files (6642MB) in the snapshot of them 2460 (6218MB) are new.
File chunks: 1383 (6649MB) total, of them 243 are new (Size of the chunks is 1202MB and 1123MB uploaded).
So even though I added 6218MB to the snapshot, due to deduplication only 1202MB was new content and of them only 1123MB was uploaded due to compression, is it right?

> duplicacy check -stats
Storage set to b2://snapshot1-storage
Listing all chunks
All chunks referenced by snapshot snapshot1 at revision 1 exist
All chunks referenced by snapshot snapshot1 at revision 2 exist
All chunks referenced by snapshot snapshot1 at revision 3 exist
All chunks referenced by snapshot snapshot1 at revision 4 exist
All chunks referenced by snapshot snapshot1 at revision 5 exist
Snapshot snapshot1 at revision 1: 1045 files (202,980K bytes), 179,490K total chunk bytes, 104K unique chunk bytes
Snapshot snapshot1 at revision 2: 1042 files (198,424K bytes), 179,489K total chunk bytes, 104K unique chunk bytes
Snapshot snapshot1 at revision 3: 1446 files (201,274K bytes), 180,699K total chunk bytes, 145K unique chunk bytes
Snapshot snapshot1 at revision 4: 1648 files (434,562K bytes), 401,626K total chunk bytes, 169K unique chunk bytes
Snapshot snapshot1 at revision 5: 4108 files (6,642M bytes), 5,555M total chunk bytes, 5,163M unique chunk bytes
Snapshot snapshot1 all revisions: 5,555M total chunk bytes, 5,555M unique chunk bytes

The results of the check command seems to conflict with the results of the backup commands.

On revision 5 there are 4108 files (6642MB, which is fine), but 5555MB of storage used. Of this storage 5163MB are unique.
Subtraction of revision 4 from 5 (unique bytes or not), gives around 5000MB of upload, which seems to conflict with the results of the backup command, can you explain (Can you also better explain the meaning of unique bytes)?

Thank you very much.

#2

The only explanation I can think of is that there is another repository backing up to the same storage with which the repository ‘snapshot1’ shares lots of chunks.

In this case, the -a option should be provided to include all repositories:

duplicacy check -a -stats

If there is only one repository then this is definitely a bug.

#3

Hello and thank you for your reply.

The results of

duplicacy check -stats

and

duplicacy check -a -stats

are the same. There is no other repository (I didn’t backup any other files to this bucket).
Also, here are the results of two more backup and check commands:

Backup for E:\Data at revision 8 completed
Files: 8445 total, 11,949M bytes; 2555 new, 4,758M bytes
File chunks: 2420 total, 11,957M bytes; 727 new, 3,770M bytes, 3,716M bytes uploaded
Metadata chunks: 4 total, 2,446K bytes; 4 new, 2,446K bytes, 1,033K bytes uploaded
All chunks: 2424 total, 11,959M bytes; 731 new, 3,773M bytes, 3,717M bytes uploaded
Total running time: 12:24:15

Snapshot snapshot1 at revision 1: 1045 files (202,980K bytes), 179,490K total chunk bytes, 104K unique chunk bytes
Snapshot snapshot1 at revision 2: 1042 files (198,424K bytes), 179,489K total chunk bytes, 104K unique chunk bytes
Snapshot snapshot1 at revision 3: 1446 files (201,274K bytes), 180,699K total chunk bytes, 145K unique chunk bytes
Snapshot snapshot1 at revision 4: 1648 files (434,562K bytes), 401,626K total chunk bytes, 169K unique chunk bytes
Snapshot snapshot1 at revision 5: 4108 files (6,642M bytes), 5,555M total chunk bytes, 507K unique chunk bytes
Snapshot snapshot1 at revision 6: 4109 files (6,643M bytes), 5,556M total chunk bytes, 508K unique chunk bytes
Snapshot snapshot1 at revision 7: 5890 files (7,191M bytes), 5,948M total chunk bytes, 714K unique chunk bytes
Snapshot snapshot1 at revision 8: 8445 files (11,949M bytes), 10,639M total chunk bytes, 4,691M unique chunk bytes
Snapshot snapshot1 all revisions: 10,641M total chunk bytes, 10,641M unique chunk bytes

and second backup and check:

Backup for E:\Data at revision 11 completed
Files: 9113 total, 12,958M bytes; 4 new, 9,408K bytes
File chunks: 2620 total, 12,985M bytes; 4 new, 9,408K bytes, 7,978K bytes uploaded
Metadata chunks: 3 total, 2,662K bytes; 3 new, 2,662K bytes, 1,122K bytes uploaded
All chunks: 2623 total, 12,988M bytes; 7 new, 12,071K bytes, 9,100K bytes uploaded
Total running time: 00:02:09

Snapshot snapshot1 at revision 1: 1045 files (202,980K bytes), 179,490K total chunk bytes, 104K unique chunk bytes
Snapshot snapshot1 at revision 2: 1042 files (198,424K bytes), 179,489K total chunk bytes, 104K unique chunk bytes
Snapshot snapshot1 at revision 3: 1446 files (201,274K bytes), 180,699K total chunk bytes, 145K unique chunk bytes
Snapshot snapshot1 at revision 4: 1648 files (434,562K bytes), 401,626K total chunk bytes, 169K unique chunk bytes
Snapshot snapshot1 at revision 5: 4108 files (6,642M bytes), 5,555M total chunk bytes, 507K unique chunk bytes
Snapshot snapshot1 at revision 6: 4109 files (6,643M bytes), 5,556M total chunk bytes, 508K unique chunk bytes
Snapshot snapshot1 at revision 7: 5890 files (7,191M bytes), 5,948M total chunk bytes, 714K unique chunk bytes
Snapshot snapshot1 at revision 8: 8445 files (11,949M bytes), 10,639M total chunk bytes, 1,033K unique chunk bytes
Snapshot snapshot1 at revision 9: 8445 files (11,949M bytes), 10,647M total chunk bytes, 1,033K unique chunk bytes
Snapshot snapshot1 at revision 10: 9110 files (13,027M bytes), 11,595M total chunk bytes, 70,292K unique chunk bytes
Snapshot snapshot1 at revision 11: 9113 files (12,958M bytes), 11,535M total chunk bytes, 9,100K unique chunk bytes
Snapshot snapshot1 all revisions: 11,608M total chunk bytes, 11,608M unique chunk bytes

Notice how the last revision in the results of the check command is always seems to be off and changes when there’s a newer revision (probably to the correct byte count). This indeed might be a bug.

#4 
> duplicacy backup -stats
Backup for E:\Data at revision 5 completed
Files: 4108 total, 6,642M bytes; 2460 new, 6,218M bytes
File chunks: 1383 total, 6,649M bytes; 243 new, 1,202M bytes, 1,123M bytes uploaded
Metadata chunks: 3 total, 1,160K bytes; 3 new, 1,160K bytes, 507K bytes uploaded
All chunks: 1386 total, 6,650M bytes; 246 new, 1,203M bytes, 1,123M bytes uploaded
Total running time: 03:52:53

It looks like there was an incomplete backup before this backup. Notice that only 246 out of 1386 chunks were new, and the total size of revision 5 is much larger than that of revision 4, so there must be a previous backup that uploaded most of remain chunks but somehow failed to complete.

Those unique chunk bytes look ok to me. Every time you upload a new revision, it will share some chunks with previous revisions so the unique chunk bytes of previous revisions will drop.

1 Like
#5

Thank you for your answer. Indeed there was an incomplete backup (actually several) since I use duplicacy on a home computer with standard residential internet connection which tends to stall. I now understand the reason for this discrepancy.

Thank you also for the explanation about the unique bytes.