Occupied storage is almost twice as large as duplicacy backup

Hi

I searched the forum and found some similar problems, but no solution that could help me.

I’m using SFTP for two different backups jobs to store the data on a remote server.

According to check (log attached below) and duplicacy portal the backup occupies about 2.5TB.
However, my storage provider says that I occupy 4.5TB and according to size check via SFTP I came to the same value.

I have tried to delete as many snapshots as possible, but this has brought little change, although that should be far more than 1TB. Am I doing something wrong that the chunks are not deleted correctly after the prune job?

I am using two prune jobs with the following options:
-keep 0:180 -keep 7:30 -keep 1:7 -id truenas_personal -threads 4
-keep 0:30 -keep 7:7 -keep 1:1 -id pve_snapshots -threads 4

(I also tried the “-collect-only” and then “-delete-only” option, because I thought there might have any fossils)

Running check command from /cache/localhost/all
Options: [-log check -storage duplicacy -a -tabular]
2022-07-29 22:25:39.901 INFO STORAGE_SET Storage set to sftp://xxxxxx@xxxxxx.xxxxxx.xx:23/duplicacy
2022-07-29 22:25:40.584 INFO SNAPSHOT_CHECK Listing all chunks
2022-07-29 22:31:07.881 INFO SNAPSHOT_CHECK 2 snapshots and 40 revisions
2022-07-29 22:31:07.888 INFO SNAPSHOT_CHECK Total chunk size is 2332G in 498469 chunks
2022-07-29 22:31:08.869 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 1 exist
2022-07-29 22:31:09.826 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 7 exist
2022-07-29 22:31:10.718 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 11 exist
2022-07-29 22:31:11.587 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 12 exist
2022-07-29 22:31:12.453 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 13 exist
2022-07-29 22:31:13.348 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 14 exist
2022-07-29 22:31:14.214 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 15 exist
2022-07-29 22:31:15.077 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 16 exist
2022-07-29 22:31:16.037 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 17 exist
2022-07-29 22:31:17.004 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 18 exist
2022-07-29 22:31:17.940 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 19 exist
2022-07-29 22:31:18.867 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 20 exist
2022-07-29 22:31:19.820 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 21 exist
2022-07-29 22:31:20.769 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 22 exist
2022-07-29 22:31:21.711 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 23 exist
2022-07-29 22:31:22.661 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 24 exist
2022-07-29 22:31:23.647 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 25 exist
2022-07-29 22:31:24.597 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 26 exist
2022-07-29 22:31:25.536 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 27 exist
2022-07-29 22:31:26.493 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 28 exist
2022-07-29 22:31:27.440 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 29 exist
2022-07-29 22:31:28.375 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 30 exist
2022-07-29 22:31:29.317 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 31 exist
2022-07-29 22:31:30.270 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 32 exist
2022-07-29 22:31:31.217 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 33 exist
2022-07-29 22:31:32.174 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 34 exist
2022-07-29 22:31:33.129 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 35 exist
2022-07-29 22:31:34.081 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 36 exist
2022-07-29 22:31:35.017 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 37 exist
2022-07-29 22:31:35.958 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 38 exist
2022-07-29 22:31:36.976 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 39 exist
2022-07-29 22:31:38.013 INFO SNAPSHOT_CHECK All chunks referenced by snapshot truenas_personal at revision 40 exist
2022-07-29 22:31:38.382 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 28 exist
2022-07-29 22:31:38.881 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 30 exist
2022-07-29 22:31:39.292 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 31 exist
2022-07-29 22:31:39.710 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 32 exist
2022-07-29 22:31:40.097 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 33 exist
2022-07-29 22:31:40.485 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 34 exist
2022-07-29 22:31:40.870 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 35 exist
2022-07-29 22:31:41.252 INFO SNAPSHOT_CHECK All chunks referenced by snapshot pve_snapshots at revision 36 exist
2022-07-29 22:32:41.905 INFO SNAPSHOT_CHECK 
          snap | rev |                          | files |    bytes | chunks |    bytes |   uniq |    bytes |    new |    bytes |
 pve_snapshots |  28 | @ 2022-07-21 01:03       |   126 | 579,982M | 117565 | 581,452M |  14419 |  71,848M | 117565 | 581,452M |
 pve_snapshots |  30 | @ 2022-07-23 01:04       |   126 | 582,132M | 117971 | 583,606M |      4 |   6,993K |  14825 |  74,002M |
 pve_snapshots |  31 | @ 2022-07-24 01:03       |   126 | 588,810M | 119163 | 590,304M |      5 |   4,373K |  15021 |  74,456M |
 pve_snapshots |  32 | @ 2022-07-25 01:03       |   126 | 591,289M | 119913 | 592,794M |      4 |   3,382K |  15344 |  74,856M |
 pve_snapshots |  33 | @ 2022-07-26 01:03       |   126 | 593,219M | 120135 | 594,728M |      0 |        0 |  15220 |  75,362M |
 pve_snapshots |  34 | @ 2022-07-27 01:03       |   126 | 593,219M | 120135 | 594,728M |      0 |        0 |      0 |        0 |
 pve_snapshots |  35 | @ 2022-07-28 09:23       |   126 | 595,253M | 120682 | 596,766M |      0 |        0 |  15368 |  76,031M |
 pve_snapshots |  36 | @ 2022-07-29 01:03       |   126 | 595,253M | 120682 | 596,766M |      0 |        0 |      0 |        0 |
 pve_snapshots | all |                          |       |          | 193343 | 956,162M | 193342 | 956,162M |        |          |

             snap | rev |                          |  files | bytes | chunks | bytes |   uniq |    bytes |    new |    bytes |
 truenas_personal |   1 | @ 2022-06-19 09:28 -hash | 164543 | 1334G | 261052 | 1227G |     49 | 101,228K | 261052 |    1227G |
 truenas_personal |   7 | @ 2022-06-26 01:00       | 167966 | 1339G | 261879 | 1231G |     45 | 138,301K |    876 |   3,659M |
 truenas_personal |  11 | @ 2022-06-30 01:00       | 168006 | 1339G | 261886 | 1231G |     38 | 113,036K |     52 | 154,301K |
 truenas_personal |  12 | @ 2022-07-01 01:00       | 168026 | 1339G | 261890 | 1231G |     33 | 101,661K |     43 | 123,819K |
 truenas_personal |  13 | @ 2022-07-02 01:00       | 168034 | 1339G | 261890 | 1231G |     26 |  94,196K |     33 | 111,515K |
 truenas_personal |  14 | @ 2022-07-03 11:28       | 168086 | 1339G | 261896 | 1231G |     24 |  76,261K |     37 | 112,165K |
 truenas_personal |  15 | @ 2022-07-04 01:00       | 168096 | 1339G | 261905 | 1231G |     31 |  85,623K |     33 |  93,888K |
 truenas_personal |  16 | @ 2022-07-05 01:00       | 168144 | 1339G | 261893 | 1231G |     24 |  88,880K |     28 | 109,583K |
 truenas_personal |  17 | @ 2022-07-06 01:00       | 168681 | 1456G | 285567 | 1340G |     38 |  87,217K |  23726 | 112,378M |
 truenas_personal |  18 | @ 2022-07-07 01:00       | 168749 | 1456G | 285568 | 1340G |     30 | 104,252K |     45 | 135,794K |
 truenas_personal |  19 | @ 2022-07-08 01:00       | 174896 | 1475G | 286353 | 1344G |     24 |  96,958K |    816 |   4,272M |
 truenas_personal |  20 | @ 2022-07-09 01:00       | 175222 | 1475G | 286373 | 1344G |     27 |  99,738K |     51 | 166,678K |
 truenas_personal |  21 | @ 2022-07-10 01:00       | 175235 | 1475G | 286383 | 1344G |     33 |  93,063K |     41 | 127,544K |
 truenas_personal |  22 | @ 2022-07-11 01:00       | 175280 | 1475G | 286386 | 1345G |     30 |  92,681K |     46 | 157,592K |
 truenas_personal |  23 | @ 2022-07-12 01:00       | 175386 | 1475G | 286394 | 1345G |     34 |  91,068K |     43 | 110,372K |
 truenas_personal |  24 | @ 2022-07-13 01:00       | 175403 | 1475G | 286387 | 1345G |     22 |  81,332K |     27 | 103,070K |
 truenas_personal |  25 | @ 2022-07-14 01:00       | 175407 | 1475G | 286395 | 1345G |     31 | 105,878K |     33 | 115,580K |
 truenas_personal |  26 | @ 2022-07-15 01:00       | 175497 | 1475G | 286384 | 1345G |     22 |  85,751K |     36 | 133,675K |
 truenas_personal |  27 | @ 2022-07-16 01:00       | 175586 | 1475G | 286401 | 1345G |     29 |  78,708K |     39 | 119,305K |
 truenas_personal |  28 | @ 2022-07-17 01:00       | 175591 | 1475G | 286402 | 1345G |     27 |  94,070K |     30 |  99,873K |
 truenas_personal |  29 | @ 2022-07-18 01:00       | 175593 | 1475G | 286410 | 1345G |     28 |  96,061K |     38 | 111,277K |
 truenas_personal |  30 | @ 2022-07-19 01:00       | 175753 | 1475G | 286409 | 1345G |     22 |  80,279K |     31 | 106,569K |
 truenas_personal |  31 | @ 2022-07-20 01:00       | 175775 | 1475G | 286426 | 1345G |     37 | 101,359K |     44 | 116,304K |
 truenas_personal |  32 | @ 2022-07-21 01:00       | 175877 | 1475G | 286419 | 1345G |     28 |  94,179K |     32 | 108,194K |
 truenas_personal |  33 | @ 2022-07-22 01:00       | 175928 | 1475G | 286425 | 1345G |     34 |  97,640K |     41 | 113,727K |
 truenas_personal |  34 | @ 2022-07-23 01:00       | 176454 | 1475G | 286443 | 1345G |     21 |  83,608K |     63 | 238,342K |
 truenas_personal |  35 | @ 2022-07-24 01:00       | 176460 | 1475G | 286451 | 1345G |     27 |  98,346K |     30 | 101,876K |
 truenas_personal |  36 | @ 2022-07-25 01:00       | 176473 | 1475G | 286451 | 1345G |     27 |  89,773K |     32 | 107,944K |
 truenas_personal |  37 | @ 2022-07-26 01:00       | 176481 | 1475G | 286447 | 1345G |     22 |  96,849K |     25 | 101,903K |
 truenas_personal |  38 | @ 2022-07-27 01:00       | 176490 | 1475G | 286449 | 1345G |     22 |  88,458K |     24 | 101,399K |
 truenas_personal |  39 | @ 2022-07-28 01:00       | 176552 | 1571G | 304037 | 1395G |     24 |  85,208K |  17636 |  51,331M |
 truenas_personal |  40 | @ 2022-07-29 01:00       | 176625 | 1571G | 304046 | 1395G |     33 | 112,455K |     33 | 112,455K |
 truenas_personal | all |                          |        |       | 305116 | 1398G | 305115 |    1398G |        |          |

Thanks for your help

-Patrick

Tis likely you have a lot of unreferenced chunks - you can run prune -exhaustive to clear them up.

2 Likes

Thank you very much! That took a while but helped alot.

I totally misunderstood this function. Since i did not have any incomplete backups and did not see any warnings about incomplete chunks in my logs - not even with -collect-only, I did not try this function (use sparingly, according to the instructions).

What is best practice to prune my backups? Use -exhaustive once a week? As fast as the data has grown I could imagine that once a month is even too little.

If this would be a repeatable problem (rapid accumulation of unreferenced chunks), you may want to try to figure out why is that happening. Outside of incomplete backups or manual manipulations with the storage (e.g. removing stuff under /snapshots etc), I don’t quite see why you should be accumulating several TBs worth of unreferenced chunks.

IMHO it is better to fix the problem at the source rather than fighting the results with exhaustive pruning.

1 Like

Thank you sevimo. I will keep watching this, but I have no idea how this happend.

Quite interesting was that the 2TB extra data comes from the snapshot “truenas_personal” (total size 1.4TB, see above in the log). However, on average only about 100MB of new data is added daily. That’s what surprises me, that in a little over a month such a big overhead has emerged.