Serious problem with missing chunks

[bn1]

I have setup 13 backups configured, and a local storage using SFTP to a NAS (currently 2.87TB).

I did my initial backup a few days ago, and then for some I did a second backup. I then ran a check. The check worked fine:

Running check command from /cache/localhost/all
Options: [-log check -storage local -a -tabular]
2019-08-11 07:28:27.693 INFO STORAGE_SET Storage set to sftp://backup@mynas.local//share/CACHEDEV1_DATA/duplicacy
2019-08-11 07:28:28.149 INFO SNAPSHOT_CHECK Listing all chunks
2019-08-11 07:28:53.698 INFO SNAPSHOT_CHECK 13 snapshots and 19 revisions
2019-08-11 07:28:53.706 INFO SNAPSHOT_CHECK Total chunk size is 2671G in 564597 chunks
2019-08-11 07:28:53.795 INFO SNAPSHOT_CHECK All chunks referenced by snapshot appdata at revision 1 exist
2019-08-11 07:28:53.815 INFO SNAPSHOT_CHECK All chunks referenced by snapshot aaaaaa at revision 1 exist
2019-08-11 07:28:53.834 INFO SNAPSHOT_CHECK All chunks referenced by snapshot aaaaaa at revision 2 exist
2019-08-11 07:28:53.840 INFO SNAPSHOT_CHECK All chunks referenced by snapshot bbbbbb at revision 1 exist
2019-08-11 07:28:53.845 INFO SNAPSHOT_CHECK All chunks referenced by snapshot bbbbbb at revision 2 exist
2019-08-11 07:28:54.057 INFO SNAPSHOT_CHECK All chunks referenced by snapshot cccccc at revision 1 exist
2019-08-11 07:28:54.096 INFO SNAPSHOT_CHECK All chunks referenced by snapshot books at revision 1 exist
2019-08-11 07:28:54.140 INFO SNAPSHOT_CHECK All chunks referenced by snapshot install at revision 1 exist
2019-08-11 07:28:54.289 INFO SNAPSHOT_CHECK All chunks referenced by snapshot nvr at revision 1 exist
2019-08-11 07:28:54.290 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dddddd at revision 1 exist
2019-08-11 07:28:54.290 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dddddd at revision 2 exist
2019-08-11 07:28:54.291 INFO SNAPSHOT_CHECK All chunks referenced by snapshot eeeeee at revision 1 exist
2019-08-11 07:28:54.291 INFO SNAPSHOT_CHECK All chunks referenced by snapshot eeeeee at revision 2 exist
2019-08-11 07:28:54.328 INFO SNAPSHOT_CHECK All chunks referenced by snapshot ffffff at revision 1 exist
2019-08-11 07:28:54.368 INFO SNAPSHOT_CHECK All chunks referenced by snapshot ffffff at revision 2 exist
2019-08-11 07:28:54.368 INFO SNAPSHOT_CHECK All chunks referenced by snapshot gggggg at revision 1 exist
2019-08-11 07:28:54.368 INFO SNAPSHOT_CHECK All chunks referenced by snapshot gggggg at revision 2 exist
2019-08-11 07:28:54.453 INFO SNAPSHOT_CHECK All chunks referenced by snapshot music at revision 1 exist
2019-08-11 07:28:56.082 INFO SNAPSHOT_CHECK All chunks referenced by snapshot photos at revision 1 exist
2019-08-11 07:29:00.312 INFO SNAPSHOT_CHECK 
   snap | rev |                          | files |  bytes | chunks |  bytes | uniq |  bytes |  new |  bytes |
 bbbbbb |   1 | @ 2019-08-05 05:31 -hash |  4437 | 9,800M |   1617 | 7,473M |    3 |   586K | 1617 | 7,473M |
 bbbbbb |   2 | @ 2019-08-05 15:27       |  4422 | 9,800M |   1618 | 7,473M |    4 |   773K |    4 |   773K |
 bbbbbb | all |                          |       |        |   1621 | 7,474M | 1381 | 6,254M |      |        |

    snap | rev |                          | files |    bytes | chunks |    bytes |  uniq |    bytes |   new |    bytes |
 cccccc |   1 | @ 2019-08-05 05:38 -hash | 11064 | 293,700M |  55988 | 276,496M | 54837 | 271,044M | 55988 | 276,496M |
 cccccc | all |                          |       |          |  55988 | 276,496M | 54837 | 271,044M |       |          |

  snap | rev |                          | files |   bytes | chunks |   bytes |  uniq |   bytes |   new |   bytes |
 books |   1 | @ 2019-08-05 10:14 -hash | 20497 | 68,148M |  12913 | 60,424M | 12878 | 60,305M | 12913 | 60,424M |
 books | all |                          |       |         |  12913 | 60,424M | 12878 | 60,305M |       |         |

    snap | rev |                          | files |   bytes | chunks |   bytes |  uniq |   bytes |   new |   bytes |
 install |   1 | @ 2019-08-05 11:25 -hash |  3365 | 76,352M |  13978 | 66,502M | 13886 | 66,076M | 13978 | 66,502M |
 install | all |                          |       |         |  13978 | 66,502M | 13886 | 66,076M |       |         |

  snap | rev |                          | files |    bytes | chunks |    bytes |  uniq |    bytes |   new |    bytes |
   nvr |   1 | @ 2019-08-05 15:27 -hash |  3539 | 162,724M |  33165 | 163,243M | 33165 | 163,243M | 33165 | 163,243M |
   nvr | all |                          |       |          |  33165 | 163,243M | 33165 | 163,243M |       |          |

    snap | rev |                          |  files |   bytes | chunks |   bytes |  uniq |   bytes |   new |   bytes |
 appdata |   1 | @ 2019-08-08 02:41 -hash | 305454 | 80,634M |  10156 | 42,059M | 10052 | 41,626M | 10156 | 42,059M |
 appdata | all |                          |        |         |  10156 | 42,059M | 10052 | 41,626M |       |         |

  snap | rev |                          | files |   bytes | chunks |   bytes | uniq |   bytes |  new |   bytes |
   aaaaaa |   1 | @ 2019-08-05 02:51 -hash | 43176 | 29,628M |   5890 | 26,809M |    9 |  4,575K | 5890 | 26,809M |
   aaaaaa |   2 | @ 2019-08-05 11:24       | 43122 | 29,629M |   5886 | 26,809M |    5 |  4,668K |    5 |  4,668K |
   aaaaaa | all |                          |       |         |   5895 | 26,813M | 4937 | 22,259M |      |         |

    snap | rev |                          | files |    bytes | chunks |    bytes | uniq |    bytes | new |    bytes |
 eeeeee |   1 | @ 2019-08-05 02:50 -hash |  2754 | 557,144K |     75 | 288,450K |    3 |     282K |  75 | 288,450K |
 eeeeee |   2 | @ 2019-08-05 11:25       |  2753 | 559,082K |     76 | 289,497K |    4 |   1,329K |   4 |   1,329K |
 eeeeee | all |                          |       |          |     79 | 289,779K |   75 | 277,032K |     |          |

     snap | rev |                          | files |   bytes | chunks |   bytes | uniq |   bytes |   new |   bytes |
 ffffff |   1 | @ 2019-08-05 04:23 -hash | 14463 | 66,835M |  13383 | 64,933M |    2 |  1,557K | 13383 | 64,933M |
 ffffff |   2 | @ 2019-08-05 12:36       | 14461 | 66,835M |  13382 | 64,933M |    1 |  1,559K |     1 |  1,559K |
 ffffff | all |                          |       |         |  13384 | 64,934M | 7405 | 36,430M |       |         |

  snap | rev |                          | files |   bytes | chunks |   bytes | uniq |   bytes | new |   bytes |
 gggggg |   1 | @ 2019-08-05 05:34 -hash |    99 | 52,709K |     14 | 43,129K |    1 |     12K |  14 | 43,129K |
 gggggg |   2 | @ 2019-08-05 05:38       |    99 | 52,709K |     14 | 43,129K |    1 |     12K |   1 |     12K |
 gggggg | all |                          |       |         |     15 | 43,141K |   15 | 43,141K |     |         |

  snap | rev |                          | files |    bytes | chunks |    bytes |  uniq |    bytes |   new |    bytes |
 music |   1 | @ 2019-08-05 12:36 -hash | 23163 | 149,283M |  29318 | 132,946M | 29284 | 132,829M | 29318 | 132,946M |
 music | all |                          |       |          |  29318 | 132,946M | 29284 | 132,829M |       |          |

   snap | rev |                          |  files | bytes | chunks | bytes |   uniq | bytes |    new | bytes |
 photos |   1 | @ 2019-08-05 18:33 -hash | 167194 | 1923G | 384802 | 1834G | 376577 | 1795G | 384802 | 1834G |
 photos | all |                          |        |       | 384802 | 1834G | 376577 | 1795G |        |       |

    snap | rev |                          | files |    bytes | chunks |    bytes | uniq |    bytes | new |    bytes |
 dddddd |   1 | @ 2019-08-05 02:48 -hash |   400 | 256,483K |     59 | 220,747K |    1 |      50K |  59 | 220,747K |
 dddddd |   2 | @ 2019-08-05 12:36       |   400 | 256,483K |     59 | 220,747K |    1 |      50K |   1 |      50K |
 dddddd | all |                          |       |          |     60 | 220,797K |   60 | 220,797K |     |          |

Now whenever I do another backup, the backups run fine, but a check after reports missing chunks.

Running check command from /cache/localhost/all
Options: [-log check -storage local -a -tabular]
2019-08-11 07:40:20.755 INFO STORAGE_SET Storage set to sftp://backup@mynas.local//share/CACHEDEV1_DATA/duplicacy
2019-08-11 07:40:21.075 INFO SNAPSHOT_CHECK Listing all chunks
2019-08-11 07:41:48.439 INFO SNAPSHOT_CHECK 13 snapshots and 32 revisions
2019-08-11 07:41:48.447 INFO SNAPSHOT_CHECK Total chunk size is 2671G in 564694 chunks
2019-08-11 07:41:48.485 INFO SNAPSHOT_CHECK All chunks referenced by snapshot ffffff at revision 1 exist
2019-08-11 07:41:48.538 INFO SNAPSHOT_CHECK All chunks referenced by snapshot ffffff at revision 2 exist
2019-08-11 07:41:48.563 FATAL DOWNLOAD_CHUNK Chunk 91f11084ff78f06ec9fb14660b024817ee7c6de41e4dd44aa387ffb2da08fe99 can't be found

I went through each snapshot in the storage folder and deleted the one from today as recommended here. I did this individually for each backup and then ran the check again, and each time got similar missing chunk errors until I had removed all of todays snapshots. Eventually the check worked OK.

I then ran the backups again - this ran fairly quickly as most of the chunks would have already existed. The backups all succeeded, but afterwards the check failed with the same problem with missing chunks.

A few points:

• the prune logs do not show any files being pruned (nothing is old enough yet)
• no other clients are using the backup repository (have never configured any)
• I’m using duplicacy web edition in docker, running under unraid
• the NAS is a QNAP, and I’m accessing it on a local LAN using SFTP.
• the initial backup was actually done using SMB (mapped to a local folder in unraid and passed through to the docker container). I then deleted the storage and added it back using the same name and SFTP.
• my logs have been modified to remove personal information (backup names)

I have to admit I’m a bit concerned about the safety of using duplicacy ongoing. I hope it’s something specific to my setup and easily fixed as I like it a lot!

Any suggestions appreciated!

[bn1]

Since posting this, I deleted the storage and re-added it using a local mount point mapped to an SMB share. Everything now works as expected - backups and checks run OK.

So this problem is either a bug with SFTP storage, a problem with my NAS’s implementation of SFTP, or an issue with changing the access method of the existing storage.

[Droolio]

I don’t really have any concrete suggestions but I’d be inclined to run the check twice to see if it reported the same chunk missing. To make sure it really went missing and wasn’t something that went wrong with the check.

[stan464]

Hi Mate, Derail post here.

Could you drop me a Guide if there is one Avail to add Duplicacy Web in UnRAID, as i use UnRAID Myself and looking at this for a replacement for Duplicati.

Thanks Mate!

[bn1]

I’m using the docker image from saspus/duplicacy-web

It would be nice if an official image was offered, but this one seems to work well.

I simply added created the docker container and added the following settings:

A few notes:

• localtime and timezone are passed in, but don’t seem to work. The container runs at UTC.
• “backups” should be mapped to where you want to store your backups. Mine is a remote NFS share locally mounted.
• I have “data” mapped to my entire user share root, which is a bit scary as the container can access everything. Kinda necessary for a backup really but you should add it as RO.

I will see about getting this added to unraid community applications.

As a fast track, you can unzip the attchment and save to /boot/config/plugins/dockerMan/templates-user

You can then just add a docker container, select “my-duplicacy” from the user templates, customise the settings for you, and start.

my-duplicacy.zip (1.2 KB)