Failed to decrypt the file: unexpected EOF

[36017c54763338ab10ea]

I have an issue with check and prune commands to one of mine SFTP VPS storage backends I call “hhsto”. When I try to do a check the log looks like this:

Running check command from /cache/localhost/all
Options: [-log check -storage HHSTO -a -fossils -files -chunks -stats -tabular -a -tabular]
2020-08-12 21:23:34.280 INFO STORAGE_SET Storage set to sftp://user@hhsto.example.com/backup/duplicacy
2020-08-12 21:23:35.898 INFO SNAPSHOT_CHECK Listing all chunks
2020-08-12 21:25:01.048 WARN SNAPSHOT_CHECK Chunk a694bb068b34e174677826b7f92c80abade7d729db709c32afcd66cdd852415e has a size of 0
2020-08-12 21:25:01.051 WARN SNAPSHOT_CHECK Chunk ea134c99f4320fc4bf83922cec65afd319b174b24e1a0c3e1ac4480101f051a5 has a size of 0
2020-08-12 21:25:01.052 WARN SNAPSHOT_CHECK Chunk be3d3dd98c256bd32583cb8551a76d5bb1e74f774ad3c9d1964f2c5dcc1f6757 has a size of 0
2020-08-12 21:25:01.053 WARN SNAPSHOT_CHECK Chunk 50af88b54fd6223c92a0d3e57635e83d0044bd6f8db7eb8bb0f7e27dfa431a79 has a size of 0
2020-08-12 21:25:01.053 WARN SNAPSHOT_CHECK Chunk 50aef3d4201a059a006e3a8209c73378a3bc52c944da46620718d13ded73f528 has a size of 0
2020-08-12 21:25:01.053 WARN SNAPSHOT_CHECK Chunk 8742899fb5287d40e56dacf01f07c8057765a3c7711252c1e726d2276f2628da has a size of 0
2020-08-12 21:25:01.054 WARN SNAPSHOT_CHECK Chunk aa2d47eed2680993d43af3102257ec9d7baf03b5b24ed6232a967a2712148232 has a size of 0
2020-08-12 21:25:01.058 WARN SNAPSHOT_CHECK Chunk 5187857210180b3f74c44e7257fe191acc0e383584db6e15f7497095aa662b99 has a size of 0
2020-08-12 21:25:01.062 WARN SNAPSHOT_CHECK Chunk 3f176ea24a3fe3972935729aa6c254cf567dc7df8f6797350b82434ef3c15398 has a size of 0
2020-08-12 21:25:01.063 WARN SNAPSHOT_CHECK Chunk 7ce10914e8a2a224e8aa3e1d4c492290b5737bd70021db1693894aed5b200ed8 has a size of 0
2020-08-12 21:25:01.064 WARN SNAPSHOT_CHECK Chunk 880837a15175a4ffbd17c6b0d3f400b873a605712388af1845a15e7a50b0ece9 has a size of 0
2020-08-12 21:25:01.066 WARN SNAPSHOT_CHECK Chunk 1272822968ce9e7b3e27624743823121e3681bbf8555d0922a0b4c8c815eca0a has a size of 0
2020-08-12 21:25:01.071 WARN SNAPSHOT_CHECK Chunk 27b0a78bb7783090245a90f01600d1ef7d3a9c6246e2c68aaf06c0ca25af1072 has a size of 0
2020-08-12 21:25:01.077 WARN SNAPSHOT_CHECK Chunk 07b56de36f2c917e2b1a70193ae3c67d500bd0ef0f094d065042d487417ee796 has a size of 0
2020-08-12 21:25:01.080 WARN SNAPSHOT_CHECK Chunk c3012d1aacb0eb0e57f745de92feb50cbdf23358285fff21c8d1e8c49345fbbc has a size of 0
2020-08-12 21:25:01.080 WARN SNAPSHOT_CHECK Chunk d6e3f4624f980eee85afc4de1a8de58efff6256802e7058721f6d524cd309aa2 has a size of 0
2020-08-12 21:25:01.081 WARN SNAPSHOT_CHECK Chunk a8da1daf03d3fcab1e58c460901fe3ae3ec5968f1994a8269fafb5173d2bf5dd has a size of 0
2020-08-12 21:25:01.083 WARN SNAPSHOT_CHECK Chunk 93e13bfd9e1495a4f6b534921eceb68142cb8e2e8628720fecace0860b20ad1e has a size of 0
2020-08-12 21:25:01.088 WARN SNAPSHOT_CHECK Chunk e220937f8ead8db24d1cb87d2070ab49f9af6edf58e029e6ff829b47d52a2c18 has a size of 0
2020-08-12 21:25:01.088 WARN SNAPSHOT_CHECK Chunk e256fd50bc2144cdbab99373b70c9499e9832a01d4f32ce22d985123c39ef812 has a size of 0
2020-08-12 21:25:28.070 ERROR DOWNLOAD_DECRYPT Failed to decrypt the file snapshots/UHS-backup/1286: unexpected EOF
Failed to decrypt the file snapshots/UHS-backup/1286: unexpected EOF

I suspect that the node that this VPS is hosted on had a data corruption dude to some issue a while back the hosting company had. But not sure on this.

Is there a way (try to) repair the backup? Or is it easier to just reset and start over (I have multiple storage backends)? In either case, how should I proceed in order to get the backups back up and running?

I have searched the forum, but I haven’t found a post with a solution.

[saspus]

Here is the post with solution

Chunks have size of zero

You’ll need to manually delete these chunk files from the storage, and then follow the guide on Fix missing chunks. Which storage backend are you using?

[36017c54763338ab10ea]

Thank you for the tip! However in my case deleting chunks and try to recreate it was way too time consuming (I spent around 1 hour to find and delete chunks). So I deleted the snapshots instead, and this is how I did that:

1. Turned off backups on all clients to the problematic storage.
2. check -storage *storage* -a and deleted whatever snapshots that gives a “Failed to decrypt” or “Missing chunks” error on the storage side. In my case, eg. snapshots/UHS-backup/1286.
3. prune -storage *storage* -exhaustive -exclusive -keep....
4. Go back to step 2 and do the same steps again until I no longer getting errors.
5. Start manual backup/copy job, check for errors. Enable backups on all clients again.

Something I found a bit odd I didn’t find any documentation how to delete a chunk, the logs didn’t provide a path where it was located. That was a bit problematic to manually reconstruct hundreds of rm commands, so I ended up with to write a regex to create the path:

Match Chunk ([a-f0-9]{2})([a-f0-9]+) has
Replace rm chunks/$1/$2
Result: a694bb068b34e174677826b7f92c80abade7d729db709c32afcd66cdd852415e => rm chunks/a6/94bb068b34e174677826b7f92c80abade7d729db709c32afcd66cdd852415e

Anyhow, it works now and I lost just a few revisions in the process that I have backed up in other storages anyway.

[gchen]

How many revisions did you remove? I should have asked you to check the timestamps of those chunk files to see If they were created around the same time.

[36017c54763338ab10ea]

I did remove around 150 revisions, that included one Backup ID that I needed to remove all snapshots for some reason. I didn’t look at the timestamps, by my impression was that it was random chunks that was missing.