Checks and prunes fail, missing chunk folder, " structure needs cleaning"

Support
#1

Hi all, no luck finding a similar issue, I had to restart my offsite backup from scratch due to a buddy getting hit with some malware and wanting to ensure my backup server at his place was clean.

Unfortunately after about a week of backing up, I’ve gotten an error, where a folder from the chunks section is missing.

I’ve tried persist, fossils,and a few other commands for checking and pruning, pruning goes through without error after deleting the cache and re-indexing everything, however check still complains about the missing folder.

I’ve also tried recreating the folder, and that did not work as it still says it’s missing.

At this point I’m kind of at a loss as to what to try, really hoping I don’t need to delete my entire backup and start from scratch. This software has been bulletproof for me, except for this error being stubborn as hell, I haven’t had to try and repair any problems before.

Running check command from /cache/localhost/all
Options: [-log check -storage HavenServer -fossils -a -tabular]
2024-07-02 00:52:24.923 INFO STORAGE_SET Storage set to sftp://QuantaServer@172.16.1.232:2022/Duplicacy
2024-07-02 00:52:24.952 INFO SNAPSHOT_CHECK Listing all chunks
2024-07-02 00:52:32.427 ERROR LIST_FILES Failed to list the directory chunks/32/: sftp: "readdirent /unraid/HavenServer Private/Duplicacy/chunks/32: structure needs cleaning" (SSH_FX_FAILURE)
Failed to list the directory chunks/32/: sftp: "readdirent /unraid/HavenServer Private/Duplicacy/chunks/32: structure needs cleaning" (SSH_FX_FAILURE)

I did get some read errors on one drive on my backup server, though it passed a smart test, perhaps that’s related, it should have been writing to an nvme cache first though, so I’m just not sure how I can tell duplicacy to forcibly recreate this section.

I’ve not seen this specific errors before so I’m not sure if setting a different backup ID like this would work Fix missing chunks

#2

Not much luck sleeping so I’ve been working at it some more with no luck. I tried to create a new backup of the same directory with a different ID but that immediately fails because of that missing folder… uhg.

#3

This is indicative of filesystem corruption. You may need to fix it first and then address missing chunks. See chkdsk on windows or fsck on Linux.

#4

Ah, damn, thanks. Looks like there are errors on one of the drives that unraid didn’t flag/warn me about for some reason, oddly not the suspect one though…

Working on running the checks and repairs on that, afterwards, should a standard backup just work correctly? Of course that does now worry me that I can’t trust this backup at all… though I guess the check should compare the hash of all files and mark anything bad for removal/replacement?

#5

I would delete the verified_chunks file if it exists and run check -chunks to verify integrity of each chunk.

If bad chunks are found — depending on how many, there are a few avenues to restore the health of the datastore; either by pruning affected snapshots or attempting to recreate the bad chunks by deleting them and making a full backup to a temporary snapshot ID