ERROR CHUNK_MAKER input/output error

Hello all,

since few days im getting this error - the file is not in use and i can access it. Has anybody an idea?

2021-10-07 21:53:41.721 DEBUG CHUNK_DUPLICATE Chunk ef0a6a02e621f9e683930337b32c898be3990925ed6ffe506241e61bf45a7f2a already exists
2021-10-07 21:53:41.721 DEBUG CHUNK_CACHE Skipped chunk ef0a6a02e621f9e683930337b32c898be3990925ed6ffe506241e61bf45a7f2a in cache
2021-10-07 21:53:41.721 INFO UPLOAD_PROGRESS Skipped chunk 1 size 204488, 200KB/s 121 days 18:35:22 0.0%
2021-10-07 21:53:42.067 ERROR CHUNK_MAKER Failed to read 0 bytes: read /media/pi/BLABLA: input/output error

What is this? SMB mount of a usb hard drive attached to raspberry pi?

What file? How are you trying to access it? From the same mount?

Do you have enough free space on your local machine for temporary chunk file? On a target machine?

hi,
thank you for reply!

/media/pi/BLABLA is the SMB mount of a usb hard drive attached to raspberry pi :slight_smile:

the file on SMB mount which the error message point is not in use and i can access (open it with nano).

i have enough space…

here is more log…

Fr 8. Okt 20:18:31 CEST 2021 Backing up /media/pi/Backup/BackupPANGPC …
2021-10-08 20:18:31.474 INFO STORAGE_SET Storage set to sftp://127.0.0.1:2022//Backup1/
2021-10-08 20:18:31.476 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_SSH_KEY_FILE
2021-10-08 20:18:33.613 DEBUG STORAGE_NESTING Chunk read levels: [1], write level: 1
2021-10-08 20:18:33.617 INFO CONFIG_INFO Compression level: 100
2021-10-08 20:18:33.617 INFO CONFIG_INFO Average chunk size: 4194304
2021-10-08 20:18:33.617 INFO CONFIG_INFO Maximum chunk size: 4194304
2021-10-08 20:18:33.617 INFO CONFIG_INFO Minimum chunk size: 4194304
2021-10-08 20:18:33.617 INFO CONFIG_INFO Chunk seed: 6475706c6963616379
2021-10-08 20:18:33.617 TRACE CONFIG_INFO Hash key: 6475706c6963616379
2021-10-08 20:18:33.618 TRACE CONFIG_INFO ID key: 6475706c6963616379
2021-10-08 20:18:33.618 DEBUG BACKUP_PARAMETERS top: /media/pi/BLABLA/BLABLABLA, quick: true, tag:
2021-10-08 20:18:33.618 TRACE SNAPSHOT_DOWNLOAD_LATEST Downloading latest revision for snapshot mywork
2021-10-08 20:18:33.618 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot mywork
2021-10-08 20:18:34.392 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/mywork/21 from the snapshot cache
2021-10-08 20:18:34.394 DEBUG CHUNK_CACHE Chunk 7028f22b606a7363a42c054755dbd842b5a5528074e93e047517df65e9309d41 has been loaded from the snapshot cache
2021-10-08 20:18:34.395 DEBUG DOWNLOAD_FETCH Fetching chunk 85580cbbe81f9848145f24482c4d72a80957083050d51c97f9cf1714f35881a3
2021-10-08 20:18:34.532 DEBUG CHUNK_CACHE Chunk 85580cbbe81f9848145f24482c4d72a80957083050d51c97f9cf1714f35881a3 has been loaded from the snapshot cache
2021-10-08 20:18:34.546 DEBUG DOWNLOAD_FETCH Fetching chunk 9f1b04d3147b939bdf577f3f69696ab9d1906e44d281d7f28c5eff220129b9e9
2021-10-08 20:18:34.627 DEBUG CHUNK_CACHE Chunk 9f1b04d3147b939bdf577f3f69696ab9d1906e44d281d7f28c5eff220129b9e9 has been loaded from the snapshot cache
2021-10-08 20:18:34.662 DEBUG DOWNLOAD_FETCH Fetching chunk 0148e57d4063c050e5ed5f4cfa6419b83446c3354502acaf166b329e5b5143d8
2021-10-08 20:18:34.772 DEBUG CHUNK_CACHE Chunk 0148e57d4063c050e5ed5f4cfa6419b83446c3354502acaf166b329e5b5143d8 has been loaded from the snapshot cache
2021-10-08 20:18:34.831 DEBUG DOWNLOAD_FETCH Fetching chunk fc8e1908558c43446b6b5ed9d22c5884f83e8f552fef1e109203edc9f10f0eb9
2021-10-08 20:18:34.912 DEBUG CHUNK_CACHE Chunk fc8e1908558c43446b6b5ed9d22c5884f83e8f552fef1e109203edc9f10f0eb9 has been loaded from the snapshot cache
2021-10-08 20:18:34.993 DEBUG DOWNLOAD_FETCH Fetching chunk cd319ce5a002394150bf4a6c0177f95158b695d319a3251415dd4166bc64c949
2021-10-08 20:18:35.035 DEBUG CHUNK_CACHE Chunk cd319ce5a002394150bf4a6c0177f95158b695d319a3251415dd4166bc64c949 has been loaded from the snapshot cache
2021-10-08 20:18:36.328 DEBUG DOWNLOAD_FETCH Fetching chunk 778bb80594d189df2f226b4263fd44a8afe4a7b45cf07ce73c01ddd62fb279d9
2021-10-08 20:18:36.371 DEBUG CHUNK_CACHE Chunk 778bb80594d189df2f226b4263fd44a8afe4a7b45cf07ce73c01ddd62fb279d9 has been loaded from the snapshot cache
2021-10-08 20:18:36.684 INFO BACKUP_START Last backup at revision 21 found
2021-10-08 20:18:36.684 INFO BACKUP_INDEXING Indexing /media/pi/BLABLA/BLABLABLA
2021-10-08 20:18:36.684 INFO SNAPSHOT_FILTER Parsing filter file /home/pi/duplicacyfolger/filters
2021-10-08 20:18:36.684 DEBUG REGEX_DEBUG There are 0 compiled regular expressions stored
2021-10-08 20:18:36.684 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2021-10-08 20:18:36.684 DEBUG LIST_ENTRIES Listing
2021-10-08 20:18:39.374 TRACE PACK_START Packing Backupfile.yyy
2021-10-08 20:18:39.374 INFO BACKUP_THREADS Use 4 uploading threads
2021-10-08 20:18:39.483 INFO PACK_END Packed Backupfile.yyy (209434)
2021-10-08 20:18:39.483 TRACE PACK_START Packing Byckupfile1.yyy
2021-10-08 20:18:39.777 ERROR CHUNK_MAKER Failed to read 0 bytes: read /media/pi/BLABLA/BLABLABLA/Backupfile1.yyy: input/output error
Storage set to sftp://127.0.0.1:2022//Backup1/
Keep no snapshots older than 30 days
No snapshot to delete
Fr 8. Okt 20:18:39 CEST 2021 Stopped backing up /media/pi/BLABLA/BLABLABLA …

Can you manually copy the file somewhere? e.g. cp /media/pi/BLABLA/BLABLABLA/Backupfile1.yyy /tmp/? Nano may be showing you the beginning for the file, but if the bad sector is somewhere in the middle you man not necessarily stumble on it.

you are right… there is an input/output error… :frowning:
What should i do? just delete the file and continue with that device or can i do some sort of repair oder “deactivate” of the bad sectors?

In my honest opinion – you should stop backing up to a single usb drive much less connected to a raspberry pi.

Backup should be more reliable than your source media – when that fails, your backup should be there to save the day.

RPI is an unshielded prototyping devices, very susceptible to ESD and power fluctuations. USB storage stack is high power low performance slowpoke; USB drives in the enclosure are least reliable in the world: they are lowest binned devices, USB enclosure power delivery is crap, power and thermal management is from flawed to nonexistent; and even if that was not the case – single HDD will rot and degrade over time no matter what; you can’t write data to the disk and expect to read it tomorrow. That’s not how modern drives work. The industry decided it’s cheaper to make shittier disks but regain reliability by using them in redundant clusters. Overall cost is lower and reliability is higher. You can’t trust single hard drive anymore and the higher write density is – the less you can trust it.

You need to have some sort of redundancy with healing (raid array comes to mind). This will save you from bad sector on one of the drives ruining your day but will do nothing to address bit rot (no checksumming so not clear which copy of mismatched data to trust). To protect against that you need BTRFS or ZFS array with checksumming enabled and periodic scrub scheduled that to read every sector, verify checksums, and correct inconsistencies. And then you may consider this a half-decent local backup (don’t forget ECC RAM too for good measure). In addition, data shall be actively maintained to be viable (periodic scrub is a must).

What you can do short term? Format your drive with BTRFS with single disk DUP profile (see Manpage/mkfs.btrfs - btrfs Wiki). This will halve the available space on your drive but will write data with redundancy so that bad and rotten sectors will be detectable and correctable during periodic scrub. But this won’t save you from HDD failure. Also, you would need to enable ERC/TLER on the disk so that read on bad sector fails and not time out.

Middle-term? get a NAS with BTFS or ZFS and NAS or enterprise grade disk (for TLER/ERC) to avoid reinventing the wheel.

Long term – just backup to the commercial cloud, where other people’s full time job is to keep your data viable and available. Keeping data alive is hard. Outsource that menial task! And it will be significantly cheaper because unlike your one-off contraption in the closet they do it on scale.

What about Duplicacy’s erasure coding you say? It depends. It may save you in some cases. It does reduce risk somewhat and will help if bad sector happens to be somewhere inside chunk files. But if that happens elsewhere, or corrupts filesystem, etc – it does not really help and therefore IMO is not worth the efforts.

2 Likes

Saspus, thank you very very much! this was eye opening!
i cannot say it clear enough: THANK YOU!

:pray: :pray: :pray:

so my journey with this begins again with more clarity :wink:

1 Like

Replace the drive. A Raspberry Pi is a solid device and external HDDs are perfectly fine for backup - *so long as you copy it (with Duplicacy) to another, more reliable, destination (i.e. cloud storage) - and monitor this process so you know when something goes wrong.

The process of copying data from the local backup storage validates the integrity of the data, since it has to unpack and repack chunks for the destination. Though you should always run periodic test restores as well (from all backup stores).

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.