FATAL DOWNLOAD_CHUNK Chunk can't be found (GCD)

Support
#1

2019-12-20 13:23:30.267 FATAL DOWNLOAD_CHUNK Chunk 3bdaf10360dd3814efcdae4f87d0dc228f7df41e8e4b9fceb505b4dd6b114757 can’t be found

Duplicacy Web Edition (same thing happens at command line)
Google Drive Storage
Scheduled tasks
3 repositories
1 check
1 prune (that hasn’t pruned anything yet)

Run checks on each of the 3 repositories reports all chunks exist for each revision.
Run check -a and it reports a missing chunk ??

After more backups, the chunk reported missing is different.

The issue being, the overnight scheduled check is reporting the check failed, so I can’t be sure of the integrity of the backup.

#2

A check script that highlights the behaviour. This is a new(ish) store and only has these 3 snapshots.

#!/bin/bash
export DUPLICACY_GOOGLEDRIVE_GCD_TOKEN=D:\\PATH\\TO\\gcd-token.json
cd /cygdrive/c/Users/Austin/.duplicacy-web && cd repositories/localhost && {
  ( cd ./0 && duplicacy check )
  ( cd ./1 && duplicacy check )
  ( cd ./2 && duplicacy check )
  ( cd ./all && duplicacy check -a )
}

check 0

Repository set to D:/DOCS
Storage set to gcd://BACKUPS/DUPLICACY
Enter storage password:
Listing all chunks
1 snapshots and 2 revisions
Total chunk size is 36,939M in 7924 chunks
All chunks referenced by snapshot CUBE-DOCS at revision 1 exist
All chunks referenced by snapshot CUBE-DOCS at revision 2 exist

check 1

Repository set to D:\MUSIC\iTunes\iTunes Media\Music
Storage set to gcd://Backups/DUPLICACY
Listing all chunks
1 snapshots and 6 revisions
Total chunk size is 36,939M in 7924 chunks
All chunks referenced by snapshot MUSIC at revision 1 exist
All chunks referenced by snapshot MUSIC at revision 2 exist
All chunks referenced by snapshot MUSIC at revision 3 exist
All chunks referenced by snapshot MUSIC at revision 4 exist
All chunks referenced by snapshot MUSIC at revision 5 exist
All chunks referenced by snapshot MUSIC at revision 6 exist

check 2

Repository set to D:\DEV
Storage set to gcd://Backups/DUPLICACY
Listing all chunks
1 snapshots and 6 revisions
Total chunk size is 36,939M in 7924 chunks
All chunks referenced by snapshot DEV at revision 1 exist
All chunks referenced by snapshot DEV at revision 2 exist
All chunks referenced by snapshot DEV at revision 3 exist
All chunks referenced by snapshot DEV at revision 4 exist
All chunks referenced by snapshot DEV at revision 5 exist
All chunks referenced by snapshot DEV at revision 6 exist

check -a

Storage set to gcd://Backups/DUPLICACY
Listing all chunks
3 snapshots and 14 revisions
Total chunk size is 36,939M in 7924 chunks
All chunks referenced by snapshot CUBE-DOCS at revision 1 exist
All chunks referenced by snapshot CUBE-DOCS at revision 2 exist
Chunk 27a736298a5102cae231894c14bb403c4d7a2e372951e9a817abeb8960f7fd0b can't be found

The check -a is aborting saying it can’t find this chunk.

#3

Can you try cd ./all && duplicacy -d check -a? My guess is that there may be another snapshot id that has the missing chunks.

#4

Another snapshot id was my initial thought, but the storage only has the three snapshots being checked.

$ duplicacy -d check -a
Storage set to gcd://Backups/DUPLICACY
Reading the environment variable DUPLICACY_GOOGLEDRIVE_GCD_TOKEN
Reading gcd_token from keychain/keyring
Reading the environment variable DUPLICACY_GOOGLEDRIVE_GCD_TOKEN
Chunk read levels: [1], write level: 1
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
id: , revisions: [], tag: , showStatistics: false, checkFiles: false, searchFossils: false, resurrect: false
Listing all chunks
Listing chunks/
Listing all snapshot ids
Listing revisions for snapshot CUBE-DOCS
Loaded file snapshots/CUBE-DOCS/1 from the snapshot cache
Loaded file snapshots/CUBE-DOCS/2 from the snapshot cache
Loaded file snapshots/CUBE-DOCS/3 from the snapshot cache
Loaded file snapshots/CUBE-DOCS/4 from the snapshot cache
Listing revisions for snapshot DEV
Loaded file snapshots/DEV/1 from the snapshot cache
Loaded file snapshots/DEV/2 from the snapshot cache
Loaded file snapshots/DEV/3 from the snapshot cache
Loaded file snapshots/DEV/4 from the snapshot cache
Loaded file snapshots/DEV/5 from the snapshot cache
Loaded file snapshots/DEV/6 from the snapshot cache
Loaded file snapshots/DEV/7 from the snapshot cache
Loaded file snapshots/DEV/8 from the snapshot cache
Listing revisions for snapshot MUSIC
Loaded file snapshots/MUSIC/1 from the snapshot cache
Loaded file snapshots/MUSIC/2 from the snapshot cache
Loaded file snapshots/MUSIC/3 from the snapshot cache
Loaded file snapshots/MUSIC/4 from the snapshot cache
Loaded file snapshots/MUSIC/5 from the snapshot cache
Loaded file snapshots/MUSIC/6 from the snapshot cache
Loaded file snapshots/MUSIC/7 from the snapshot cache
Loaded file snapshots/MUSIC/8 from the snapshot cache
3 snapshots and 20 revisions
Total chunk size is 37,495M in 8044 chunks
Chunk 27a736298a5102cae231894c14bb403c4d7a2e372951e9a817abeb8960f7fd0b can't be found

A second run, gave slightly different result

$ duplicacy -d check -a
Storage set to gcd://Backups/DUPLICACY
Reading the environment variable DUPLICACY_GOOGLEDRIVE_GCD_TOKEN
Reading gcd_token from keychain/keyring
Reading the environment variable DUPLICACY_GOOGLEDRIVE_GCD_TOKEN
Chunk read levels: [1], write level: 1
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
id: , revisions: [], tag: , showStatistics: false, checkFiles: false, searchFossils: false, resurrect: false
Listing all chunks
Listing chunks/
Listing all snapshot ids
Listing revisions for snapshot MUSIC
Loaded file snapshots/MUSIC/1 from the snapshot cache
Loaded file snapshots/MUSIC/2 from the snapshot cache
Loaded file snapshots/MUSIC/3 from the snapshot cache
Loaded file snapshots/MUSIC/4 from the snapshot cache
Loaded file snapshots/MUSIC/5 from the snapshot cache
Loaded file snapshots/MUSIC/6 from the snapshot cache
Loaded file snapshots/MUSIC/7 from the snapshot cache
Loaded file snapshots/MUSIC/8 from the snapshot cache
Listing revisions for snapshot CUBE-DOCS
Loaded file snapshots/CUBE-DOCS/1 from the snapshot cache
Loaded file snapshots/CUBE-DOCS/2 from the snapshot cache
Loaded file snapshots/CUBE-DOCS/3 from the snapshot cache
Loaded file snapshots/CUBE-DOCS/4 from the snapshot cache
Listing revisions for snapshot DEV
Loaded file snapshots/DEV/1 from the snapshot cache
Loaded file snapshots/DEV/2 from the snapshot cache
Loaded file snapshots/DEV/3 from the snapshot cache
Loaded file snapshots/DEV/4 from the snapshot cache
Loaded file snapshots/DEV/5 from the snapshot cache
Loaded file snapshots/DEV/6 from the snapshot cache
Loaded file snapshots/DEV/7 from the snapshot cache
Loaded file snapshots/DEV/8 from the snapshot cache
3 snapshots and 20 revisions
Total chunk size is 37,495M in 8044 chunks
Chunk ba1e216d4b7fd6d034019fd6ac7cea35c63227b10ce71a516e1359d0576386e2 has been loaded from the snapshot cache
All chunks referenced by snapshot CUBE-DOCS at revision 1 exist
Fetching chunk 30fc14bd211783c36b10d71d3bf3cb875eb0b6a01d032c81ddcfbe49fa0527db
Chunk 30fc14bd211783c36b10d71d3bf3cb875eb0b6a01d032c81ddcfbe49fa0527db has been loaded from the snapshot cache
All chunks referenced by snapshot CUBE-DOCS at revision 2 exist
Fetching chunk 30fc14bd211783c36b10d71d3bf3cb875eb0b6a01d032c81ddcfbe49fa0527db
Chunk 30fc14bd211783c36b10d71d3bf3cb875eb0b6a01d032c81ddcfbe49fa0527db has been loaded from the snapshot cache
All chunks referenced by snapshot CUBE-DOCS at revision 3 exist
Fetching chunk 30fc14bd211783c36b10d71d3bf3cb875eb0b6a01d032c81ddcfbe49fa0527db
Chunk 30fc14bd211783c36b10d71d3bf3cb875eb0b6a01d032c81ddcfbe49fa0527db has been loaded from the snapshot cache
All chunks referenced by snapshot CUBE-DOCS at revision 4 exist
Fetching chunk 27a736298a5102cae231894c14bb403c4d7a2e372951e9a817abeb8960f7fd0b
Chunk 27a736298a5102cae231894c14bb403c4d7a2e372951e9a817abeb8960f7fd0b can't be found
#5

Remove the cache directory all/.duplicacy/cache and try again. One of the snapshot files stored in the cache may be different from what is in the storage.

2 Likes
#6

Removing cache fixed it. Thanks.

1 Like
#7

For anyone: Feel free to use the :heart: button on the posts that you found useful.

For the OP of any #support topic: you can mark the post that solved your issue by ticking the :checked: under the post. That of course may include your own post :slight_smile: