Interrupted dry-run backup causes missing chunks in subsequent backup

Bug
#1

Please describe what you are doing to trigger the bug:

  1. Initiate a dry-run backup from the CLI. While the backup is uploading chunks, type CTRL-C to terminate the backup operation.
  2. Perform a normal backup.
  3. Perform a check operation.

Please describe what you expect to happen (but doesn’t):
All chunks are present.

Please describe what actually happens (the wrong behaviour):
Some chunks are missing.
Subsequent backup operations create revisions which continue to have the same chunks missing.

I believe this occurs because interruption of the dry-run backup creates files incomplete_chunks and incomplete_snapshot within the cache/$SNAPSHOT directory. These files should not be created during a dry-run operation.

Tested with Duplicacy 3.2.3 on Linux using local storage.

#2

This makes no sense, because dry run does not upload anything. That’s the whole point of dry run.

Backup, dry run or not, does not delete anything from the destination storage.

Interrupted prune, on the other hand, can result in ghost snapshots left in the repository.

#3

@saspus Not sure I understand your response. I agree that it makes no sense, which is why I reported it as a bug.

I’m not claiming that “backup -dry-run” deletes anything from the storage. But It does, in fact, create the two files mentioned in the cache directory if it is interrupted (that is the bug).

Those files apparently (I have not reviewed the code to verify this) indicate to the subsequent backup that some chunks have already been uploaded, when in fact they have not because it was indeed a dry-run.

I confirmed this by deleting those files prior to starting the live backup operation, and in that case there were no missing chunks.

#4

Sorry, I’m not doubting that this is what you see. I was trying to say that missing chunks are likely a result of something else, not dry-run backup.

Do you reliably reproduce this?

i.e.

  1. check storage – good.
  2. Run backup dry run
  3. Interrupt backup
  4. check storage – bad?

That does look like a bug. Duplicacy is supposed to use files in the cache directory only if they exist on the target. This is necessary to support concurrent scenarios. If the file exists in the cache but does not exist on the target – it shall be treated as if it’s missing.

if presence of some stuff in the cache directory causes check to malfunction - it’s a problem.

#5

@saspus Yes, I reproduced it about 6 times and found that it occurs consistently. I started the sequence each time with an empty cache directory and and an empty storage directory (no need for your step 1).

The live backup completes with no missing chunks in either of the following cases:

  • I let the dry-run backup run to completion prior to the live backup.
  • I interrupt the dry-run backup and then delete the two cache files prior to the live backup.

Just to clarify, the sequence to reproduce the problem is:

  1. Clear cache and storage directory
  2. Start dry-run backup and interrupt it during chunk upload.
  3. Run live backup to completion.
  4. Run check command and observe missing chunk errors.
2 Likes
#6

I couldn’t reproduce this bug. How many files do you have in the source directory?

#7

There are 14698 files in the repository.

To demonstrate the issue, please save the bash script below to test.sh and run:

  • test.sh -dry-run # hit CTRL-C while chunks are uploading
  • test.sh # let this run till completion
#!/bin/bash

repository_path=/old/pcm_temp
snapshot=test1
storage_dir=/tmp/duplicacy_test1/

mkdir -p $storage_dir
cd $repository_path

if [[ "$1" == "-dry-run" ]]; then
  rm -rf $storage_dir $repository_path/.duplicacy
  duplicacy init  $snapshot $storage_dir
  duplicacy -log backup -stats -dry-run
else  
  duplicacy -log backup -stats
  duplicacy check -tabular
fi

Following is the output of the dry-run. Note the last line “Incomplete snapshot saved to /old/pcm_temp/.duplicacy/cache/default/incomplete_snapshot”.

root@vps3:/old/pcm_temp$ test.sh -dry-run
/old/pcm_temp will be backed up to /tmp/duplicacy_test1/ with id test1
2024-07-11 16:22:39.721 INFO STORAGE_SET Storage set to /tmp/duplicacy_test1/
2024-07-11 16:22:39.729 INFO BACKUP_START No previous backup found
2024-07-11 16:22:39.729 INFO BACKUP_INDEXING Indexing /old/pcm_temp
2024-07-11 16:22:39.729 INFO SNAPSHOT_FILTER Parsing filter file /old/pcm_temp/.duplicacy/filters
2024-07-11 16:22:39.729 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2024-07-11 16:22:39.938 INFO UPLOAD_PROGRESS Uploaded chunk 0 size 4275726, 4.08MB/s 00:02:17 0.7%
2024-07-11 16:22:39.955 INFO UPLOAD_PROGRESS Uploaded chunk 1 size 2298091, 6.27MB/s 00:01:29 1.1%
2024-07-11 16:22:39.970 INFO UPLOAD_PROGRESS Uploaded chunk 2 size 1844406, 8.03MB/s 00:01:09 1.4%
2024-07-11 16:22:39.995 INFO UPLOAD_PROGRESS Uploaded chunk 3 size 2327391, 10.25MB/s 00:00:54 1.8%
2024-07-11 16:22:40.019 INFO UPLOAD_PROGRESS Uploaded chunk 4 size 4575914, 14.61MB/s 00:00:38 2.6%
2024-07-11 16:22:40.082 INFO UPLOAD_PROGRESS Uploaded chunk 5 size 5486889, 19.84MB/s 00:00:28 3.5%
2024-07-11 16:22:40.098 INFO UPLOAD_PROGRESS Uploaded chunk 6 size 2013870, 21.77MB/s 00:00:25 3.8%
2024-07-11 16:22:40.115 INFO UPLOAD_PROGRESS Uploaded chunk 7 size 2914412, 24.54MB/s 00:00:22 4.3%
2024-07-11 16:22:40.181 INFO UPLOAD_PROGRESS Uploaded chunk 8 size 5673358, 29.95MB/s 00:00:18 5.3%
2024-07-11 16:22:40.213 INFO UPLOAD_PROGRESS Uploaded chunk 9 size 4386120, 34.14MB/s 00:00:16 6.0%
2024-07-11 16:22:40.275 INFO UPLOAD_PROGRESS Uploaded chunk 10 size 6193328, 40.04MB/s 00:00:14 7.1%
2024-07-11 16:22:40.296 INFO UPLOAD_PROGRESS Uploaded chunk 11 size 1929588, 41.88MB/s 00:00:13 7.4%
2024-07-11 16:22:40.339 INFO UPLOAD_PROGRESS Uploaded chunk 12 size 4034618, 45.73MB/s 00:00:12 8.1%
2024-07-11 16:22:40.393 INFO UPLOAD_PROGRESS Uploaded chunk 13 size 5995783, 51.45MB/s 00:00:10 9.1%
2024-07-11 16:22:40.400 INFO UPLOAD_PROGRESS Uploaded chunk 14 size 1735242, 53.11MB/s 00:00:10 9.4%
2024-07-11 16:22:40.421 INFO UPLOAD_PROGRESS Uploaded chunk 15 size 5384844, 58.24MB/s 00:00:09 10.3%
2024-07-11 16:22:40.432 INFO UPLOAD_PROGRESS Uploaded chunk 16 size 2577408, 60.70MB/s 00:00:09 10.8%
2024-07-11 16:22:40.437 INFO UPLOAD_PROGRESS Uploaded chunk 17 size 1313069, 61.95MB/s 00:00:09 11.0%
2024-07-11 16:22:40.481 INFO UPLOAD_PROGRESS Uploaded chunk 18 size 4167523, 65.93MB/s 00:00:08 11.7%
2024-07-11 16:22:40.491 INFO UPLOAD_PROGRESS Uploaded chunk 19 size 2290757, 68.11MB/s 00:00:08 12.1%
2024-07-11 16:22:40.542 INFO UPLOAD_PROGRESS Uploaded chunk 20 size 3898146, 71.83MB/s 00:00:07 12.8%
2024-07-11 16:22:40.556 INFO UPLOAD_PROGRESS Uploaded chunk 21 size 1985096, 73.72MB/s 00:00:07 13.1%
2024-07-11 16:22:40.568 INFO UPLOAD_PROGRESS Uploaded chunk 22 size 1862161, 75.50MB/s 00:00:07 13.4%
^C2024-07-11 16:22:41.238 INFO INCOMPLETE_SAVE Incomplete snapshot saved to /old/pcm_temp/.duplicacy/cache/default/incomplete_snapshot

Following is the output of the live run. Note:

  • The fourth line “Previous incomplete backup contains 14698 files and 23 chunks”.
  • The last line: “Some chunks referenced by some snapshots do not exist in the storage”.
root@vps3:/old/pcm_temp$ test.sh
2024-07-11 16:22:51.613 INFO STORAGE_SET Storage set to /tmp/duplicacy_test1/
2024-07-11 16:22:51.621 INFO BACKUP_START No previous backup found
2024-07-11 16:22:51.635 INFO INCOMPLETE_LOAD Previous incomplete backup contains 14698 files and 23 chunks
2024-07-11 16:22:51.635 INFO BACKUP_INDEXING Indexing /old/pcm_temp
2024-07-11 16:22:51.635 INFO SNAPSHOT_FILTER Parsing filter file /old/pcm_temp/.duplicacy/filters
2024-07-11 16:22:51.635 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2024-07-11 16:22:51.898 INFO UPLOAD_PROGRESS Uploaded chunk 0 size 1831513, 1.75MB/s 00:04:37 0.3%
2024-07-11 16:22:51.976 INFO UPLOAD_PROGRESS Uploaded chunk 1 size 6002362, 7.47MB/s 00:01:04 1.5%
2024-07-11 16:22:52.030 INFO UPLOAD_PROGRESS Uploaded chunk 2 size 6231912, 13.41MB/s 00:00:36 2.7%
2024-07-11 16:22:52.121 INFO UPLOAD_PROGRESS Uploaded chunk 3 size 4478429, 17.69MB/s 00:00:27 3.6%
2024-07-11 16:22:52.131 INFO UPLOAD_PROGRESS Uploaded chunk 4 size 1197414, 18.83MB/s 00:00:25 3.8%
2024-07-11 16:22:52.159 INFO UPLOAD_PROGRESS Uploaded chunk 5 size 3636664, 22.30MB/s 00:00:21 4.5%
2024-07-11 16:22:52.184 INFO UPLOAD_PROGRESS Uploaded chunk 6 size 3034136, 25.19MB/s 00:00:19 5.1%
2024-07-11 16:22:52.197 INFO UPLOAD_PROGRESS Uploaded chunk 7 size 1552316, 26.67MB/s 00:00:18 5.4%
2024-07-11 16:22:52.209 INFO UPLOAD_PROGRESS Uploaded chunk 8 size 1419480, 28.02MB/s 00:00:17 5.7%
2024-07-11 16:22:52.217 INFO UPLOAD_PROGRESS Uploaded chunk 9 size 1355307, 29.32MB/s 00:00:16 6.0%
2024-07-11 16:22:52.276 INFO UPLOAD_PROGRESS Uploaded chunk 10 size 6747241, 35.75MB/s 00:00:13 7.3%
2024-07-11 16:22:52.283 INFO UPLOAD_PROGRESS Uploaded chunk 11 size 1425444, 37.11MB/s 00:00:13 7.6%
2024-07-11 16:22:52.294 INFO UPLOAD_PROGRESS Uploaded chunk 12 size 2877315, 39.85MB/s 00:00:12 8.2%
2024-07-11 16:22:52.305 INFO UPLOAD_PROGRESS Uploaded chunk 13 size 2136932, 41.89MB/s 00:00:11 8.6%
2024-07-11 16:22:52.534 INFO UPLOAD_PROGRESS Uploaded chunk 14 size 16033339, 57.18MB/s 00:00:08 11.7%
2024-07-11 16:22:52.539 INFO UPLOAD_PROGRESS Uploaded chunk 15 size 1112768, 58.24MB/s 00:00:08 12.0%
2024-07-11 16:22:52.546 INFO UPLOAD_PROGRESS Uploaded chunk 16 size 1722878, 59.89MB/s 00:00:08 12.3%
2024-07-11 16:22:52.703 INFO UPLOAD_PROGRESS Uploaded chunk 17 size 15672429, 74.83MB/s 00:00:06 15.4%
2024-07-11 16:22:52.711 INFO UPLOAD_PROGRESS Uploaded chunk 18 size 2215693, 76.95MB/s 00:00:06 15.8%
2024-07-11 16:22:52.794 INFO UPLOAD_PROGRESS Uploaded chunk 19 size 2056253, 78.91MB/s 00:00:06 16.2%
2024-07-11 16:22:52.818 INFO UPLOAD_PROGRESS Uploaded chunk 20 size 6185115, 84.81MB/s 00:00:05 17.4%
2024-07-11 16:22:52.825 INFO UPLOAD_PROGRESS Uploaded chunk 21 size 1585291, 86.32MB/s 00:00:05 17.7%
2024-07-11 16:22:52.913 INFO UPLOAD_PROGRESS Uploaded chunk 22 size 11178639, 96.98MB/s 00:00:05 19.9%
2024-07-11 16:22:52.921 INFO UPLOAD_PROGRESS Uploaded chunk 23 size 1554128, 98.46MB/s 00:00:04 20.2%
2024-07-11 16:22:52.946 INFO UPLOAD_PROGRESS Uploaded chunk 24 size 5105079, 103.33MB/s 00:00:04 21.2%
2024-07-11 16:22:52.964 INFO UPLOAD_PROGRESS Uploaded chunk 25 size 4400381, 107.53MB/s 00:00:04 22.1%
2024-07-11 16:22:52.999 INFO UPLOAD_PROGRESS Uploaded chunk 26 size 5169997, 112.46MB/s 00:00:04 23.1%
2024-07-11 16:22:53.006 INFO UPLOAD_PROGRESS Uploaded chunk 27 size 1437569, 56.91MB/s 00:00:07 23.4%
2024-07-11 16:22:53.057 INFO UPLOAD_PROGRESS Uploaded chunk 28 size 6622079, 60.07MB/s 00:00:07 24.7%
2024-07-11 16:22:53.133 INFO UPLOAD_PROGRESS Uploaded chunk 29 size 10209771, 64.94MB/s 00:00:06 26.7%
2024-07-11 16:22:53.234 INFO UPLOAD_PROGRESS Uploaded chunk 30 size 13344445, 71.30MB/s 00:00:05 29.3%
2024-07-11 16:22:53.276 INFO UPLOAD_PROGRESS Uploaded chunk 31 size 8239662, 75.23MB/s 00:00:05 31.0%
2024-07-11 16:22:53.320 INFO UPLOAD_PROGRESS Uploaded chunk 32 size 7032812, 78.59MB/s 00:00:05 32.3%
2024-07-11 16:22:53.330 INFO UPLOAD_PROGRESS Uploaded chunk 33 size 2955945, 79.99MB/s 00:00:05 32.9%
2024-07-11 16:22:53.387 INFO UPLOAD_PROGRESS Uploaded chunk 34 size 6877758, 83.27MB/s 00:00:04 34.3%
2024-07-11 16:22:53.392 INFO UPLOAD_PROGRESS Uploaded chunk 35 size 1321264, 83.90MB/s 00:00:04 34.5%
2024-07-11 16:22:53.402 INFO UPLOAD_PROGRESS Uploaded chunk 36 size 2844167, 85.26MB/s 00:00:04 35.1%
2024-07-11 16:22:53.500 INFO UPLOAD_PROGRESS Uploaded chunk 37 size 11205211, 90.60MB/s 00:00:04 37.3%
2024-07-11 16:22:53.611 INFO UPLOAD_PROGRESS Uploaded chunk 38 size 11698589, 96.18MB/s 00:00:04 39.6%
2024-07-11 16:22:53.765 INFO UPLOAD_PROGRESS Uploaded chunk 39 size 16777216, 104.18MB/s 00:00:03 42.9%
2024-07-11 16:22:53.776 INFO UPLOAD_PROGRESS Uploaded chunk 40 size 3174269, 105.70MB/s 00:00:03 43.5%
2024-07-11 16:22:53.783 INFO UPLOAD_PROGRESS Skipped chunk 42 size 2877315, 107.07MB/s 00:00:03 44.1%
2024-07-11 16:22:53.785 INFO UPLOAD_PROGRESS Uploaded chunk 41 size 2061258, 108.05MB/s 00:00:03 44.5%
2024-07-11 16:22:53.795 INFO UPLOAD_PROGRESS Skipped chunk 43 size 2136932, 109.07MB/s 00:00:03 44.9%
2024-07-11 16:22:53.941 INFO UPLOAD_PROGRESS Uploaded chunk 44 size 12364350, 114.96MB/s 00:00:03 47.3%
2024-07-11 16:22:53.947 INFO UPLOAD_PROGRESS Skipped chunk 47 size 1112768, 115.50MB/s 00:00:03 47.6%
2024-07-11 16:22:53.948 INFO UPLOAD_PROGRESS Uploaded chunk 45 size 1175697, 116.06MB/s 00:00:03 47.8%
2024-07-11 16:22:53.956 INFO UPLOAD_PROGRESS Skipped chunk 48 size 1722878, 116.88MB/s 00:00:03 48.1%
2024-07-11 16:22:53.957 INFO UPLOAD_PROGRESS Uploaded chunk 46 size 2596390, 118.12MB/s 00:00:03 48.6%
2024-07-11 16:22:54.047 INFO UPLOAD_PROGRESS Uploaded chunk 49 size 8304747, 81.38MB/s 00:00:03 50.3%
2024-07-11 16:22:54.076 INFO UPLOAD_PROGRESS Skipped chunk 51 size 2215693, 82.09MB/s 00:00:03 50.7%
2024-07-11 16:22:54.085 INFO UPLOAD_PROGRESS Uploaded chunk 50 size 7364333, 84.43MB/s 00:00:03 52.1%
2024-07-11 16:22:54.089 INFO UPLOAD_PROGRESS Skipped chunk 52 size 2056253, 85.08MB/s 00:00:03 52.6%
2024-07-11 16:22:54.130 INFO UPLOAD_PROGRESS Skipped chunk 53 size 6185115, 87.05MB/s 00:00:03 53.8%
2024-07-11 16:22:54.139 INFO UPLOAD_PROGRESS Skipped chunk 54 size 1585291, 87.55MB/s 00:00:03 54.1%
2024-07-11 16:22:54.263 INFO UPLOAD_PROGRESS Skipped chunk 55 size 11178639, 91.11MB/s 00:00:03 56.3%
2024-07-11 16:22:54.290 INFO UPLOAD_PROGRESS Uploaded chunk 56 size 2658946, 91.95MB/s 00:00:03 56.8%
2024-07-11 16:22:54.317 INFO UPLOAD_PROGRESS Uploaded chunk 57 size 3574002, 93.09MB/s 00:00:03 57.5%
2024-07-11 16:22:54.346 INFO UPLOAD_PROGRESS Skipped chunk 59 size 1437569, 93.55MB/s 00:00:03 57.8%
2024-07-11 16:22:54.357 INFO UPLOAD_PROGRESS Uploaded chunk 58 size 5222985, 95.21MB/s 00:00:03 58.8%
2024-07-11 16:22:54.413 INFO UPLOAD_PROGRESS Uploaded chunk 60 size 6617486, 97.31MB/s 00:00:02 60.1%
2024-07-11 16:22:54.464 INFO UPLOAD_PROGRESS Skipped chunk 61 size 10209771, 100.55MB/s 00:00:02 62.1%
2024-07-11 16:22:54.557 INFO UPLOAD_PROGRESS Skipped chunk 62 size 13344445, 104.80MB/s 00:00:02 64.7%
2024-07-11 16:22:54.657 INFO UPLOAD_PROGRESS Skipped chunk 63 size 8239662, 107.42MB/s 00:00:02 66.4%
2024-07-11 16:22:54.703 INFO UPLOAD_PROGRESS Skipped chunk 64 size 7032812, 109.65MB/s 00:00:02 67.7%
2024-07-11 16:22:54.721 INFO UPLOAD_PROGRESS Skipped chunk 65 size 2955945, 110.59MB/s 00:00:02 68.3%
2024-07-11 16:22:54.769 INFO UPLOAD_PROGRESS Skipped chunk 66 size 6877758, 112.78MB/s 00:00:02 69.7%
2024-07-11 16:22:54.775 INFO UPLOAD_PROGRESS Skipped chunk 67 size 1321264, 113.20MB/s 00:00:02 69.9%
2024-07-11 16:22:54.795 INFO UPLOAD_PROGRESS Skipped chunk 68 size 2844167, 114.10MB/s 00:00:02 70.5%
2024-07-11 16:22:54.871 INFO UPLOAD_PROGRESS Skipped chunk 69 size 11205211, 117.66MB/s 00:00:02 72.7%
2024-07-11 16:22:54.901 INFO UPLOAD_PROGRESS Uploaded chunk 70 size 3154577, 118.67MB/s 00:00:02 73.3%
2024-07-11 16:22:54.983 INFO UPLOAD_PROGRESS Uploaded chunk 71 size 9084545, 121.55MB/s 00:00:01 75.1%
2024-07-11 16:22:55.162 INFO UPLOAD_PROGRESS Uploaded chunk 72 size 16777216, 95.17MB/s 00:00:02 78.4%
2024-07-11 16:22:55.196 INFO UPLOAD_PROGRESS Uploaded chunk 73 size 3845546, 96.08MB/s 00:00:02 79.2%
2024-07-11 16:22:55.208 INFO UPLOAD_PROGRESS Uploaded chunk 74 size 2403229, 96.66MB/s 00:00:02 79.6%
2024-07-11 16:22:55.222 INFO UPLOAD_PROGRESS Uploaded chunk 75 size 3892510, 97.58MB/s 00:00:01 80.4%
2024-07-11 16:22:55.503 INFO UPLOAD_PROGRESS Uploaded chunk 76 size 8502329, 99.61MB/s 00:00:01 82.1%
2024-07-11 16:22:55.511 INFO UPLOAD_PROGRESS Uploaded chunk 77 size 2016426, 100.09MB/s 00:00:01 82.5%
2024-07-11 16:22:55.546 INFO UPLOAD_PROGRESS Uploaded chunk 78 size 10127863, 102.51MB/s 00:00:01 84.4%
2024-07-11 16:22:55.550 INFO UPLOAD_PROGRESS Uploaded chunk 79 size 1054734, 102.76MB/s 00:00:01 84.7%
2024-07-11 16:22:55.597 INFO UPLOAD_PROGRESS Uploaded chunk 80 size 8554693, 104.80MB/s 00:00:01 86.3%
2024-07-11 16:22:55.649 INFO UPLOAD_PROGRESS Uploaded chunk 81 size 5134119, 106.02MB/s 00:00:01 87.3%
2024-07-11 16:22:55.660 INFO UPLOAD_PROGRESS Uploaded chunk 82 size 1410834, 106.36MB/s 00:00:01 87.6%
2024-07-11 16:22:55.704 INFO UPLOAD_PROGRESS Uploaded chunk 83 size 5098685, 107.57MB/s 00:00:01 88.6%
2024-07-11 16:22:55.748 INFO UPLOAD_PROGRESS Uploaded chunk 84 size 3935138, 108.51MB/s 00:00:01 89.4%
2024-07-11 16:22:55.774 INFO UPLOAD_PROGRESS Uploaded chunk 85 size 2855230, 109.19MB/s 00:00:01 90.0%
2024-07-11 16:22:55.861 INFO UPLOAD_PROGRESS Uploaded chunk 86 size 9638665, 111.49MB/s 00:00:01 91.9%
2024-07-11 16:22:55.927 INFO UPLOAD_PROGRESS Uploaded chunk 87 size 8777508, 113.58MB/s 00:00:01 93.6%
2024-07-11 16:22:55.953 INFO UPLOAD_PROGRESS Uploaded chunk 88 size 4399394, 114.63MB/s 00:00:01 94.4%
2024-07-11 16:22:55.988 INFO UPLOAD_PROGRESS Uploaded chunk 89 size 4072060, 115.60MB/s 00:00:01 95.2%
2024-07-11 16:22:56.005 INFO UPLOAD_PROGRESS Uploaded chunk 90 size 2256990, 92.91MB/s 00:00:01 95.7%
2024-07-11 16:22:56.029 INFO UPLOAD_PROGRESS Uploaded chunk 91 size 2732508, 93.43MB/s 00:00:01 96.2%
2024-07-11 16:22:56.051 INFO UPLOAD_PROGRESS Uploaded chunk 92 size 3152317, 94.04MB/s 00:00:01 96.8%
2024-07-11 16:22:56.082 INFO UPLOAD_PROGRESS Uploaded chunk 93 size 3963821, 94.79MB/s 00:00:01 97.6%
2024-07-11 16:22:56.117 INFO UPLOAD_PROGRESS Uploaded chunk 94 size 4773318, 95.70MB/s 00:00:01 98.6%
2024-07-11 16:22:56.331 INFO UPLOAD_FILE <REDACTED>
2024-07-11 16:22:56.363 INFO BACKUP_END Backup for /old/pcm_temp at revision 1 completed
2024-07-11 16:22:56.364 INFO BACKUP_STATS Files: 14698 total, 567,268K bytes; 7196 new, 489,993K bytes
2024-07-11 16:22:56.364 INFO BACKUP_STATS File chunks: 118 total, 567,301K bytes; 76 new, 395,716K bytes, 325,949K bytes uploaded
2024-07-11 16:22:56.364 INFO BACKUP_STATS Metadata chunks: 4 total, 2,566K bytes; 4 new, 2,566K bytes, 1,272K bytes uploaded
2024-07-11 16:22:56.364 INFO BACKUP_STATS All chunks: 122 total, 569,868K bytes; 80 new, 398,282K bytes, 327,221K bytes uploaded
2024-07-11 16:22:56.364 INFO BACKUP_STATS Total running time: 00:00:05

Storage set to /tmp/duplicacy_test1/
Listing all chunks
1 snapshots and 1 revisions
Total chunk size is 327,221K in 80 chunks
Chunk 859631742736a16d6509264de8663e748665a93894a5f75027cc4d33a775159b referenced by snapshot test1 at revision 1 does not exist
Chunk b62262be30c4e25fb5436918017b260d8c48b27e0dcf353964e304f3d451331d referenced by snapshot test1 at revision 1 does not exist
Chunk 777c1d10b1fe27ca98da47964c22ec7772e61999a8525a43000b3474e5a63e97 referenced by snapshot test1 at revision 1 does not exist
Chunk 062ef78398f13ea5f6770297962759387ccdaa03e0d5f3ef76918ce5b2e7f636 referenced by snapshot test1 at revision 1 does not exist
Chunk 2dd6519c40b103fb4ec3cae86a39e636c211d8a5b736ec80e2f04d50ee3c88f7 referenced by snapshot test1 at revision 1 does not exist
Chunk 91eee81f1d2d9876bb59491473577ea940c934acb21402b33b922a1f0f28fbfd referenced by snapshot test1 at revision 1 does not exist
Chunk 49b70807fd3fa21c141bdceea7f8f7c7adc37bf8ed0044af6c4e0bdfc2a46b9c referenced by snapshot test1 at revision 1 does not exist
Chunk 8958511ef524ac68d739b287b07ff30c606c15efcb4fb7015227c3f395f9b52e referenced by snapshot test1 at revision 1 does not exist
Chunk 9d4ceee579b5abac3ce2c9a7744ffdbcb4d45fb00f80bbabdc164430e769a85b referenced by snapshot test1 at revision 1 does not exist
Chunk 66ef72effeb165decbcef11bc1661432d9d322ab5e4cf4563d8f0b21ebfde5b1 referenced by snapshot test1 at revision 1 does not exist
Chunk 714c0604008d9f546e1b791afef50ef48b221dd7ec14034080ed71807b7293e2 referenced by snapshot test1 at revision 1 does not exist
Chunk 1d39b4239c874d6ce6c4db516eb2be0400aae287e8018cf8a305389243ff2e6d referenced by snapshot test1 at revision 1 does not exist
Chunk 4fbf3306ad600200e4b7b6cdb669954e850cfcecabe18c4f0a3f7758da338b2c referenced by snapshot test1 at revision 1 does not exist
Chunk 67aac079d6f11438e0b2bc5608d6678c28a893e0b8b24d29dae77d7889b1ac49 referenced by snapshot test1 at revision 1 does not exist
Chunk ad8d27bae7cf707436f6a31ee12c188ae4a11db5faba49b11bc7526cb8c58bd3 referenced by snapshot test1 at revision 1 does not exist
Chunk 449f89a40b702ec1e79a010f016d70c3f97719912ec9745c5894967cf2e7c53b referenced by snapshot test1 at revision 1 does not exist
Chunk 2972ac5177b23fb2220e37a1907c5486be81a838bcb4fac705c66b6cd7a39209 referenced by snapshot test1 at revision 1 does not exist
Chunk 4e655368a5a930865790ad3d36b03a1992ca4769cba776b1c21122df7c091fd7 referenced by snapshot test1 at revision 1 does not exist
Chunk ffecdf89da3b42f650d2a4cae5c5fcda058edd13ef99b228f6d8989d41ee8b19 referenced by snapshot test1 at revision 1 does not exist
Chunk c504ae5d566dacade066cd8f272bccac821fdb2e9480146771ea28c8dd344174 referenced by snapshot test1 at revision 1 does not exist
Chunk 410f38b21b94639b2a80d598f528a30c501aae1444a2f760fe2ab60429e7feb8 referenced by snapshot test1 at revision 1 does not exist
Chunk ca594f092f6c129838a982a6328185f5f18c517444e53af41f63c883c10aa271 referenced by snapshot test1 at revision 1 does not exist
Chunk de57f019b23c569764f65dd155155f7f11d76c5557b2270d9f97ff04398314cc referenced by snapshot test1 at revision 1 does not exist
Some chunks referenced by snapshot test1 at revision 1 are missing
Some chunks referenced by some snapshots do not exist in the storage

Tested with Duplicacy 3.2.3 (254953) on Linux.

#8

Thank you for your detailed instructions! I was able to reproduce the bug and find out what was wrong.

The fixed has been checked in: Don't save the incomplete snapshot for a dry run · gilbertchen/duplicacy@69f5d2f · GitHub

4 Likes
#9

Hi @gchen any date for a duplicacy cli release with that fix ? (And other fix too :slight_smile: )

#10

@gchen probably breaking convention asking for a release date but as this fix and few others are in the commit list but haven’t been released yet.

Concerned I may break the backup by getting an aborted dry run :cry:

+1 vote for a command line release :+1:

Thanks

1 Like
#11

@gchen, most users lack the knowledge to build the executable from the source code, which is why these requests are common. I believe releasing a new CLI version would be great.

1 Like
#12

In addition, why not setup nightly build pipeline so that people who are so inclined can download and use bleeding edge without needing to bug the developer every time.

1 Like
#13

I’ll make a new CLI version by next week. The night build pipeline may require some extra work, mostly because the mac versions need to be code signed and notarized which can only be done on my own machine.

4 Likes
#14

@gchen this is great news. As I use dry-run to test things I just automatically think I can do anything and it will do no harm but this bug in particularly may be giving me false security. I am surprised I haven’t broken things already but the last time I tested my backup everything was restorable. Thanks again.

#15

Great news, and thanks for your work.

#16

Could you not use a self-hosted runner in your pipeline?

Incidentally, there’s a number of dependencies with high severity vulns - as you can see from my cloned repo (I just enabled dependabot). Nothing too critical on the face of it tho. Since you’re intending to release a new CLI soon, might be worth bumping some of these library versions (and migrate jwt-go to golang-jwt etc.)?

1 Like