OneDrive Business: ERROR DOWNLOAD_CHUNK

[stefan1]

Hi there,

I now tried the third time to backup to onedrive for business but it always fails on restore.

After I try to restore some things from my 4TB NAS Backup (using docker container), I always got errors downloading chunk files. These files where displayed inside the onedrive webinterface, but not in OneDrive Client (Windows) and seems to be invisible to other software too. So i followed the guide for missing chunks (deleted it inside the webinterface and let Duplicacy recreate it as theses files never change). This seems to work but after that the next was not possible to download. So I played this game fo about 5 times and then deleted the 4TB archive.

As the first upload took about 2 weeks, I decided to start with smaller parts. So I moved all files to another folder and then copied about 40-100 GB files in the backup location and started a backup. Everything seems to be fine again and after an initial test with 5GB everything could be restored. But after I had about 400GB I tried to restore everything again and as described before I got the identical szenario - file not found.

After that I decided to start from a fresh duplicacy installtion. So i removed all configs, caches and logs and configured it from new. But shortly after that - same problem.

So my question now is: Can someone please help me? As I am using this software now to backup my NAS, Webserver and a important PC, I need a solution for this. (All checks said “everything is ok”.)

Environment:
Docker Image: Docker Hub (running on QNAP)
Web-UI: 1.4.0
Commandline: 2.6.1

Logfile of the last restore process:

2020-08-21 02:31:34.201 INFO DOWNLOAD_PROGRESS Downloaded chunk 554 size 3710943, 4.24MB/s 1 day 01:41:06 0.4%
2020-08-21 02:31:34.311 INFO ONEDRIVE_RETRY Response code: 429; retry after 1203 milliseconds
2020-08-21 02:31:34.383 INFO ONEDRIVE_RETRY Response code: 429; retry after 1017 milliseconds
2020-08-21 02:31:34.525 INFO ONEDRIVE_RETRY Response code: 429; retry after 1068 milliseconds
2020-08-21 02:31:34.640 INFO ONEDRIVE_RETRY Response code: 429; retry after 1183 milliseconds
2020-08-21 02:31:34.855 INFO ONEDRIVE_RETRY Response code: 429; retry after 682 milliseconds
2020-08-21 02:31:35.109 INFO ONEDRIVE_RETRY Response code: 429; retry after 853 milliseconds
2020-08-21 02:31:35.271 INFO ONEDRIVE_RETRY Response code: 429; retry after 3665 milliseconds
2020-08-21 02:31:35.597 INFO ONEDRIVE_RETRY Response code: 429; retry after 1240 milliseconds
2020-08-21 02:31:35.831 INFO ONEDRIVE_RETRY Response code: 429; retry after 1985 milliseconds
2020-08-21 02:31:36.040 INFO ONEDRIVE_RETRY Response code: 429; retry after 4420 milliseconds
2020-08-21 02:31:36.043 INFO ONEDRIVE_RETRY Response code: 429; retry after 2279 milliseconds
2020-08-21 02:31:36.079 INFO ONEDRIVE_RETRY Response code: 429; retry after 248 milliseconds
2020-08-21 02:31:36.585 INFO ONEDRIVE_RETRY Response code: 429; retry after 1242 milliseconds
2020-08-21 02:31:36.794 INFO ONEDRIVE_RETRY Response code: 429; retry after 2189 milliseconds
2020-08-21 02:31:36.794 INFO ONEDRIVE_RETRY Response code: 429; retry after 6912 milliseconds
2020-08-21 02:31:37.325 INFO ONEDRIVE_RETRY Response code: 429; retry after 2955 milliseconds
2020-08-21 02:31:37.902 INFO DOWNLOAD_PROGRESS Downloaded chunk 558 size 7459906, 4.23MB/s 1 day 01:45:57 0.4%
2020-08-21 02:31:38.156 INFO ONEDRIVE_RETRY Response code: 429; retry after 6920 milliseconds
2020-08-21 02:31:38.222 INFO ONEDRIVE_RETRY Response code: 429; retry after 5743 milliseconds
2020-08-21 02:31:38.440 INFO ONEDRIVE_RETRY Response code: 429; retry after 7781 milliseconds
2020-08-21 02:31:39.398 INFO ONEDRIVE_RETRY Response code: 429; retry after 2489 milliseconds
2020-08-21 02:31:39.515 INFO ONEDRIVE_RETRY Response code: 429; retry after 636 milliseconds
2020-08-21 02:31:40.606 INFO ONEDRIVE_RETRY Response code: 429; retry after 13690 milliseconds
2020-08-21 02:31:40.665 INFO ONEDRIVE_RETRY Response code: 429; retry after 6685 milliseconds
2020-08-21 02:31:40.857 INFO ONEDRIVE_RETRY Response code: 429; retry after 9891 milliseconds
2020-08-21 02:31:42.280 INFO ONEDRIVE_RETRY Response code: 429; retry after 16709 milliseconds
2020-08-21 02:31:44.369 INFO ONEDRIVE_RETRY Response code: 429; retry after 391 milliseconds
2020-08-21 02:31:45.330 INFO ONEDRIVE_RETRY Response code: 429; retry after 31476 milliseconds
2020-08-21 02:31:45.506 INFO ONEDRIVE_RETRY Response code: 429; retry after 9829 milliseconds
2020-08-21 02:31:45.556 INFO DOWNLOAD_PROGRESS Downloaded chunk 551 size 4659049, 4.16MB/s 1 day 02:11:31 0.4%
2020-08-21 02:31:46.586 INFO ONEDRIVE_RETRY Response code: 429; retry after 11087 milliseconds
2020-08-21 02:31:47.698 INFO ONEDRIVE_RETRY Response code: 429; retry after 1357 milliseconds
2020-08-21 02:31:49.475 INFO ONEDRIVE_RETRY Response code: 429; retry after 6519 milliseconds
2020-08-21 02:31:51.159 INFO ONEDRIVE_RETRY Response code: 429; retry after 31573 milliseconds
2020-08-21 02:31:54.719 INFO ONEDRIVE_RETRY Response code: 429; retry after 62032 milliseconds
2020-08-21 02:31:55.658 INFO ONEDRIVE_RETRY Response code: 429; retry after 23900 milliseconds
2020-08-21 02:31:56.432 INFO ONEDRIVE_RETRY Response code: 429; retry after 3305 milliseconds
2020-08-21 02:31:58.127 INFO ONEDRIVE_RETRY Response code: 429; retry after 7341 milliseconds
2020-08-21 02:31:59.611 INFO ONEDRIVE_RETRY Response code: 429; retry after 56508 milliseconds
2020-08-21 02:32:00.380 INFO ONEDRIVE_RETRY Response code: 429; retry after 30198 milliseconds
2020-08-21 02:32:06.112 INFO ONEDRIVE_RETRY Response code: 429; retry after 6468 milliseconds
2020-08-21 02:32:13.122 INFO ONEDRIVE_RETRY Response code: 429; retry after 59312 milliseconds
2020-08-21 02:32:17.056 INFO ONEDRIVE_RETRY Response code: 429; retry after 20730 milliseconds
2020-08-21 02:32:20.144 INFO ONEDRIVE_RETRY Response code: 429; retry after 35710 milliseconds
2020-08-21 02:32:23.348 INFO ONEDRIVE_RETRY Response code: 429; retry after 45548 milliseconds
2020-08-21 02:32:30.579 ERROR DOWNLOAD_CHUNK Failed to download the chunk 93c108ec7eb7e1dc2a7cfa6852b0bc1e013cbcb7eb278821a37b83fb177790a3: Maximum number of retries reached

[gchen]

A response code of 429 means too many requests. How many threads did you use and what was the download rate at the beginning?

[stefan1]

I used 10 threads, because if I just use one it only has about 1MB/s to restore.

Now I removed all commandline parameters and tried to restore, but I only get this:

2020-08-21 11:59:18.441 INFO REPOSITORY_SET Repository set to X:/Sam2
2020-08-21 11:59:18.489 INFO STORAGE_SET Storage set to odb://QNAP_Duplicacy
2020-08-21 11:59:20.190 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2020-08-21 11:59:21.834 INFO RESTORE_INPLACE Forcing in-place mode with a non-default preference path
2020-08-21 11:59:23.529 INFO SNAPSHOT_FILTER Parsing filter file \\?\C:\Users\stefa\.duplicacy-web\repositories\localhost\restore\.duplicacy\filters
2020-08-21 11:59:23.530 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2020-08-21 11:59:23.632 INFO RESTORE_START Restoring X:/Sam2 to revision 6
2020-08-21 12:00:02.182 INFO DOWNLOAD_PROGRESS Downloaded chunk 1 size 3859035, 97KB/s 41 days 19:03:25 0.0%
2020-08-21 12:00:04.764 WARN DOWNLOAD_RETRY Failed to decrypt the chunk 0b8bf46dc01e62b224985bcf4fa7b82bddf7ca3517f950ac348ba04e6c798cf7: cipher: message authentication failed; retrying
2020-08-21 12:00:06.047 WARN DOWNLOAD_RETRY Failed to decrypt the chunk 0b8bf46dc01e62b224985bcf4fa7b82bddf7ca3517f950ac348ba04e6c798cf7: cipher: message authentication failed; retrying
2020-08-21 12:00:07.471 WARN DOWNLOAD_RETRY Failed to decrypt the chunk 0b8bf46dc01e62b224985bcf4fa7b82bddf7ca3517f950ac348ba04e6c798cf7: cipher: message authentication failed; retrying
2020-08-21 12:00:08.748 ERROR DOWNLOAD_DECRYPT Failed to decrypt the chunk 0b8bf46dc01e62b224985bcf4fa7b82bddf7ca3517f950ac348ba04e6c798cf7: cipher: message authentication failed

This is the same I had bevore. If i delete 0b8bf46dc01e62b224985bcf4fa7b82bddf7ca3517f950ac348ba04e6c798cf7 and let the backup recreate it, there will be another one which is broken.

Is OneDrive for Business tested and ok?

[Droolio]

Simply deleting the bad chunk won’t force the next backup to recreate it. As far as Duplicacy is concerned, it’s still there - it would be too expensive to check the existance for every chunk when running backups. All chunks referenced in the last snapshot are assumed to exist.

To fix your storage, you’ll need to change the backup ID (or add a new, temporary, backup ID - pointing to the same local repository) and run that. It’ll avoid re-uploading (actual) existing chunks and may replace your missing chunk.

There’s a possibility the chunk is bundled with older data, and may not be restored. In which case, you could discard the old backup ID and continue using the fresh new ID from then on.

There’s also the possibility you have other corrupt chunks, in which case your new backup will succeed but may still reference a bad chunk. (Duplicacy saw the chunk existed, but didn’t check that it was valid.) You may have to continue the process of deleting corrupted chunks (after re-testing with check -files/chunks or a restore) and doing the above all over.

Unfortunately, I don’t know of a good way to get a complete list of bad chunks - so that you can perform the fix in one go. Duplicacy exits when encountering bad chunks, although can skip and ignore missing chunks when running check til the end. (I wonder if it can skip both missing and bad chunks when doing check -chunks?)

The good news is, doing a fresh backup or restore shouldn’t re-upload or re-download unnecessary data. On backup, chunks that exist on the storage should be skipped (even with a fresh ID). On restore, you can restore to an existing restore location, and all previously restored files will be skipped. So you should be able to fix the storage after a few iterations of the above, using a temporary restore location. Avoid the temptation to ‘start from scratch’ - the storage can be fixed, with a few tricks, and it’s useful to learn how.

One thing you’ll want to do after fixing a storage is to run prune -exhaustive and to check -all with either -chunks or -files -r <last> - to remove bad chunks and identify and remove bad snapshots.

As to whether OneDrive is suitable or not. Personally, I wouldn’t use OneDrive personal or business. I’ve just not had good experiences with their sync process.

[stefan1]

@Droolio thank you very much for your detailed information!

I try now the check and prune commands if I can recover my initial Backup with 4TB. All in all I had never troubles with OneDrive for Businness. And as this is a cheap cloud storage for me (25TB is enought), it is perfect.

The check -chunks seems to download every chunk - so it could take a while with about 5-10 MB/s. I write again as soon as it has finished.

But all in all i am searching for a good backup solution for my QNAP Nas. And using Docker and Duplicacy seemed to be a really good solution. At the moment I am not sure if it is because if a backup solution says “backup is ok”, then it should be fine. But something happened which prevents duplicacy from working as usual. Maybe it is not able to handle errors during upload? Or could it happen if you stop a running backup process? Dont know.

[stefan1]

Ok I tried it multiple times to upload using OneDrive for Business. The upload is going very well - there are no errors.

But as soon as I download anything I get chunk errors as described above. If I try to check using ‘-chunks’ or ‘-files’ it starts to check all chunks, and as soon as it starts downloading chunks it fails.

In my eyes there is a bug or error in the connector for OneDrive for Business. At the moment I tried it with different storages on my account and made sure that no other machine connects to that account. But none of the storages worked.

Even if I only used it now for about 2 months and already bought 3 licenses for it - i am loving this tool. But as it is not working I cannot use it anymore till there is a solution. During test now I uploaded about 10TB with a customer internet connection with only 5MB/s at maximum. That took weeks and now all is garbage.

I read about the SFTP bug which ignored upload errors. Could it be, that this (or something like that) is happening here too?

All in all it would be great if the dev team could take a look at this problem.

[gchen]

OneDrive for Business does use a different API to upload large files unlike OneDrive for Personal. But I double checked the code and there isn’t anything obviously wrong like the sftp bug. Besides, what happened in the sftp backend is unlikely to happen in http-based backends like OneDrive, because the content length is always passed to the server when uploading a file so the file would never get truncated.

Would you be interested to test New Feature: Erasure Coding that can protect against corrupt chunks? I would start with a smaller backup, something in the a few hundred GB range.

[stefan1]

Would love to test anything which should help!

At the moment I am using it inside a docker container with the webinterface. Is there a tutorial how to enable/use this with the webinterface?

(And the webinterface is using version 2.6.1 - don’t know how to force update to a newer or beta version which contains this.)

[stefan1]

@gchen I now set up the Erasure Coding and backed up a 250GB folder. The Backup process was without any errors but it seems that restore is not working as good (see log files below).

The error code 429 is telling me during the test restore process, that the connection is throttled. I search for this error code and it seems like, that OneDrive is throttling connections from unknown / unverified tools during business hours.

If this happens, it seems that there is a retry value which also seems to be used by Duplicacy to wait before connecting again. But after all the restore hits the maximum number of retries which exist the restore process.

At the moment I do not know what to do, but maybe you have an idea. Maybe it is possible to add some configurable waiting time after X throttling errors happen? E.g. if maximum errors by code 429 are reached, wait 2 minutes and then try again?!?

Links:
https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

Log:

2020-09-11 13:43:35.839 INFO ONEDRIVE_RETRY Response code: 429; retry after 384 milliseconds
2020-09-11 13:43:36.400 INFO ONEDRIVE_RETRY Response code: 429; retry after 1235 milliseconds
2020-09-11 13:43:37.808 INFO ONEDRIVE_RETRY Response code: 429; retry after 2028 milliseconds
2020-09-11 13:43:40.565 INFO DOWNLOAD_PROGRESS Downloaded chunk 6614 size 6003182, 3.24MB/s 1 day 01:32:28 1.7%
2020-09-11 13:43:41.182 INFO DOWNLOAD_PROGRESS Downloaded chunk 6615 size 3080450, 3.24MB/s 1 day 01:32:32 1.7%
2020-09-11 13:43:41.851 INFO DOWNLOAD_PROGRESS Downloaded chunk 6616 size 2287418, 3.24MB/s 1 day 01:31:53 1.7%
2020-09-11 13:43:42.172 INFO ONEDRIVE_RETRY Response code: 429; retry after 53 milliseconds
2020-09-11 13:43:42.737 INFO DOWNLOAD_PROGRESS Downloaded chunk 6617 size 2244674, 3.24MB/s 1 day 01:32:12 1.7%
2020-09-11 13:43:43.040 INFO ONEDRIVE_RETRY Response code: 429; retry after 21 milliseconds
2020-09-11 13:43:43.231 INFO ONEDRIVE_RETRY Response code: 429; retry after 945 milliseconds
2020-09-11 13:43:44.554 INFO DOWNLOAD_PROGRESS Downloaded chunk 6618 size 1776449, 3.23MB/s 1 day 01:33:35 1.7%
2020-09-11 13:43:45.610 INFO DOWNLOAD_PROGRESS Downloaded chunk 6619 size 10444153, 3.24MB/s 1 day 01:31:34 1.7%
2020-09-11 13:43:45.951 INFO ONEDRIVE_RETRY Response code: 429; retry after 877 milliseconds
2020-09-11 13:43:47.017 INFO ONEDRIVE_RETRY Response code: 429; retry after 1371 milliseconds
2020-09-11 13:43:48.910 INFO DOWNLOAD_PROGRESS Downloaded chunk 6620 size 3287056, 3.23MB/s 1 day 01:33:28 1.7%
2020-09-11 13:43:49.674 INFO DOWNLOAD_PROGRESS Downloaded chunk 6621 size 4883920, 3.23MB/s 1 day 01:33:01 1.7%
2020-09-11 13:43:50.640 INFO DOWNLOAD_PROGRESS Downloaded chunk 6622 size 2338204, 3.23MB/s 1 day 01:33:18 1.7%
2020-09-11 13:43:51.309 INFO DOWNLOAD_PROGRESS Downloaded chunk 6623 size 3319022, 3.23MB/s 1 day 01:33:18 1.7%
2020-09-11 13:43:51.893 INFO DOWNLOAD_PROGRESS Downloaded chunk 6624 size 2487499, 3.23MB/s 1 day 01:32:36 1.7%
2020-09-11 13:43:52.186 INFO ONEDRIVE_RETRY Response code: 429; retry after 309 milliseconds
2020-09-11 13:43:52.664 INFO ONEDRIVE_RETRY Response code: 429; retry after 1376 milliseconds
2020-09-11 13:43:54.256 INFO ONEDRIVE_RETRY Response code: 429; retry after 3927 milliseconds
2020-09-11 13:43:58.356 INFO ONEDRIVE_RETRY Response code: 429; retry after 6969 milliseconds
2020-09-11 13:44:05.504 INFO ONEDRIVE_RETRY Response code: 429; retry after 3240 milliseconds
2020-09-11 13:44:08.921 INFO ONEDRIVE_RETRY Response code: 429; retry after 19729 milliseconds
2020-09-11 13:44:28.711 INFO ONEDRIVE_RETRY Response code: 429; retry after 58922 milliseconds
2020-09-11 13:45:27.978 INFO ONEDRIVE_RETRY Response code: 429; retry after 87885 milliseconds
2020-09-11 13:46:55.863 ERROR DOWNLOAD_CHUNK Failed to download the chunk 1d6e7723ba5a4031b3ef610d8261cf3d52d4b6d83adb13d392a9149483de5835: Maximum number of retries reached
\ No newline at end of selection

[gchen]

We can try 2 things:

• Wait for the delay specified by the Retry-After http header
• Fill in the User-Agent information to decorate the traffic.

I’ll send you a new binary with these improvements over the weekend.

In the meantime, can you double check the storage to make sure the storage is protected by erasure coding? If the web GUI still uses the old CLI to run the backup it may ignore the erasure coding setting. You can download any chunk file (under chunks) or any snapshot file (under snapshots) and view it in binary mode. The file should start with duplicacy\003.

[stefan1]

Hi @gchen - first thank you very much for your help.

I double checked the storage by downloading a chunk file (random) and the latest snapshot file. But as I opened both they started with “duplicacy.&.” so no “\003” (or is this the encoded “&”?). If this is not correct I would need a example how to enable it using the Webinterface (last time I created a storage space using the 2.6.10 commandline with “-erasure-coding 8:3” and enabled nesting usinga the “nesting” config file (see below), then I added it to the webinterface by providing the onedrive file and the path I provided as storage location by the commandline tool).

About the restore process:
Yesterday I restarted the restore process again. And it went fine till midnight (see last loglines below, restored over 50%). At the moment (0 o’clock), my webserver is starting backups with duplicacy. At this time the restore process stopped working. Maybe it has something to do with the throttling. How manyconnections is Duplicacy making for backup and for restore?

"nesting" file:

{
“read-levels”: [2],
“write-level”: 2
}

restore log file as it stopped working:

…
2020-09-12 00:03:48.427 INFO DOWNLOAD_PROGRESS Downloaded chunk 35111 size 1416372, 6.04MB/s 05:44:26 57.4%
2020-09-12 00:03:49.035 INFO DOWNLOAD_PROGRESS Downloaded chunk 35112 size 1488371, 6.04MB/s 05:44:27 57.4%
2020-09-12 00:03:49.689 INFO DOWNLOAD_PROGRESS Downloaded chunk 35113 size 1623428, 6.04MB/s 05:44:26 57.4%
2020-09-12 00:03:50.450 INFO DOWNLOAD_PROGRESS Downloaded chunk 35114 size 4615029, 6.04MB/s 05:44:26 57.4%
2020-09-12 00:03:51.064 INFO DOWNLOAD_PROGRESS Downloaded chunk 35115 size 1725453, 6.04MB/s 05:44:26 57.4%
2020-09-12 00:03:51.687 INFO DOWNLOAD_PROGRESS Downloaded chunk 35116 size 2709343, 6.04MB/s 05:44:25 57.4%
2020-09-12 00:03:52.793 INFO DOWNLOAD_PROGRESS Downloaded chunk 35117 size 16777216, 6.04MB/s 05:44:21 57.4%
2020-09-12 00:03:53.773 INFO DOWNLOAD_PROGRESS Downloaded chunk 35118 size 14062801, 6.04MB/s 05:44:18 57.4%
2020-09-12 00:03:54.405 INFO DOWNLOAD_PROGRESS Downloaded chunk 35119 size 3593659, 6.04MB/s 05:44:18 57.4%
2020-09-12 00:03:55.218 INFO DOWNLOAD_PROGRESS Downloaded chunk 35120 size 11076002, 6.04MB/s 05:44:16 57.4%
2020-09-12 00:03:56.352 INFO DOWNLOAD_PROGRESS Downloaded chunk 35121 size 16777216, 6.04MB/s 05:44:12 57.4%
2020-09-12 00:03:56.946 INFO DOWNLOAD_PROGRESS Downloaded chunk 35122 size 1260199, 6.04MB/s 05:44:12 57.4%
2020-09-12 00:03:57.935 INFO DOWNLOAD_PROGRESS Downloaded chunk 35123 size 3182914, 6.04MB/s 05:44:11 57.4%
2020-09-12 00:03:58.828 INFO DOWNLOAD_PROGRESS Downloaded chunk 35124 size 16777216, 6.04MB/s 05:44:08 57.4%
2020-09-12 00:03:59.633 INFO DOWNLOAD_PROGRESS Downloaded chunk 35125 size 14451745, 6.04MB/s 05:44:04 57.4%
2020-09-12 00:04:00.198 INFO DOWNLOAD_PROGRESS Downloaded chunk 35126 size 4076734, 6.04MB/s 05:44:04 57.4%
2020-09-12 00:04:00.889 INFO DOWNLOAD_PROGRESS Downloaded chunk 35127 size 4706470, 6.04MB/s 05:44:03 57.4%
2020-09-12 00:04:01.581 INFO DOWNLOAD_PROGRESS Downloaded chunk 35128 size 2296952, 6.04MB/s 05:44:03 57.4%
2020-09-12 00:04:02.318 INFO DOWNLOAD_PROGRESS Downloaded chunk 35129 size 16777216, 6.04MB/s 05:43:59 57.4%
2020-09-12 00:04:02.746 INFO DOWNLOAD_PROGRESS Downloaded chunk 35130 size 2073865, 6.04MB/s 05:43:58 57.4%
2020-09-12 00:04:03.156 INFO DOWNLOAD_PROGRESS Downloaded chunk 35131 size 1323783, 6.04MB/s 05:43:59 57.4%
2020-09-12 00:04:03.541 INFO DOWNLOAD_PROGRESS Downloaded chunk 35132 size 2816354, 6.04MB/s 05:43:58 57.4%
2020-09-12 00:04:03.994 INFO DOWNLOAD_PROGRESS Downloaded chunk 35133 size 3656824, 6.04MB/s 05:43:57 57.4%
2020-09-12 00:04:04.453 INFO DOWNLOAD_PROGRESS Downloaded chunk 35134 size 3111257, 6.04MB/s 05:43:57 57.4%
2020-09-12 00:04:04.949 INFO DOWNLOAD_PROGRESS Downloaded chunk 35135 size 7810063, 6.04MB/s 05:43:55 57.4%
2020-09-12 00:04:05.527 INFO DOWNLOAD_PROGRESS Downloaded chunk 35136 size 5273520, 6.04MB/s 05:43:54 57.4%
2020-09-12 00:04:06.010 INFO DOWNLOAD_PROGRESS Downloaded chunk 35137 size 5054427, 6.04MB/s 05:43:53 57.4%
2020-09-12 00:04:06.614 INFO DOWNLOAD_PROGRESS Downloaded chunk 35138 size 6461668, 6.04MB/s 05:43:52 57.4%
2020-09-12 00:04:07.169 INFO DOWNLOAD_PROGRESS Downloaded chunk 35139 size 5621154, 6.04MB/s 05:43:51 57.4%
2020-09-12 00:04:07.778 INFO DOWNLOAD_PROGRESS Downloaded chunk 35140 size 3311416, 6.04MB/s 05:43:50 57.4%
2020-09-12 00:04:08.698 INFO DOWNLOAD_PROGRESS Downloaded chunk 35141 size 7193597, 6.04MB/s 05:43:49 57.4%
2020-09-12 00:04:09.390 INFO DOWNLOAD_PROGRESS Downloaded chunk 35142 size 2361422, 6.04MB/s 05:43:49 57.4%
2020-09-12 00:04:09.962 INFO DOWNLOAD_PROGRESS Downloaded chunk 35143 size 2284590, 6.04MB/s 05:43:48 57.4%
2020-09-12 00:04:10.521 INFO DOWNLOAD_PROGRESS Downloaded chunk 35144 size 1082763, 6.04MB/s 05:43:48 57.4%
2020-09-12 00:04:11.077 INFO DOWNLOAD_PROGRESS Downloaded chunk 35145 size 2817214, 6.04MB/s 05:43:48 57.4%
2020-09-12 00:04:11.555 INFO DOWNLOAD_PROGRESS Downloaded chunk 35146 size 1522183, 6.04MB/s 05:43:48 57.5%
2020-09-12 00:04:12.381 INFO DOWNLOAD_PROGRESS Downloaded chunk 35147 size 5038056, 6.04MB/s 05:43:47 57.5%
2020-09-12 00:04:13.076 INFO DOWNLOAD_PROGRESS Downloaded chunk 35148 size 2954338, 6.04MB/s 05:43:47 57.5%
2020-09-12 00:04:14.164 INFO DOWNLOAD_PROGRESS Downloaded chunk 35149 size 6046783, 6.04MB/s 05:43:46 57.5%
2020-09-12 00:04:15.190 INFO DOWNLOAD_PROGRESS Downloaded chunk 35150 size 10748113, 6.04MB/s 05:43:44 57.5%
2020-09-12 00:04:15.899 INFO DOWNLOAD_PROGRESS Downloaded chunk 35151 size 3628008, 6.04MB/s 05:43:43 57.5%
2020-09-12 00:04:16.458 INFO DOWNLOAD_PROGRESS Downloaded chunk 35152 size 1396657, 6.04MB/s 05:43:44 57.5%
2020-09-12 00:04:17.018 INFO DOWNLOAD_PROGRESS Downloaded chunk 35153 size 1847018, 6.04MB/s 05:43:44 57.5%
2020-09-12 00:04:17.751 INFO DOWNLOAD_PROGRESS Downloaded chunk 35154 size 5801133, 6.04MB/s 05:43:42 57.5%
2020-09-12 00:04:18.332 INFO DOWNLOAD_PROGRESS Downloaded chunk 35155 size 1602859, 6.04MB/s 05:43:42 57.5%
2020-09-12 00:04:19.229 INFO DOWNLOAD_PROGRESS Downloaded chunk 35156 size 7596185, 6.04MB/s 05:43:41 57.5%
2020-09-12 00:04:20.175 INFO DOWNLOAD_PROGRESS Downloaded chunk 35157 size 7644886, 6.04MB/s 05:43:40 57.5%
2020-09-12 00:04:20.970 INFO DOWNLOAD_PROGRESS Downloaded chunk 35158 size 4637446, 6.04MB/s 05:43:39 57.5%
2020-09-12 00:04:21.689 INFO DOWNLOAD_PROGRESS Downloaded chunk 35159 size 4628599, 6.04MB/s 05:43:38 57.5%
2020-09-12 00:04:22.328 INFO DOWNLOAD_PROGRESS Downloaded chunk 35160 size 4112023, 6.04MB/s 05:43:38 57.5%
2020-09-12 00:04:23.006 INFO DOWNLOAD_PROGRESS Downloaded chunk 35161 size 3373923, 6.04MB/s 05:43:37 57.5%
2020-09-12 00:04:23.622 INFO DOWNLOAD_PROGRESS Downloaded chunk 35162 size 1185331, 6.04MB/s 05:43:37 57.5%
2020-09-12 00:04:24.439 INFO DOWNLOAD_PROGRESS Downloaded chunk 35163 size 3320402, 6.04MB/s 05:43:37 57.5%
2020-09-12 00:04:25.307 INFO DOWNLOAD_PROGRESS Downloaded chunk 35164 size 6208676, 6.04MB/s 05:43:36 57.5%
2020-09-12 00:04:25.908 INFO DOWNLOAD_PROGRESS Downloaded chunk 35165 size 1202977, 6.04MB/s 05:43:36 57.5%
2020-09-12 00:04:26.490 INFO DOWNLOAD_PROGRESS Downloaded chunk 35166 size 1373799, 6.04MB/s 05:43:36 57.5%
2020-09-12 00:04:26.995 INFO DOWNLOAD_PROGRESS Downloaded chunk 35167 size 1264274, 6.04MB/s 05:43:36 57.5%
2020-09-12 00:04:27.702 INFO DOWNLOAD_PROGRESS Downloaded chunk 35168 size 5087403, 6.04MB/s 05:43:35 57.5%
2020-09-12 00:04:28.849 INFO DOWNLOAD_PROGRESS Downloaded chunk 35169 size 11744426, 6.04MB/s 05:43:32 57.5%
2020-09-12 00:04:29.466 INFO DOWNLOAD_PROGRESS Downloaded chunk 35170 size 1058705, 6.04MB/s 05:43:33 57.5%
2020-09-12 00:04:29.932 INFO DOWNLOAD_PROGRESS Downloaded chunk 35171 size 1085673, 6.04MB/s 05:43:33 57.5%
2020-09-12 00:04:30.715 INFO DOWNLOAD_PROGRESS Downloaded chunk 35172 size 2929823, 6.04MB/s 05:43:33 57.5%
2020-09-12 00:04:31.266 INFO DOWNLOAD_PROGRESS Downloaded chunk 35173 size 1758719, 6.04MB/s 05:43:33 57.5%
2020-09-12 00:04:31.991 INFO DOWNLOAD_PROGRESS Downloaded chunk 35174 size 3146386, 6.04MB/s 05:43:32 57.5%
2020-09-12 00:04:32.875 INFO DOWNLOAD_PROGRESS Downloaded chunk 35175 size 7050538, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:33.426 INFO DOWNLOAD_PROGRESS Downloaded chunk 35176 size 1192552, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:33.741 INFO ONEDRIVE_RETRY Response code: 429; retry after 49 milliseconds
2020-09-12 00:04:34.352 INFO DOWNLOAD_PROGRESS Downloaded chunk 35177 size 1743318, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:34.909 INFO DOWNLOAD_PROGRESS Downloaded chunk 35178 size 2624870, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:35.500 INFO DOWNLOAD_PROGRESS Downloaded chunk 35179 size 1512224, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:36.298 INFO DOWNLOAD_PROGRESS Downloaded chunk 35180 size 5043399, 6.04MB/s 05:43:30 57.5%
2020-09-12 00:04:36.640 INFO ONEDRIVE_RETRY Response code: 429; retry after 445 milliseconds
2020-09-12 00:04:37.606 INFO DOWNLOAD_PROGRESS Downloaded chunk 35181 size 3343539, 6.04MB/s 05:43:30 57.5%
2020-09-12 00:04:37.914 INFO ONEDRIVE_RETRY Response code: 429; retry after 299 milliseconds
2020-09-12 00:04:38.537 INFO DOWNLOAD_PROGRESS Downloaded chunk 35182 size 1238098, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:38.830 INFO ONEDRIVE_RETRY Response code: 429; retry after 811 milliseconds
2020-09-12 00:04:40.192 INFO DOWNLOAD_PROGRESS Downloaded chunk 35183 size 4109828, 6.04MB/s 05:43:31 57.5%
2020-09-12 00:04:41.270 INFO DOWNLOAD_PROGRESS Downloaded chunk 35184 size 11282465, 6.04MB/s 05:43:29 57.5%
2020-09-12 00:04:41.849 INFO DOWNLOAD_PROGRESS Downloaded chunk 35185 size 1760807, 6.04MB/s 05:43:28 57.5%
2020-09-12 00:04:42.179 INFO ONEDRIVE_RETRY Response code: 429; retry after 834 milliseconds
2020-09-12 00:04:43.445 INFO DOWNLOAD_PROGRESS Downloaded chunk 35186 size 2131752, 6.04MB/s 05:43:29 57.5%
2020-09-12 00:04:44.132 INFO DOWNLOAD_PROGRESS Downloaded chunk 35187 size 2854613, 6.04MB/s 05:43:29 57.5%
2020-09-12 00:04:44.701 INFO DOWNLOAD_PROGRESS Downloaded chunk 35188 size 1922306, 6.04MB/s 05:43:28 57.5%
2020-09-12 00:04:45.621 INFO DOWNLOAD_PROGRESS Downloaded chunk 35189 size 5741124, 6.04MB/s 05:43:28 57.5%
2020-09-12 00:04:46.320 INFO DOWNLOAD_PROGRESS Downloaded chunk 35190 size 4132079, 6.04MB/s 05:43:27 57.5%
2020-09-12 00:04:47.337 INFO DOWNLOAD_PROGRESS Downloaded chunk 35191 size 8326319, 6.04MB/s 05:43:26 57.5%
2020-09-12 00:04:47.975 INFO DOWNLOAD_PROGRESS Downloaded chunk 35192 size 1734424, 6.04MB/s 05:43:25 57.5%
2020-09-12 00:04:48.292 INFO ONEDRIVE_RETRY Response code: 429; retry after 639 milliseconds
2020-09-12 00:04:49.358 INFO DOWNLOAD_PROGRESS Downloaded chunk 35193 size 2438615, 6.04MB/s 05:43:26 57.5%
2020-09-12 00:04:49.659 INFO ONEDRIVE_RETRY Response code: 429; retry after 282 milliseconds
2020-09-12 00:04:50.687 INFO DOWNLOAD_PROGRESS Downloaded chunk 35194 size 5850813, 6.04MB/s 05:43:25 57.5%
2020-09-12 00:04:51.416 INFO DOWNLOAD_PROGRESS Downloaded chunk 35195 size 4727156, 6.04MB/s 05:43:25 57.5%
2020-09-12 00:04:52.284 INFO DOWNLOAD_PROGRESS Downloaded chunk 35196 size 6380431, 6.04MB/s 05:43:24 57.5%
2020-09-12 00:04:52.795 INFO DOWNLOAD_PROGRESS Downloaded chunk 35197 size 1259464, 6.04MB/s 05:43:23 57.5%
2020-09-12 00:04:53.327 INFO DOWNLOAD_PROGRESS Downloaded chunk 35198 size 1541345, 6.04MB/s 05:43:24 57.5%
2020-09-12 00:04:53.644 INFO ONEDRIVE_RETRY Response code: 429; retry after 767 milliseconds
2020-09-12 00:04:55.408 INFO DOWNLOAD_PROGRESS Downloaded chunk 35199 size 7109923, 6.04MB/s 05:43:23 57.5%
2020-09-12 00:04:56.836 INFO DOWNLOAD_PROGRESS Downloaded chunk 35200 size 14481291, 6.04MB/s 05:43:20 57.5%
2020-09-12 00:04:57.585 INFO DOWNLOAD_PROGRESS Downloaded chunk 35201 size 3947710, 6.04MB/s 05:43:19 57.5%
2020-09-12 00:04:58.523 INFO DOWNLOAD_PROGRESS Downloaded chunk 35202 size 10387433, 6.04MB/s 05:43:17 57.5%
2020-09-12 00:04:59.559 INFO DOWNLOAD_PROGRESS Downloaded chunk 35203 size 9416112, 6.04MB/s 05:43:16 57.5%
2020-09-12 00:05:00.096 INFO DOWNLOAD_PROGRESS Downloaded chunk 35204 size 1892251, 6.04MB/s 05:43:16 57.5%
2020-09-12 00:05:00.612 INFO DOWNLOAD_PROGRESS Downloaded chunk 35205 size 1118766, 6.04MB/s 05:43:15 57.5%
2020-09-12 00:05:00.812 INFO ONEDRIVE_RETRY Response code: 429; retry after 527 milliseconds
2020-09-12 00:05:01.571 INFO ONEDRIVE_RETRY Response code: 429; retry after 17 milliseconds
2020-09-12 00:05:01.920 INFO ONEDRIVE_RETRY Response code: 429; retry after 2666 milliseconds
2020-09-12 00:05:05.376 INFO DOWNLOAD_PROGRESS Downloaded chunk 35206 size 6859018, 6.04MB/s 05:43:17 57.5%
2020-09-12 00:05:06.706 INFO DOWNLOAD_PROGRESS Downloaded chunk 35207 size 9949032, 6.04MB/s 05:43:15 57.5%
2020-09-12 00:05:07.728 INFO DOWNLOAD_PROGRESS Downloaded chunk 35208 size 6833673, 6.04MB/s 05:43:14 57.5%
2020-09-12 00:05:08.677 INFO DOWNLOAD_PROGRESS Downloaded chunk 35209 size 8516398, 6.04MB/s 05:43:12 57.5%
2020-09-12 00:05:09.207 INFO DOWNLOAD_PROGRESS Downloaded chunk 35210 size 1481221, 6.04MB/s 05:43:13 57.5%
2020-09-12 00:05:09.505 INFO ONEDRIVE_RETRY Response code: 429; retry after 185 milliseconds
2020-09-12 00:05:10.388 INFO DOWNLOAD_PROGRESS Downloaded chunk 35211 size 5774538, 6.04MB/s 05:43:12 57.5%
2020-09-12 00:05:11.449 INFO DOWNLOAD_PROGRESS Downloaded chunk 35212 size 11444141, 6.04MB/s 05:43:10 57.5%
2020-09-12 00:05:11.952 INFO DOWNLOAD_PROGRESS Downloaded chunk 35213 size 1073539, 6.04MB/s 05:43:09 57.5%
2020-09-12 00:05:12.635 INFO DOWNLOAD_PROGRESS Downloaded chunk 35214 size 4023362, 6.04MB/s 05:43:09 57.5%
2020-09-12 00:05:13.112 INFO DOWNLOAD_PROGRESS Downloaded chunk 35215 size 1356783, 6.04MB/s 05:43:09 57.6%
2020-09-12 00:05:13.946 INFO DOWNLOAD_PROGRESS Downloaded chunk 35216 size 7185033, 6.04MB/s 05:43:07 57.6%
2020-09-12 00:05:14.506 INFO DOWNLOAD_PROGRESS Downloaded chunk 35217 size 1829347, 6.04MB/s 05:43:08 57.6%
2020-09-12 00:05:15.182 INFO DOWNLOAD_PROGRESS Downloaded chunk 35218 size 4715347, 6.04MB/s 05:43:07 57.6%
2020-09-12 00:05:16.284 INFO DOWNLOAD_PROGRESS Downloaded chunk 35219 size 11459080, 6.04MB/s 05:43:05 57.6%
2020-09-12 00:05:16.856 INFO DOWNLOAD_PROGRESS Downloaded chunk 35220 size 2587998, 6.04MB/s 05:43:04 57.6%
2020-09-12 00:05:17.433 INFO DOWNLOAD_PROGRESS Downloaded chunk 35221 size 2082286, 6.04MB/s 05:43:04 57.6%
2020-09-12 00:05:18.378 INFO DOWNLOAD_PROGRESS Downloaded chunk 35222 size 9282136, 6.04MB/s 05:43:02 57.6%
2020-09-12 00:05:19.102 INFO DOWNLOAD_PROGRESS Downloaded chunk 35223 size 4214735, 6.04MB/s 05:43:02 57.6%
2020-09-12 00:05:20.525 INFO DOWNLOAD_PROGRESS Downloaded chunk 35224 size 16777216, 6.04MB/s 05:42:58 57.6%
2020-09-12 00:05:21.200 INFO DOWNLOAD_PROGRESS Downloaded chunk 35225 size 2312255, 6.04MB/s 05:42:58 57.6%
2020-09-12 00:05:21.711 INFO DOWNLOAD_PROGRESS Downloaded chunk 35226 size 1202871, 6.04MB/s 05:42:58 57.6%
2020-09-12 00:05:22.805 INFO DOWNLOAD_PROGRESS Downloaded chunk 35227 size 7624707, 6.04MB/s 05:42:56 57.6%
2020-09-12 00:05:23.483 INFO DOWNLOAD_PROGRESS Downloaded chunk 35228 size 3886551, 6.04MB/s 05:42:56 57.6%
2020-09-12 00:05:24.376 INFO DOWNLOAD_PROGRESS Downloaded chunk 35229 size 7363530, 6.04MB/s 05:42:55 57.6%
2020-09-12 00:05:25.261 INFO DOWNLOAD_PROGRESS Downloaded chunk 35230 size 6932267, 6.04MB/s 05:42:54 57.6%
2020-09-12 00:05:25.924 INFO DOWNLOAD_PROGRESS Downloaded chunk 35231 size 3921742, 6.04MB/s 05:42:53 57.6%
2020-09-12 00:05:26.433 INFO DOWNLOAD_PROGRESS Downloaded chunk 35232 size 1652802, 6.04MB/s 05:42:53 57.6%
2020-09-12 00:05:27.289 INFO DOWNLOAD_PROGRESS Downloaded chunk 35233 size 7863702, 6.04MB/s 05:42:51 57.6%
2020-09-12 00:05:27.937 INFO DOWNLOAD_PROGRESS Downloaded chunk 35234 size 1372554, 6.04MB/s 05:42:51 57.6%
2020-09-12 00:05:28.759 INFO DOWNLOAD_PROGRESS Downloaded chunk 35235 size 4481033, 6.04MB/s 05:42:51 57.6%
2020-09-12 00:05:29.229 INFO DOWNLOAD_PROGRESS Downloaded chunk 35236 size 1122103, 6.04MB/s 05:42:51 57.6%
2020-09-12 00:05:30.177 INFO DOWNLOAD_PROGRESS Downloaded chunk 35237 size 8455612, 6.04MB/s 05:42:49 57.6%
2020-09-12 00:05:30.569 INFO ONEDRIVE_RETRY Response code: 429; retry after 816 milliseconds
2020-09-12 00:05:31.454 INFO ONEDRIVE_RETRY Response code: 429; retry after 61 milliseconds
2020-09-12 00:05:31.840 INFO ONEDRIVE_RETRY Response code: 429; retry after 2639 milliseconds
2020-09-12 00:05:34.724 INFO ONEDRIVE_RETRY Response code: 429; retry after 2407 milliseconds
2020-09-12 00:05:37.324 INFO ONEDRIVE_RETRY Response code: 429; retry after 5612 milliseconds
2020-09-12 00:05:43.093 INFO ONEDRIVE_RETRY Response code: 429; retry after 23965 milliseconds
2020-09-12 00:06:07.252 INFO ONEDRIVE_RETRY Response code: 429; retry after 10663 milliseconds
2020-09-12 00:06:17.975 INFO ONEDRIVE_RETRY Response code: 429; retry after 30599 milliseconds
2020-09-12 00:06:48.574 ERROR DOWNLOAD_CHUNK Failed to download the chunk 1f82ccc1a3ed0ade3fde96f4e738b0047ac3ca80094fd9c78c8d7d17a8f24f51: Maximum number of retries reached
\ No newline at end of selection

[gchen]

Non-printable ascii characters are all showing up as ., so you can’t tell if it is \001 or \003. To show the file in binary mode, on maxOS or linux run xxd:

xxd chunk_file | head

On Windows run certutil to encode into a file and open the file:

certutil -encodehex chunk_file output

By default Duplicacy uses only 1 thread for backup or restore, but you can specify the number of threads with the -threads option.

[stefan1]

I tried:

• HEX editor inside Visual Studio Code (Windows)
• HxD Hex editor (Windows)
• certutil -encodehex chunk_file output (Windows)
• xxd chunk_file | head (Ubuntu 20.04 inside Windows and Ubuntu 18.04 on a real server)

I tried that on the snapshot and the chunk file an all I get is:

stefan@UncleSam-MSI:/mnt/c/Users/stefa/Downloads$ xxd 3 | head
00000000: 6475 706c 6963 6163 7903 2609 0000 0000  duplicacy.&.....
00000010: 0000 0800 0300 2d09 ec73 dbcb 5c99 8f80  ......-..s..\...
...
stefan@UncleSam-MSI:/mnt/c/Users/stefa/Downloads$ xxd 8e87c9baa499a578bbc7b2d2aa779bd9a109bae4844aa7b8e55f0d4e27ba | head
00000000: 6475 706c 6963 6163 7903 2615 2300 0000  duplicacy.&.#...
00000010: 0000 0800 0300 0e15 01be 9df0 fe2d b1f7  .............-..
...

Also tried to download other snapshots and chunk files - same result.

As my stoarge is encrypted I think I can send you 1-2 chunk files to test it.

[gchen]

00000000: 6475 706c 6963 6163 7903

This is the banner duplicacy\003 which means erasure coding was turned on during the backup. So you’re good.

[gchen]

It turns out that OneDrive for Business will keep the partial file when an upload is aborted, even if the size of the file is less than the specified content length in the http header. So if a backup is interrupted by ctrl-C or an error, the files that are still being transferred will become corrupted.

I’ve submitted a bug report to OneDrive.

This doesn’t happen with OneDrive for Personal, or the upload by session API (which is used when the file is larger than 4MB).

If they can’t fix this bug in this week, I’ll make a new CLI release that uses the upload by session API for all files.

[stefan1]

Wow thank you very much for checking that! I am looking forward for the solution

[stefan1]

Do you have any update regarding OneDrive behaviour?

[gchen]

The commit to switch to the upload by session API has already been checked in: Fix OneDrive Business and improve retry mechanism. · gilbertchen/duplicacy@eecbb8f · GitHub so we don’t need their fix.

I planed to release a new CLI release last week but unfortunately it got delayed since I still have a few changes that I wanted to make. I promise it will be available by the end of this week.

[gchen]

Microsoft finally fixed the issue: The simple upload API of OneDrive for Business does not respect the content length and saves a partial file · Issue #1366 · OneDrive/onedrive-api-docs · GitHub

The CLI has been using the upload by session API for a while so I don’t think it is necessary to switch back.