[Duplicacy Web Edition 1.2.0]: OneDrive 429 errors

ttm · 11 February 2020 20:29

Please describe what you are doing to trigger the bug:
Backing up to OneDrive results in failing after abound an hour with the following error messages (personal data removed):

Running backup command from /root/[SNIP] to back up [SNIP]
Options: [-log backup -storage [SNIP] -threads 8 -stats]
2020-02-11 19:36:36.944 INFO REPOSITORY_SET Repository set to [SNIP]
2020-02-11 19:36:36.944 INFO STORAGE_SET Storage set to [SNIP]
2020-02-11 19:36:41.134 INFO BACKUP_START No previous backup found
2020-02-11 19:36:41.134 INFO BACKUP_INDEXING Indexing [SNIP]
2020-02-11 19:36:41.134 INFO SNAPSHOT_FILTER Parsing filter file [SNIP]
2020-02-11 19:36:41.134 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2020-02-11 19:36:59.497 INFO BACKUP_LIST Listing all chunks
2020-02-11 19:38:44.743 INFO BACKUP_THREADS Use 8 uploading threads
2020-02-11 19:43:00.711 INFO ONEDRIVE_RETRY Response code: 503; retry after 41 milliseconds
2020-02-11 19:51:54.640 INFO ONEDRIVE_RETRY Response code: 503; retry after 77 milliseconds
2020-02-11 20:12:37.178 INFO ONEDRIVE_RETRY Response code: 429; retry after 693 milliseconds
2020-02-11 20:12:37.378 INFO ONEDRIVE_RETRY Response code: 429; retry after 602 milliseconds
2020-02-11 20:12:37.860 INFO ONEDRIVE_RETRY Response code: 429; retry after 748 milliseconds
2020-02-11 20:12:37.961 INFO ONEDRIVE_RETRY Response code: 429; retry after 314 milliseconds
2020-02-11 20:12:37.979 INFO ONEDRIVE_RETRY Response code: 429; retry after 735 milliseconds
2020-02-11 20:12:39.303 INFO ONEDRIVE_RETRY Response code: 429; retry after 785 milliseconds
2020-02-11 20:12:39.335 INFO ONEDRIVE_RETRY Response code: 429; retry after 967 milliseconds
2020-02-11 20:12:40.275 INFO ONEDRIVE_RETRY Response code: 429; retry after 631 milliseconds
2020-02-11 20:12:40.594 INFO ONEDRIVE_RETRY Response code: 429; retry after 463 milliseconds
2020-02-11 20:12:41.696 INFO ONEDRIVE_RETRY Response code: 429; retry after 6456 milliseconds
2020-02-11 20:12:42.294 INFO ONEDRIVE_RETRY Response code: 429; retry after 1230 milliseconds
2020-02-11 20:12:43.814 INFO ONEDRIVE_RETRY Response code: 429; retry after 1369 milliseconds
2020-02-11 20:12:45.361 INFO ONEDRIVE_RETRY Response code: 429; retry after 10 milliseconds
2020-02-11 20:12:45.575 INFO ONEDRIVE_RETRY Response code: 429; retry after 485 milliseconds
2020-02-11 20:12:45.667 INFO ONEDRIVE_RETRY Response code: 429; retry after 12 milliseconds
2020-02-11 20:12:46.047 INFO ONEDRIVE_RETRY Response code: 429; retry after 2233 milliseconds
2020-02-11 20:12:46.225 INFO ONEDRIVE_RETRY Response code: 429; retry after 1295 milliseconds
2020-02-11 20:12:46.448 INFO ONEDRIVE_RETRY Response code: 429; retry after 1225 milliseconds
2020-02-11 20:12:47.734 INFO ONEDRIVE_RETRY Response code: 429; retry after 141 milliseconds
2020-02-11 20:12:48.618 INFO ONEDRIVE_RETRY Response code: 429; retry after 4593 milliseconds
2020-02-11 20:12:50.855 INFO ONEDRIVE_RETRY Response code: 429; retry after 1889 milliseconds
2020-02-11 20:12:50.943 INFO ONEDRIVE_RETRY Response code: 429; retry after 2135 milliseconds
2020-02-11 20:12:51.143 INFO ONEDRIVE_RETRY Response code: 429; retry after 1962 milliseconds
2020-02-11 20:12:53.244 INFO ONEDRIVE_RETRY Response code: 429; retry after 1208 milliseconds
2020-02-11 20:12:53.244 INFO ONEDRIVE_RETRY Response code: 429; retry after 1013 milliseconds
2020-02-11 20:12:53.693 INFO ONEDRIVE_RETRY Response code: 429; retry after 1140 milliseconds
2020-02-11 20:12:55.487 INFO ONEDRIVE_RETRY Response code: 429; retry after 30487 milliseconds
2020-02-11 20:12:56.034 INFO ONEDRIVE_RETRY Response code: 429; retry after 21415 milliseconds
2020-02-11 20:12:57.027 INFO ONEDRIVE_RETRY Response code: 429; retry after 1019 milliseconds
2020-02-11 20:12:58.408 INFO ONEDRIVE_RETRY Response code: 429; retry after 6170 milliseconds
2020-02-11 20:12:58.677 INFO ONEDRIVE_RETRY Response code: 429; retry after 3932 milliseconds
2020-02-11 20:12:58.878 INFO ONEDRIVE_RETRY Response code: 429; retry after 4759 milliseconds
2020-02-11 20:12:59.648 INFO ONEDRIVE_RETRY Response code: 429; retry after 1476 milliseconds
2020-02-11 20:13:00.205 INFO ONEDRIVE_RETRY Response code: 429; retry after 1782 milliseconds
2020-02-11 20:13:00.300 INFO ONEDRIVE_RETRY Response code: 429; retry after 2847 milliseconds
2020-02-11 20:13:03.989 INFO ONEDRIVE_RETRY Response code: 429; retry after 15340 milliseconds
2020-02-11 20:13:07.256 INFO ONEDRIVE_RETRY Response code: 429; retry after 1221 milliseconds
2020-02-11 20:13:08.854 INFO ONEDRIVE_RETRY Response code: 429; retry after 1801 milliseconds
2020-02-11 20:13:09.277 INFO ONEDRIVE_RETRY Response code: 429; retry after 15887 milliseconds
2020-02-11 20:13:11.421 INFO ONEDRIVE_RETRY Response code: 429; retry after 9711 milliseconds
2020-02-11 20:13:11.481 INFO ONEDRIVE_RETRY Response code: 429; retry after 351 milliseconds
2020-02-11 20:13:13.578 INFO ONEDRIVE_RETRY Response code: 429; retry after 4508 milliseconds
2020-02-11 20:13:14.873 INFO ONEDRIVE_RETRY Response code: 429; retry after 5317 milliseconds
2020-02-11 20:13:15.432 INFO ONEDRIVE_RETRY Response code: 429; retry after 1467 milliseconds
2020-02-11 20:13:19.531 INFO ONEDRIVE_RETRY Response code: 429; retry after 8182 milliseconds
2020-02-11 20:13:19.912 INFO ONEDRIVE_RETRY Response code: 429; retry after 47115 milliseconds
2020-02-11 20:13:20.687 INFO ONEDRIVE_RETRY Response code: 429; retry after 23639 milliseconds
2020-02-11 20:13:24.147 INFO ONEDRIVE_RETRY Response code: 429; retry after 9626 milliseconds
2020-02-11 20:13:24.268 INFO ONEDRIVE_RETRY Response code: 429; retry after 18959 milliseconds
2020-02-11 20:13:24.587 INFO ONEDRIVE_RETRY Response code: 429; retry after 20559 milliseconds
2020-02-11 20:13:26.332 INFO ONEDRIVE_RETRY Response code: 429; retry after 35065 milliseconds
2020-02-11 20:13:26.341 INFO ONEDRIVE_RETRY Response code: 429; retry after 12220 milliseconds
2020-02-11 20:13:29.420 INFO ONEDRIVE_RETRY Response code: 429; retry after 28937 milliseconds
2020-02-11 20:13:36.519 INFO ONEDRIVE_RETRY Response code: 429; retry after 26725 milliseconds
2020-02-11 20:13:39.772 INFO ONEDRIVE_RETRY Response code: 429; retry after 28808 milliseconds
2020-02-11 20:13:44.210 INFO ONEDRIVE_RETRY Response code: 429; retry after 62292 milliseconds
2020-02-11 20:13:44.327 ERROR UPLOAD_CHUNK Failed to upload the chunk [SNIP]: Maximum number of retries reached
2020-02-11 20:13:44.361 INFO INCOMPLETE_SAVE Incomplete snapshot saved to [SNIP].duplicacy/incomplete
Failed to upload the chunk [SNIP]: Maximum number of retries reached

Please describe what you expect to happen (but doesn’t):
A full backup to OneDrive.

Please describe what actually happens (the wrong behaviour):
See above.

I have no idea what causes the 429 errors. Backup via Duplicati to OneDrive works fine.
My internet connection is also fine during backup.
The error also occured with Duplicacy Web Edition 1.1.0 and threads set to standard setting.

ttm · 11 February 2020 20:32

Just happened again:
started @ 2020-02-11 20:28:29.557
failed @ 2020-02-11 21:30:16.261
Log entries are similar.

Trying now again with -debug

Additional information: I am trying to backup about 500 GB split into small (1-5 MB) files (about 350000).

ttm · 11 February 2020 21:48

Log with -debug enabled. Sadly I can’t see any useful information. Personal data replaced by [SNIP]

2020-02-11 22:33:34.602 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.613 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/0e/c80db8d0e173eafa758b22ad6d875d6a4b5b04a75b8af8f400f56c6ad4ad99:/content
2020-02-11 22:33:34.737 DEBUG CHUNK_UPLOAD Chunk bcfc18ae3ed1bb5ee70928f314c66ff4fe7acdade7c8f9978e791406491e25f3 has been uploaded
2020-02-11 22:33:34.737 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/60/9b9314bdfa72f159ecc08a383c079e4fcb5d6789a4ac54f12fad31223e8b69?select=id,name,size,folder
2020-02-11 22:33:34.746 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.746 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.770 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.770 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.771 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.771 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.823 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.823 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.843 DEBUG CHUNK_UPLOAD Chunk 44aabe6d8bf3b99939d9cd1137ae1441504c1170093dc5c9c4767e27c92c7ce9 has been uploaded
2020-02-11 22:33:34.843 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/41/639bc47a9222064155ae2410091289babbba981ef484e823368e7addd3fbb7?select=id,name,size,folder
2020-02-11 22:33:34.843 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.843 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.845 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.845 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.859 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.860 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.869 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.869 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.871 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.871 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.875 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.875 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.883 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.883 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.886 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.886 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.887 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.887 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.897 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.897 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.908 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.908 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.920 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.920 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.923 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:34.923 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:34.992 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:33:35.447 DEBUG CHUNK_UPLOAD Chunk fd34572ed96ce489a92ebbc12bfb1342f6b72ea2062788420e7f5af16c32bd79 has been uploaded
2020-02-11 22:33:35.447 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/5a/9925aebcd275c241f59d99a33c0e16ef978dd6d19f8b21a0ad01f2d4974ad2?select=id,name,size,folder
2020-02-11 22:33:35.458 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/60/9b9314bdfa72f159ecc08a383c079e4fcb5d6789a4ac54f12fad31223e8b69:/content
2020-02-11 22:33:35.499 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.499 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.508 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.508 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.521 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.521 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.532 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.532 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.549 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.549 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.577 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.577 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.665 DEBUG CHUNK_UPLOAD Chunk a1dfadef89e484ad0e4e804bd0ae28b505bad504713d4cde7277606e9bc5e422 has been uploaded
2020-02-11 22:33:35.665 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/72/9d9bab9965b7f5c10cfad058c3f4a64b107db02a8d41501c2278c39953a75b?select=id,name,size,folder
2020-02-11 22:33:35.668 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.668 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.706 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.706 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.707 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/41/639bc47a9222064155ae2410091289babbba981ef484e823368e7addd3fbb7:/content
2020-02-11 22:33:35.708 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.708 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.718 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.718 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.721 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.721 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:35.753 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:35.754 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.041 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/5a/9925aebcd275c241f59d99a33c0e16ef978dd6d19f8b21a0ad01f2d4974ad2:/content
2020-02-11 22:33:36.313 DEBUG CHUNK_UPLOAD Chunk 8d7a25676c3f1c010992820e994c2852603bad95d8807e53903a80b0e59671dc has been uploaded
2020-02-11 22:33:36.313 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610?select=id,name,size,folder
2020-02-11 22:33:36.361 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.361 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.366 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/72/9d9bab9965b7f5c10cfad058c3f4a64b107db02a8d41501c2278c39953a75b:/content
2020-02-11 22:33:36.390 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.390 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.393 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.393 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.744 DEBUG CHUNK_UPLOAD Chunk 40aff74aba36b1259371178ab372c3cc5a578333f77ac8d83743a20741d4bd44 has been uploaded
2020-02-11 22:33:36.744 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c?select=id,name,size,folder
2020-02-11 22:33:36.744 DEBUG CHUNK_UPLOAD Chunk 0ec80db8d0e173eafa758b22ad6d875d6a4b5b04a75b8af8f400f56c6ad4ad99 has been uploaded
2020-02-11 22:33:36.744 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/76/09c27fffa4769d123707896921cddd0f5ffc3f35e52c7acc9dc0264489f43e?select=id,name,size,folder
2020-02-11 22:33:36.772 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.772 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.773 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.773 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.795 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.795 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.817 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.817 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.821 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.821 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.871 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.871 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.875 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.875 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.886 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.886 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.891 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.891 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.902 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.902 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.957 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.957 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.969 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.969 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.980 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.980 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.984 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.984 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.994 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.994 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:36.999 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:36.999 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:37.018 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:37.018 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:37.248 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:33:37.668 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/76/09c27fffa4769d123707896921cddd0f5ffc3f35e52c7acc9dc0264489f43e:/content
2020-02-11 22:33:39.123 DEBUG CHUNK_UPLOAD Chunk 609b9314bdfa72f159ecc08a383c079e4fcb5d6789a4ac54f12fad31223e8b69 has been uploaded
2020-02-11 22:33:39.123 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673?select=id,name,size,folder
2020-02-11 22:33:39.192 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:33:40.052 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:33:41.713 DEBUG CHUNK_UPLOAD Chunk 41639bc47a9222064155ae2410091289babbba981ef484e823368e7addd3fbb7 has been uploaded
2020-02-11 22:33:41.718 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d?select=id,name,size,folder
2020-02-11 22:33:41.725 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:41.725 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:41.739 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:41.739 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:41.762 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:41.762 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:41.765 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:41.765 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:41.768 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:41.768 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.224 DEBUG CHUNK_UPLOAD Chunk 7609c27fffa4769d123707896921cddd0f5ffc3f35e52c7acc9dc0264489f43e has been uploaded
2020-02-11 22:33:43.224 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943?select=id,name,size,folder
2020-02-11 22:33:43.232 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:43.232 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.264 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:43.265 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.269 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:43.269 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.271 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:43.272 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.328 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:43.328 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.331 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:43.331 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:43.853 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:33:44.189 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:33:44.760 DEBUG CHUNK_UPLOAD Chunk 5a9925aebcd275c241f59d99a33c0e16ef978dd6d19f8b21a0ad01f2d4974ad2 has been uploaded
2020-02-11 22:33:44.760 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9?select=id,name,size,folder
2020-02-11 22:33:44.767 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:44.767 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:45.677 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:33:45.839 DEBUG CHUNK_UPLOAD Chunk 729d9bab9965b7f5c10cfad058c3f4a64b107db02a8d41501c2278c39953a75b has been uploaded
2020-02-11 22:33:45.839 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:33:45.846 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:45.846 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:45.849 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:45.849 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:45.879 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:45.879 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:45.883 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:45.883 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:45.884 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:45.884 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:45.885 INFO PACK_END Packed [SNIP] ([SNIP])
2020-02-11 22:33:45.885 TRACE PACK_START Packing [SNIP]
2020-02-11 22:33:46.340 INFO ONEDRIVE_RETRY Response code: 429; retry after 728 milliseconds
2020-02-11 22:33:46.930 INFO ONEDRIVE_RETRY Response code: 429; retry after 354 milliseconds
2020-02-11 22:33:46.980 INFO ONEDRIVE_RETRY Response code: 429; retry after 565 milliseconds
2020-02-11 22:33:47.003 INFO ONEDRIVE_RETRY Response code: 429; retry after 578 milliseconds
2020-02-11 22:33:47.068 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:33:47.142 INFO ONEDRIVE_RETRY Response code: 429; retry after 113 milliseconds
2020-02-11 22:33:47.255 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:33:47.284 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:33:47.545 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:33:47.579 INFO ONEDRIVE_RETRY Response code: 429; retry after 202 milliseconds
2020-02-11 22:33:47.581 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:33:47.781 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:33:47.792 INFO ONEDRIVE_RETRY Response code: 429; retry after 763 milliseconds
2020-02-11 22:33:48.007 INFO ONEDRIVE_RETRY Response code: 429; retry after 877 milliseconds
2020-02-11 22:33:48.458 INFO ONEDRIVE_RETRY Response code: 429; retry after 2730 milliseconds
2020-02-11 22:33:48.555 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:33:48.884 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:33:51.188 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:33:51.545 INFO ONEDRIVE_RETRY Response code: 429; retry after 478 milliseconds
2020-02-11 22:33:51.699 INFO ONEDRIVE_RETRY Response code: 429; retry after 7907 milliseconds
2020-02-11 22:33:51.745 INFO ONEDRIVE_RETRY Response code: 429; retry after 907 milliseconds
2020-02-11 22:33:51.945 INFO ONEDRIVE_RETRY Response code: 429; retry after 1910 milliseconds
2020-02-11 22:33:52.023 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:33:52.652 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:33:52.755 INFO ONEDRIVE_RETRY Response code: 429; retry after 1442 milliseconds
2020-02-11 22:33:53.855 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:33:54.197 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:33:55.893 INFO ONEDRIVE_RETRY Response code: 429; retry after 3782 milliseconds
2020-02-11 22:33:56.722 INFO ONEDRIVE_RETRY Response code: 429; retry after 3497 milliseconds
2020-02-11 22:33:56.945 INFO ONEDRIVE_RETRY Response code: 429; retry after 1084 milliseconds
2020-02-11 22:33:58.029 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:33:58.248 INFO ONEDRIVE_RETRY Response code: 429; retry after 225 milliseconds
2020-02-11 22:33:58.474 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:33:59.606 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:33:59.675 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:33:59.886 INFO ONEDRIVE_RETRY Response code: 429; retry after 521 milliseconds
2020-02-11 22:34:00.104 INFO ONEDRIVE_RETRY Response code: 429; retry after 11438 milliseconds
2020-02-11 22:34:00.219 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:34:00.407 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:34:01.007 INFO ONEDRIVE_RETRY Response code: 429; retry after 3641 milliseconds
2020-02-11 22:34:02.298 INFO ONEDRIVE_RETRY Response code: 429; retry after 7658 milliseconds
2020-02-11 22:34:03.603 INFO ONEDRIVE_RETRY Response code: 429; retry after 2421 milliseconds
2020-02-11 22:34:04.087 INFO ONEDRIVE_RETRY Response code: 429; retry after 3407 milliseconds
2020-02-11 22:34:04.649 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:34:04.943 INFO ONEDRIVE_RETRY Response code: 429; retry after 1279 milliseconds
2020-02-11 22:34:06.024 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:34:06.187 INFO ONEDRIVE_RETRY Response code: 429; retry after 2904 milliseconds
2020-02-11 22:34:06.222 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:34:07.494 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:34:07.756 INFO ONEDRIVE_RETRY Response code: 429; retry after 4004 milliseconds
2020-02-11 22:34:08.530 INFO ONEDRIVE_RETRY Response code: 429; retry after 772 milliseconds
2020-02-11 22:34:09.091 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:34:09.302 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:34:09.957 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:34:10.728 INFO ONEDRIVE_RETRY Response code: 429; retry after 823 milliseconds
2020-02-11 22:34:11.151 INFO ONEDRIVE_RETRY Response code: 429; retry after 2147 milliseconds
2020-02-11 22:34:11.543 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:34:11.551 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:34:11.760 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:34:12.080 INFO ONEDRIVE_RETRY Response code: 429; retry after 5107 milliseconds
2020-02-11 22:34:12.201 INFO ONEDRIVE_RETRY Response code: 429; retry after 9940 milliseconds
2020-02-11 22:34:13.298 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:34:14.907 INFO ONEDRIVE_RETRY Response code: 429; retry after 15557 milliseconds
2020-02-11 22:34:15.292 INFO ONEDRIVE_RETRY Response code: 429; retry after 27211 milliseconds
2020-02-11 22:34:17.188 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:34:17.265 INFO ONEDRIVE_RETRY Response code: 429; retry after 1451 milliseconds
2020-02-11 22:34:17.699 INFO ONEDRIVE_RETRY Response code: 429; retry after 63140 milliseconds
2020-02-11 22:34:18.198 INFO ONEDRIVE_RETRY Response code: 429; retry after 2618 milliseconds
2020-02-11 22:34:18.716 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:34:19.219 INFO ONEDRIVE_RETRY Response code: 429; retry after 4621 milliseconds
2020-02-11 22:34:19.513 INFO ONEDRIVE_RETRY Response code: 429; retry after 6028 milliseconds
2020-02-11 22:34:20.816 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:34:22.141 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:34:22.444 INFO ONEDRIVE_RETRY Response code: 429; retry after 4740 milliseconds
2020-02-11 22:34:23.420 INFO ONEDRIVE_RETRY Response code: 429; retry after 5500 milliseconds
2020-02-11 22:34:23.573 INFO ONEDRIVE_RETRY Response code: 429; retry after 14324 milliseconds
2020-02-11 22:34:23.840 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:34:25.542 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:34:25.999 INFO ONEDRIVE_RETRY Response code: 429; retry after 13521 milliseconds
2020-02-11 22:34:27.185 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:34:28.920 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:34:29.099 INFO ONEDRIVE_RETRY Response code: 429; retry after 6817 milliseconds
2020-02-11 22:34:30.464 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:34:31.752 INFO ONEDRIVE_RETRY Response code: 429; retry after 10011 milliseconds
2020-02-11 22:34:31.872 INFO ONEDRIVE_RETRY Response code: 429; retry after 7850 milliseconds
2020-02-11 22:34:32.446 INFO ONEDRIVE_RETRY Response code: 429; retry after 60364 milliseconds
2020-02-11 22:34:35.916 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:34:37.897 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:34:38.336 INFO ONEDRIVE_RETRY Response code: 429; retry after 20580 milliseconds
2020-02-11 22:34:39.520 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:34:39.570 INFO ONEDRIVE_RETRY Response code: 429; retry after 3013 milliseconds
2020-02-11 22:34:39.722 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:34:41.763 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:34:42.043 INFO ONEDRIVE_RETRY Response code: 429; retry after 4548 milliseconds
2020-02-11 22:34:42.337 INFO ONEDRIVE_RETRY Response code: 429; retry after 13665 milliseconds
2020-02-11 22:34:42.503 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:34:42.583 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/97/885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c:/content
2020-02-11 22:34:46.019 INFO ONEDRIVE_RETRY Response code: 429; retry after 62849 milliseconds
2020-02-11 22:34:46.592 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/ba/16863d5373fe266e5f9e61825d094ec6fdcd6bead882e5ce3f98be12fd2f4d:/content
2020-02-11 22:34:46.712 INFO ONEDRIVE_RETRY Response code: 429; retry after 23481 milliseconds
2020-02-11 22:34:46.782 INFO ONEDRIVE_RETRY Response code: 429; retry after 54939 milliseconds
2020-02-11 22:34:47.436 INFO ONEDRIVE_RETRY Response code: 429; retry after 85118 milliseconds
2020-02-11 22:34:56.002 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c0/c3d9ec0f10414775e8abfaaf4c2ceff2584ad70b537515a8e22706bcf5a33c:/content
2020-02-11 22:34:58.182 INFO ONEDRIVE_RETRY Response code: 429; retry after 60616 milliseconds
2020-02-11 22:34:58.916 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:35:01.305 INFO ONEDRIVE_RETRY Response code: 429; retry after 37298 milliseconds
2020-02-11 22:35:10.194 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:35:12.878 INFO ONEDRIVE_RETRY Response code: 429; retry after 15883 milliseconds
2020-02-11 22:35:20.839 DEBUG ONEDRIVE_CALL GET https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/e4/feae13ede68561a183f0ec8efb28b6b4c12a72b2af353b79cd98929005387c?select=id,name,size,folder
2020-02-11 22:35:20.884 INFO ONEDRIVE_RETRY Response code: 429; retry after 59151 milliseconds
2020-02-11 22:35:28.761 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/06/afac7251db941d30e6415a2df6d0cb2787364bbcbf5902d246f3cda4ed2673:/content
2020-02-11 22:35:31.452 INFO ONEDRIVE_RETRY Response code: 429; retry after 97609 milliseconds
2020-02-11 22:35:32.810 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/7f/ab45f4b28ef88929981976982cbe623e61305ea09d92201d2349d15c987cd9:/content
2020-02-11 22:35:33.247 INFO ONEDRIVE_RETRY Response code: 429; retry after 126006 milliseconds
2020-02-11 22:35:38.603 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/1e/2db545edb703c308c3be5efd41708b1224c42db14626a64bee8f501fd6b610:/content
2020-02-11 22:35:41.027 INFO ONEDRIVE_RETRY Response code: 429; retry after 79280 milliseconds
2020-02-11 22:35:41.721 ERROR UPLOAD_CHUNK Failed to upload the chunk 97885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c: Maximum number of retries reached
2020-02-11 22:35:48.868 DEBUG ONEDRIVE_CALL PUT https://api.onedrive.com/v1.0/drive/root:/[SNIP]/chunks/c4/7a2956b1bcb8b8ddda2c463e7f80295775b23772118684a8b0d1f21a412943:/content
2020-02-11 22:35:49.582 INFO ONEDRIVE_RETRY Response code: 429; retry after 83066 milliseconds
2020-02-11 22:35:50.331 INFO INCOMPLETE_SAVE Incomplete snapshot saved to /.duplicacy-web/repositories/[SNIP]/.duplicacy/incomplete
Failed to upload the chunk 97885114c84b8d99223af33a967d9e8e520e5f1141e3d2874a3ef9f28bf5a68c: Maximum number of retries reached

gchen · 11 February 2020 22:07

HTTP code 409 means too many requests. That means OneDrive is rate limiting your connections.

ttm · 11 February 2020 22:15

Thanks for the answer.
I can’t see any mistake on my end since I backed up the same data to OneDrive with Duplicati without any issue. Maybe it is OneDrive Dev App related?

Update: I used the default chunk size (50 MB) in Duplicati. So there are less file operations than with Duplicacy, right? Maybe thats the reason why there wasn’t any issue with Duplicati?

Using Duplicacy with only 1 thread did sadly also result in the same 429 error as stated above after about one hour.

It seems that Arq Backup is reacting to 429 errors with sleep and try again as stated in the release notes (https://www.arqbackup.com/download/arq5_release_notes.html). Maybe Duplicacy’s retry interval is too short?

gchen · 12 February 2020 03:39

I wonder if you’re hitting per-user daily cap. Is Duplicati working if you try it now?

Maybe wait until tomorrow and only use one thread.

ttm · 12 February 2020 08:44

I just tried this and yes, Duplicati is working just fine after getting the 429 errors in Duplicacy.
A daily cap also doesn’t make sense since I am able to restart the backup with Duplicacy after it was aborted. Every time on the same day. Then it would backup another hour before getting the 429 errors again.

I am ok with rate limiting, but that the backup gets aborted instead of waiting an amount of time isn’t that good for serious backup use.

As stated above 1 thread will also lead to the 429 errors after 1 hour.

My current workaround for the initial backup is to schedule it in an 2 hour intervall, so it will continue after it failed.
Maybe this helps Avoid getting throttled or blocked in SharePoint Online | Microsoft Docs since it’s the same API?

Btw. Arq Backup with OneDrive is also working fine and Arq is using a similiar chunk size to Duplicacy.

gchen · 13 February 2020 02:38

I’ll submit a commit tomorrow to improve the retry mechanism for OneDrive.

ttm · 13 February 2020 18:22

Thanks!
If v1.2.1. was the new version with the commit it sadly didn’t change anything for me. Tried it today.

ttm · 15 February 2020 22:08

After some more testing by initializing a new folder as new Storage on my OneDrive and creating a full new backup it seems to work without any 429 errors. I’ll keep testing it…