OneDrive timeouts

iamgreen · 12 December 2019 18:58

I’m constantly getting timeouts when backing up to OneDrive. Do a Check: timeout. Try a Backup: timeout or connection reset by peer. Sometimes it starts working after an hour or two, but today I haven’t been able to get the backup to work at all.

Does anyone else have trouble with OneDrive? The weird thing is that once it starts working, it seems to work fine. One backup took over 3 days. I had to try several times to get it started, but once it was going it worked for the whole 3 days.

Here’s a typical Check failure:

Running check command from /home/dog/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage onedrive-test -a -tabular]
2019-12-12 03:29:32.641 INFO STORAGE_SET Storage set to one://box_backup
2019-12-12 03:30:02.642 ERROR STORAGE_CREATE Failed to load the OneDrive storage at one://box_backup: Get https://api.onedrive.com/v1.0/drive/root:/box_backup?select=id,name,size,folder: dial tcp 13.107.42.12:443: i/o timeout
Failed to load the OneDrive storage at one://box_backup: Get https://api.onedrive.com/v1.0/drive/root:/box_backup?select=id,name,size,folder: dial tcp 13.107.42.12:443: i/o timeout

Here’s a typical Backup failure:

Running backup command from /home/dog/.duplicacy-web/repositories/localhost/2 to back up /media/Pictures
Options: [-log backup -storage onedrive-test -stats]
2019-12-12 15:44:57.036 INFO REPOSITORY_SET Repository set to /media/Pictures
2019-12-12 15:44:57.037 INFO STORAGE_SET Storage set to one://box_backup
2019-12-12 15:45:22.635 INFO BACKUP_START No previous backup found
2019-12-12 15:45:22.635 INFO BACKUP_INDEXING Indexing /media/Pictures
2019-12-12 15:45:22.635 INFO SNAPSHOT_FILTER Parsing filter file /home/dog/.duplicacy-web/repositories/localhost/2/.duplicacy/filters
2019-12-12 15:45:22.635 INFO SNAPSHOT_FILTER Loaded 4 include/exclude pattern(s)
2019-12-12 15:56:50.019 INFO INCOMPLETE_LOAD Incomplete snapshot loaded from /home/dog/.duplicacy-web/repositories/localhost/2/.duplicacy/incomplete
2019-12-12 15:56:50.019 INFO BACKUP_LIST Listing all chunks
2019-12-12 15:57:20.019 INFO ONEDRIVE_RETRY Get https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks:/children?top=1000&select=name,size,folder: dial tcp 13.107.42.12:443: i/o timeout; retry after 344 milliseconds
2019-12-12 16:00:36.591 INFO ONEDRIVE_RETRY Get https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/11:/children?top=1000&select=name,size,folder: read tcp 192.168.22.19:58104->13.107.42.12:443: read: connection reset by peer; retry after 340 milliseconds
2019-12-12 16:06:02.364 INFO ONEDRIVE_RETRY Get https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/fb:/children?top=1000&select=name,size,folder: read tcp 192.168.22.19:58124->13.107.42.12:443: read: connection reset by peer; retry after 838 milliseconds
2019-12-12 16:06:18.396 INFO FILE_SKIP Skipped 6 files from previous incomplete backup
2019-12-12 16:10:20.176 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: http2: stream closed; retry after 291 milliseconds
2019-12-12 16:10:25.349 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: write tcp 192.168.22.19:58156->13.107.42.12:443: write: connection reset by peer; retry after 478 milliseconds
2019-12-12 16:11:56.065 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: stream error: stream ID 1; CANCEL; retry after 3096 milliseconds
2019-12-12 16:14:01.301 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: write tcp 192.168.22.19:58242->13.107.42.12:443: write: connection reset by peer; retry after 4182 milliseconds
2019-12-12 16:15:32.381 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: stream error: stream ID 1; CANCEL; retry after 7691 milliseconds
2019-12-12 16:17:38.080 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: write tcp 192.168.22.19:58258->13.107.42.12:443: write: connection reset by peer; retry after 22741 milliseconds
2019-12-12 16:19:21.626 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: stream error: stream ID 1; CANCEL; retry after 10670 milliseconds
2019-12-12 16:21:19.673 INFO ONEDRIVE_RETRY Put https://api.onedrive.com/v1.0/drive/root:/box_backup/chunks/8e/8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777:/content: stream error: stream ID 1; CANCEL; retry after 51003 milliseconds
2019-12-12 16:22:10.676 ERROR UPLOAD_CHUNK Failed to upload the chunk 8e8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777: Maximum number of retries reached
2019-12-12 16:22:10.916 INFO INCOMPLETE_SAVE Incomplete snapshot saved to /home/dog/.duplicacy-web/repositories/localhost/2/.duplicacy/incomplete
Failed to upload the chunk 8e8bb8b16d6a502b4e4d6fad5e8c5af7e936b60598c6680eadd45b9ab64c4777: Maximum number of retries reached

I’m using the Web version on Ubuntu 18.04.3 LTS.

TheBestPessimist · 13 December 2019 11:02

From my understanding “connection reset by peer” means that “the other end” (not yours) has just “thrown its hands in the air and gave up”.

Could it be that onedrive has some issues?

iamgreen · 13 December 2019 17:03

That’s why I’m wondering if anyone else has problems with OneDrive. It’s been having these timeouts on and off for a week (the whole time I’ve been testing Duplicacy). I don’t know if Duplicacy is doing something wrong, OneDrive is acting up, or …?

My plan was to use Duplicacy to back up to the 1TB of storage per account provided by Office 365. That would make the price of Office not a bad deal. Now I’m wondering if I need to re-think things.

gchen · 13 December 2019 17:54

Duplicacy tries up to 8 times when there is an error, which may not be enough:

github.com

gilbertchen/duplicacy/blob/d43fe1a282e4c7e46ab2b3eb2b3849c9e55652d1/src/duplicacy_oneclient.go#L78


	client.RefreshToken(false)


	return client, nil
}


func (client *OneDriveClient) call(url string, method string, input interface{}, contentType string) (io.ReadCloser, int64, error) {


	var response *http.Response


	backoff := 1
	for i := 0; i < 8; i++ {


		LOG_DEBUG("ONEDRIVE_CALL", "%s %s", method, url)


		var inputReader io.Reader


		switch input.(type) {
		default:
			jsonInput, err := json.Marshal(input)
			if err != nil {
				return nil, 0, err

TheBestPessimist · 13 December 2019 18:14

Next day edit: i appologise about the phrasing below, as it sounds rather aggressive . Next time i do this, please ping me on that. I still stand by my idea though: having a global setting for retries would make things much more user-friendly.

@gchen i have asked this multiple times: all retries (all of them) should be increased and respect a setting in the repository. In this way flaky connections can just increase that number (and the sleep for the rate limiting should also respect this setting)

towerbr · 13 December 2019 19:14

I agree, any hardcoded parameter will always be a “source” of support demands.

gchen · 15 December 2019 01:32

A universal retry parameter for all backends would be a nice to have feature, but this is only possible with backends implemented bottom up via http apis (OneDrive being one of them). For backends that rely on third-party libraries, this may not be supported.