Backup failing for Google Cloud Storage

Support
#1

When backing up to Google Cloud Storage I keep receiving timeouts that stop the backup process.

I receive a dialog along the lines of
ERROR Failed to upload chunk [somechunkhash]: Put https://storage.googleapis.com/[location]/[chunkHash] dial tcp [someIPAddress/notMine]:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

This past time I received 3 put messages like this and 1 head message. All in the same dialog.

The log does not reflect these messages though. The last line is simply
Failed to find the path for chunk [someChunkHash]: Head https://storage.googleapis.com/[location]/[chunkHash]: dial tcp [someIPAddress/NotMine]

The file before the error was slightly larger if that makes any difference (I do not remember the sizes of the last file in the log before this particular occurrence). 5,512,787,968 bytes. I have gigabit upload speed. Let me know if you need anymore information.

#2

I’m running some tests to reproduce it.

#3

All my tests passed without errors.

This looks more like temporary network issues. Did these errors happen when the backup was starting, or in the middle after the backup had been running for a while?

#4

I received this error on 2 different computers on 4 or 5 separate occasions now. I am using coldline storage. I have not had troubles with cloudberry using a similar setup. It happens after the backup has been running for a little while. The error interrupts the backup and it does not completely finish. I would say I’ve barely gotten through my larger backup, but the smaller one did eventually finish after several runs. If the exception could at least be handled in a way that the backup would continue to run, that would be nice. It does make me a little worried about the integrity of the upload though. Is there a way to verify already uploaded data?

#5

It’s not always a big file. This past time it was after a 39 MB file.

#6

I didn’t run the tests against the coldline storage. I’ll try it now.

You can run this command to verify existing backups:

duplicacy check -files
#7

Any luck reproducing this?

I have gigabit internet (up and down). If you are having troubles reproducing it, I wonder if it could be a race condition that is harder to observe. I could try limiting the upload speed, though would prefer not to limit it at all.

It could for instance be a connection time out since my pc may be spending more time generating packets than uploading, leaving the connection idle for too long.

#8

Maybe not. Looking at the log files using a text editor, these lines are within a minute of each other.

2017-04-24 05:08:15.445 INFO UPLOAD_PROGRESS Uploaded chunk 109527 size 2638680, 46.39MB/s 1 day 05:50:44 8.6%

2017-04-24 05:08:36.468 ERROR UPLOAD_CHUNK Failed to upload the chunk 3dd5fb094eb7cc81c96b3adb9adb59d6967a253dc3a0aff22f355ab00050ac78: Put https://storage.googleapis.com/[redacted]/[redacted]/chunks/3dd5fb094eb7cc81c96b3adb9adb59d6967a253dc3a0aff22f355ab00050ac78: dial tcp 173.194.219.128:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

#9

Perhaps it is still a rate issue though. Sort of like DOS prevention or something. To get an idea of how many requests I might be making per second, see the following 3 minutes worth of logs.

I also found this documentation about possible errors using Google Cloud Storage API and possible error 429–Too Many Requests
https://cloud.google.com/storage/docs/json_api/v1/status-codes#429_Too_Many_Requests

It may be that the interoperability just doesn’t return a pretty error message and cuts the connection.

Based on the following from that documentation

Trying to mutate a bucket’s or object’s metadata more than once per second.

Trying to compose objects at over 200 components per second.

It sounds to me like, if the name is not checked for with each upload (limit 2 per second), then it should be fine (because my 1 computer could never reach 200 puts per second). I may be interpreting this incorrectly though. I am not familiar with this stuff.

edit: Reading this again, I don’t know that a check if file exists should count as a mutate/ modify. I do have a sneaky suspicion that I’m getting hit with some sort of rate limit though. As mentioned below, I will let you know if applying a rate limit helps.

Does passing in the “exclusive” flag skip the existing file check? I may just need to set this up using cli so I can pass that in if so.
Limiting the speed would be the other option, but I couldn’t really guarantee a number of API calls via bandwidth.

2017-04-21 20:33:00.037 INFO UPLOAD_PROGRESS Uploaded chunk 58966 size 1272683, 45.44MB/s 1 day 07:41:08 5.0%
2017-04-21 20:33:00.719 INFO UPLOAD_PROGRESS Uploaded chunk 58967 size 5919729, 45.45MB/s 1 day 07:41:05 5.0%
2017-04-21 20:33:01.098 INFO UPLOAD_PROGRESS Uploaded chunk 58968 size 10984396, 45.44MB/s 1 day 07:41:20 5.0%
2017-04-21 20:33:01.417 INFO UPLOAD_PROGRESS Uploaded chunk 58969 size 6616330, 45.44MB/s 1 day 07:41:17 5.0%
2017-04-21 20:33:01.740 INFO UPLOAD_PROGRESS Uploaded chunk 58970 size 4770930, 45.44MB/s 1 day 07:41:15 5.0%
2017-04-21 20:33:02.040 INFO UPLOAD_PROGRESS Uploaded chunk 58971 size 2209917, 45.43MB/s 1 day 07:41:33 5.0%
2017-04-21 20:33:02.341 INFO UPLOAD_PROGRESS Uploaded chunk 58972 size 1951206, 45.43MB/s 1 day 07:41:32 5.0%
2017-04-21 20:33:02.639 INFO UPLOAD_PROGRESS Uploaded chunk 58973 size 3481554, 45.43MB/s 1 day 07:41:31 5.0%
2017-04-21 20:33:03.072 INFO UPLOAD_PROGRESS Uploaded chunk 58974 size 14677418, 45.43MB/s 1 day 07:41:43 5.0%
2017-04-21 20:33:03.470 INFO UPLOAD_PROGRESS Uploaded chunk 58975 size 12931356, 45.43MB/s 1 day 07:41:38 5.0%
2017-04-21 20:33:03.470 INFO UPLOAD_PROGRESS Skipped chunk 58977 size 2425844, 45.43MB/s 1 day 07:41:37 5.0%
2017-04-21 20:33:03.591 INFO UPLOAD_PROGRESS Skipped chunk 58978 size 5970259, 45.43MB/s 1 day 07:41:35 5.0%
2017-04-21 20:33:03.625 INFO UPLOAD_PROGRESS Skipped chunk 58979 size 2089306, 45.43MB/s 1 day 07:41:34 5.0%
2017-04-21 20:33:03.817 INFO UPLOAD_PROGRESS Uploaded chunk 58976 size 9302021, 45.43MB/s 1 day 07:41:30 5.0%
2017-04-21 20:33:04.383 INFO UPLOAD_PROGRESS Uploaded chunk 58980 size 1207809, 45.43MB/s 1 day 07:41:48 5.0%
2017-04-21 20:33:04.383 INFO UPLOAD_PROGRESS Skipped chunk 58982 size 11185413, 45.43MB/s 1 day 07:41:44 5.0%
2017-04-21 20:33:04.537 INFO UPLOAD_PROGRESS Skipped chunk 58983 size 7455233, 45.43MB/s 1 day 07:41:40 5.0%
2017-04-21 20:33:04.715 INFO UPLOAD_PROGRESS Skipped chunk 58984 size 9700973, 45.43MB/s 1 day 07:41:36 5.0%
2017-04-21 20:33:04.770 INFO UPLOAD_PROGRESS Uploaded chunk 58981 size 3867690, 45.43MB/s 1 day 07:41:35 5.0%
2017-04-21 20:33:04.777 INFO UPLOAD_PROGRESS Skipped chunk 58985 size 3793858, 45.43MB/s 1 day 07:41:33 5.0%
2017-04-21 20:33:04.856 INFO UPLOAD_PROGRESS Skipped chunk 58986 size 4036820, 45.43MB/s 1 day 07:41:31 5.0%
2017-04-21 20:33:04.882 INFO UPLOAD_PROGRESS Skipped chunk 58987 size 1576452, 45.43MB/s 1 day 07:41:31 5.0%
2017-04-21 20:33:05.031 INFO UPLOAD_PROGRESS Skipped chunk 58989 size 4563305, 45.43MB/s 1 day 07:41:48 5.0%
2017-04-21 20:33:05.062 INFO UPLOAD_PROGRESS Skipped chunk 58990 size 2049545, 45.43MB/s 1 day 07:41:47 5.0%
2017-04-21 20:33:05.169 INFO UPLOAD_PROGRESS Skipped chunk 58991 size 5731868, 45.43MB/s 1 day 07:41:45 5.0%
2017-04-21 20:33:05.404 INFO UPLOAD_PROGRESS Uploaded chunk 58988 size 2269745, 45.43MB/s 1 day 07:41:44 5.0%
2017-04-21 20:33:05.777 INFO UPLOAD_PROGRESS Uploaded chunk 58992 size 3021416, 45.43MB/s 1 day 07:41:42 5.0%
2017-04-21 20:33:05.777 INFO UPLOAD_PROGRESS Skipped chunk 58994 size 9304856, 45.43MB/s 1 day 07:41:38 5.0%
2017-04-21 20:33:06.045 INFO UPLOAD_PROGRESS Uploaded chunk 58993 size 2440414, 45.42MB/s 1 day 07:41:56 5.0%
2017-04-21 20:33:06.379 INFO UPLOAD_PROGRESS Uploaded chunk 58995 size 7914152, 45.42MB/s 1 day 07:41:53 5.0%
2017-04-21 20:33:06.628 INFO UPLOAD_PROGRESS Uploaded chunk 58996 size 2440427, 45.43MB/s 1 day 07:41:52 5.0%
2017-04-21 20:33:06.920 INFO UPLOAD_PROGRESS Uploaded chunk 58997 size 2688368, 45.43MB/s 1 day 07:41:51 5.0%
2017-04-21 20:33:07.212 INFO UPLOAD_PROGRESS Uploaded chunk 58998 size 3971656, 45.42MB/s 1 day 07:42:08 5.0%
2017-04-21 20:33:07.459 INFO UPLOAD_PROGRESS Uploaded chunk 58999 size 2741504, 45.42MB/s 1 day 07:42:07 5.0%
2017-04-21 20:33:07.687 INFO UPLOAD_PROGRESS Uploaded chunk 59000 size 1207927, 45.42MB/s 1 day 07:42:07 5.0%
2017-04-21 20:33:08.036 INFO UPLOAD_PROGRESS Uploaded chunk 59001 size 11725219, 45.41MB/s 1 day 07:42:21 5.0%
2017-04-21 20:33:08.960 INFO UPLOAD_PROGRESS Uploaded chunk 59002 size 10897283, 45.42MB/s 1 day 07:42:16 5.0%
2017-04-21 20:33:09.206 INFO UPLOAD_PROGRESS Uploaded chunk 59003 size 1460328, 45.41MB/s 1 day 07:42:34 5.0%
2017-04-21 20:33:09.487 INFO UPLOAD_PROGRESS Uploaded chunk 59004 size 2801078, 45.41MB/s 1 day 07:42:33 5.0%
2017-04-21 20:33:09.862 INFO UPLOAD_PROGRESS Uploaded chunk 59005 size 3401958, 45.41MB/s 1 day 07:42:32 5.0%
2017-04-21 20:33:10.138 INFO UPLOAD_PROGRESS Uploaded chunk 59006 size 1706023, 45.40MB/s 1 day 07:42:50 5.0%
2017-04-21 20:33:10.436 INFO UPLOAD_PROGRESS Uploaded chunk 59007 size 1088772, 45.40MB/s 1 day 07:42:50 5.0%
2017-04-21 20:33:10.705 INFO UPLOAD_PROGRESS Uploaded chunk 59008 size 2108760, 45.40MB/s 1 day 07:42:49 5.0%
2017-04-21 20:33:11.037 INFO UPLOAD_PROGRESS Uploaded chunk 59009 size 4573017, 45.40MB/s 1 day 07:43:06 5.0%
2017-04-21 20:33:11.339 INFO UPLOAD_PROGRESS Uploaded chunk 59010 size 2583117, 45.40MB/s 1 day 07:43:05 5.0%
2017-04-21 20:33:11.677 INFO UPLOAD_PROGRESS Uploaded chunk 59011 size 2804437, 45.40MB/s 1 day 07:43:04 5.0%
2017-04-21 20:33:12.109 INFO UPLOAD_PROGRESS Uploaded chunk 59012 size 6914123, 45.39MB/s 1 day 07:43:20 5.0%
2017-04-21 20:33:12.446 INFO UPLOAD_PROGRESS Uploaded chunk 59013 size 1575288, 45.39MB/s 1 day 07:43:19 5.0%
2017-04-21 20:33:12.677 INFO UPLOAD_PROGRESS Uploaded chunk 59014 size 1377728, 45.39MB/s 1 day 07:43:18 5.0%
2017-04-21 20:33:13.132 INFO UPLOAD_PROGRESS Uploaded chunk 59015 size 1484582, 45.38MB/s 1 day 07:43:37 5.0%
2017-04-21 20:33:13.654 INFO UPLOAD_PROGRESS Uploaded chunk 59016 size 11566296, 45.38MB/s 1 day 07:43:32 5.0%
2017-04-21 20:33:14.141 INFO UPLOAD_PROGRESS Uploaded chunk 59017 size 9329107, 45.38MB/s 1 day 07:43:47 5.0%
2017-04-21 20:33:14.435 INFO UPLOAD_PROGRESS Uploaded chunk 59018 size 2417138, 45.38MB/s 1 day 07:43:46 5.0%
2017-04-21 20:33:14.889 INFO UPLOAD_PROGRESS Uploaded chunk 59019 size 13809365, 45.38MB/s 1 day 07:43:40 5.0%
2017-04-21 20:33:15.223 INFO UPLOAD_PROGRESS Uploaded chunk 59020 size 3774990, 45.37MB/s 1 day 07:43:58 5.0%
2017-04-21 20:33:15.575 INFO UPLOAD_PROGRESS Uploaded chunk 59021 size 3672965, 45.37MB/s 1 day 07:43:56 5.0%
2017-04-21 20:33:15.787 INFO UPLOAD_PROGRESS Uploaded chunk 59022 size 1091372, 45.38MB/s 1 day 07:43:56 5.0%
2017-04-21 20:33:15.787 INFO UPLOAD_PROGRESS Skipped chunk 59024 size 1210841, 45.38MB/s 1 day 07:43:55 5.0%
2017-04-21 20:33:15.808 INFO UPLOAD_PROGRESS Skipped chunk 59025 size 1518294, 45.38MB/s 1 day 07:43:54 5.0%
2017-04-21 20:33:16.098 INFO UPLOAD_PROGRESS Uploaded chunk 59023 size 3221036, 45.37MB/s 1 day 07:44:12 5.0%
2017-04-21 20:33:16.408 INFO UPLOAD_PROGRESS Uploaded chunk 59026 size 6954244, 45.37MB/s 1 day 07:44:09 5.0%
2017-04-21 20:33:16.706 INFO UPLOAD_PROGRESS Uploaded chunk 59027 size 1503095, 45.37MB/s 1 day 07:44:08 5.0%
2017-04-21 20:33:16.999 INFO UPLOAD_PROGRESS Uploaded chunk 59028 size 4056558, 45.37MB/s 1 day 07:44:07 5.0%
2017-04-21 20:33:17.380 INFO UPLOAD_PROGRESS Uploaded chunk 59029 size 5281851, 45.36MB/s 1 day 07:44:24 5.0%
2017-04-21 20:33:17.599 INFO UPLOAD_PROGRESS Uploaded chunk 59030 size 1178119, 45.36MB/s 1 day 07:44:23 5.0%
2017-04-21 20:33:18.228 INFO UPLOAD_PROGRESS Uploaded chunk 59031 size 10634466, 45.36MB/s 1 day 07:44:38 5.0%
2017-04-21 20:33:18.530 INFO UPLOAD_PROGRESS Uploaded chunk 59032 size 1804842, 45.36MB/s 1 day 07:44:37 5.0%
2017-04-21 20:33:18.841 INFO UPLOAD_PROGRESS Uploaded chunk 59033 size 5112876, 45.36MB/s 1 day 07:44:35 5.0%
2017-04-21 20:33:19.128 INFO UPLOAD_PROGRESS Uploaded chunk 59034 size 3685344, 45.35MB/s 1 day 07:44:52 5.0%
2017-04-21 20:33:19.520 INFO UPLOAD_PROGRESS Uploaded chunk 59035 size 7319901, 45.35MB/s 1 day 07:44:49 5.0%
2017-04-21 20:33:19.882 INFO UPLOAD_PROGRESS Uploaded chunk 59036 size 5167928, 45.35MB/s 1 day 07:44:47 5.0%
2017-04-21 20:33:20.340 INFO UPLOAD_PROGRESS Uploaded chunk 59037 size 7722673, 45.35MB/s 1 day 07:45:03 5.0%
2017-04-21 20:33:20.581 INFO UPLOAD_PROGRESS Uploaded chunk 59038 size 1748492, 45.35MB/s 1 day 07:45:02 5.0%
2017-04-21 20:33:20.582 INFO UPLOAD_PROGRESS Skipped chunk 59040 size 1804842, 45.35MB/s 1 day 07:45:01 5.0%
2017-04-21 20:33:21.057 INFO UPLOAD_PROGRESS Uploaded chunk 59039 size 5685196, 45.34MB/s 1 day 07:45:18 5.0%
2017-04-21 20:33:21.468 INFO UPLOAD_PROGRESS Uploaded chunk 59041 size 3899178, 45.34MB/s 1 day 07:45:16 5.0%
2017-04-21 20:33:21.938 INFO UPLOAD_PROGRESS Uploaded chunk 59042 size 1213698, 45.34MB/s 1 day 07:45:16 5.0%
2017-04-21 20:33:22.228 INFO UPLOAD_PROGRESS Uploaded chunk 59043 size 5340565, 45.34MB/s 1 day 07:45:32 5.0%
2017-04-21 20:33:22.481 INFO UPLOAD_PROGRESS Uploaded chunk 59044 size 1199504, 45.34MB/s 1 day 07:45:32 5.0%
2017-04-21 20:33:22.482 INFO UPLOAD_PROGRESS Skipped chunk 59046 size 1804842, 45.34MB/s 1 day 07:45:31 5.0%
2017-04-21 20:33:22.778 INFO UPLOAD_PROGRESS Uploaded chunk 59045 size 4821003, 45.34MB/s 1 day 07:45:29 5.0%
2017-04-21 20:33:23.034 INFO UPLOAD_PROGRESS Uploaded chunk 59047 size 3371478, 45.33MB/s 1 day 07:45:47 5.0%
2017-04-21 20:33:23.319 INFO UPLOAD_PROGRESS Uploaded chunk 59048 size 1741398, 45.33MB/s 1 day 07:45:46 5.0%
2017-04-21 20:33:23.634 INFO UPLOAD_PROGRESS Uploaded chunk 59049 size 6316801, 45.33MB/s 1 day 07:45:43 5.0%
2017-04-21 20:33:24.160 INFO UPLOAD_PROGRESS Uploaded chunk 59050 size 14360197, 45.33MB/s 1 day 07:45:56 5.0%
2017-04-21 20:33:24.509 INFO UPLOAD_PROGRESS Uploaded chunk 59051 size 6649865, 45.33MB/s 1 day 07:45:53 5.0%
2017-04-21 20:33:24.818 INFO UPLOAD_PROGRESS Uploaded chunk 59052 size 8696336, 45.33MB/s 1 day 07:45:50 5.0%
2017-04-21 20:33:25.067 INFO UPLOAD_PROGRESS Uploaded chunk 59053 size 1966136, 45.32MB/s 1 day 07:46:08 5.0%
2017-04-21 20:33:25.532 INFO UPLOAD_PROGRESS Uploaded chunk 59054 size 9397132, 45.32MB/s 1 day 07:46:04 5.0%
2017-04-21 20:33:25.968 INFO UPLOAD_PROGRESS Uploaded chunk 59055 size 14621740, 45.33MB/s 1 day 07:45:58 5.0%
2017-04-21 20:33:26.369 INFO UPLOAD_PROGRESS Uploaded chunk 59056 size 4806236, 45.32MB/s 1 day 07:46:15 5.0%
2017-04-21 20:33:26.725 INFO UPLOAD_PROGRESS Uploaded chunk 59057 size 8327735, 45.32MB/s 1 day 07:46:11 5.0%
2017-04-21 20:33:27.011 INFO UPLOAD_PROGRESS Uploaded chunk 59058 size 4013463, 45.31MB/s 1 day 07:46:28 5.0%
2017-04-21 20:33:27.708 INFO UPLOAD_PROGRESS Uploaded chunk 59059 size 9942676, 45.31MB/s 1 day 07:46:24 5.0%
2017-04-21 20:33:28.226 INFO UPLOAD_PROGRESS Uploaded chunk 59060 size 9333426, 45.31MB/s 1 day 07:46:39 5.0%
2017-04-21 20:33:28.617 INFO UPLOAD_PROGRESS Uploaded chunk 59061 size 10220200, 45.31MB/s 1 day 07:46:35 5.0%
2017-04-21 20:33:29.143 INFO UPLOAD_PROGRESS Uploaded chunk 59062 size 8630475, 45.30MB/s 1 day 07:46:50 5.0%
2017-04-21 20:33:29.454 INFO UPLOAD_PROGRESS Uploaded chunk 59063 size 6653975, 45.31MB/s 1 day 07:46:47 5.0%
2017-04-21 20:33:29.947 INFO UPLOAD_PROGRESS Uploaded chunk 59064 size 8232034, 45.31MB/s 1 day 07:46:44 5.0%
2017-04-21 20:33:30.333 INFO UPLOAD_PROGRESS Uploaded chunk 59065 size 3373375, 45.30MB/s 1 day 07:47:02 5.0%
2017-04-21 20:33:30.579 INFO UPLOAD_PROGRESS Uploaded chunk 59066 size 1282448, 45.30MB/s 1 day 07:47:01 5.0%
2017-04-21 20:33:31.014 INFO UPLOAD_PROGRESS Uploaded chunk 59067 size 9373441, 45.29MB/s 1 day 07:47:16 5.0%
2017-04-21 20:33:31.317 INFO UPLOAD_PROGRESS Uploaded chunk 59068 size 1739316, 45.29MB/s 1 day 07:47:15 5.0%
2017-04-21 20:33:31.639 INFO UPLOAD_PROGRESS Uploaded chunk 59069 size 2421590, 45.29MB/s 1 day 07:47:14 5.0%
2017-04-21 20:33:31.926 INFO UPLOAD_PROGRESS Uploaded chunk 59070 size 1395614, 45.29MB/s 1 day 07:47:14 5.0%
2017-04-21 20:33:32.318 INFO UPLOAD_PROGRESS Uploaded chunk 59071 size 5718454, 45.29MB/s 1 day 07:47:30 5.0%
2017-04-21 20:33:32.754 INFO UPLOAD_PROGRESS Uploaded chunk 59072 size 11920131, 45.29MB/s 1 day 07:47:25 5.0%
2017-04-21 20:33:33.086 INFO UPLOAD_PROGRESS Uploaded chunk 59073 size 4554269, 45.28MB/s 1 day 07:47:42 5.0%
2017-04-21 20:33:33.552 INFO UPLOAD_PROGRESS Uploaded chunk 59074 size 8835313, 45.28MB/s 1 day 07:47:39 5.0%
2017-04-21 20:33:34.012 INFO UPLOAD_PROGRESS Uploaded chunk 59075 size 3611584, 45.28MB/s 1 day 07:47:56 5.0%
2017-04-21 20:33:34.321 INFO UPLOAD_PROGRESS Uploaded chunk 59076 size 5583100, 45.28MB/s 1 day 07:47:54 5.0%
2017-04-21 20:33:34.557 INFO UPLOAD_PROGRESS Uploaded chunk 59077 size 1888287, 45.28MB/s 1 day 07:47:53 5.0%
2017-04-21 20:33:35.325 INFO UPLOAD_PROGRESS Uploaded chunk 59078 size 2042239, 45.27MB/s 1 day 07:48:11 5.0%
2017-04-21 20:33:35.593 INFO UPLOAD_PROGRESS Uploaded chunk 59079 size 3020220, 45.27MB/s 1 day 07:48:10 5.0%
2017-04-21 20:33:35.918 INFO UPLOAD_PROGRESS Uploaded chunk 59080 size 3699206, 45.27MB/s 1 day 07:48:08 5.0%
2017-04-21 20:33:36.185 INFO UPLOAD_PROGRESS Uploaded chunk 59081 size 1292687, 45.27MB/s 1 day 07:48:27 5.0%
2017-04-21 20:33:36.461 INFO UPLOAD_PROGRESS Uploaded chunk 59082 size 2806041, 45.27MB/s 1 day 07:48:25 5.0%
2017-04-21 20:33:36.918 INFO UPLOAD_PROGRESS Uploaded chunk 59083 size 8764221, 45.27MB/s 1 day 07:48:22 5.0%
2017-04-21 20:33:37.342 INFO UPLOAD_PROGRESS Uploaded chunk 59084 size 6095347, 45.26MB/s 1 day 07:48:38 5.0%
2017-04-21 20:33:37.604 INFO UPLOAD_PROGRESS Uploaded chunk 59085 size 2691408, 45.26MB/s 1 day 07:48:37 5.0%
2017-04-21 20:33:38.048 INFO UPLOAD_PROGRESS Uploaded chunk 59086 size 8398847, 45.25MB/s 1 day 07:48:52 5.0%
2017-04-21 20:33:38.436 INFO UPLOAD_PROGRESS Uploaded chunk 59087 size 5188830, 45.26MB/s 1 day 07:48:50 5.0%
2017-04-21 20:33:38.759 INFO UPLOAD_PROGRESS Uploaded chunk 59088 size 3116646, 45.26MB/s 1 day 07:48:49 5.0%
2017-04-21 20:33:39.351 INFO UPLOAD_PROGRESS Uploaded chunk 59089 size 2516145, 45.25MB/s 1 day 07:49:07 5.0%
2017-04-21 20:33:39.713 INFO UPLOAD_PROGRESS Uploaded chunk 59090 size 1554663, 45.25MB/s 1 day 07:49:06 5.0%
2017-04-21 20:33:40.038 INFO UPLOAD_PROGRESS Uploaded chunk 59091 size 1632559, 45.24MB/s 1 day 07:49:24 5.0%
2017-04-21 20:33:40.816 INFO UPLOAD_PROGRESS Uploaded chunk 59092 size 6186519, 45.24MB/s 1 day 07:49:22 5.0%
2017-04-21 20:33:41.325 INFO UPLOAD_PROGRESS Uploaded chunk 59093 size 7763392, 45.24MB/s 1 day 07:49:37 5.0%
2017-04-21 20:33:41.829 INFO UPLOAD_PROGRESS Uploaded chunk 59094 size 9184048, 45.24MB/s 1 day 07:49:34 5.0%
2017-04-21 20:33:42.097 INFO UPLOAD_PROGRESS Uploaded chunk 59095 size 1753242, 45.23MB/s 1 day 07:49:52 5.0%
2017-04-21 20:33:42.360 INFO UPLOAD_PROGRESS Uploaded chunk 59096 size 1914000, 45.23MB/s 1 day 07:49:51 5.0%
2017-04-21 20:33:42.668 INFO UPLOAD_PROGRESS Uploaded chunk 59097 size 3284890, 45.23MB/s 1 day 07:49:50 5.0%
2017-04-21 20:33:42.924 INFO UPLOAD_PROGRESS Uploaded chunk 59098 size 1450370, 45.23MB/s 1 day 07:49:49 5.0%
2017-04-21 20:33:43.348 INFO UPLOAD_PROGRESS Uploaded chunk 59099 size 4449164, 45.23MB/s 1 day 07:50:06 5.0%
2017-04-21 20:33:44.238 INFO UPLOAD_PROGRESS Uploaded chunk 59100 size 2151933, 45.22MB/s 1 day 07:50:24 5.0%
2017-04-21 20:33:44.965 INFO UPLOAD_PROGRESS Uploaded chunk 59101 size 1339583, 45.22MB/s 1 day 07:50:24 5.0%
2017-04-21 20:33:45.308 INFO UPLOAD_PROGRESS Uploaded chunk 59102 size 2391997, 45.21MB/s 1 day 07:50:42 5.0%
2017-04-21 20:33:45.612 INFO UPLOAD_PROGRESS Uploaded chunk 59103 size 6087603, 45.21MB/s 1 day 07:50:39 5.0%
2017-04-21 20:33:46.021 INFO UPLOAD_PROGRESS Uploaded chunk 59104 size 5432484, 45.21MB/s 1 day 07:50:56 5.0%
2017-04-21 20:33:46.303 INFO UPLOAD_PROGRESS Uploaded chunk 59105 size 1421986, 45.21MB/s 1 day 07:50:55 5.0%
2017-04-21 20:33:46.693 INFO UPLOAD_PROGRESS Uploaded chunk 59106 size 5540417, 45.21MB/s 1 day 07:50:53 5.0%
2017-04-21 20:33:46.942 INFO UPLOAD_PROGRESS Uploaded chunk 59107 size 1097974, 45.21MB/s 1 day 07:50:52 5.0%
2017-04-21 20:33:47.351 INFO UPLOAD_PROGRESS Uploaded chunk 59108 size 1991058, 45.20MB/s 1 day 07:51:10 5.0%
2017-04-21 20:33:48.033 INFO UPLOAD_PROGRESS Uploaded chunk 59109 size 3032007, 45.19MB/s 1 day 07:51:28 5.0%
2017-04-21 20:33:48.326 INFO UPLOAD_PROGRESS Uploaded chunk 59110 size 2499713, 45.19MB/s 1 day 07:51:27 5.0%
2017-04-21 20:33:48.607 INFO UPLOAD_PROGRESS Uploaded chunk 59111 size 1339822, 45.19MB/s 1 day 07:51:26 5.0%
2017-04-21 20:33:49.022 INFO UPLOAD_PROGRESS Uploaded chunk 59112 size 2165280, 45.19MB/s 1 day 07:51:44 5.0%
2017-04-21 20:33:49.342 INFO UPLOAD_PROGRESS Uploaded chunk 59113 size 4545606, 45.19MB/s 1 day 07:51:43 5.0%
2017-04-21 20:33:49.885 INFO UPLOAD_PROGRESS Uploaded chunk 59114 size 4109052, 45.19MB/s 1 day 07:51:41 5.0%
2017-04-21 20:33:50.387 INFO UPLOAD_PROGRESS Uploaded chunk 59115 size 6809613, 45.18MB/s 1 day 07:51:57 5.0%
2017-04-21 20:33:50.659 INFO UPLOAD_PROGRESS Uploaded chunk 59116 size 1350219, 45.18MB/s 1 day 07:51:56 5.0%
2017-04-21 20:33:50.969 INFO UPLOAD_PROGRESS Uploaded chunk 59117 size 2907562, 45.18MB/s 1 day 07:51:55 5.0%
2017-04-21 20:33:51.320 INFO UPLOAD_PROGRESS Uploaded chunk 59118 size 2736925, 45.17MB/s 1 day 07:52:13 5.0%
2017-04-21 20:33:51.618 INFO UPLOAD_PROGRESS Uploaded chunk 59119 size 3360472, 45.18MB/s 1 day 07:52:11 5.0%
2017-04-21 20:33:52.079 INFO UPLOAD_PROGRESS Uploaded chunk 59120 size 12678942, 45.17MB/s 1 day 07:52:25 5.0%
2017-04-21 20:33:52.634 INFO UPLOAD_PROGRESS Uploaded chunk 59121 size 5733345, 45.17MB/s 1 day 07:52:23 5.0%
2017-04-21 20:33:53.032 INFO UPLOAD_PROGRESS Uploaded chunk 59122 size 1315102, 45.16MB/s 1 day 07:52:41 5.0%
2017-04-21 20:33:53.517 INFO UPLOAD_PROGRESS Uploaded chunk 59123 size 9799751, 45.16MB/s 1 day 07:52:37 5.0%
2017-04-21 20:33:53.851 INFO UPLOAD_PROGRESS Uploaded chunk 59124 size 3445414, 45.17MB/s 1 day 07:52:35 5.0%
2017-04-21 20:33:54.113 INFO UPLOAD_PROGRESS Uploaded chunk 59125 size 2004711, 45.16MB/s 1 day 07:52:54 5.0%
2017-04-21 20:33:54.505 INFO UPLOAD_PROGRESS Uploaded chunk 59126 size 1798549, 45.16MB/s 1 day 07:52:53 5.0%
2017-04-21 20:33:54.859 INFO UPLOAD_PROGRESS Uploaded chunk 59127 size 4555948, 45.16MB/s 1 day 07:52:51 5.0%
2017-04-21 20:33:55.256 INFO UPLOAD_PROGRESS Uploaded chunk 59128 size 5752677, 45.15MB/s 1 day 07:53:07 5.0%
2017-04-21 20:33:55.823 INFO UPLOAD_PROGRESS Uploaded chunk 59129 size 11337568, 45.15MB/s 1 day 07:53:03 5.0%
2017-04-21 20:33:56.090 INFO UPLOAD_PROGRESS Uploaded chunk 59130 size 1455943, 45.15MB/s 1 day 07:53:21 5.0%
2017-04-21 20:33:56.326 INFO UPLOAD_PROGRESS Uploaded chunk 59131 size 1310487, 45.15MB/s 1 day 07:53:20 5.0%
2017-04-21 20:33:56.797 INFO UPLOAD_PROGRESS Uploaded chunk 59132 size 6070650, 45.15MB/s 1 day 07:53:18 5.0%
2017-04-21 20:33:57.187 INFO UPLOAD_PROGRESS Uploaded chunk 59133 size 13364843, 45.14MB/s 1 day 07:53:31 5.0%
2017-04-21 20:33:58.621 INFO UPLOAD_PROGRESS Uploaded chunk 59134 size 4358840, 45.14MB/s 1 day 07:53:48 5.0%
2017-04-21 20:33:59.126 INFO UPLOAD_PROGRESS Uploaded chunk 59135 size 6532545, 45.13MB/s 1 day 07:54:04 5.0%
2017-04-21 20:33:59.878 INFO UPLOAD_PROGRESS Uploaded chunk 59136 size 2472706, 45.13MB/s 1 day 07:54:03 5.0%
2017-04-21 20:34:00.139 INFO UPLOAD_PROGRESS Uploaded chunk 59137 size 1430654, 45.12MB/s 1 day 07:54:22 5.0%
2017-04-21 20:34:00.369 INFO UPLOAD_PROGRESS Uploaded chunk 59138 size 1291554, 45.12MB/s 1 day 07:54:21 5.0%
2017-04-21 20:34:00.650 INFO UPLOAD_PROGRESS Uploaded chunk 59139 size 1449501, 45.12MB/s 1 day 07:54:21 5.0%
2017-04-21 20:34:00.982 INFO UPLOAD_PROGRESS Uploaded chunk 59140 size 9179590, 45.13MB/s 1 day 07:54:17 5.0%
2017-04-21 20:34:01.262 INFO UPLOAD_PROGRESS Uploaded chunk 59141 size 1637793, 45.12MB/s 1 day 07:54:35 5.0%
2017-04-21 20:34:01.578 INFO UPLOAD_PROGRESS Uploaded chunk 59142 size 5396599, 45.12MB/s 1 day 07:54:33 5.0%
2017-04-21 20:34:02.108 INFO UPLOAD_PROGRESS Uploaded chunk 59143 size 4194154, 45.11MB/s 1 day 07:54:50 5.0%
2017-04-21 20:34:02.837 INFO UPLOAD_PROGRESS Uploaded chunk 59144 size 11234754, 45.11MB/s 1 day 07:54:45 5.0%
2017-04-21 20:34:03.117 INFO UPLOAD_PROGRESS Uploaded chunk 59145 size 1689423, 45.11MB/s 1 day 07:55:03 5.0%
2017-04-21 20:34:03.392 INFO UPLOAD_PROGRESS Uploaded chunk 59146 size 1964860, 45.11MB/s 1 day 07:55:03 5.0%
2017-04-21 20:34:04.205 INFO UPLOAD_PROGRESS Uploaded chunk 59147 size 5454846, 45.10MB/s 1 day 07:55:19 5.0%
2017-04-21 20:34:04.604 INFO UPLOAD_PROGRESS Uploaded chunk 59148 size 3236323, 45.10MB/s 1 day 07:55:18 5.0%
2017-04-21 20:34:04.892 INFO UPLOAD_PROGRESS Uploaded chunk 59149 size 1838063, 45.10MB/s 1 day 07:55:17 5.0%
2017-04-21 20:34:05.385 INFO UPLOAD_PROGRESS Uploaded chunk 59150 size 4471455, 45.09MB/s 1 day 07:55:34 5.0%
2017-04-21 20:34:05.729 INFO UPLOAD_PROGRESS Uploaded chunk 59151 size 2746065, 45.09MB/s 1 day 07:55:33 5.0%
2017-04-21 20:34:06.446 INFO UPLOAD_PROGRESS Uploaded chunk 59152 size 14537990, 45.09MB/s 1 day 07:55:46 5.0%
2017-04-21 20:34:06.702 INFO UPLOAD_PROGRESS Uploaded chunk 59153 size 1292713, 45.09MB/s 1 day 07:55:45 5.0%
2017-04-21 20:34:07.518 INFO UPLOAD_PROGRESS Uploaded chunk 59154 size 9211910, 45.08MB/s 1 day 07:56:00 5.0%
2017-04-21 20:34:07.842 INFO UPLOAD_PROGRESS Uploaded chunk 59155 size 2773644, 45.08MB/s 1 day 07:55:59 5.0%
2017-04-21 20:34:08.187 INFO UPLOAD_PROGRESS Uploaded chunk 59156 size 6943017, 45.08MB/s 1 day 07:56:15 5.0%
2017-04-21 20:34:08.482 INFO UPLOAD_PROGRESS Uploaded chunk 59157 size 2603510, 45.08MB/s 1 day 07:56:14 5.0%
2017-04-21 20:34:08.791 INFO UPLOAD_PROGRESS Uploaded chunk 59158 size 6884787, 45.08MB/s 1 day 07:56:11 5.0%
2017-04-21 20:34:09.073 INFO UPLOAD_PROGRESS Uploaded chunk 59159 size 2098281, 45.07MB/s 1 day 07:56:29 5.0%
2017-04-21 20:34:09.417 INFO UPLOAD_PROGRESS Uploaded chunk 59160 size 1786951, 45.07MB/s 1 day 07:56:28 5.0%
2017-04-21 20:34:09.853 INFO UPLOAD_PROGRESS Uploaded chunk 59161 size 3415003, 45.07MB/s 1 day 07:56:27 5.0%
2017-04-21 20:34:10.229 INFO UPLOAD_PROGRESS Uploaded chunk 59162 size 4886306, 45.07MB/s 1 day 07:56:44 5.0%
2017-04-21 20:34:10.571 INFO UPLOAD_PROGRESS Uploaded chunk 59163 size 1307286, 45.07MB/s 1 day 07:56:43 5.0%
2017-04-21 20:34:11.305 INFO UPLOAD_PROGRESS Uploaded chunk 59164 size 15635282, 45.06MB/s 1 day 07:56:56 5.0%
2017-04-21 20:34:11.575 INFO UPLOAD_PROGRESS Uploaded chunk 59165 size 2273763, 45.06MB/s 1 day 07:56:55 5.0%
2017-04-21 20:34:11.953 INFO UPLOAD_PROGRESS Uploaded chunk 59166 size 5081883, 45.06MB/s 1 day 07:56:52 5.0%
2017-04-21 20:34:12.280 INFO UPLOAD_PROGRESS Uploaded chunk 59167 size 4691735, 45.06MB/s 1 day 07:57:09 5.0%
2017-04-21 20:34:12.720 INFO UPLOAD_PROGRESS Uploaded chunk 59168 size 9218096, 45.06MB/s 1 day 07:57:06 5.0%
2017-04-21 20:34:13.046 INFO UPLOAD_PROGRESS Uploaded chunk 59169 size 5000462, 45.05MB/s 1 day 07:57:22 5.0%
2017-04-21 20:34:13.513 INFO UPLOAD_PROGRESS Uploaded chunk 59170 size 11056200, 45.05MB/s 1 day 07:57:18 5.0%
2017-04-21 20:34:14.059 INFO UPLOAD_PROGRESS Uploaded chunk 59171 size 10067450, 45.05MB/s 1 day 07:57:32 5.0%
2017-04-21 20:34:14.325 INFO UPLOAD_PROGRESS Uploaded chunk 59172 size 1927196, 45.05MB/s 1 day 07:57:32 5.0%
2017-04-21 20:34:14.685 INFO UPLOAD_PROGRESS Uploaded chunk 59173 size 9481864, 45.05MB/s 1 day 07:57:28 5.0%
2017-04-21 20:34:15.017 INFO UPLOAD_PROGRESS Uploaded chunk 59174 size 4376217, 45.04MB/s 1 day 07:57:45 5.0%
2017-04-21 20:34:15.777 INFO UPLOAD_PROGRESS Uploaded chunk 59175 size 3077451, 45.04MB/s 1 day 07:57:43 5.0%
2017-04-21 20:34:16.065 INFO UPLOAD_PROGRESS Uploaded chunk 59176 size 2239451, 45.04MB/s 1 day 07:58:01 5.0%
2017-04-21 20:34:16.426 INFO UPLOAD_PROGRESS Uploaded chunk 59177 size 4510627, 45.04MB/s 1 day 07:57:59 5.0%
2017-04-21 20:34:16.671 INFO UPLOAD_PROGRESS Uploaded chunk 59178 size 1052689, 45.04MB/s 1 day 07:57:59 5.0%
2017-04-21 20:34:17.052 INFO UPLOAD_PROGRESS Uploaded chunk 59179 size 16777216, 45.03MB/s 1 day 07:58:11 5.0%
2017-04-21 20:34:17.609 INFO UPLOAD_PROGRESS Uploaded chunk 59180 size 7957347, 45.03MB/s 1 day 07:58:07 5.0%
2017-04-21 20:34:17.820 INFO UPLOAD_PROGRESS Uploaded chunk 59181 size 1259814, 45.03MB/s 1 day 07:58:07 5.0%
2017-04-21 20:34:18.298 INFO UPLOAD_PROGRESS Uploaded chunk 59182 size 4796350, 45.03MB/s 1 day 07:58:24 5.0%
2017-04-21 20:34:18.623 INFO UPLOAD_PROGRESS Uploaded chunk 59183 size 1800536, 45.03MB/s 1 day 07:58:23 5.0%
2017-04-21 20:34:18.875 INFO UPLOAD_PROGRESS Uploaded chunk 59184 size 2598102, 45.03MB/s 1 day 07:58:22 5.0%
2017-04-21 20:34:19.190 INFO UPLOAD_PROGRESS Uploaded chunk 59185 size 2004606, 45.02MB/s 1 day 07:58:40 5.0%
2017-04-21 20:34:19.547 INFO UPLOAD_PROGRESS Uploaded chunk 59186 size 12143358, 45.02MB/s 1 day 07:58:35 5.0%
2017-04-21 20:34:19.907 INFO UPLOAD_PROGRESS Uploaded chunk 59187 size 3552693, 45.02MB/s 1 day 07:58:33 5.0%
2017-04-21 20:34:20.166 INFO UPLOAD_PROGRESS Uploaded chunk 59188 size 1904980, 45.02MB/s 1 day 07:58:52 5.0%
2017-04-21 20:34:20.452 INFO UPLOAD_PROGRESS Uploaded chunk 59189 size 3471989, 45.02MB/s 1 day 07:58:50 5.0%
2017-04-21 20:34:20.668 INFO UPLOAD_PROGRESS Uploaded chunk 59190 size 1367721, 45.02MB/s 1 day 07:58:49 5.0%
2017-04-21 20:34:21.033 INFO UPLOAD_PROGRESS Uploaded chunk 59191 size 3615536, 45.01MB/s 1 day 07:59:07 5.0%
2017-04-21 20:34:21.505 INFO UPLOAD_PROGRESS Uploaded chunk 59192 size 8563848, 45.01MB/s 1 day 07:59:03 5.0%
2017-04-21 20:34:21.805 INFO UPLOAD_PROGRESS Uploaded chunk 59193 size 2318424, 45.01MB/s 1 day 07:59:02 5.0%
2017-04-21 20:34:22.161 INFO UPLOAD_PROGRESS Uploaded chunk 59194 size 3433244, 45.00MB/s 1 day 07:59:20 5.0%
2017-04-21 20:34:22.628 INFO UPLOAD_PROGRESS Uploaded chunk 59195 size 12190394, 45.01MB/s 1 day 07:59:15 5.0%
2017-04-21 20:34:23.067 INFO UPLOAD_PROGRESS Uploaded chunk 59196 size 10606612, 45.00MB/s 1 day 07:59:29 5.0%
2017-04-21 20:34:23.382 INFO UPLOAD_PROGRESS Uploaded chunk 59197 size 4772312, 45.00MB/s 1 day 07:59:27 5.0%
2017-04-21 20:34:23.650 INFO UPLOAD_PROGRESS Uploaded chunk 59198 size 2978391, 45.00MB/s 1 day 07:59:26 5.0%
2017-04-21 20:34:24.133 INFO UPLOAD_PROGRESS Uploaded chunk 59199 size 2161113, 44.99MB/s 1 day 07:59:44 5.0%
2017-04-21 20:34:24.415 INFO UPLOAD_PROGRESS Uploaded chunk 59200 size 2331237, 44.99MB/s 1 day 07:59:43 5.0%
2017-04-21 20:34:25.464 INFO UPLOAD_PROGRESS Uploaded chunk 59201 size 7682132, 44.99MB/s 1 day 07:59:58 5.0%
2017-04-21 20:34:25.730 INFO UPLOAD_PROGRESS Uploaded chunk 59202 size 2251923, 44.99MB/s 1 day 07:59:58 5.0%
2017-04-21 20:34:26.066 INFO UPLOAD_PROGRESS Uploaded chunk 59203 size 5336368, 44.98MB/s 1 day 08:00:14 5.0%
2017-04-21 20:34:26.482 INFO UPLOAD_PROGRESS Uploaded chunk 59204 size 4355002, 44.98MB/s 1 day 08:00:12 5.0%
2017-04-21 20:34:26.760 INFO UPLOAD_PROGRESS Uploaded chunk 59205 size 1978540, 44.98MB/s 1 day 08:00:12 5.0%
2017-04-21 20:34:27.109 INFO UPLOAD_PROGRESS Uploaded chunk 59206 size 6126527, 44.98MB/s 1 day 08:00:28 5.0%
2017-04-21 20:34:27.922 INFO UPLOAD_PROGRESS Uploaded chunk 59207 size 7926936, 44.98MB/s 1 day 08:00:25 5.0%
2017-04-21 20:34:28.240 INFO UPLOAD_PROGRESS Uploaded chunk 59208 size 4351119, 44.97MB/s 1 day 08:00:42 5.0%
2017-04-21 20:34:28.526 INFO UPLOAD_PROGRESS Uploaded chunk 59209 size 2791963, 44.97MB/s 1 day 08:00:40 5.0%
2017-04-21 20:34:28.816 INFO UPLOAD_PROGRESS Uploaded chunk 59210 size 2556666, 44.97MB/s 1 day 08:00:39 5.0%
2017-04-21 20:34:29.005 INFO UPLOAD_PROGRESS Uploaded chunk 59211 size 1335773, 44.97MB/s 1 day 08:00:58 5.0%
2017-04-21 20:34:29.414 INFO UPLOAD_PROGRESS Uploaded chunk 59212 size 8144009, 44.97MB/s 1 day 08:00:54 5.0%
2017-04-21 20:34:29.692 INFO UPLOAD_PROGRESS Uploaded chunk 59213 size 1374462, 44.97MB/s 1 day 08:00:54 5.0%
2017-04-21 20:34:29.949 INFO UPLOAD_PROGRESS Uploaded chunk 59214 size 1570213, 44.97MB/s 1 day 08:00:53 5.0%
2017-04-21 20:34:30.258 INFO UPLOAD_PROGRESS Uploaded chunk 59215 size 6112459, 44.96MB/s 1 day 08:01:09 5.0%
2017-04-21 20:34:30.511 INFO UPLOAD_PROGRESS Uploaded chunk 59216 size 3022322, 44.96MB/s 1 day 08:01:08 5.0%
2017-04-21 20:34:30.796 INFO UPLOAD_PROGRESS Uploaded chunk 59217 size 5560553, 44.96MB/s 1 day 08:01:06 5.0%
2017-04-21 20:34:31.081 INFO UPLOAD_PROGRESS Uploaded chunk 59218 size 1203171, 44.95MB/s 1 day 08:01:24 5.0%
2017-04-21 20:34:31.916 INFO UPLOAD_PROGRESS Uploaded chunk 59219 size 6533205, 44.96MB/s 1 day 08:01:21 5.0%
2017-04-21 20:34:32.157 INFO UPLOAD_PROGRESS Uploaded chunk 59220 size 2529564, 44.95MB/s 1 day 08:01:39 5.0%
2017-04-21 20:34:32.669 INFO UPLOAD_PROGRESS Uploaded chunk 59221 size 3428021, 44.95MB/s 1 day 08:01:38 5.0%
2017-04-21 20:34:33.076 INFO UPLOAD_PROGRESS Uploaded chunk 59222 size 1524419, 44.94MB/s 1 day 08:01:56 5.0%
2017-04-21 20:34:33.376 INFO UPLOAD_PROGRESS Uploaded chunk 59223 size 2143596, 44.94MB/s 1 day 08:01:55 5.0%
2017-04-21 20:34:33.707 INFO UPLOAD_PROGRESS Uploaded chunk 59224 size 1980546, 44.94MB/s 1 day 08:01:54 5.0%
2017-04-21 20:34:33.928 INFO UPLOAD_PROGRESS Uploaded chunk 59225 size 1141332, 44.94MB/s 1 day 08:01:54 5.0%
2017-04-21 20:34:34.523 INFO UPLOAD_PROGRESS Uploaded chunk 59226 size 6431884, 44.94MB/s 1 day 08:02:10 5.0%
2017-04-21 20:34:34.816 INFO UPLOAD_PROGRESS Uploaded chunk 59227 size 1757256, 44.94MB/s 1 day 08:02:09 5.0%
2017-04-21 20:34:35.261 INFO UPLOAD_PROGRESS Uploaded chunk 59228 size 15506718, 44.93MB/s 1 day 08:02:22 5.0%
2017-04-21 20:34:35.571 INFO UPLOAD_PROGRESS Uploaded chunk 59229 size 1701030, 44.93MB/s 1 day 08:02:21 5.0%
2017-04-21 20:34:35.859 INFO UPLOAD_PROGRESS Uploaded chunk 59230 size 2667067, 44.93MB/s 1 day 08:02:20 5.0%
2017-04-21 20:34:36.195 INFO UPLOAD_PROGRESS Uploaded chunk 59231 size 7263689, 44.93MB/s 1 day 08:02:36 5.0%
2017-04-21 20:34:36.660 INFO UPLOAD_PROGRESS Uploaded chunk 59232 size 8006133, 44.93MB/s 1 day 08:02:32 5.0%
2017-04-21 20:34:37.299 INFO UPLOAD_PROGRESS Uploaded chunk 59233 size 1060309, 44.92MB/s 1 day 08:02:51 5.0%
2017-04-21 20:34:37.557 INFO UPLOAD_PROGRESS Uploaded chunk 59234 size 1175082, 44.92MB/s 1 day 08:02:50 5.0%
2017-04-21 20:34:37.920 INFO UPLOAD_PROGRESS Uploaded chunk 59235 size 6197524, 44.92MB/s 1 day 08:02:48 5.0%
2017-04-21 20:34:38.184 INFO UPLOAD_PROGRESS Uploaded chunk 59236 size 1480101, 44.91MB/s 1 day 08:03:06 5.0%
2017-04-21 20:34:38.477 INFO UPLOAD_PROGRESS Uploaded chunk 59237 size 1609310, 44.91MB/s 1 day 08:03:05 5.0%
2017-04-21 20:34:38.777 INFO UPLOAD_PROGRESS Uploaded chunk 59238 size 2276899, 44.92MB/s 1 day 08:03:04 5.0%
2017-04-21 20:34:39.138 INFO UPLOAD_PROGRESS Uploaded chunk 59239 size 2956458, 44.91MB/s 1 day 08:03:22 5.0%
2017-04-21 20:34:39.630 INFO UPLOAD_PROGRESS Uploaded chunk 59240 size 4701185, 44.91MB/s 1 day 08:03:20 5.0%
2017-04-21 20:34:40.229 INFO UPLOAD_PROGRESS Uploaded chunk 59241 size 3912436, 44.90MB/s 1 day 08:03:37 5.0%
2017-04-21 20:34:40.570 INFO UPLOAD_PROGRESS Uploaded chunk 59242 size 5179430, 44.90MB/s 1 day 08:03:35 5.0%
2017-04-21 20:34:41.172 INFO UPLOAD_PROGRESS Uploaded chunk 59243 size 1069155, 44.90MB/s 1 day 08:03:53 5.0%
2017-04-21 20:34:41.408 INFO UPLOAD_PROGRESS Uploaded chunk 59244 size 1302476, 44.90MB/s 1 day 08:03:53 5.0%
2017-04-21 20:34:41.798 INFO UPLOAD_PROGRESS Uploaded chunk 59245 size 2839146, 44.90MB/s 1 day 08:03:52 5.0%
2017-04-21 20:34:42.035 INFO UPLOAD_PROGRESS Uploaded chunk 59246 size 1373668, 44.89MB/s 1 day 08:04:10 5.0%
2017-04-21 20:34:42.304 INFO UPLOAD_PROGRESS Uploaded chunk 59247 size 1762186, 44.89MB/s 1 day 08:04:09 5.0%
2017-04-21 20:34:42.917 INFO UPLOAD_PROGRESS Uploaded chunk 59248 size 4488537, 44.89MB/s 1 day 08:04:07 5.0%
2017-04-21 20:34:43.273 INFO UPLOAD_PROGRESS Uploaded chunk 59249 size 1902100, 44.88MB/s 1 day 08:04:25 5.0%
2017-04-21 20:34:43.687 INFO UPLOAD_PROGRESS Uploaded chunk 59250 size 5029334, 44.88MB/s 1 day 08:04:23 5.0%
2017-04-21 20:34:44.404 INFO UPLOAD_PROGRESS Uploaded chunk 59251 size 1761414, 44.88MB/s 1 day 08:04:41 5.0%
2017-04-21 20:34:45.175 INFO UPLOAD_PROGRESS Uploaded chunk 59252 size 3374710, 44.87MB/s 1 day 08:04:59 5.0%
2017-04-21 20:34:45.570 INFO UPLOAD_PROGRESS Uploaded chunk 59253 size 6299601, 44.87MB/s 1 day 08:04:56 5.0%
2017-04-21 20:34:46.209 INFO UPLOAD_PROGRESS Uploaded chunk 59254 size 2353321, 44.86MB/s 1 day 08:05:14 5.0%
2017-04-21 20:34:46.499 INFO UPLOAD_PROGRESS Uploaded chunk 59255 size 5080241, 44.86MB/s 1 day 08:05:12 5.0%
2017-04-21 20:34:46.915 INFO UPLOAD_PROGRESS Uploaded chunk 59256 size 4929023, 44.87MB/s 1 day 08:05:10 5.0%
2017-04-21 20:34:47.248 INFO UPLOAD_PROGRESS Uploaded chunk 59257 size 5690342, 44.86MB/s 1 day 08:05:26 5.0%
2017-04-21 20:34:47.571 INFO UPLOAD_PROGRESS Uploaded chunk 59258 size 2214148, 44.86MB/s 1 day 08:05:25 5.0%
2017-04-21 20:34:48.113 INFO UPLOAD_PROGRESS Uploaded chunk 59259 size 16777216, 44.85MB/s 1 day 08:05:37 5.0%
2017-04-21 20:34:48.383 INFO UPLOAD_PROGRESS Uploaded chunk 59260 size 2702731, 44.86MB/s 1 day 08:05:36 5.0%
2017-04-21 20:34:48.735 INFO UPLOAD_PROGRESS Uploaded chunk 59261 size 2745000, 44.86MB/s 1 day 08:05:35 5.0%
2017-04-21 20:34:49.072 INFO UPLOAD_PROGRESS Uploaded chunk 59262 size 5632847, 44.85MB/s 1 day 08:05:51 5.0%
2017-04-21 20:34:49.540 INFO UPLOAD_PROGRESS Uploaded chunk 59263 size 4671684, 44.85MB/s 1 day 08:05:49 5.0%
2017-04-21 20:34:49.863 INFO UPLOAD_PROGRESS Uploaded chunk 59264 size 6202246, 44.85MB/s 1 day 08:05:47 5.0%
2017-04-21 20:34:50.165 INFO UPLOAD_PROGRESS Uploaded chunk 59265 size 3761859, 44.84MB/s 1 day 08:06:04 5.0%
2017-04-21 20:34:50.483 INFO UPLOAD_PROGRESS Uploaded chunk 59266 size 4744087, 44.84MB/s 1 day 08:06:02 5.0%
2017-04-21 20:34:50.725 INFO UPLOAD_PROGRESS Uploaded chunk 59267 size 1345077, 44.85MB/s 1 day 08:06:02 5.0%
2017-04-21 20:34:51.047 INFO UPLOAD_PROGRESS Uploaded chunk 59268 size 3705805, 44.84MB/s 1 day 08:06:19 5.0%
2017-04-21 20:34:51.334 INFO UPLOAD_PROGRESS Uploaded chunk 59269 size 2723989, 44.84MB/s 1 day 08:06:18 5.0%
2017-04-21 20:34:51.718 INFO UPLOAD_PROGRESS Uploaded chunk 59270 size 9241830, 44.84MB/s 1 day 08:06:14 5.0%
2017-04-21 20:34:52.081 INFO UPLOAD_PROGRESS Uploaded chunk 59271 size 10825079, 44.83MB/s 1 day 08:06:28 5.0%
2017-04-21 20:34:52.442 INFO UPLOAD_PROGRESS Uploaded chunk 59272 size 4347315, 44.84MB/s 1 day 08:06:26 5.0%
2017-04-21 20:34:52.750 INFO UPLOAD_PROGRESS Uploaded chunk 59273 size 3489918, 44.84MB/s 1 day 08:06:25 5.0%
2017-04-21 20:34:53.223 INFO UPLOAD_PROGRESS Uploaded chunk 59274 size 11061532, 44.83MB/s 1 day 08:06:39 5.0%
2017-04-21 20:34:53.646 INFO UPLOAD_PROGRESS Uploaded chunk 59275 size 11127358, 44.83MB/s 1 day 08:06:34 5.0%
2017-04-21 20:34:53.959 INFO UPLOAD_PROGRESS Uploaded chunk 59276 size 6344638, 44.83MB/s 1 day 08:06:32 5.0%
2017-04-21 20:34:54.743 INFO UPLOAD_PROGRESS Uploaded chunk 59277 size 3491750, 44.83MB/s 1 day 08:06:49 5.0%
2017-04-21 20:34:55.066 INFO UPLOAD_PROGRESS Uploaded chunk 59278 size 3252529, 44.82MB/s 1 day 08:07:07 5.0%
2017-04-21 20:34:55.390 INFO UPLOAD_PROGRESS Uploaded chunk 59279 size 1891031, 44.82MB/s 1 day 08:07:06 5.0%
2017-04-21 20:34:56.050 INFO UPLOAD_PROGRESS Uploaded chunk 59280 size 1320529, 44.81MB/s 1 day 08:07:24 5.0%
2017-04-21 20:34:56.606 INFO UPLOAD_PROGRESS Uploaded chunk 59281 size 6419629, 44.81MB/s 1 day 08:07:21 5.0%
2017-04-21 20:34:57.038 INFO UPLOAD_PROGRESS Uploaded chunk 59282 size 6143516, 44.81MB/s 1 day 08:07:38 5.0%
2017-04-21 20:34:57.246 INFO UPLOAD_PROGRESS Uploaded chunk 59283 size 1369271, 44.81MB/s 1 day 08:07:37 5.0%
2017-04-21 20:34:58.029 INFO UPLOAD_PROGRESS Uploaded chunk 59284 size 6358333, 44.80MB/s 1 day 08:07:53 5.0%
2017-04-21 20:34:58.337 INFO UPLOAD_PROGRESS Uploaded chunk 59285 size 5033308, 44.80MB/s 1 day 08:07:51 5.0%
2017-04-21 20:34:58.826 INFO UPLOAD_PROGRESS Uploaded chunk 59286 size 15782120, 44.80MB/s 1 day 08:07:45 5.0%
2017-04-21 20:34:59.159 INFO UPLOAD_PROGRESS Uploaded chunk 59287 size 5772771, 44.80MB/s 1 day 08:08:01 5.0%
2017-04-21 20:34:59.798 INFO UPLOAD_PROGRESS Uploaded chunk 59288 size 3563753, 44.80MB/s 1 day 08:07:59 5.0%
2017-04-21 20:35:00.259 INFO UPLOAD_PROGRESS Uploaded chunk 59289 size 2363738, 44.79MB/s 1 day 08:08:17 5.0%
2017-04-21 20:35:00.677 INFO UPLOAD_PROGRESS Uploaded chunk 59290 size 7231768, 44.79MB/s 1 day 08:08:14 5.0%
2017-04-21 20:35:01.060 INFO UPLOAD_PROGRESS Uploaded chunk 59291 size 6932970, 44.79MB/s 1 day 08:08:30 5.0%
2017-04-21 20:35:01.310 INFO UPLOAD_PROGRESS Uploaded chunk 59292 size 1049935, 44.79MB/s 1 day 08:08:30 5.0%
2017-04-21 20:35:01.614 INFO UPLOAD_PROGRESS Uploaded chunk 59293 size 1836511, 44.79MB/s 1 day 08:08:29 5.0%
2017-04-21 20:35:01.901 INFO UPLOAD_PROGRESS Uploaded chunk 59294 size 4396311, 44.79MB/s 1 day 08:08:27 5.0%
2017-04-21 20:35:02.723 INFO UPLOAD_PROGRESS Uploaded chunk 59295 size 12971577, 44.78MB/s 1 day 08:08:41 5.0%
2017-04-21 20:35:03.010 INFO UPLOAD_PROGRESS Uploaded chunk 59296 size 3066663, 44.78MB/s 1 day 08:08:58 5.0%
2017-04-21 20:35:03.318 INFO UPLOAD_PROGRESS Uploaded chunk 59297 size 9049475, 44.78MB/s 1 day 08:08:54 5.0%
2017-04-21 20:35:03.616 INFO UPLOAD_PROGRESS Uploaded chunk 59298 size 2916772, 44.78MB/s 1 day 08:08:53 5.0%
2017-04-21 20:35:03.974 INFO UPLOAD_PROGRESS Uploaded chunk 59299 size 3427307, 44.78MB/s 1 day 08:08:52 5.0%
2017-04-21 20:35:04.292 INFO UPLOAD_PROGRESS Uploaded chunk 59300 size 5225129, 44.77MB/s 1 day 08:09:08 5.0%
2017-04-21 20:35:04.604 INFO UPLOAD_PROGRESS Uploaded chunk 59301 size 3302682, 44.77MB/s 1 day 08:09:07 5.0%
2017-04-21 20:35:04.909 INFO UPLOAD_PROGRESS Uploaded chunk 59302 size 3675885, 44.77MB/s 1 day 08:09:05 5.0%
2017-04-21 20:35:05.318 INFO UPLOAD_PROGRESS Uploaded chunk 59303 size 11566724, 44.77MB/s 1 day 08:09:19 5.0%
2017-04-21 20:35:05.590 INFO UPLOAD_PROGRESS Uploaded chunk 59304 size 1912119, 44.77MB/s 1 day 08:09:19 5.0%
2017-04-21 20:35:05.963 INFO UPLOAD_PROGRESS Uploaded chunk 59305 size 5041189, 44.77MB/s 1 day 08:09:16 5.0%
2017-04-21 20:35:06.375 INFO UPLOAD_PROGRESS Uploaded chunk 59306 size 5331078, 44.76MB/s 1 day 08:09:33 5.0%
2017-04-21 20:35:07.089 INFO UPLOAD_PROGRESS Uploaded chunk 59307 size 9312536, 44.76MB/s 1 day 08:09:48 5.0%
2017-04-21 20:35:07.380 INFO UPLOAD_PROGRESS Uploaded chunk 59308 size 3206006, 44.76MB/s 1 day 08:09:47 5.0%
2017-04-21 20:35:07.792 INFO UPLOAD_PROGRESS Uploaded chunk 59309 size 4739855, 44.76MB/s 1 day 08:09:45 5.0%
2017-04-21 20:35:08.201 INFO UPLOAD_PROGRESS Uploaded chunk 59310 size 5043663, 44.75MB/s 1 day 08:10:01 5.0%
2017-04-21 20:35:08.496 INFO UPLOAD_PROGRESS Uploaded chunk 59311 size 2802235, 44.75MB/s 1 day 08:10:00 5.0%
2017-04-21 20:35:08.840 INFO UPLOAD_PROGRESS Uploaded chunk 59312 size 1583232, 44.75MB/s 1 day 08:09:59 5.0%
2017-04-21 20:35:09.120 INFO UPLOAD_PROGRESS Uploaded chunk 59313 size 2011926, 44.74MB/s 1 day 08:10:17 5.0%
2017-04-21 20:35:09.390 INFO UPLOAD_PROGRESS Uploaded chunk 59314 size 2354926, 44.74MB/s 1 day 08:10:16 5.0%
2017-04-21 20:35:09.716 INFO UPLOAD_PROGRESS Uploaded chunk 59315 size 3672839, 44.74MB/s 1 day 08:10:15 5.0%
2017-04-21 20:35:09.996 INFO UPLOAD_PROGRESS Uploaded chunk 59316 size 1641258, 44.75MB/s 1 day 08:10:14 5.0%
2017-04-21 20:35:10.324 INFO UPLOAD_PROGRESS Uploaded chunk 59317 size 3038780, 44.74MB/s 1 day 08:10:32 5.0%
2017-04-21 20:35:10.689 INFO UPLOAD_PROGRESS Uploaded chunk 59318 size 6903283, 44.74MB/s 1 day 08:10:29 5.0%
2017-04-21 20:35:10.973 INFO UPLOAD_PROGRESS Uploaded chunk 59319 size 6273325, 44.74MB/s 1 day 08:10:26 5.0%
2017-04-21 20:35:11.233 INFO UPLOAD_PROGRESS Uploaded chunk 59320 size 1207956, 44.73MB/s 1 day 08:10:45 5.0%
2017-04-21 20:35:11.775 INFO UPLOAD_PROGRESS Uploaded chunk 59321 size 16777216, 44.74MB/s 1 day 08:10:38 5.0%
2017-04-21 20:35:12.082 INFO UPLOAD_PROGRESS Uploaded chunk 59322 size 3208151, 44.73MB/s 1 day 08:10:55 5.0%
2017-04-21 20:35:12.382 INFO UPLOAD_PROGRESS Uploaded chunk 59323 size 3986250, 44.73MB/s 1 day 08:10:53 5.0%
2017-04-21 20:35:13.278 INFO UPLOAD_PROGRESS Uploaded chunk 59324 size 8843295, 44.72MB/s 1 day 08:11:08 5.0%
2017-04-21 20:35:13.628 INFO UPLOAD_PROGRESS Uploaded chunk 59325 size 7289096, 44.73MB/s 1 day 08:11:05 5.0%
2017-04-21 20:35:14.041 INFO UPLOAD_PROGRESS Uploaded chunk 59326 size 8519794, 44.72MB/s 1 day 08:11:21 5.0%
2017-04-21 20:35:14.315 INFO UPLOAD_PROGRESS Uploaded chunk 59327 size 2537740, 44.72MB/s 1 day 08:11:20 5.0%
2017-04-21 20:35:14.546 INFO UPLOAD_PROGRESS Uploaded chunk 59328 size 1235747, 44.72MB/s 1 day 08:11:19 5.0%
2017-04-21 20:35:15.150 INFO UPLOAD_PROGRESS Uploaded chunk 59329 size 2736366, 44.71MB/s 1 day 08:11:37 5.0%
2017-04-21 20:35:15.372 INFO UPLOAD_PROGRESS Uploaded chunk 59330 size 1073257, 44.71MB/s 1 day 08:11:36 5.0%
2017-04-21 20:35:15.738 INFO UPLOAD_PROGRESS Uploaded chunk 59331 size 1568240, 44.71MB/s 1 day 08:11:36 5.0%
2017-04-21 20:35:16.686 INFO UPLOAD_PROGRESS Uploaded chunk 59332 size 5401919, 44.71MB/s 1 day 08:11:52 5.0%
2017-04-21 20:35:17.281 INFO UPLOAD_PROGRESS Uploaded chunk 59333 size 6793071, 44.70MB/s 1 day 08:12:08 5.0%
2017-04-21 20:35:17.569 INFO UPLOAD_PROGRESS Uploaded chunk 59334 size 2260606, 44.70MB/s 1 day 08:12:07 5.0%
2017-04-21 20:35:18.073 INFO UPLOAD_PROGRESS Uploaded chunk 59335 size 3757268, 44.69MB/s 1 day 08:12:25 5.0%
2017-04-21 20:35:18.356 INFO UPLOAD_PROGRESS Uploaded chunk 59336 size 5653447, 44.70MB/s 1 day 08:12:22 5.0%
2017-04-21 20:35:19.136 INFO UPLOAD_PROGRESS Uploaded chunk 59337 size 1259944, 44.69MB/s 1 day 08:12:40 5.0%
2017-04-21 20:35:19.549 INFO UPLOAD_PROGRESS Uploaded chunk 59338 size 4018508, 44.69MB/s 1 day 08:12:39 5.0%
2017-04-21 20:35:20.825 INFO UPLOAD_PROGRESS Uploaded chunk 59339 size 16749196, 44.68MB/s 1 day 08:12:51 5.0%
2017-04-21 20:35:21.640 INFO UPLOAD_PROGRESS Uploaded chunk 59340 size 1086159, 44.68MB/s 1 day 08:13:09 5.0%
2017-04-21 20:35:21.961 INFO UPLOAD_PROGRESS Uploaded chunk 59341 size 3060887, 44.68MB/s 1 day 08:13:08 5.0%
2017-04-21 20:35:22.222 INFO UPLOAD_PROGRESS Uploaded chunk 59342 size 1368246, 44.67MB/s 1 day 08:13:26 5.0%
2017-04-21 20:35:22.463 INFO UPLOAD_PROGRESS Uploaded chunk 59343 size 1477415, 44.67MB/s 1 day 08:13:25 5.0%
2017-04-21 20:35:22.827 INFO UPLOAD_PROGRESS Uploaded chunk 59344 size 7469508, 44.67MB/s 1 day 08:13:22 5.0%
2017-04-21 20:35:23.234 INFO UPLOAD_PROGRESS Uploaded chunk 59345 size 7834814, 44.67MB/s 1 day 08:13:38 5.0%
2017-04-21 20:35:23.527 INFO UPLOAD_PROGRESS Uploaded chunk 59346 size 2685010, 44.67MB/s 1 day 08:13:37 5.0%
2017-04-21 20:35:23.813 INFO UPLOAD_PROGRESS Uploaded chunk 59347 size 6620309, 44.67MB/s 1 day 08:13:34 5.0%
2017-04-21 20:35:24.133 INFO UPLOAD_PROGRESS Uploaded chunk 59348 size 1753335, 44.66MB/s 1 day 08:13:52 5.0%
2017-04-21 20:35:24.450 INFO UPLOAD_PROGRESS Uploaded chunk 59349 size 2623671, 44.66MB/s 1 day 08:13:51 5.0%
2017-04-21 20:35:24.807 INFO UPLOAD_PROGRESS Uploaded chunk 59350 size 8892905, 44.66MB/s 1 day 08:13:47 5.0%
2017-04-21 20:35:25.073 INFO UPLOAD_PROGRESS Uploaded chunk 59351 size 4300881, 44.66MB/s 1 day 08:14:04 5.0%
2017-04-21 20:35:25.479 INFO UPLOAD_PROGRESS Uploaded chunk 59352 size 5901166, 44.66MB/s 1 day 08:14:02 5.0%
2017-04-21 20:35:25.798 INFO UPLOAD_PROGRESS Uploaded chunk 59353 size 1736387, 44.66MB/s 1 day 08:14:01 5.0%
2017-04-21 20:35:26.220 INFO UPLOAD_PROGRESS Uploaded chunk 59354 size 7721799, 44.65MB/s 1 day 08:14:16 5.0%
2017-04-21 20:35:26.509 INFO UPLOAD_PROGRESS Uploaded chunk 59355 size 1296861, 44.65MB/s 1 day 08:14:16 5.0%
2017-04-21 20:35:26.811 INFO UPLOAD_PROGRESS Uploaded chunk 59356 size 4415019, 44.65MB/s 1 day 08:14:14 5.0%
2017-04-21 20:35:27.161 INFO UPLOAD_PROGRESS Uploaded chunk 59357 size 10119040, 44.65MB/s 1 day 08:14:29 5.0%
2017-04-21 20:35:27.626 INFO UPLOAD_PROGRESS Uploaded chunk 59358 size 1760170, 44.65MB/s 1 day 08:14:28 5.0%
2017-04-21 20:35:27.957 INFO UPLOAD_PROGRESS Uploaded chunk 59359 size 4949714, 44.65MB/s 1 day 08:14:26 5.0%
2017-04-21 20:35:28.532 INFO UPLOAD_PROGRESS Uploaded chunk 59360 size 4738202, 44.64MB/s 1 day 08:14:43 5.0%
2017-04-21 20:35:28.846 INFO UPLOAD_PROGRESS Uploaded chunk 59361 size 3597468, 44.64MB/s 1 day 08:14:41 5.0%
2017-04-21 20:35:29.105 INFO UPLOAD_PROGRESS Uploaded chunk 59362 size 3117711, 44.63MB/s 1 day 08:14:59 5.0%
2017-04-21 20:35:29.480 INFO UPLOAD_PROGRESS Uploaded chunk 59363 size 8546368, 44.64MB/s 1 day 08:14:55 5.0%
2017-04-21 20:35:29.739 INFO UPLOAD_PROGRESS Uploaded chunk 59364 size 2436279, 44.64MB/s 1 day 08:14:54 5.0%
2017-04-21 20:35:30.024 INFO UPLOAD_PROGRESS Uploaded chunk 59365 size 1334111, 44.63MB/s 1 day 08:15:12 5.0%
2017-04-21 20:35:30.333 INFO UPLOAD_PROGRESS Uploaded chunk 59366 size 3703255, 44.63MB/s 1 day 08:15:11 5.0%
2017-04-21 20:35:30.659 INFO UPLOAD_PROGRESS Uploaded chunk 59367 size 5931823, 44.63MB/s 1 day 08:15:08 5.0%
2017-04-21 20:35:30.918 INFO UPLOAD_PROGRESS Uploaded chunk 59368 size 1281719, 44.63MB/s 1 day 08:15:08 5.0%
2017-04-21 20:35:31.578 INFO UPLOAD_PROGRESS Uploaded chunk 59369 size 1732396, 44.62MB/s 1 day 08:15:26 5.0%
2017-04-21 20:35:31.956 INFO UPLOAD_PROGRESS Uploaded chunk 59370 size 3245720, 44.62MB/s 1 day 08:15:24 5.0%
2017-04-21 20:35:32.286 INFO UPLOAD_PROGRESS Uploaded chunk 59371 size 2508298, 44.62MB/s 1 day 08:15:42 5.0%
2017-04-21 20:35:32.603 INFO UPLOAD_PROGRESS Uploaded chunk 59372 size 2072819, 44.62MB/s 1 day 08:15:41 5.0%
2017-04-21 20:35:32.934 INFO UPLOAD_PROGRESS Uploaded chunk 59373 size 3100242, 44.62MB/s 1 day 08:15:40 5.0%
2017-04-21 20:35:33.388 INFO UPLOAD_PROGRESS Uploaded chunk 59374 size 5583249, 44.61MB/s 1 day 08:15:56 5.0%
2017-04-21 20:35:33.879 INFO UPLOAD_PROGRESS Uploaded chunk 59375 size 9445292, 44.61MB/s 1 day 08:15:52 5.0%
2017-04-21 20:35:34.591 INFO UPLOAD_PROGRESS Uploaded chunk 59376 size 1188284, 44.61MB/s 1 day 08:16:11 5.0%
2017-04-21 20:35:34.966 INFO UPLOAD_PROGRESS Uploaded chunk 59377 size 4411869, 44.61MB/s 1 day 08:16:09 5.0%
2017-04-21 20:35:35.304 INFO UPLOAD_PROGRESS Uploaded chunk 59378 size 4729668, 44.60MB/s 1 day 08:16:26 5.0%
2017-04-21 20:35:35.774 INFO UPLOAD_PROGRESS Uploaded chunk 59379 size 7320787, 44.60MB/s 1 day 08:16:23 5.0%
2017-04-21 20:35:36.413 INFO UPLOAD_PROGRESS Uploaded chunk 59380 size 11910505, 44.60MB/s 1 day 08:16:36 5.0%
2017-04-21 20:35:36.795 INFO UPLOAD_PROGRESS Uploaded chunk 59381 size 6839865, 44.60MB/s 1 day 08:16:34 5.0%
2017-04-21 20:35:37.154 INFO UPLOAD_PROGRESS Uploaded chunk 59382 size 8110329, 44.59MB/s 1 day 08:16:49 5.0%
2017-04-21 20:35:37.922 INFO UPLOAD_PROGRESS Uploaded chunk 59383 size 6891667, 44.59MB/s 1 day 08:16:46 5.0%
2017-04-21 20:35:38.398 INFO UPLOAD_PROGRESS Uploaded chunk 59384 size 15926337, 44.59MB/s 1 day 08:16:58 5.0%
2017-04-21 20:35:38.676 INFO UPLOAD_PROGRESS Uploaded chunk 59385 size 1428290, 44.59MB/s 1 day 08:16:58 5.0%
2017-04-21 20:35:38.928 INFO UPLOAD_PROGRESS Uploaded chunk 59386 size 1999383, 44.59MB/s 1 day 08:16:57 5.0%
2017-04-21 20:35:39.259 INFO UPLOAD_PROGRESS Uploaded chunk 59387 size 2224763, 44.58MB/s 1 day 08:17:15 5.0%
2017-04-21 20:35:39.649 INFO UPLOAD_PROGRESS Uploaded chunk 59388 size 4061201, 44.58MB/s 1 day 08:17:13 5.0%
2017-04-21 20:35:40.067 INFO UPLOAD_PROGRESS Uploaded chunk 59389 size 8453644, 44.58MB/s 1 day 08:17:28 5.0%
2017-04-21 20:35:40.772 INFO UPLOAD_PROGRESS Uploaded chunk 59390 size 4829961, 44.58MB/s 1 day 08:17:26 5.0%
2017-04-21 20:35:41.110 INFO UPLOAD_PROGRESS Uploaded chunk 59391 size 1559360, 44.57MB/s 1 day 08:17:44 5.0%
2017-04-21 20:35:41.416 INFO UPLOAD_PROGRESS Uploaded chunk 59392 size 4149901, 44.57MB/s 1 day 08:17:42 5.0%
2017-04-21 20:35:41.664 INFO UPLOAD_PROGRESS Uploaded chunk 59393 size 1251265, 44.57MB/s 1 day 08:17:42 5.0%
2017-04-21 20:35:41.953 INFO UPLOAD_PROGRESS Uploaded chunk 59394 size 4668736, 44.57MB/s 1 day 08:17:40 5.0%
2017-04-21 20:35:42.376 INFO UPLOAD_PROGRESS Uploaded chunk 59395 size 5861196, 44.56MB/s 1 day 08:17:56 5.0%
2017-04-21 20:35:42.625 INFO UPLOAD_PROGRESS Uploaded chunk 59396 size 1576950, 44.56MB/s 1 day 08:17:56 5.0%
2017-04-21 20:35:42.989 INFO UPLOAD_PROGRESS Uploaded chunk 59397 size 3488592, 44.57MB/s 1 day 08:17:54 5.0%
2017-04-21 20:35:43.514 INFO UPLOAD_PROGRESS Uploaded chunk 59398 size 2059012, 44.56MB/s 1 day 08:18:12 5.0%
2017-04-21 20:35:44.289 INFO UPLOAD_PROGRESS Uploaded chunk 59399 size 1404570, 44.55MB/s 1 day 08:18:30 5.0%
2017-04-21 20:35:47.029 INFO UPLOAD_PROGRESS Uploaded chunk 59400 size 4333447, 44.53MB/s 1 day 08:19:25 5.0%
2017-04-21 20:35:47.316 INFO UPLOAD_PROGRESS Uploaded chunk 59401 size 3206442, 44.53MB/s 1 day 08:19:24 5.0%
2017-04-21 20:35:49.975 INFO UPLOAD_PROGRESS Uploaded chunk 59402 size 8557405, 44.52MB/s 1 day 08:19:58 5.0%
2017-04-21 20:35:50.340 INFO UPLOAD_PROGRESS Uploaded chunk 59403 size 11486958, 44.51MB/s 1 day 08:20:12 5.0%
2017-04-21 20:36:11.347 ERROR UPLOAD_CHUNK Failed to upload the chunk
#10

That does not seem to be an option for the backup part of the CLI version. Also, it appears I would lose the ability to send e-mail notifications upon failure if I used the CLI version (and scheduling via a windows service, but I could run it via Task Scheduler).

For now I will limit my upload speed, but perhaps a api rate limit can be imposed for GCP or even better, an “exclusive” option for backups that will not perform the “file exists” api call when uploading which could also save some money on api calls (maybe not enough to matter).

edit: I will let you know if the rate limit helps.

#11

This is definitely caused by rate limiting on the server side.

Duplicacy uses goamz to access Google Cloud Storage. I think its retry mechanism can be improved to handle this rate limiting issue.

#12

Thanks for the confirmation.

I set a rate limit of 10000 KBps (should be 80 Mbps or roughly 10 MBps). I am still copying at speeds between 50 and 60 MBps. This isn’t the first time I’ve had troubles getting settings from the GUI to apply to the backup process. I also had troubles with frequency.

I checked the schedule file in the .duplicacy folder and it seems to have applied okay.

{
   "backup_scheduled" : true,
   "rate_limit" : "10000",
   "retires" : 3,
   "enable_vss" : true,
   "email_options" : {
      "subject" : "",
      "from" : "",
      "to" : "",
      "username" : "",
      "server" : "",
      "failure_only" : false
   },
   "backup_frequency" : 86400,
   "prune_enabled" : true,
   "backup_threads" : 1,
   "prune_parameters" : [
      0,
      120,
      7,
      92,
      1,
      91
   ],
   "email_enabled" : false,
   "backup_time" : 60677,
   "backup_days" : 127
}
#13

Please try the latest build at https://github.com/gilbertchen/duplicacy-cli/releases

The option for rate limiting is -limit-rate. However, this only limits the rate of transferring bytes over the network. Since many chunks already exist on the storage, they are actually not uploaded but they are counted in the upload speed being displayed. Therefore, those upload speeds are alway significantly higher than the rate limit.

#14

Still failing at 10000 KBps. Same error message.

I don’t know if this log info helps, but here it is.

2017-04-25 00:26:00.858 INFO UPLOAD_PROGRESS Uploaded chunk 114061 size 9710648, 40.24MB/s 1 day 10:15:32 9.0%
2017-04-25 00:26:02.016 INFO UPLOAD_PROGRESS Uploaded chunk 114062 size 3121269, 40.23MB/s 1 day 10:15:51 9.0%
2017-04-25 00:26:03.054 INFO UPLOAD_PROGRESS Uploaded chunk 114063 size 5961670, 40.23MB/s 1 day 10:16:00 9.0%
2017-04-25 00:26:04.843 INFO UPLOAD_PROGRESS Uploaded chunk 114064 size 16777216, 40.22MB/s 1 day 10:16:06 9.0%
2017-04-25 00:26:05.832 INFO UPLOAD_PROGRESS Uploaded chunk 114065 size 7156805, 40.22MB/s 1 day 10:16:14 9.0%
2017-04-25 00:26:06.224 INFO UPLOAD_PROGRESS Uploaded chunk 114066 size 2208678, 40.22MB/s 1 day 10:16:23 9.0%
2017-04-25 00:26:07.887 INFO UPLOAD_PROGRESS Uploaded chunk 114067 size 15252565, 40.22MB/s 1 day 10:16:29 9.0%
2017-04-25 00:26:08.589 INFO UPLOAD_PROGRESS Uploaded chunk 114068 size 5652306, 40.21MB/s 1 day 10:16:38 9.0%
2017-04-25 00:26:09.062 INFO UPLOAD_PROGRESS Uploaded chunk 114069 size 3153143, 40.21MB/s 1 day 10:16:47 9.0%
2017-04-25 00:26:09.347 INFO UPLOAD_PROGRESS Uploaded chunk 114070 size 1307461, 40.21MB/s 1 day 10:16:47 9.0%
2017-04-25 00:26:09.841 INFO UPLOAD_PROGRESS Uploaded chunk 114071 size 3102476, 40.21MB/s 1 day 10:16:46 9.0%
2017-04-25 00:26:10.377 INFO UPLOAD_PROGRESS Uploaded chunk 114072 size 3376395, 40.21MB/s 1 day 10:16:55 9.0%
2017-04-25 00:26:10.812 INFO UPLOAD_PROGRESS Uploaded chunk 114073 size 2695242, 40.21MB/s 1 day 10:16:54 9.0%
2017-04-25 00:26:11.241 INFO UPLOAD_PROGRESS Uploaded chunk 114074 size 2599460, 40.21MB/s 1 day 10:17:04 9.0%
2017-04-25 00:26:12.472 INFO UPLOAD_PROGRESS Uploaded chunk 114075 size 11243435, 40.20MB/s 1 day 10:17:11 9.0%
2017-04-25 00:26:12.954 INFO UPLOAD_PROGRESS Uploaded chunk 114076 size 3004988, 40.20MB/s 1 day 10:17:10 9.0%
2017-04-25 00:26:13.540 INFO UPLOAD_PROGRESS Uploaded chunk 114077 size 4538367, 40.20MB/s 1 day 10:17:19 9.0%
2017-04-25 00:26:14.511 INFO UPLOAD_PROGRESS Uploaded chunk 114078 size 6338832, 40.20MB/s 1 day 10:17:27 9.0%
2017-04-25 00:26:15.315 INFO UPLOAD_PROGRESS Uploaded chunk 114079 size 6562701, 40.19MB/s 1 day 10:17:35 9.0%
2017-04-25 00:26:16.067 INFO UPLOAD_PROGRESS Uploaded chunk 114080 size 3029580, 40.19MB/s 1 day 10:17:45 9.0%
2017-04-25 00:26:17.009 INFO UPLOAD_PROGRESS Uploaded chunk 114081 size 1926082, 40.19MB/s 1 day 10:17:54 9.0%
2017-04-25 00:26:17.504 INFO UPLOAD_PROGRESS Uploaded chunk 114082 size 2824735, 40.19MB/s 1 day 10:17:53 9.0%
2017-04-25 00:26:18.521 INFO UPLOAD_PROGRESS Uploaded chunk 114083 size 8780378, 40.19MB/s 1 day 10:18:01 9.0%
2017-04-25 00:26:18.913 INFO UPLOAD_PROGRESS Uploaded chunk 114084 size 1983981, 40.19MB/s 1 day 10:18:01 9.0%
2017-04-25 00:26:20.186 INFO UPLOAD_PROGRESS Uploaded chunk 114085 size 11246045, 40.18MB/s 1 day 10:18:18 9.0%
2017-04-25 00:26:20.737 INFO UPLOAD_PROGRESS Uploaded chunk 114086 size 1305542, 40.18MB/s 1 day 10:18:17 9.0%
2017-04-25 00:26:22.050 INFO UPLOAD_PROGRESS Uploaded chunk 114087 size 11646296, 40.18MB/s 1 day 10:18:35 9.0%
2017-04-25 00:26:22.919 INFO UPLOAD_PROGRESS Uploaded chunk 114088 size 7269006, 40.18MB/s 1 day 10:18:33 9.0%
2017-04-25 00:26:23.521 INFO UPLOAD_PROGRESS Uploaded chunk 114089 size 4780615, 40.17MB/s 1 day 10:18:41 9.0%
2017-04-25 00:26:23.821 INFO UPLOAD_PROGRESS Uploaded chunk 114090 size 1163609, 40.17MB/s 1 day 10:18:41 9.0%
2017-04-25 00:26:24.642 INFO UPLOAD_PROGRESS Uploaded chunk 114091 size 6456576, 40.17MB/s 1 day 10:18:49 9.0%
2017-04-25 00:26:26.208 INFO UPLOAD_PROGRESS Uploaded chunk 114092 size 14300276, 40.16MB/s 1 day 10:19:06 9.0%
2017-04-25 00:26:27.304 INFO UPLOAD_PROGRESS Uploaded chunk 114093 size 9713777, 40.16MB/s 1 day 10:19:13 9.0%
2017-04-25 00:26:27.575 INFO UPLOAD_PROGRESS Uploaded chunk 114094 size 1096103, 40.16MB/s 1 day 10:19:13 9.0%
2017-04-25 00:26:28.109 INFO UPLOAD_PROGRESS Uploaded chunk 114095 size 3860574, 40.16MB/s 1 day 10:19:22 9.0%
2017-04-25 00:26:28.885 INFO UPLOAD_PROGRESS Uploaded chunk 114096 size 2888449, 40.16MB/s 1 day 10:19:21 9.0%
2017-04-25 00:26:30.172 INFO UPLOAD_PROGRESS Uploaded chunk 114097 size 11611887, 40.15MB/s 1 day 10:19:38 9.0%
2017-04-25 00:26:31.570 INFO UPLOAD_PROGRESS Uploaded chunk 114098 size 13059774, 40.15MB/s 1 day 10:19:45 9.0%
2017-04-25 00:26:32.015 INFO UPLOAD_PROGRESS Uploaded chunk 114099 size 3023350, 40.15MB/s 1 day 10:19:54 9.0%
2017-04-25 00:26:32.737 INFO UPLOAD_PROGRESS Uploaded chunk 114100 size 5565408, 40.15MB/s 1 day 10:19:53 9.0%
2017-04-25 00:26:33.769 INFO UPLOAD_PROGRESS Uploaded chunk 114101 size 9220030, 40.15MB/s 1 day 10:20:00 9.0%
2017-04-25 00:26:34.786 INFO UPLOAD_PROGRESS Uploaded chunk 114102 size 3301334, 40.14MB/s 1 day 10:20:09 9.0%
2017-04-25 00:26:35.218 INFO UPLOAD_PROGRESS Uploaded chunk 114103 size 2431739, 40.14MB/s 1 day 10:20:19 9.0%
2017-04-25 00:26:35.840 INFO UPLOAD_PROGRESS Uploaded chunk 114104 size 4356261, 40.14MB/s 1 day 10:20:18 9.0%
2017-04-25 00:26:36.145 INFO UPLOAD_PROGRESS Uploaded chunk 114105 size 1054372, 40.14MB/s 1 day 10:20:27 9.0%
2017-04-25 00:26:37.415 INFO UPLOAD_PROGRESS Uploaded chunk 114106 size 11418260, 40.14MB/s 1 day 10:20:34 9.0%
2017-04-25 00:26:38.287 INFO UPLOAD_PROGRESS Uploaded chunk 114107 size 7084017, 40.13MB/s 1 day 10:20:43 9.0%
2017-04-25 00:26:39.102 INFO UPLOAD_PROGRESS Uploaded chunk 114108 size 6445595, 40.13MB/s 1 day 10:20:51 9.0%
2017-04-25 00:26:39.751 INFO UPLOAD_PROGRESS Uploaded chunk 114109 size 3899959, 40.13MB/s 1 day 10:20:50 9.0%
2017-04-25 00:26:40.103 INFO UPLOAD_PROGRESS Uploaded chunk 114110 size 1691228, 40.13MB/s 1 day 10:21:00 9.0%
2017-04-25 00:26:40.702 INFO UPLOAD_PROGRESS Uploaded chunk 114111 size 4889830, 40.13MB/s 1 day 10:20:58 9.0%
2017-04-25 00:26:41.137 INFO UPLOAD_PROGRESS Uploaded chunk 114112 size 2494939, 40.12MB/s 1 day 10:21:08 9.0%
2017-04-25 00:26:41.522 INFO UPLOAD_PROGRESS Uploaded chunk 114113 size 1930320, 40.12MB/s 1 day 10:21:07 9.0%
2017-04-25 00:26:42.158 INFO UPLOAD_PROGRESS Uploaded chunk 114114 size 5146836, 40.12MB/s 1 day 10:21:16 9.0%
2017-04-25 00:26:42.976 INFO UPLOAD_PROGRESS Uploaded chunk 114115 size 6789722, 40.12MB/s 1 day 10:21:14 9.0%
2017-04-25 00:26:44.156 INFO UPLOAD_PROGRESS Uploaded chunk 114116 size 10255870, 40.12MB/s 1 day 10:21:31 9.0%
2017-04-25 00:26:44.460 INFO UPLOAD_PROGRESS Uploaded chunk 114117 size 1125725, 40.12MB/s 1 day 10:21:31 9.0%
2017-04-25 00:26:44.867 INFO UPLOAD_PROGRESS Uploaded chunk 114118 size 1793175, 40.12MB/s 1 day 10:21:31 9.0%
2017-04-25 00:26:45.183 INFO UPLOAD_PROGRESS Uploaded chunk 114119 size 1626726, 40.11MB/s 1 day 10:21:40 9.0%
2017-04-25 00:26:47.060 INFO UPLOAD_PROGRESS Uploaded chunk 114120 size 16777216, 40.11MB/s 1 day 10:21:56 9.0%
2017-04-25 00:26:47.411 INFO UPLOAD_PROGRESS Uploaded chunk 114121 size 1831570, 40.11MB/s 1 day 10:21:55 9.0%
2017-04-25 00:26:47.966 INFO UPLOAD_PROGRESS Uploaded chunk 114122 size 3822479, 40.11MB/s 1 day 10:21:54 9.0%
2017-04-25 00:26:48.522 INFO UPLOAD_PROGRESS Uploaded chunk 114123 size 3972755, 40.11MB/s 1 day 10:22:03 9.0%
2017-04-25 00:26:50.112 INFO UPLOAD_PROGRESS Uploaded chunk 114124 size 14756914, 40.10MB/s 1 day 10:22:20 9.0%
2017-04-25 00:26:50.678 INFO UPLOAD_PROGRESS Uploaded chunk 114125 size 1055295, 40.10MB/s 1 day 10:22:19 9.0%
2017-04-25 00:26:51.946 INFO UPLOAD_PROGRESS Uploaded chunk 114126 size 11608608, 40.10MB/s 1 day 10:22:26 9.0%
2017-04-25 00:26:52.457 INFO UPLOAD_PROGRESS Uploaded chunk 114127 size 3623483, 40.10MB/s 1 day 10:22:35 9.0%
2017-04-25 00:26:53.869 INFO UPLOAD_PROGRESS Uploaded chunk 114128 size 12629679, 40.09MB/s 1 day 10:22:42 9.0%
2017-04-25 00:26:54.285 INFO UPLOAD_PROGRESS Uploaded chunk 114129 size 2081237, 40.09MB/s 1 day 10:22:52 9.0%
2017-04-25 00:26:54.873 INFO UPLOAD_PROGRESS Uploaded chunk 114130 size 3056207, 40.09MB/s 1 day 10:22:51 9.0%
2017-04-25 00:26:55.333 INFO UPLOAD_PROGRESS Uploaded chunk 114131 size 3219691, 40.09MB/s 1 day 10:23:00 9.0%
2017-04-25 00:26:55.677 INFO UPLOAD_PROGRESS Uploaded chunk 114132 size 1481414, 40.09MB/s 1 day 10:23:00 9.0%
2017-04-25 00:26:56.202 INFO UPLOAD_PROGRESS Uploaded chunk 114133 size 3662514, 40.08MB/s 1 day 10:23:09 9.0%
2017-04-25 00:26:56.606 INFO UPLOAD_PROGRESS Uploaded chunk 114134 size 2271229, 40.08MB/s 1 day 10:23:08 9.0%
2017-04-25 00:26:58.216 INFO UPLOAD_PROGRESS Uploaded chunk 114135 size 15007075, 40.08MB/s 1 day 10:23:24 9.0%
2017-04-25 00:26:58.825 INFO UPLOAD_PROGRESS Uploaded chunk 114136 size 4775203, 40.08MB/s 1 day 10:23:23 9.0%
2017-04-25 00:26:59.467 INFO UPLOAD_PROGRESS Uploaded chunk 114137 size 3594666, 40.08MB/s 1 day 10:23:32 9.0%
2017-04-25 00:26:59.808 INFO UPLOAD_PROGRESS Uploaded chunk 114138 size 1758991, 40.08MB/s 1 day 10:23:32 9.0%
2017-04-25 00:27:00.343 INFO UPLOAD_PROGRESS Uploaded chunk 114139 size 3764498, 40.07MB/s 1 day 10:23:41 9.0%
2017-04-25 00:27:00.952 INFO UPLOAD_PROGRESS Uploaded chunk 114140 size 1848469, 40.07MB/s 1 day 10:23:40 9.0%
2017-04-25 00:27:02.123 INFO UPLOAD_PROGRESS Uploaded chunk 114141 size 10159956, 40.07MB/s 1 day 10:23:57 9.0%
2017-04-25 00:27:02.520 INFO UPLOAD_PROGRESS Uploaded chunk 114142 size 2312260, 40.07MB/s 1 day 10:23:57 9.0%
2017-04-25 00:27:03.449 INFO UPLOAD_PROGRESS Uploaded chunk 114143 size 7909862, 40.07MB/s 1 day 10:24:05 9.0%
2017-04-25 00:27:03.888 INFO UPLOAD_PROGRESS Uploaded chunk 114144 size 2067322, 40.07MB/s 1 day 10:24:04 9.0%
2017-04-25 00:27:04.889 INFO UPLOAD_PROGRESS Uploaded chunk 114145 size 8675153, 40.06MB/s 1 day 10:24:12 9.0%
2017-04-25 00:27:05.320 INFO UPLOAD_PROGRESS Uploaded chunk 114146 size 2313180, 40.06MB/s 1 day 10:24:21 9.0%
2017-04-25 00:27:05.946 INFO UPLOAD_PROGRESS Uploaded chunk 114147 size 2802832, 40.06MB/s 1 day 10:24:21 9.0%
2017-04-25 00:27:06.781 INFO UPLOAD_PROGRESS Uploaded chunk 114148 size 6352424, 40.06MB/s 1 day 10:24:29 9.0%
2017-04-25 00:27:07.403 INFO UPLOAD_PROGRESS Uploaded chunk 114149 size 5126147, 40.06MB/s 1 day 10:24:38 9.0%
2017-04-25 00:27:07.722 INFO UPLOAD_PROGRESS Uploaded chunk 114150 size 1716056, 40.06MB/s 1 day 10:24:37 9.0%
2017-04-25 00:27:08.166 INFO UPLOAD_PROGRESS Uploaded chunk 114151 size 3062477, 40.05MB/s 1 day 10:24:47 9.0%
2017-04-25 00:27:08.598 INFO UPLOAD_PROGRESS Uploaded chunk 114152 size 1911408, 40.05MB/s 1 day 10:24:46 9.0%
2017-04-25 00:27:09.414 INFO UPLOAD_PROGRESS Uploaded chunk 114153 size 1204325, 40.05MB/s 1 day 10:24:56 9.0%
2017-04-25 00:27:10.097 INFO UPLOAD_PROGRESS Uploaded chunk 114154 size 5384198, 40.05MB/s 1 day 10:25:04 9.0%
2017-04-25 00:27:10.716 INFO UPLOAD_PROGRESS Uploaded chunk 114155 size 4332229, 40.05MB/s 1 day 10:25:03 9.0%
2017-04-25 00:27:12.046 INFO UPLOAD_PROGRESS Uploaded chunk 114156 size 12038008, 40.04MB/s 1 day 10:25:20 9.0%
2017-04-25 00:27:12.716 INFO UPLOAD_PROGRESS Uploaded chunk 114157 size 2685237, 40.04MB/s 1 day 10:25:19 9.0%
2017-04-25 00:27:13.043 INFO UPLOAD_PROGRESS Uploaded chunk 114158 size 1614499, 40.04MB/s 1 day 10:25:29 9.0%
2017-04-25 00:27:13.334 INFO UPLOAD_PROGRESS Uploaded chunk 114159 size 1060100, 40.04MB/s 1 day 10:25:29 9.0%
2017-04-25 00:27:13.829 INFO UPLOAD_PROGRESS Uploaded chunk 114160 size 3631357, 40.04MB/s 1 day 10:25:28 9.0%
2017-04-25 00:27:14.443 INFO UPLOAD_PROGRESS Uploaded chunk 114161 size 4318202, 40.04MB/s 1 day 10:25:37 9.0%
2017-04-25 00:27:15.273 INFO UPLOAD_PROGRESS Uploaded chunk 114162 size 7139175, 40.03MB/s 1 day 10:25:45 9.0%
2017-04-25 00:27:15.869 INFO UPLOAD_PROGRESS Uploaded chunk 114163 size 4229144, 40.03MB/s 1 day 10:25:44 9.0%
2017-04-25 00:27:16.536 INFO UPLOAD_PROGRESS Uploaded chunk 114164 size 5224605, 40.03MB/s 1 day 10:25:52 9.0%
2017-04-25 00:27:17.774 INFO UPLOAD_PROGRESS Uploaded chunk 114165 size 11492457, 40.03MB/s 1 day 10:25:59 9.0%
2017-04-25 00:27:18.834 INFO UPLOAD_PROGRESS Uploaded chunk 114166 size 9342994, 40.03MB/s 1 day 10:26:07 9.0%
2017-04-25 00:27:19.600 INFO UPLOAD_PROGRESS Uploaded chunk 114167 size 5620592, 40.02MB/s 1 day 10:26:15 9.0%
2017-04-25 00:27:19.990 INFO UPLOAD_PROGRESS Uploaded chunk 114168 size 2189957, 40.02MB/s 1 day 10:26:15 9.0%
2017-04-25 00:27:20.684 INFO UPLOAD_PROGRESS Uploaded chunk 114169 size 5035737, 40.02MB/s 1 day 10:26:24 9.0%
2017-04-25 00:27:21.156 INFO UPLOAD_PROGRESS Uploaded chunk 114170 size 2858383, 40.02MB/s 1 day 10:26:33 9.0%
2017-04-25 00:27:21.485 INFO UPLOAD_PROGRESS Uploaded chunk 114171 size 1398414, 40.02MB/s 1 day 10:26:33 9.0%
2017-04-25 00:27:21.763 INFO UPLOAD_PROGRESS Uploaded chunk 114172 size 1105100, 40.02MB/s 1 day 10:26:32 9.0%
2017-04-25 00:27:22.130 INFO UPLOAD_PROGRESS Uploaded chunk 114173 size 1792441, 40.01MB/s 1 day 10:26:42 9.0%
2017-04-25 00:27:23.051 INFO UPLOAD_PROGRESS Uploaded chunk 114174 size 7923683, 40.01MB/s 1 day 10:26:50 9.0%
2017-04-25 00:27:23.997 INFO UPLOAD_PROGRESS Uploaded chunk 114175 size 7082990, 40.01MB/s 1 day 10:26:48 9.0%
2017-04-25 00:27:24.580 INFO UPLOAD_PROGRESS Uploaded chunk 114176 size 4503195, 40.01MB/s 1 day 10:26:57 9.0%
2017-04-25 00:27:25.167 INFO UPLOAD_PROGRESS Uploaded chunk 114177 size 4598736, 40.01MB/s 1 day 10:27:06 9.0%
2017-04-25 00:27:25.760 INFO UPLOAD_PROGRESS Uploaded chunk 114178 size 4446811, 40.01MB/s 1 day 10:27:04 9.0%
2017-04-25 00:27:26.202 INFO UPLOAD_PROGRESS Uploaded chunk 114179 size 2834262, 40.00MB/s 1 day 10:27:14 9.0%
2017-04-25 00:27:26.805 INFO UPLOAD_PROGRESS Uploaded chunk 114180 size 4119379, 40.00MB/s 1 day 10:27:13 9.0%
2017-04-25 00:27:27.146 INFO UPLOAD_PROGRESS Uploaded chunk 114181 size 1128394, 40.00MB/s 1 day 10:27:22 9.0%
2017-04-25 00:27:27.721 INFO UPLOAD_PROGRESS Uploaded chunk 114182 size 4108835, 40.00MB/s 1 day 10:27:21 9.0%
2017-04-25 00:27:28.161 INFO UPLOAD_PROGRESS Uploaded chunk 114183 size 2383326, 40.00MB/s 1 day 10:27:31 9.0%
2017-04-25 00:27:28.828 INFO UPLOAD_PROGRESS Uploaded chunk 114184 size 2241554, 40.00MB/s 1 day 10:27:30 9.0%
2017-04-25 00:27:29.310 INFO UPLOAD_PROGRESS Uploaded chunk 114185 size 3381778, 40.00MB/s 1 day 10:27:39 9.0%
2017-04-25 00:27:30.932 INFO UPLOAD_PROGRESS Uploaded chunk 114186 size 15326831, 39.99MB/s 1 day 10:27:45 9.0%
2017-04-25 00:27:31.576 INFO UPLOAD_PROGRESS Uploaded chunk 114187 size 5102852, 39.99MB/s 1 day 10:27:54 9.0%
2017-04-25 00:27:32.496 INFO UPLOAD_PROGRESS Uploaded chunk 114188 size 7535392, 39.99MB/s 1 day 10:28:02 9.0%
2017-04-25 00:27:32.923 INFO UPLOAD_PROGRESS Uploaded chunk 114189 size 2096902, 39.99MB/s 1 day 10:28:01 9.0%
2017-04-25 00:27:33.540 INFO UPLOAD_PROGRESS Uploaded chunk 114190 size 2043588, 39.98MB/s 1 day 10:28:11 9.0%
2017-04-25 00:27:34.110 INFO UPLOAD_PROGRESS Uploaded chunk 114191 size 4607881, 39.98MB/s 1 day 10:28:20 9.0%
2017-04-25 00:27:34.789 INFO UPLOAD_PROGRESS Uploaded chunk 114192 size 5486218, 39.98MB/s 1 day 10:28:18 9.0%
2017-04-25 00:27:35.402 INFO UPLOAD_PROGRESS Uploaded chunk 114193 size 4411294, 39.98MB/s 1 day 10:28:27 9.0%
2017-04-25 00:27:35.752 INFO UPLOAD_PROGRESS Uploaded chunk 114194 size 1585039, 39.98MB/s 1 day 10:28:27 9.0%
2017-04-25 00:27:56.749 ERROR UPLOAD_CHUNK Failed to upload the chunk 7708989e2e517dd31b0baed6733c6797193c0a1865431ec5b300cdf0ea977001: Put https://storage.googleapis.com/[redacted]/[redacted]/chunks/7708989e2e517dd31b0baed6733c6797193c0a1865431ec5b300cdf0ea977001: dial tcp 173.194.219.128:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
#15

Are you running the latest 1.2.5 version? The time between the failure and the last successful upload was about 21 seconds (27:35 -> 27:56), but in the latest 1.2.5 version the retry mechanism has been changed to allow for 8 attempts and a maximum wait of 5 minutes.

#16

I was using the GUI version. I downloaded the CLI version you mentioned after the test with the GUI version where I throttled the upload speed. The CLI version doesn’t seem to produce the text log, but it also failed. I did at least use the -stats flag.

I didn’t see a way to turn on the log similar to the gui version in the documentation.

Terminal out put of
\Scripts\Duplicacy\duplicacy_win_x64_1.2.5.exe backup -vss -stats -limit-rate 12500

Skipped chunk 3329 size 11383365, 42.08MB/s 1 day 10:19:54 0.3%
Skipped chunk 3330 size 1210868, 42.08MB/s 1 day 10:19:45 0.3%
Skipped chunk 3331 size 6784692, 42.10MB/s 1 day 10:18:56 0.3%
Skipped chunk 3332 size 2596051, 42.10MB/s 1 day 10:18:37 0.3%
Skipped chunk 3333 size 1692088, 42.11MB/s 1 day 10:18:25 0.3%
Skipped chunk 3334 size 7616082, 42.13MB/s 1 day 10:17:30 0.3%
Skipped chunk 3335 size 1433062, 42.13MB/s 1 day 10:17:20 0.3%
Skipped chunk 3336 size 8380075, 42.15MB/s 1 day 10:16:20 0.3%
Skipped chunk 3337 size 3969406, 42.05MB/s 1 day 10:21:08 0.3%
Skipped chunk 3338 size 1237734, 42.05MB/s 1 day 10:20:59 0.3%
Skipped chunk 3339 size 13561563, 42.09MB/s 1 day 10:19:22 0.3%
Skipped chunk 3340 size 1069825, 42.09MB/s 1 day 10:19:14 0.3%
Skipped chunk 3341 size 1177936, 42.09MB/s 1 day 10:19:05 0.3%
Skipped chunk 3342 size 1902182, 42.10MB/s 1 day 10:18:52 0.3%
Failed to find the path for the chunk e5f69f43124ce626ca70dc8b5c6579bd39718a9cfb
c591bc80c9c0a076f411b8: Head https://storage.googleapis.com/[redacted]/[redacted]/chunks/e5f69f43124ce626ca70dc8b5c6579bd39718a9cfbc591bc80c9c0a076f411b8: dial
 tcp 74.125.21.128:443: connectex: A connection attempt failed because the conne
cted party did not properly respond after a period of time, or established conne
ction failed because connected host has failed to respond.
#17

This may be a wild shot in the dark, but another idea occurs to me.

Okay, so it looks like the app is trying to dial tcp 74.125.21.128:443 directly. Is the ip address resolved and stored somewhere for repeat usage throughout the backup process? If so, maybe it needs to be re-resolved. I just performed some rudimentary test where I ran nslookup and pinged storage.googleapis.com a few times in a few different ways. I noticed that each time I pinged it, I sometimes resolved a different address (see output below).

Perhaps google tries to load balance by dropping connections on one IP, forcing the clients to choose the next ip available in the list. This is just speculation though.


“A client program picking an IP address from a list (of A/AAAA-records) will generally try the addresses in the order they where returned by the DNS server (round robin). If the client cannot connect to the first IP address returned, it should try the second and so on. For example all major browsers do this, however many other Internet client programs “forget” this step and fail if they cannot connect to the first IP address.” 
>ping storage.googleapis.com

Pinging storage.l.googleusercontent.com [173.194.219.128] with 32 bytes of data:

Reply from 173.194.219.128: bytes=32 time=5ms TTL=45
Reply from 173.194.219.128: bytes=32 time=4ms TTL=45
Reply from 173.194.219.128: bytes=32 time=4ms TTL=45
Reply from 173.194.219.128: bytes=32 time=4ms TTL=45

Ping statistics for 173.194.219.128:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 4ms, Maximum = 5ms, Average = 4ms

-------------------------

>ping storage.googleapis.com

Pinging storage.l.googleusercontent.com [74.125.21.128] with 32 bytes of data:
Reply from 74.125.21.128: bytes=32 time=5ms TTL=45
Reply from 74.125.21.128: bytes=32 time=4ms TTL=45
Reply from 74.125.21.128: bytes=32 time=5ms TTL=45
Reply from 74.125.21.128: bytes=32 time=5ms TTL=45

Ping statistics for 74.125.21.128:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 4ms, Maximum = 5ms, Average = 4ms
#18

Can you run the CLI version with the -log option to see if the retry mechanism is working:

\Scripts\Duplicacy\duplicacy_win_x64_1.2.5.exe -log backup -vss -stats

The -limit-rate option is not needed because it doesn’t actually limit the API call rates.

It is possible that they do rate limiting by forcing you to switch to a different IP, but this should have been handled gracefully by the retry mechanism. I think the right solution is to use the official google client to access Google Cloud Storage, instead of relying on a third-party S3 client like goamz.

#19

Ah. Running it again with -log, though it is obviously going to fail again, we can at least look for the change you are expecting.

I agree that using the Google Cloud official API’s instead of S3 interoperability would be a great thing to have in there. I used Cloudberry with Google Cloud Storage which doesn’t use S3 interoperability with no problems and it completed the 5TB initial backup in less than 24 hours. I went to sleep and woke up and it was done. (that said, I would much rather use this). If we can’t get this working as is, I would be willing to stick around to see the completion of an official Google Storage API integration (depending on how long that would take).

#20

It doesn’t look like it is working at all now.

2017-04-25 19:59:07.346 INFO UPLOAD_PROGRESS Skipped chunk 10537 size 3900210, 3
9.17MB/s 1 day 12:37:34 0.9%
2017-04-25 19:59:07.402 INFO UPLOAD_PROGRESS Skipped chunk 10538 size 2112008, 3
9.17MB/s 1 day 12:37:29 0.9%
2017-04-25 19:59:07.491 INFO UPLOAD_PROGRESS Skipped chunk 10539 size 3530716, 3
9.18MB/s 1 day 12:37:21 0.9%
2017-04-25 19:59:07.657 INFO UPLOAD_PROGRESS Skipped chunk 10540 size 8385868, 3
9.18MB/s 1 day 12:37:00 0.9%
2017-04-25 19:59:07.728 INFO UPLOAD_PROGRESS Skipped chunk 10541 size 3612533, 3
9.18MB/s 1 day 12:36:51 0.9%
2017-04-25 19:59:07.950 INFO UPLOAD_PROGRESS Skipped chunk 10542 size 7704403, 3
9.19MB/s 1 day 12:36:32 0.9%
2017-04-25 19:59:08.023 INFO UPLOAD_PROGRESS Skipped chunk 10543 size 5135929, 3
9.16MB/s 1 day 12:37:59 0.9%
2017-04-25 19:59:08.201 INFO UPLOAD_PROGRESS Skipped chunk 10544 size 9810784, 3
9.17MB/s 1 day 12:37:35 0.9%
2017-04-25 19:59:08.277 INFO UPLOAD_PROGRESS Skipped chunk 10545 size 3463709, 3
9.17MB/s 1 day 12:37:27 0.9%
2017-04-25 19:59:08.566 INFO UPLOAD_PROGRESS Skipped chunk 10546 size 16777216,
39.19MB/s 1 day 12:36:46 0.9%
2017-04-25 19:59:08.643 INFO UPLOAD_PROGRESS Skipped chunk 10547 size 2249968, 3
9.19MB/s 1 day 12:36:40 0.9%
2017-04-25 19:59:08.751 INFO UPLOAD_PROGRESS Skipped chunk 10548 size 5949424, 3
9.19MB/s 1 day 12:36:26 0.9%
2017-04-25 19:59:08.859 INFO UPLOAD_PROGRESS Skipped chunk 10549 size 5380412, 3
9.20MB/s 1 day 12:36:13 0.9%
2017-04-25 19:59:29.899 ERROR UPLOAD_CHUNK Failed to find the path for the chunk
 14cc12c7da7ecd9745bc46d64e58d6fe7211b5a541a260ec65cf5582649a7eb7: Head https://
storage.googleapis.com/[redacted]/[redacted]/chunks/14cc12c7da7ecd9745bc46d64e5
8d6fe7211b5a541a260ec65cf5582649a7eb7: dial tcp 74.125.21.128:443: connectex: A
connection attempt failed because the connected party did not properly respond a
fter a period of time, or established connection failed because connected host h
as failed to respond.