Googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded, Maximum number of retries reached (backoff: 64, attempts: 15)

Support
#1

Hi,

I’ve been trying to perform an initial backup of about 4.5TB and 500k files to Google Drive for several days now, starting with 8 threads, then 6, and now even just 1, and every time I inevitably run into this at some point:

[2019-08-12 03:22:32 PDT] Uploaded chunk 38492 size 181015734, 20.86MB/s 23:19:18 63.4% 
[2019-08-12 03:31:54 PDT] [0] Maximum number of retries reached (backoff: 64, attempts: 15)
[2019-08-12 03:31:54 PDT] Failed to upload the chunk 9560062f248c9fac8743f024b64b19ea3591748b933a51c84bf7a1b92efb1685: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
[2019-08-12 03:32:05 PDT] Incomplete snapshot saved to preferences/incomplete

It’s not clear to me what exactly causes this as duplicacy doesn’t print the API calls, but I’d like to figure out mitigation steps so that I could let duplicacy back up at full speed with 4-8 threads without fear and have it properly back off and retry past these 15 tries and 64 seconds.

Is the number of steps and max backoff something I could tweak?

Alternatively, is there a notion of quotas that duplicacy could be told about so that it could attempt to adjust for them by slowing down?

What should we do?

Thank you.

Prune failure on Google team drives
#2

What you could try is use Google Drive File Stream for the initial upload.

You can also use a filters file to add one portion at a time of the whole backup set, so that you split the huge 4.5 tb into more manageable chunks.

Sidenote: i have had a lot of problems with gdrive rate limiting, and not everything that is rate-limited is shown in their docs.

What :d: should do is allow us to set the number of retries (instead of the default 15), but @gchen doesn’t seem to agree :frowning:

#3

I don’t think we should be required to change backup types or use filters in order to complete backups - there should be a better solution that takes account the touchiness of Drive rate-limiting.

I ran the 8 threads with debug and trace to see exactly how the backoff works, and what I found was that duplicacy doubles the backoff every retry but only until 64s, then retries up to 15 times and dies as soon as one of the upload threads gets to the 15th count. 64 is reached at attempt 6, and all subsequent attempts also use 64. I’d love to tweak both of these and bump them up.

[2019-08-12 08:33:15 PDT] 2019-08-12 08:33:15.895 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 0.75 seconds (backoff: 2, attempts: 1)
[2019-08-12 08:33:16 PDT] 2019-08-12 08:33:16.786 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 2.87 seconds (backoff: 4, attempts: 2)
[2019-08-12 08:33:19 PDT] 2019-08-12 08:33:19.786 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 9.98 seconds (backoff: 8, attempts: 3)
[2019-08-12 08:33:29 PDT] 2019-08-12 08:33:29.881 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 12.17 seconds (backoff: 16, attempts: 4)
[2019-08-12 08:33:42 PDT] 2019-08-12 08:33:42.154 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 31.67 seconds (backoff: 32, attempts: 5)
[2019-08-12 08:34:13 PDT] 2019-08-12 08:34:13.932 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 40.41 seconds (backoff: 64, attempts: 6)
[2019-08-12 08:34:54 PDT] 2019-08-12 08:34:54.448 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 17.20 seconds (backoff: 64, attempts: 7)
[2019-08-12 08:35:11 PDT] 2019-08-12 08:35:11.754 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 41.78 seconds (backoff: 64, attempts: 8)
[2019-08-12 08:35:53 PDT] 2019-08-12 08:35:53.647 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 15.30 seconds (backoff: 64, attempts: 9)
[2019-08-12 08:36:09 PDT] 2019-08-12 08:36:09.048 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 28.52 seconds (backoff: 64, attempts: 10)
[2019-08-12 08:36:37 PDT] 2019-08-12 08:36:37.684 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 73.47 seconds (backoff: 64, attempts: 11)
[2019-08-12 08:37:51 PDT] 2019-08-12 08:37:51.269 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 49.39 seconds (backoff: 64, attempts: 12)
[2019-08-12 08:38:40 PDT] 2019-08-12 08:38:40.935 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 9.18 seconds (backoff: 64, attempts: 13)
[2019-08-12 08:38:50 PDT] 2019-08-12 08:38:50.232 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 61.55 seconds (backoff: 64, attempts: 14)
[2019-08-12 08:39:51 PDT] 2019-08-12 08:39:51.902 DEBUG GCD_RETRY [5] User rate limit exceeded.; retrying after 29.41 seconds (backoff: 64, attempts: 15)
[2019-08-12 08:40:21 PDT] 2019-08-12 08:40:21.426 INFO GCD_RETRY [5] Maximum number of retries reached (backoff: 64, attempts: 15)

The “Error 403: User rate limit exceeded., userRateLimitExceeded” is specifically about the per-user quota, which is 1000 reqs per 100 seconds per user. I’m at a loss about why we’re running into the quota even with exponential backoff and just several threads (heck, I tried 8, 6, 4, 2, and even 1, and eventually ran into this during the backup).

The worst part is the per-user usage isn’t displayed in the Cloud Console for some reason:

image
image.png982×225 16.1 KB

@gchen Will you allow us to raise the limits to make duplicacy more robust please as with higher backoffs, eventually the API will allow calls again?

#4

After digging around further, it looks like duplicacy doesn’t use Google Cloud Platform Drive API credentials and instead integrates with Drive directly. It’s quite possible that the user quota I mentioned only applies to GCP and not Drive.

Is there a way to use the Cloud Platform Drive API instead? I think that’d resolve the problem.

#5

I’m backing up on Linux, I don’t think this would work, would it?

#6

Found Upload files to Google Drive - G Suite Admin Help, which seems to suggest 750GB/day is the limit.

Individual users can only upload 750 GB each day between My Drive and all shared drives. Users who reach the 750-GB limit or upload a file larger than 750 GB will be blocked from uploading additional files that day.

Does this mean this limit applies whether or not the Drive v3 API is used?

#7

Yes this is a well-known limit for Google Drive. Only way to bypass it is more user accounts, with a team drive. Or wait another day. :slight_smile:

#8

I’ve now arrived to the same conclusion after reading Sending data TO Gdrive: 403: Rate Limit Exceeded - rclone forum. How frustrating.

Regardless, being able to tweak exponential backoff max time and # of retries would make dealing with this process automatic. Hope @gchen agrees.

#9

The limits are defined here: duplicacy/duplicacy_gcdstorage.go at bb58f42a3717bd143af7c0c9b749dfe86c7a68af · gilbertchen/duplicacy · GitHub.

#10

I’ve just restarted the 60% complete 4.5TB backup for like the 10th time, and now it’s uploading a lot of chunks again rather than skipping them. I feel like at this rate, without the larger backoff and more retries, it’ll never actually complete.

#11

It will skip them but if you’re hitting the 750GB/day limit, you may need to give it some time to calm down, like a full 24 hours, before resuming. More backoff and retries won’t help if you’re hitting that limit.

#12

I think it will definitely help because the backup will continue rather than start over.

For example, I keep restarting the backup every day, and instead of skipping a lot of chunks, the longer the pause between backups, the more chunks are reuploaded instead of being skipped.

For example: the backup ran to 48% last night and stopped at 3am (previously I was able to get it to as high as 60%). At 8am, I restarted it, and already many initial chunks said uploaded instead of skipped:

[2019-08-13 08:22:40 PDT] Listing all chunks
[2019-08-13 08:29:52 PDT] Skipped 582731 files from previous incomplete backup
[2019-08-13 08:29:52 PDT] Use 4 uploading threads
[2019-08-13 08:29:56 PDT] Skipped chunk 2 size 60780518, 14.49MB/s 17:37:13 0.0%
[2019-08-13 08:29:59 PDT] Skipped chunk 3 size 166162161, 30.92MB/s 08:15:26 0.0%
[2019-08-13 08:30:00 PDT] Skipped chunk 4 size 33179333, 31.01MB/s 08:13:58 0.0%
[2019-08-13 08:30:00 PDT] Skipped chunk 5 size 37355359, 35.46MB/s 07:11:55 0.0%
[2019-08-13 08:30:02 PDT] Skipped chunk 7 size 17242555, 30.01MB/s 08:30:19 0.0%
[2019-08-13 08:30:03 PDT] Uploaded chunk 1 size 103348254, 36.25MB/s 07:02:32 0.0%
[2019-08-13 08:30:05 PDT] Uploaded chunk 6 size 33483828, 33.13MB/s 07:42:19 0.0%
[2019-08-13 08:30:06 PDT] Uploaded chunk 8 size 27896830, 32.66MB/s 07:48:53 0.0%
[2019-08-13 08:30:08 PDT] Skipped chunk 9 size 268435456, 44.58MB/s 05:43:26 0.0%
[2019-08-13 08:30:12 PDT] Skipped chunk 10 size 165201060, 43.54MB/s 05:51:34 0.0%
[2019-08-13 08:30:13 PDT] Skipped chunk 11 size 33117451, 42.97MB/s 05:56:13 0.0%
[2019-08-13 08:30:14 PDT] Skipped chunk 12 size 19390558, 41.86MB/s 06:05:41 0.1%
[2019-08-13 08:30:15 PDT] Skipped chunk 13 size 58737214, 42.47MB/s 06:00:21 0.1%
[2019-08-13 08:30:17 PDT] Skipped chunk 14 size 80045607, 42.13MB/s 06:03:16 0.1%
[2019-08-13 08:30:22 PDT] Skipped chunk 15 size 177101917, 40.74MB/s 06:15:36 0.1%
[2019-08-13 08:30:32 PDT] Uploaded chunk 17 size 63400839, 32.06MB/s 07:57:10 0.1%
[2019-08-13 08:30:32 PDT] Uploaded chunk 19 size 46820720, 33.18MB/s 07:41:06 0.1%
[2019-08-13 08:30:33 PDT] Uploaded chunk 16 size 99820662, 34.69MB/s 07:20:57 0.1%
[2019-08-13 08:30:33 PDT] Uploaded chunk 18 size 71985168, 35.50MB/s 07:10:52 0.1%
[2019-08-13 08:30:37 PDT] Uploaded chunk 20 size 45592751, 34.10MB/s 07:28:33 0.1%
[2019-08-13 08:30:40 PDT] Uploaded chunk 22 size 79612882, 33.55MB/s 07:35:51 0.1%
[2019-08-13 08:30:44 PDT] Uploaded chunk 24 size 45138352, 31.80MB/s 08:00:58 0.1%
[2019-08-13 08:30:44 PDT] Uploaded chunk 21 size 173008834, 34.97MB/s 07:17:15 0.1%
[2019-08-13 08:30:48 PDT] Uploaded chunk 26 size 27600928, 32.94MB/s 07:44:09 0.2%
[2019-08-13 08:30:49 PDT] Uploaded chunk 25 size 62839868, 33.42MB/s 07:37:33 0.2%
[2019-08-13 08:30:50 PDT] Uploaded chunk 27 size 76625982, 34.10MB/s 07:28:20 0.2%
[2019-08-13 08:30:54 PDT] Uploaded chunk 23 size 172804205, 34.56MB/s 07:22:19 0.2%
[2019-08-13 08:30:58 PDT] Uploaded chunk 31 size 36910783, 33.00MB/s 07:43:13 0.2%
[2019-08-13 08:31:06 PDT] Uploaded chunk 30 size 88912128, 30.58MB/s 08:19:51 0.2%
[2019-08-13 08:31:07 PDT] Uploaded chunk 29 size 244635407, 33.28MB/s 07:39:08 0.2%
[2019-08-13 08:31:09 PDT] Uploaded chunk 32 size 102043296, 33.68MB/s 07:33:39 0.2%
[2019-08-13 08:31:10 PDT] Uploaded chunk 33 size 36717785, 33.70MB/s 07:33:24 0.2%
[2019-08-13 08:31:12 PDT] Uploaded chunk 34 size 78548417, 33.79MB/s 07:32:06 0.2%
[2019-08-13 08:31:17 PDT] Uploaded chunk 36 size 53356755, 32.40MB/s 07:51:27 0.2%
[2019-08-13 08:31:23 PDT] Uploaded chunk 28 size 80264516, 31.11MB/s 08:11:03 0.3%
[2019-08-13 08:31:23 PDT] Uploaded chunk 35 size 223258293, 33.45MB/s 07:36:35 0.3%
[2019-08-13 08:31:28 PDT] Uploaded chunk 39 size 57763404, 32.28MB/s 07:53:05 0.3%
[2019-08-13 08:31:28 PDT] Uploaded chunk 38 size 85548438, 33.13MB/s 07:40:54 0.3%
[2019-08-13 08:31:29 PDT] Uploaded chunk 40 size 44283456, 33.22MB/s 07:39:35 0.3%
[2019-08-13 08:31:32 PDT] Skipped chunk 45 size 17961339, 32.40MB/s 07:51:17 0.3%
[2019-08-13 08:31:32 PDT] Skipped chunk 46 size 21815217, 32.60MB/s 07:48:16 0.3%
[2019-08-13 08:31:34 PDT] Uploaded chunk 43 size 29388633, 32.24MB/s 07:53:33 0.3%
[2019-08-13 08:31:35 PDT] Uploaded chunk 37 size 265250236, 34.38MB/s 07:23:54 0.3%
[2019-08-13 08:31:37 PDT] Uploaded chunk 44 size 22253349, 33.93MB/s 07:29:49 0.3%
[2019-08-13 08:31:40 PDT] Uploaded chunk 47 size 68783370, 33.59MB/s 07:34:16 0.3%
[2019-08-13 08:31:42 PDT] Uploaded chunk 42 size 204630656, 34.76MB/s 07:18:59 0.4%
[2019-08-13 08:31:49 PDT] Uploaded chunk 48 size 169936753, 34.06MB/s 07:27:51 0.4%
[2019-08-13 08:31:55 PDT] Uploaded chunk 50 size 101910932, 33.19MB/s 07:39:33 0.4%
[2019-08-13 08:31:55 PDT] Uploaded chunk 41 size 88773003, 33.88MB/s 07:30:11 0.4%
[2019-08-13 08:31:56 PDT] Uploaded chunk 51 size 103656669, 34.40MB/s 07:23:16 0.4%
[2019-08-13 08:32:01 PDT] Uploaded chunk 53 size 59969268, 33.51MB/s 07:35:01 0.4%
[2019-08-13 08:32:02 PDT] Uploaded chunk 49 size 268435456, 35.23MB/s 07:12:47 0.4%
[2019-08-13 08:32:04 PDT] Uploaded chunk 52 size 131609653, 35.64MB/s 07:07:40 0.5%
[2019-08-13 08:32:10 PDT] Uploaded chunk 55 size 88771294, 34.71MB/s 07:19:10 0.5%
[2019-08-13 08:32:12 PDT] Uploaded chunk 54 size 199911899, 35.57MB/s 07:08:23 0.5%
[2019-08-13 08:32:14 PDT] Uploaded chunk 58 size 40190493, 35.34MB/s 07:11:10 0.5%
[2019-08-13 08:32:15 PDT] Uploaded chunk 56 size 148288730, 36.08MB/s 07:02:14 0.5%
[2019-08-13 08:32:17 PDT] Uploaded chunk 59 size 74478379, 36.08MB/s 07:02:18 0.5%
[2019-08-13 08:32:20 PDT] Uploaded chunk 62 size 25833619, 35.51MB/s 07:09:00 0.5%
[2019-08-13 08:33:16 PDT] Uploaded chunk 57 size 58344064, 26.04MB/s 09:45:05 0.5%
[2019-08-13 08:33:24 PDT] Uploaded chunk 64 size 40695424, 25.24MB/s 10:03:36 0.5%
[2019-08-13 08:34:19 PDT] Uploaded chunk 63 size 268435456, 21.00MB/s 12:05:16 0.6%
[2019-08-13 08:34:23 PDT] Uploaded chunk 66 size 41078410, 20.83MB/s 12:11:00 0.6%
[2019-08-13 08:43:24 PDT] [1] Maximum number of retries reached (backoff: 64, attempts: 15)
[2019-08-13 08:43:24 PDT] Failed to upload the chunk 1580f575fc54c646969e1b3c215011a0ddf7ac57ce9c985e04f05e4cc0c96b2c: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
[2019-08-13 08:43:38 PDT] Incomplete snapshot saved to preferences/incomplete

If the backup was instead backing off, it would have continued instead of restarting and reuploading chunks over and over. It’d be much better for this backoff to automatically handle retrying than me doing it manually and losing progress.

#13

I’ve built duplicacy from scratch in a fork, going to test these changes now with our backup.

#14

But this won’t help you when you hit the 750GB limit - the backoff and retries are not meant to handle that. Nobody really know how Google implement the limit - whether they simply lock you out for an hour or so, then calculate a new rolling total over the last 24 hours. Who knows.

Why not set a -limit-rate to 8000 or 9000 or so. Problem solved. A week and then it’ll be complete.

Also, Duplicacy by and large doesn’t ‘reupload’ chunks that have already been uploaded. Those are new chunks, everything else is skipped.

#15

It’ll definitely help. The limits are published - 750GB / day, so the larger backoff will delay the uploads by enough time to proceed when there is more quota available.

It’s running now and I just ran into the quota at 94%. Now the upload threads paused for 2-4 hours and I’m waiting for them to resume and hopefully finally finish the backup by morning.

I prefer this solution - it’s more robust than artificially limiting the upload speed. I don’t understand the resistance to this approach, especially when we roughly know Google uses a rolling 750GB/24hr quota.

#16

I think I’ll modify my fork to the code to cap the max backoff at 2048 seconds and up the max retries way up. This way it’ll retry at most about every half an hour instead of exponentially backing off to too infrequent intervals.

Why can’t we just have these as parameters in the core?

Edit: Bump the max backoff way down, but up retries way up. · archon810/duplicacy@52ca17a · GitHub

#17

@archon810 you could also try this, which would be much easier to implement since after a backup is complete, there’s no need for api calls for skipping chunks

#18

OK, but again, why? It’s much simpler to leave the backup and let it complete at its own pace than babysit filters or bandwidth rates.

#19

I don’t understand the resistance to capping bandwidth to match the actual GCD bandwidth limits, but you do it your way. :slight_smile: Whichever way you cut it, you have to ‘artificially limit the upload speed’, because Google will artificially cut you off.

If you look around the internet at people that use Rclone with GCD and the 750GB limit, they mostly use the equivalent -bwlimit - although helpfully, you can also set a timetable for different speeds at different times. Now that might be a very useful feature, and relatively easy to implement.

I just don’t think modifying the existing back-off and retry parameters as being sensible for a short-term issue that might help in the initial upload phase, but could very well interfere with normal use.

1 Like
#20

Bump the max backoff way down, but up retries way up. · archon810/duplicacy@52ca17a · GitHub turned out to be the winner and I finally completed the backup without duplicacy giving up.

It’s the most robust and straightforward solution, and I am really hopeful @gchen incorporates the parameters into the core, despite the bwlimit and filter suggestions.

1 Like