GoogleAPI limiting?

Support
#1

Hi all
I have been struggeling with my Duplicacy backup for some days now.

I was wondering if it could be to somekind of a limit?

My backup is running fine but the last “Check” is failing becurse of
2024-10-14 16:24:06.287 ERROR LIST_FILES Failed to list the directory chunks/: googleapi: Error 400: Invalid Value, invalid
Failed to list the directory chunks/: googleapi: Error 400: Invalid Value, invalid

I just saw that i have about 34000 chunks?

“2024-10-14 16:22:12.958 INFO BACKUP_STATS All chunks: 34115 total, 167,795M bytes; 0 new, 0 bytes, 0 bytes uploaded”

2024-10-14 16:20:46.942 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2024-10-14 16:20:59.911 INFO BACKUP_END Backup for /Pictures at revision 1048 completed
2024-10-14 16:20:59.911 INFO BACKUP_STATS Files: 37095 total, 800,682M bytes; 0 new, 0 bytes
2024-10-14 16:20:59.911 INFO BACKUP_STATS File chunks: 165811 total, 816,181M bytes; 0 new, 0 bytes, 0 bytes uploaded
2024-10-14 16:20:59.911 INFO BACKUP_STATS Metadata chunks: 19 total, 17,892K bytes; 0 new, 0 bytes, 0 bytes uploaded
2024-10-14 16:20:59.911 INFO BACKUP_STATS All chunks: 165830 total, 816,198M bytes; 0 new, 0 bytes, 0 bytes uploaded
2024-10-14 16:20:59.911 INFO BACKUP_STATS Total running time: 00:00:14

So can google meaby limiting the “Check” ?
I tryed this with no luck

Hope to get some help.
Thanks!

#2

Can you turn on debug logging by adding -d as a global option to the check job and post the log here?

1 Like
#3

Hi gchen.

Thanks for the reply :slight_smile:

Of course!

Running check command from /cache/localhost/all
Options: [-log -d check -storage dbhansdude -a -tabular]
2024-10-14 16:45:42.155 INFO STORAGE_SET Storage set to gcd://Duplicacy
2024-10-14 16:45:42.155 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_DBHANSDUDE_GCD_TOKEN
2024-10-14 16:45:43.483 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_DBHANSDUDE_PASSWORD
2024-10-14 16:45:45.798 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2024-10-14 16:45:46.169 DEBUG STORAGE_NESTING Chunk read levels: [1], write level: 1
2024-10-14 16:45:46.181 INFO CONFIG_INFO Compression level: 100
2024-10-14 16:45:46.181 INFO CONFIG_INFO Average chunk size: 4194304
2024-10-14 16:45:46.181 INFO CONFIG_INFO Maximum chunk size: 16777216
2024-10-14 16:45:46.181 INFO CONFIG_INFO Minimum chunk size: 1048576
2024-10-14 16:45:46.181 INFO CONFIG_INFO Chunk seed: 1f107d2a01fb662002f377d518905cd6f56e3299f5d689e140c28108330957ec
2024-10-14 16:45:46.181 TRACE CONFIG_INFO Hash key: e381a789dada0990c5b2d0c071ca0a02d2b6a98b50f1fba23a3912d49e79fea2
2024-10-14 16:45:46.181 TRACE CONFIG_INFO ID key: 406f3f7adb956fbc39f1c9f26b5249ef837a7fe07f4d323476cb19eb6c949cd7
2024-10-14 16:45:46.181 TRACE CONFIG_INFO File chunks are encrypted
2024-10-14 16:45:46.181 TRACE CONFIG_INFO Metadata chunks are encrypted
2024-10-14 16:45:46.181 DEBUG LIST_PARAMETERS id: , revisions: , tag: , showStatistics: false, showTabular: true, checkFiles: false, searchFossils: false, resurrect: false
2024-10-14 16:45:46.181 INFO SNAPSHOT_CHECK Listing all chunks
2024-10-14 16:45:46.181 TRACE LIST_FILES Listing chunks/
2024-10-14 16:45:46.733 DEBUG GCD_STORAGE Listing chunks; 256 items returned
2024-10-14 16:45:47.470 DEBUG GCD_STORAGE Listing fossils; 256 items returned
2024-10-14 16:45:50.075 DEBUG GCD_STORAGE Listing chunks/6d; 1091 items returned
2024-10-14 16:45:52.809 DEBUG GCD_STORAGE Listing chunks/73; 1102 items returned
2024-10-14 16:45:55.050 DEBUG GCD_STORAGE Listing chunks/64; 1119 items returned
2024-10-14 16:45:57.270 DEBUG GCD_STORAGE Listing chunks/e9; 1087 items returned
2024-10-14 16:45:59.532 DEBUG GCD_STORAGE Listing chunks/20; 1081 items returned
2024-10-14 16:46:01.732 DEBUG GCD_STORAGE Listing chunks/40; 1016 items returned
2024-10-14 16:46:03.983 DEBUG GCD_STORAGE Listing chunks/4e; 1087 items returned
2024-10-14 16:46:06.469 DEBUG GCD_STORAGE Listing chunks/aa; 1088 items returned
2024-10-14 16:46:08.700 DEBUG GCD_STORAGE Listing chunks/30; 1052 items returned
2024-10-14 16:46:11.125 DEBUG GCD_STORAGE Listing chunks/a3; 1113 items returned
2024-10-14 16:46:13.260 DEBUG GCD_STORAGE Listing chunks/56; 1096 items returned
2024-10-14 16:46:15.280 DEBUG GCD_STORAGE Listing chunks/fe; 1141 items returned
2024-10-14 16:46:17.400 DEBUG GCD_STORAGE Listing chunks/48; 1092 items returned
2024-10-14 16:46:19.494 DEBUG GCD_STORAGE Listing chunks/85; 1088 items returned
2024-10-14 16:46:21.783 DEBUG GCD_STORAGE Listing chunks/ea; 1141 items returned
2024-10-14 16:46:24.132 DEBUG GCD_STORAGE Listing chunks/e1; 1110 items returned
2024-10-14 16:46:26.470 DEBUG GCD_STORAGE Listing chunks/62; 1119 items returned
2024-10-14 16:46:28.687 DEBUG GCD_STORAGE Listing chunks/36; 1069 items returned
2024-10-14 16:46:30.943 DEBUG GCD_STORAGE Listing chunks/23; 1140 items returned
2024-10-14 16:46:33.294 DEBUG GCD_STORAGE Listing chunks/fb; 1162 items returned
2024-10-14 16:46:36.022 DEBUG GCD_STORAGE Listing chunks/57; 1034 items returned
2024-10-14 16:46:38.138 DEBUG GCD_STORAGE Listing chunks/63; 1068 items returned
2024-10-14 16:46:40.338 DEBUG GCD_STORAGE Listing chunks/f8; 1105 items returned
2024-10-14 16:46:42.678 DEBUG GCD_STORAGE Listing chunks/19; 1029 items returned
2024-10-14 16:46:45.109 DEBUG GCD_STORAGE Listing chunks/71; 1088 items returned
2024-10-14 16:46:47.266 DEBUG GCD_STORAGE Listing chunks/f4; 1051 items returned
2024-10-14 16:46:49.093 DEBUG GCD_STORAGE Listing chunks/09; 1051 items returned
2024-10-14 16:46:51.561 DEBUG GCD_STORAGE Listing chunks/42; 1091 items returned
2024-10-14 16:46:54.201 DEBUG GCD_STORAGE Listing chunks/81; 1144 items returned
2024-10-14 16:46:56.449 DEBUG GCD_STORAGE Listing chunks/74; 1054 items returned
2024-10-14 16:46:59.054 DEBUG GCD_STORAGE Listing chunks/47; 1128 items returned
2024-10-14 16:47:00.990 DEBUG GCD_STORAGE Listing chunks/ae; 1130 items returned
2024-10-14 16:47:03.503 DEBUG GCD_STORAGE Listing chunks/e2; 1076 items returned
2024-10-14 16:47:05.607 DEBUG GCD_STORAGE Listing chunks/d8; 1068 items returned
2024-10-14 16:47:07.984 DEBUG GCD_STORAGE Listing chunks/0e; 1089 items returned
2024-10-14 16:47:10.353 DEBUG GCD_STORAGE Listing chunks/02; 1090 items returned
2024-10-14 16:47:12.610 DEBUG GCD_STORAGE Listing chunks/ce; 1097 items returned
2024-10-14 16:47:15.294 DEBUG GCD_STORAGE Listing chunks/b0; 1126 items returned
2024-10-14 16:47:17.572 DEBUG GCD_STORAGE Listing chunks/1a; 1089 items returned
2024-10-14 16:47:19.943 DEBUG GCD_STORAGE Listing chunks/e7; 1126 items returned
2024-10-14 16:47:21.985 ERROR LIST_FILES Failed to list the directory chunks/: googleapi: Error 400: Invalid Value, invalid
Failed to list the directory chunks/: googleapi: Error 400: Invalid Value, invalid

#4

Can you add -threads 4 or -threads 8 to the check job and see what happens? Let’s hope that the listing can run faster and complete before the rate limiting kicks in.

1 Like
Suddenly im getting Failed to list the directory chunks/: googleapi: Error 400: Invalid Value, invalid
#5

Hi @gchen
Yeap i just tryed with

-d -threads 8-
But got
Check storage dbhansdude, status: Invalid options, log file:

Is this correct?

-threads 8 -d | -

#6

No, -d is a global option. -threads is a command option.

1 Like
#7

You are a life saver!
Thanks!
It is working now!
Wuhu!

image
image3325×293 31.3 KB