Any tips to boost google drive performance as much as possible?

[datao]

Hello Duplicacy community, I have searched for tips regarding boosting performance to backup/restore towards google drive & not found much, so I’m making this thread to hopefully get some tips from other seasoned users that have managed it.

I use duplicacy web version on a Windows NAS to make local & cloud encrypted backup of my local NAS data. My choice of cloud is google workspace, because I happen to have a lot of space available there. Even if it’s often argued that it’s not the best cloud destination, I would like to make the best of it if possible.

In other separate usage (data that does not need versioning/deduplication and so on) I use rclone to push data to/from this google workspace and I saturate my gigabit connection with transfers easily, so I know that this is not a bandwidth problem towards Google. I have played with the ‘threads’ option and I manage to get it to transfer usually max 15MB/s (while rclone does 100+MB/s without a problem)

I thought this might have something to do with my chunk size? Duplicacy’s default is 4MiB as far as I know & sending loads of small files is slower than sending bigger files. Would it be better if I boosted this to a higher number? Would it make a difference?

Does it make any difference if the backup is encrypted or not?

Other than that I only know of -threads that I can use to influence the throughput. If anyone has some ideas to share with me I’d be happy to hear them.

Another thing I wanted to ask about is ‘copy-compatible’ - is there any disadvantage whatsoever to have my cloud backup ‘copy-compatible’ ? I think once I’ve found the ideal settings for my cloud backup, I figure I’ll just tick ‘copy-compatible’ & use the exact same settings for the local backup. Then it should always work interchangably to copy it to other local & cloud destinations right? I don’t see much benefit to using different encryption keys etc. for the different backup locations.

[datao]

I found a topic here, where a user changed the chunk size apparently to 14MB and then achieved 100MB/s speeds.

Adjust chunk size / investigate performance via Web UI

Yes; one option is deleting the contents of the old storage and initializing the storage again, but with your desired chunk size. Another is leaving your old storage as is for historical reasons and initializing the new storage with the desired chunk size in a new bucket or directory.

I guess I have some testing to do.

[saspus]

@datao

Would it be better if I boosted this to a higher number? Would it make a difference?

This strongly depends on type of data you are backing up. For static data this does not matter - you upload it once and that’s it. For static data that gets added in large chunks (e.g. media) huge chunk size will help with practically no downsides. For data that experiences small distributed changes (like text documents, code bases, other databases) there is a point where gains from throughput due to large chunks are negated by the massive amount of extra data that needs to be uploaded and stored because of said too large of the chunks size. Whether that matters of course also depends on how much of that type of data you have and how much of it changes. If too much — this can matter even if storage itself is unlimited, like Google drive.

From my limited personal experience backing up my entire iCloud folder (about 1TB) to Storj with average 32MB chunk size I did not notice any excessive space usage over last year. I don’t have 4MB chunk size based history of the same data to compare, but the size on the storage looks reasonable.

@datao

14MB

I’m pretty sure it must be power of two.

[datao]

Yes, it’s possible that user was talking about number of threads and not chunk size, not sure.

Anyway, so that’s worth a shot for me to try.

[datao]

I have been experimenting & taught myself to use the CLI & found the ‘benchmark’ feature in the CLI which was very helpful, so I have been testing towards Google Drive with many different settings, tinkering with chunk size & threads, here are some preliminary finding:

The benchmark uses 256.00M files to split into chunks as standard. I’ve monitored my NAS & no resources seem to have been capped.

In the benchmarks I consistently for some reason get poorer upload speeds than download speeds. Why this is, I don’t know. I seem to hit a cap of 4-5 threads for upload, where if I increase it more it won’t provide more speed, even less. Chunk-size has little to no influence in this with standard value (4M) or higher.

For upload, the best upload values I’ve hit are around 10-15MB/s, sometimes 20MB/s but it’s inconsistent.

For download, a threads value of 12-16 seems to be good, where higher values provide diminishing returns & not much added throughput. 35-38MB/s is about as good as it gets (pretty good and acceptable to me.)

chunk-size:
I know setting this depends on your own library file sizes, so it can be useful to first analyze what kind of filesizes you’re dealing with. I’m not going to use the fixed chunk size, as I have many smaller files.

1M. (Just to test) - provided 10-12MB/s download

4M which is the standard value provides acceptable throughput of 20MB/s in download

8M - 25MB/s

16M would let me hit the 35-38MB/s download

Higher chunk-sizes gets slightly better, but diminishing returns.

I have decided to land on the chunk-size 32M - then the smallest chunks will be 8M and the largest will be 128M (funnily enough, I see 128M is the chunk value used in my rclone setup too)

In summary:
YMMY, this is just what I concluded from my benchmarking:

For upload - threads 4-5, more doesn’t help.
For download - threads 12-16, dimishing returns from more.
chunk-size 32M (min. 8M, max 128M)