OneDrive benchmark: deleting 41 thousand temporary files

Support
,
#1

I tried to benchmark my Onedrive storage and I see something weird:

>duplicacy benchmark
Storage set to one://z-duplicacy-backups
Generating 244.14M byte random data in memory
Writing random data to local disk
Wrote 244.14M bytes in 1.96s: 124.26M/s
Reading the random data from local disk
Read 244.14M bytes in 0.80s: 305.48M/s
Split 244.14M bytes into 50 chunks without compression/encryption in 4.24s: 57.59M/s
Split 244.14M bytes into 50 chunks with compression but without encryption in 5.51s: 44.34M/s
Split 244.14M bytes into 50 chunks with compression and encryption in 12.75s: 19.15M/s
Deleting 41468 temporary files from previous benchmark runs

41 thousand “temporary files”? Is that correct in this scenario or because of some bug it’s been deleting something else too? This command was running for 15-20 minutes and then reported remote timeout

Delete https://api.onedrive.com/v1.0/drive/root:/z-duplicacy-backups/benchmark/8e/xxxxxxxxx: read tcp 192.168.1.130:62073->xx.xx.xx.xx:443:
wsarecv: An existing connection was forcibly closed by the remote host.; retry after 760 milliseconds

Remote folder /benchmark/ was empty long ago…

EDIT: It looks like other two machines backing up to the same Onedrive storage are getting connections closed now.

Benchmark command details
#3

The default number of chunks for testing is 64, unless you specify a different value with the -chunk-count option. I don’t see how it can become 41468 if you don’t specify this option.

#4

It’s finally finished (overnight).

...
Deleting 41468 temporary files from previous benchmark runs
Delete https://api.onedrive.com/v1.0/drive/root:/z-duplicacy-backups/benchmark/8e/xxxxxxxxxxx: read tcp 192.168.1.130:62073->xx.xx.xx.xx:443: wsarecv: An existing connection was forcibly closed by the remote host.; retry after 760 milliseconds
Generating 64 chunks
Uploaded 256.00M bytes in 1140.78s: 230K/s
Downloaded 256.00M bytes in 70.52s: 3.63M/s
Deleted 64 temporary files from the storage

So you may want to look at the code - what is the count associated with “temporary files from previous benchmark runs”

#5

It’s reproducible. I ran again, now it’s deleting (?) 8479 more files than previously. This is the storage where I’m actively adding files, so now I’m really concerned.

>duplicacy benchmark
Storage set to one://z-duplicacy-backups
Generating 244.14M byte random data in memory
Writing random data to local disk
Wrote 244.14M bytes in 1.80s: 135.31M/s
Reading the random data from local disk
Read 244.14M bytes in 0.23s: 1052.26M/s
Split 244.14M bytes into 50 chunks without compression/encryption in 4.10s: 59.48M/s
Split 244.14M bytes into 50 chunks with compression but without encryption in 5.46s: 44.75M/s
Split 244.14M bytes into 50 chunks with compression and encryption in 12.55s: 19.45M/s
Deleting 49947 temporary files from previous benchmark runs
...
#6

The benchmark command only attempts to delete files under the benchmark folder in the storage. Unless there is something wrong with the code that lists the chunks… I’ll check that…

#7

I found a bug in the chunk listing code in the OneDrive backend – it lists all the chunks under the chunks folder rather than the benchmark folder. This is why you got so many ‘temporary’ files. Fortunately when it attempts to delete these chunks it adds the benchmark prefix to the paths making the deletion unsuccessful.

The workaround for now is to run the benchmark command against an empty storage.

2 Likes
#8

Just to be clear: so there’s no risk of data loss because the paths for the delete calls are wrong.

1 Like