Not being able to backup with STORJ

Support
#1

Hello everyone,
A few months ago I started using duplicady and getting info about how ti works. After getting a lot of help in the forum I managed to init the storage using the STORJ native client (with 15 threads about 23mb of upload and 70 of download). However, I’ve been out of home for a few months due to work.

I’m using an average chunk size of 32 (which should be the best for STORJ) and -vss since I’m on Windows.

However, after starting the backup I don’t see much progress. I used -stat and it is showing this:

Skipped chunk 13053 size 9, 1KB/s 4445 days 04:58:13 0.0%
Uploaded chunk 13043 size 30, 1KB/s 4445 days 04:58:13 0.0%
Skipped chunk 13055 size 12, 1KB/s 4445 days 04:58:13 0.0%
Uploaded chunk 13039 size 53, 1KB/s 4445 days 04:58:13 0.0%
Uploaded chunk 13040 size 73, 1KB/s 4445 days 04:58:13 0.0%
Uploaded chunk 13041 size 19, 1KB/s 4445 days 04:58:13 0.0%
Skipped chunk 13059 size 16, 1KB/s 4445 days 04:58:13 0.0%
Skipped chunk 13060 size 19, 1KB/s 4445 days 04:58:13 0.0%
Uploaded chunk 13048 size 20, 1KB/s 4445 days 04:58:13 0.0%
Uploaded chunk 13045 size 59, 1KB/s 4441 days 12:08:25 0.0%

I took just a small fragment since it is the same all the time. However, the speed and remaining time were better when it started. Any idea of why? It is normal that it skips so many chunks?

Thanks in advance!

#2

A chunk size of 32 is too small. Did STORJ actually suggest 32 or did they mean 32M? Anyway just use the default one and it should be fine.

2 Likes
#3

The reasoning here is that Storj segment size is 64M, so the closer you get to that, the less segments will you have.

With that, related questions:

From the documentation:

  -chunk-size, -c <size>          the average size of chunks (default is 4M)
  -max-chunk-size, -max <size>    the maximum size of chunks (default is chunk-size*4)
  -min-chunk-size, -min <size>    the minimum size of chunks (default is chunk-size/4)

  1. What happens if I specify only the max size=64M? Will the average chunk size become 16?

  2. Will specifying average chunk size=32M and max=64M result in the expected shift towards larger chunks while keeping flexibility of slicing at good boundaries?

1 Like
#4

The average chunk size will remain unchanged at 16M.

That is correct.

1 Like
#5

Completely my bad, assumed that the unit was Mb.

Therefore I assume that I should re init the storage and specify -c 32000000 and -max 64000000 to optimize it for STORJ.

Thank to both of you from answering!
@saspus nice to see you again! Thanks to you I managed to understand this a bit!

Edit: It’s been a while since I looked this up, and I can’t find it now. Which was the difference between the passphrase and the password asked when -init the storage? I assume that the passphrase is the one setup on the STORJ website to access the bucket, and the password the encryption key that I have to set up? From what I’m reading I don’t find anything conclusive.

#6

It’s worse – it’s actually MiB, but command line expects M

I would specify 32M and 64M respectively. Storj wants 64MiB, 64000000 is 64MB, but if you tell duplicacy M it will understand it as MiB…( and in the UI storj will show files in MB).

Good to see you too!

1 Like
#7

Did that and everything seems working properly now:

Uploaded chunk 236 size 36581474, 22.37MB/s 05:21:29 1.8%
Uploaded chunk 239 size 39058362, 22.17MB/s 05:24:27 1.8%
Uploaded chunk 240 size 40654167, 22.21MB/s 05:23:45 1.8%
Uploaded chunk 237 size 58782670, 22.31MB/s 05:22:22 1.8%
Uploaded chunk 245 size 10659656, 22.27MB/s 05:22:50 1.8%
Uploaded chunk 241 size 24227695, 22.34MB/s 05:21:54 1.8%
Uploaded chunk 238 size 57851418, 22.37MB/s 05:21:27 1.8%

Will check tomorrow but it can probably be ready overnight. I’ll try to recover a folder to check that I did not mess anything up.
Before this I was wondering why the huge number of segments of STORJ after the benchmarks, now it is completely obvious.

Got caught with the small details, luckily the forum is one of the best I’ve encountered.
Thanks for answering so quick to both of you!

1 Like
#8

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.