Seeding duplicacy with B2 Fireball

I have about 42TB of data I need backed up, and I would like to back it up to B2. Given the volume of data involved here, I’d prefer to not do the initial backup over the internet, as that would both take awhile and tie up our network for that same time. Backblaze offers a service, called Fireball, where they will send you a Synology NAS and anything you put on there will be added to a B2 bucket. This seems like it would solve my problem, but how would duplicacy handle this? Could I just set it up to the fireball, let it do the initial backup, send it off to backblaze, and, once its in b2, then just remove/readd the storage to duplicacy, and it’ll find its data there and be good? Or is this not possible?

1 Like

Yes, it’s possible, exactly as you describe.

You can easily test it though. Create local folder (to emulate mounted fireball), configure it as target in duplicacy, and backup few files there.

Then using third party tools, like Cyberduck, upload that folder to B2 (to emulate FedEx driving that firewall to Backblaze)

Configure duplicacy with that cloud B2 folder now. Run check, run list, run restore, run backup to confirm that it works and resumes
as if nothing happened.

3 Likes

Hello, so that mostly worked. I backed up a small folder to a local folder, then deleted that storage, then copied the data to b2 (the structure in the bucket is just 3 folders, chunks and snapshots with a .duplicacy folder that is empty, plus a config file, if that’s correct), then initialized a b2 storage with the same name, and it detected as already initialized. I then did a check and that came back with an error about a slice index being out of range and the check failed. Funnily enough, the list and restore worked fine, and I can even backup fine, just the check isn’t working. Is there any way to fix that?

There should not be a .duplicacy folder on the target. It gets created on the source, not target.

I would clear local cache and retry. If it fails again — post the log up to the failure.

Ahh ok, I had just made an empty folder there assuming it may need it and my b2 client just didn’t upload hidden files. Anyway, cleared cache entirely and retried, same error

2022-08-28 14:06:43.249 INFO BACKBLAZE_URL download URL is: https://f001.backblazeb2.com
2022-08-28 14:06:43.428 INFO SNAPSHOT_CHECK Listing all chunks
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk .bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 00.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 01.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 07.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 0e.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 11.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 13.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 15.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 16.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 1a.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 1b.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 1c.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 1d.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 1e.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 20.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 21.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 22.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 24.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 29.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 2b.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 2c.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 2e.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 30.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 32.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 35.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 37.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 3e.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 3f.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 40.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 42.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 44.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 45.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 46.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 47.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 48.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 49.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 50.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 51.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 52.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 55.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 56.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 57.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 58.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 5a.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 5b.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 64.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 65.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 69.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 6c.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 6f.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 70.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 73.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 74.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 75.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 78.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 7a.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 7b.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 7e.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 7f.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 82.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 83.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 84.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 86.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 87.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 89.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 8b.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 8c.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 91.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 96.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 97.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 98.bzEmpty has a size of 0
2022-08-28 14:06:43.498 WARN SNAPSHOT_CHECK Chunk 9b.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk 9f.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk a0.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk a1.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk a4.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk a9.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk ab.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk b2.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk b5.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk b6.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk be.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk bf.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk c0.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk c1.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk c2.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk c3.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk c4.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk c6.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk ca.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk cd.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk cf.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk d0.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk d2.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk d6.bzEmpty has a size of 0
2022-08-28 14:06:43.499 WARN SNAPSHOT_CHECK Chunk d7.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk d9.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk db.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk dd.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk df.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk e2.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk e4.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk e6.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk e8.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk e9.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk ea.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk ed.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk f0.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk f1.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk f3.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk f6.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk fb.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk fc.bzEmpty has a size of 0
2022-08-28 14:06:43.500 WARN SNAPSHOT_CHECK Chunk fd.bzEmpty has a size of 0
runtime error: slice bounds out of range
goroutine 1 [running]:
runtime/debug.Stack(0x29, 0x0, 0x0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/gilbertchen/duplicacy/src.CatchLogException()
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:233 +0xee
panic(0xeddce0, 0x1af8930)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/gilbertchen/duplicacy/src.(*B2Storage).ListFiles(0xc00022d270, 0x0, 0xc000582dc0, 0x12, 0x0, 0x0, 0x13, 0x1, 0xc0004f3ca0, 0xc0001d08e0, ...)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_b2storage.go:87 +0xaaf
github.com/gilbertchen/duplicacy/src.(*SnapshotManager).ListSnapshotRevisions(0xc00013e240, 0xc000136430, 0x8, 0x5, 0xc00045ee38, 0xc0004f4960, 0x2, 0x4)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_snapshotmanager.go:538 +0x224
github.com/gilbertchen/duplicacy/src.(*SnapshotManager).CheckSnapshots(0xc00013e240, 0xc000136430, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x100, 0x1, ...)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_snapshotmanager.go:875 +0x891
main.checkSnapshots(0xc0003b2240)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:980 +0x808
github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli.Command.Run(0x101f011, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x103c455, 0x20, 0x0, ...)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli/command.go:160 +0x74d
github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli.(*App).Run(0xc00036c480, 0xc0000c2000, 0x7, 0x7, 0x0, 0x0)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli/app.go:179 +0x64f
main.main()
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:2201 +0x65b6

Do the .bzEmpty warnings matter? Those seem to be automatically created by b2 in every folder.

What happened there with a million of empty files? How did you upload data to B2? Can you download it back, and compare with the original?

So it turns out Cyberduck was creating those files for some reason, which was what I was using to upload. Using an rclone mount with rsync didn’t create them and then the check was fine! thanks for everything

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.