Unable to use swift backend with Blomp

Support
#1

Hello,

I’m using Blomp storage and they use OpenStack Swift on the backend. I was able to get it working in rclone for my HomeAssistant instance, but trying to link it into duplicacy throws an error:

2024/11/30 12:03:19 ERROR STORAGE_CREATE Failed to load the OpenStack Swift storage at swift://blompuser@email.com@authenticate.blomp.com/v2.0/blompuser@email.com?auth_version=2&tenant=storage&endpoint_type=public: Post "https://email.com/tokens": remote error: tls: unrecognized name

Blomp uses your email address as your username and container name. The issue appears to be with the username being the email address. I’m not sure what rclone does different (it appears to use ncw/swift on the backend as well). I’m not versed in ‘go’ enough, so having a bit of trouble with understanding the code.

FWIW, if I click the folder icon next to directories I do get a list of files/folders in my blomp storage, so it can authenticate, it seems to be an issue getting a token

Any help would be appreciated. I can open a github issue if needed, but thought I’d check here first.

#2

I’ve also responded to a thread on the Blomp forum with someone having issues as well.

https://support.blomp.com/viewtopic.php?p=4899&sid=9a3e605f20c64b0ac6fb4b5be9e7e4cd#p4899

@gchen, is there any additional info I can gather that may be helpful?

#3

BUMP

Found -stack global option. Adding the details from that here. Hoping to find a solution soon. I’m happy to compile and test myself, I just don’t have any experience with GO lang so I am unable to assist with fixing and submitting a pull request.

/ # /config/bin/duplicacy_linux_x64_3.2.3 -stack -log -d info -repository /cache/localhost/all swift://user@email.net@authenticate.blomp
.com/v2.0/user@email.net?auth_version=2&tenant=storage&endpoint_type=public
/ # 
[3]-  Done                       `v
/ # 2024-12-05 09:41:35.947 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_SWIFT_KEY
2024-12-05 09:41:35.991 ERROR STORAGE_CREATE Failed to load the OpenStack Swift storage at swift://user@email.net@authenticate.blomp.com/v2.0/user@email.net?auth_version=2: Post "https://email.net/tokens": remote error: tls: unrecognized name
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/gilbertchen/duplicacy/src.CatchLogException()
        /Users/gchen/zincbox/duplicacy/src/duplicacy_log.go:227 +0x53
panic({0x118c540, 0xc000420330})
        /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/gilbertchen/duplicacy/src.logf(0x2, {0x12a306f, 0xe}, {0x12e0679?, 0x1?}, {0xc0005bc838?, 0x7ffc7d565e3a?, 0x64?})
        /Users/gchen/zincbox/duplicacy/src/duplicacy_log.go:180 +0x6e5
github.com/gilbertchen/duplicacy/src.LOG_ERROR(...)
        /Users/gchen/zincbox/duplicacy/src/duplicacy_log.go:107
github.com/gilbertchen/duplicacy/src.CreateStorage({{0x129abaa, 0x7}, {0x129abaa, 0x7}, {0x0, 0x0}, {0x7ffc7d565e3a, 0x64}, 0x0, 0x0, ...}, ...)
        /Users/gchen/zincbox/duplicacy/src/duplicacy_storage.go:699 +0x3867
main.infoStorage(0xc0001c4120)
        /Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:1350 +0x3c6
github.com/gilbertchen/cli.Command.Run({{0x12978c2, 0x4}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x12db3c7, 0x30}, {0x0, ...}, ...}, ...)
        /Users/gchen/zincbox/go/pkg/mod/github.com/gilbertchen/cli@v1.2.1-0.20160223210219-1de0a1836ce9/command.go:160 +0x95a
github.com/gilbertchen/cli.(*App).Run(0xc00012b7a0, {0xc000034080, 0x8, 0x8})
        /Users/gchen/zincbox/go/pkg/mod/github.com/gilbertchen/cli@v1.2.1-0.20160223210219-1de0a1836ce9/app.go:179 +0xdc5
main.main()
        /Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2283 +0x6e2c

[2]-  Done(100)                  /config/bin/duplicacy_linux_x64_3.2.3 -stack -log -d info -repository /cache/localhost/all swift://user@email.net@authenticate.blomp.com/v2.0/user@email.net?auth_version=2
/ #

@gchen

#4

I think it is just a URL parsing issue. I’ll take a look.

#5

Thank you! Let me know if you need anything from me. I’m happy to help however I can.

#6

Hello, would you try this patch, which should support usernames containing ‘@’

#7

Same errors as previous. I could also be doing something wrong when building. Not familiar with compiling go applications. My arch is amd64. I could use windows or linux binaries to test, but my actual duplicacy install is in a docker container.

#8

I missed that the container name also has an ‘@’, so more changes are needed in the way the storage URL is parsed. Past that, I have verified that it works. I’ll try to come up with a proper patch.
Edit: no further pach needed. The code allows passing the container in the query string, and that works fine with blomp. Change your URL to this:

swift://user@email.net@authenticate.ain.net?auth_version=2&tenant=storage&endpoint_type=public&storage_dir=user@email.net

(I’ve also changed the host to “authenticate.ain.net” as described in their documentation: GitHub - blompuser/blomp-cloud-storage-docs: Blomp Cloud Storage documentatioin)

#9

Thank you. I can confirm that I was able to add the storage. I used the Web UI to do this, but it requires a container be entered into the field before it will allow the continue button to be clicked, I used the dev console in my browser to remove the 'disabled=" " ’ from the continue button, so this will likely need fixed as well.

I then configured my backups and my schedules. I started one of my backups and got the following errors in the logs. Any ideas what would cause this and how I can fix?

Running backup command from /cache/localhost/4 to back up /backuproot/VMBackup/Hass.io
Options: [-log backup -storage Blomp -stats -threads 4 -stats]
2024-12-11 11:53:56.935 INFO REPOSITORY_SET Repository set to /backuproot/VMBackup/Hass.io
2024-12-11 11:53:56.936 INFO STORAGE_SET Storage set to swift://user@email.com@authenticate.ain.net/?auth_version=2&tenant=storage&endpoint_type=public&storage_dir=user@email.com/Duplicacy
2024-12-11 11:54:04.110 INFO BACKUP_START No previous backup found
2024-12-11 11:54:04.111 INFO BACKUP_LIST Listing all chunks
2024-12-11 11:54:04.210 INFO BACKUP_INDEXING Indexing /backuproot/VMBackup/Hass.io
2024-12-11 11:54:04.210 INFO SNAPSHOT_FILTER Parsing filter file /cache/localhost/4/.duplicacy/filters
2024-12-11 11:54:04.210 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2024-12-11 11:54:56.936 ERROR UPLOAD_CHUNK Failed to upload the chunk b7058cac619699966d5b54054b4403ec36f45021d68e5d9878dd94b895509abf: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/b7/058cac619699966d5b54054b4403ec36f45021d68e5d9878dd94b895509abf": context deadline exceeded
2024-12-11 11:54:56.958 ERROR UPLOAD_CHUNK Failed to upload the chunk fce95b508eb51f0a0d121b32d85c8a0517cfc128ee22d5cc990677c5ed2a683e: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/fc/e95b508eb51f0a0d121b32d85c8a0517cfc128ee22d5cc990677c5ed2a683e": context deadline exceeded
2024-12-11 11:54:56.980 ERROR UPLOAD_CHUNK Failed to upload the chunk 819f1a76e98c45b1457f7042e6ffe7bc2192f1b5ac6a76e69294fb1185c4d6a6: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/81/9f1a76e98c45b1457f7042e6ffe7bc2192f1b5ac6a76e69294fb1185c4d6a6": context deadline exceeded
2024-12-11 11:54:56.992 ERROR UPLOAD_CHUNK Failed to upload the chunk a505b009ba25e02f401ce2301d1140350081780659c9b90ce533fe34719bc956: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/a5/05b009ba25e02f401ce2301d1140350081780659c9b90ce533fe34719bc956": context deadline exceeded
2024-12-11 11:54:57.057 INFO INCOMPLETE_SAVE Incomplete snapshot saved to /cache/localhost/4/.duplicacy/cache/Blomp/incomplete_snapshot
Failed to upload the chunk b7058cac619699966d5b54054b4403ec36f45021d68e5d9878dd94b895509abf: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/b7/058cac619699966d5b54054b4403ec36f45021d68e5d9878dd94b895509abf": context deadline exceeded Failed to upload the chunk fce95b508eb51f0a0d121b32d85c8a0517cfc128ee22d5cc990677c5ed2a683e: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/fc/e95b508eb51f0a0d121b32d85c8a0517cfc128ee22d5cc990677c5ed2a683e": context deadline exceeded Failed to upload the chunk 819f1a76e98c45b1457f7042e6ffe7bc2192f1b5ac6a76e69294fb1185c4d6a6: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/81/9f1a76e98c45b1457f7042e6ffe7bc2192f1b5ac6a76e69294fb1185c4d6a6": context deadline exceeded Failed to upload the chunk a505b009ba25e02f401ce2301d1140350081780659c9b90ce533fe34719bc956: Put "http://swiftproxy.acs.ai.net:8080/v1/AUTH_8b989f118e624ca6957e102775583f6f/user@email.com/Duplicacy/chunks/a5/05b009ba25e02f401ce2301d1140350081780659c9b90ce533fe34719bc956": context deadline exceeded
#10

Could that be a temporary issue unrelated to duplicacy? I can backup to both the root of the container and to a pseudo directory within, similar to your setup. (sorry that I don’t have better than “works for me” to offer right now!). I’m using the CLI but it’s not obvious to me that would change anything.

#11

Maybe? I can’t even init the storage via CLI right now.

/config/bin # 2024-12-12 10:20:04.542 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_BLOMP_SWIFT_KEY
2024-12-12 10:20:05.744 ERROR STORAGE_CREATE Failed to load the OpenStack Swift storage at swift://user@email.com@authenticate.blomp.com/v2.0/?auth_version=2: Response didn’t have storage url and auth token

#12

A temporary workaround (or, perhaps, permanent?) is to apply unix philosophy to this: one utility for one job.

If rclone can successfully connect to your remote — use it to serve data from it over some protocol that duplicacy works well with, e.g. s3.

You can start rclone serve in pre-backup script and shut it down in post-backup. Or have the serve session running permanently via os scheduler.

#13

I’m pretty new to rclone, so was not aware this was an option. I’ll look it over. Still very weird that the UI can init the storage, but the CLI cant. The reason I’m trying to init via CLI is because GUI does not allow adjusting chunk sizes

#14

Interesting. You may be able to find in the logs how did webUI called duplicacy CLI to init the storage. Most of the parameters it passes via environment variables though, maybe that’s the difference.

I would create a fake Duplicacy CLI that would dump environment and command line arguments into a file to see how does UI pass the data. Or use OS tracing facilities to snoop that information.

Lastly, you can init the storage locally with the desired configuration, and copy the whole thing to the target storage using some other tools.

#15

The issue should be fixed by Improve parsing logic for swift storage URLs that contain multiple '@' · gilbertchen/duplicacy@065ae50 · GitHub

The web GUI does not have this issue before all components of the storage url are entered individually so there is no need to separate them.