Problem setting up Duplicacy docker backup to S3 (minio) storage

Support
#1

I have Duplicacy running in a Docker on an Unraid server. Version 3.2.3.

I’ve built a small OpenMediaVault (OMV) NAS as a target for my Unraid backups.

I’ve had success using sftp for a storage target on OMV.

I’d like to try S3 storage as a target (via minio) on the same OMV server, but so far I haven’t had any luck.

When I attempt to create the S3 storage entry like this (picture):

S3-config
S3-config594×660 26.5 KB

I get the following error:

Failed to list the buckets: SerializationError: failed to decode REST XML response
	status code: 200, request id: 
caused by: XML syntax error on line 38: element <link> closed by </noscript>

I installed WinSCP on my Windows PC, and it connects to the minio S3 storage using the same credentials I entered in Duplicacy, so I’m stumped. I’ve actually scratched and rebuilt the OMV server a couple times thinking I screwed up there, but it seems to work.

#2

For minio you shall use minio, minios, or s3c backends. See Amazon S3 section here: Storage Backends · gilbertchen/duplicacy Wiki · GitHub

I’m not sure if web ui allows to specify this in the GUI. If it does not — create some other storage, and then replace URL in duplicacy.json

#3

As far as I can tell, with Duplicacy in a Docker, all I’ve got is the web GUI. I’ve selected minio over http (not https). Is there a way to use the Duplicacy CLI version inside the Docker container? Or see the json it’s trying to create? I’m blind to what Duplicacy is doing right now.

#4

You mean in the gui? So it’s supported in gui?

The duplicacy.json file shall be in the config folder that shall be mounted outside of the container. Depending on which container you are using this may or may not be trivial to do — but should not be difficult.

#5

Yes, here are the service provider options in the GUI:

S3-config2
S3-config2594×656 36.2 KB

I just looked at duplicacy.json. The problem is that since I cannot finish defining the S3 storage option, nothing for that option is stored in the json. (My sftp option is there, so I think I’m looking at the correct file.)

Is there a way to use the cli from the docker console? Maybe I can try this via cli to get a better idea of what the problem is.

#6

Yes, but it’s easier to do it on a separate machine. Download the CLI from the GitHub and attempt to init you minio storage in an empty folder.

But it looks like some incompatibility between the minio server and whatever client duplicacy is built with

#7

Downloaded the CLI version on my Windows PC. Here’s what happened:

E:\Kits\Duplicacy>duplicacy -d init dbtest minio://us-5224-CM4NAS@192.168.1.55/cm4-s3-bu-bucket/dbtest
Reading the environment variable DUPLICACY_S3_ID
Enter S3 Access Key ID:xxxxxxxxxxxxxxxxxxx
Reading the environment variable DUPLICACY_S3_SECRET
Enter S3 Secret Access Key:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Reading the environment variable DUPLICACY_S3_ID
Reading the environment variable DUPLICACY_S3_SECRET
Failed to download the configuration file from the storage: Failed to parse the config file: invalid character '<' looking for beginning of value
#8

Looks like some incompatibility between the client and server. @gchen?

#9 
invalid character '<' looking for beginning of value

This looks like the S3 server is sending back some random html page. Try using a s3 client to download the config file and check the content.

#10

Downloaded S3 Browser. When I try to connect to the minio instance I get this:

[I] [2025-02-24 20:29:36] Getting buckets list.. TaskID: 21
[W] [2025-02-24 20:29:36] System.Xml.XmlException: The 'link' start tag on line 38 position 3411 does not match the end tag of 'noscript'. Line 38, position 3470.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag)
   at System.Xml.XmlTextReaderImpl.ParseEndElement()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
   at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
   at System.Xml.XmlDocument.Load(XmlReader reader)
   at System.Xml.XmlDocument.LoadXml(String xml)
   at zh.a(String glu)
   at ox.Do()
   at xm.c(xg gfd) TaskID: 21

So I guess it’s minio I’m having trouble with? I have no idea where the configuration file is or how to see it. This is minio installed on OpenMediaVault, which is also new to me.

#11

I remember reading somewhere that minio sometimes returns html embedded in xml when errors occur, and perhaps this fails to parse. Maybe there is a setting to disable that, I’m not sure.

Personally, I’ve given up on minio long time ago. I could never make it stable enough for long enough.

For backing up to local server the best approach would be SMB endpoint. (Note, not mounting the share to windows and using drive letter or network disk; but rather connecting duplicacy to SMB:// endpoint directly instead; this also helps workaround windows oddity if not being able to connect to the same SMB server with multiple different credentials — Microsoft invented SMB, and yet here we are in 2025)

Side note: since you are just starting — why OMV? OMV was created to provide Linux based alternative to FreeNAS (now TrueNAS). Today TrueNAS Scale, which is Linux based, is stable, and is a commercial product used by enterprises (this helps a lot with stability). (Btw they scrapped minio from default install even on TrueNAS core some time back — I wonder why :)) either way — I would suggest exploring TrueNAS Scale, before you pour too much time into OMV. (Full disclosure — I’m a die-hard FreeBSD dude so I’m clinging to already dead TrueNAS Core, but Scale is mature enough and is the future)

#12

The “why” is a long story. My local NAS is an Unraid server, but I worry about backing it up. I have a Raspberry PI CM4 module and an Axzez carrier board with 5 SATA ports. I thought this was a good idea, but I didn’t realize the carrier board needed some kernal patches to work. They offer a desktop linux distribution and an OMV 7 distribution, so I’m trying OMV. So the Raspberry PI NAS is supposed to be a backup target for the Unraid server. Right now I’m trying to make this working with everything local, but one option later could be to move the backup target server to another location.

I have this working with sftp locally. I wanted to try S3 to see if there’s a performance difference.

All of this has felt like a bad idea. I’m also considering replacing the CM4 with a N100 motherboard so I have more options, like TrueNAS Scale.

Thanks for the pointers. I have learned quite a bit the last week or so. I’ll check a few more minio configuration settings concerning the error returns. If I figure it out and get it working I’ll post an update. Right now I’m going to assume this is a dead end and it’s back to sftp.

1 Like
#13

There are a few things to consider (tangentially related to duplicacy, so we are no too far off topic)

  • To guard against bit rot your “backup of a backup” must be rock solid: you would want to use either zfs (which TrueNAS and OMV support) or btrfs, to have self-healing. ZFS comes with a high memory requirements, that may not be feasible on pi-type devices (which is likely why Synology and Ubiquiti went btrfs route)
  • Another alternative would be to enable erasure coding in duplicacy to combat some bit rot to some degree – but this would come at the space overhead, that could be significant, depending on (proportional to ) the size of backup and erasure coding parameters.
  • These multi (2+) -SATA port devices often (I did not check the one you referenced) are based on simple bridge paired with SATA port multiplexers. That latter bit is highly concerning: there is no port multiplexer in existence that is stable let alone peformant. The prevailing recommendation from multiple storage solution vendors (and users of their forums) is to avoid them like plague in any scenarios where data integrity is important.

I was unable to find a solution in similar circumstances that would be stable, not cost arm and leg, and low power, and resorted to buying old enterprise, predominantly Intel x64 Supermicro servers from local recyclers and old datacenter components off eBay. Any low power solution immediately becomes triple digits, even on secondary market. I’m wondering what you will end up using – having low power ARM device as a file server with ZFS is an ultimate “holy grail”. My current home server with 80TB of space consumes 160W… it’s very painful with $0.52/kwh electic rates here