Duplicated Folder on USB Disk

[items]

Hi,
I’m a little bit in trouble with duplicacy. I’ve made a Unraid backup on a USB disk with Duplicacy Web and everything was fine, each folder was there in more than 90 revisions. I had to format the disk due to several reasons and I’ve started new with revision 1 but now I had duplicated folders (look at the screenshots please) . So I formatted the disk again, I’ve deleted the whole cache directory and the storage, and initialized everything again from zero. After 2 days the backup was finished, but again: There are duplicated folders. I don’t have them on my filesystem and I don’t understand, why one folder e. g. data/backup is doubled and the first “data” directory consists of just some files but the other directory contains all files. So why “data” exists two times, when this isn’t the case on my harddisk? And it is not only data, also other folders are existing twice.

I’m absolutely confused and have no idea to solve that. And I’m also afraid of loosing data. Can anybody help?

Unfortunatley I can just attach one screenshot as a new user, Sorry for that. But as you can see, I have the “data” folder twice, but with different content. The first one consists of just some files. The other one seems to be the original one with all files. But I’m not 100% sure of that.

Thanks in advance
items

Folders in revision 1:

[items]

Here is the other one with the content of the first data folder:

[items]

and here is the second folder with parts of its content:

[saspus]

• What are you backing up on unraid? Could it be there are some symlinks to the same data?

• to rule out browser shenanigans, try to list files in the command line, from another computer if you want.

• why is the folder called Time Machine? Do you keep Time Machine backups there?

[items]

Thanks saspus!
I’m backing up everything except appdata in /mnt/user. So all shares and also my Time Machine Mac Backups. I don’t have any symlink and I ran the list comand succesful yesterday, but I just see thousands of files and not just the folders. So I’ve unmounted the disk and attached it to a Linux VM but here duplicacy says that no backup exists at all on the storage.

Very frustrating, but I’m glad, that I have checked whether I have a backup that is restoreable before running into serious trouble. Currently I rsync all files in order to have a backup at all and after that I’ll investigate again. I assume that after formatting the disk something is still in the cache or in a database or whatever, so it might be good to start again from scratch.

Cheers
items

[saspus]

@items

and also my Time Machine Mac Backups

I strongly recommend against doing this. Time Machine backup is a disk image. Unless you absolutely certain that the Time Machine was unmounted at the time of backup, the image will be unusable. Instead, backup data from your Mac directly. Common approach is to backup from the machines to unraid, and then duplicacy copy snapshots to offsite storage, over the internet.

Note, I also recommend against using USB media for backups, it’s highly unreliable.

@items

I don’t have any symlink and I ran the list comand succesful yesterday, but I just see thousands of files and not just the folders

Elaborate on this one. If on another machine, you initialize the duplicacy repository in an empty folder, pointing to that USB drive, and do duplicacy list -id <same_snapshot_id_you_used_on_unraid> – you don’t see the revision that was created on unraid? Do you see a revision file under snapshots/<snapshotid>/ on your usb drive? If so, then use --files flag to see the files.

@items

So I’ve unmounted the disk and attached it to a Linux VM but here duplicacy says that no backup exists at all on the storage.

How did you initialize the repository and how did you check for presence of files?

@items

I’m glad, that I have checked whether I have a backup that is restoreable before running into serious trouble

This is the recommended practice – untested == broken.

[items]

Thanks again saspus!

It took some days to get 2 luckybackups (6TB) on differents disks to feel safe again

After that I’ve deleted the whole Duplicacy stuff again and made a test with one folder on an encrypted disk. It was absolutely no problem to restore it on another computer, so I’ve started new to backup the 3 tb agin which will also take some days. The whole stuff has been done mostly through the Web Frontend which works well and I’ll also consider your hints regarding Time Machine.
The last time I struggled with the comand line options of duplicacy but I’m not sure whether it is because I’m not really familiar with it or because of the broken backup. But I’ll try again after this backup is finished if it doesn’t work by the frontend.

Finally I think it was a configurration issue or somthing went wrong while exploring the usage of Duplicacy.

[items]

Hi again,
it took some days to make new backups and I’m still confused. I’ve mounted the disk on a VM and Duplicacy shows me exactly the folders I would have expected. Nothing was duplicated. I ran the list comand but it doesnt show folders just thousands of files. So that doesn’t help. After that I’ve installed another Docker Container (hotios repo) with Duplicacy on another port, mounted the disk here and also no duplicates were shown.

It seems that there is a glitch in the matrix, two versions of Duplicacy are showing, that there is no duplicated folder, but one container shows them. No idea what to do with that could it be a bug in your version Saspus?

This is what Hotios Duplicacy shows to me (V 1.7.2):

And this is your version saspus (V1.8.2). Some folders are duplicated, and it shows me a different content:

[Droolio]

Not that this should be happening, but I notice an appreciable gap between the folder icon and the text, when compared between the duplicates… any of your files / folder have prefixed spaces in them?

[items]

Thanks Droolio! Definateley not, have the same folders since years. No spaces at all. I suppose it is a problem with 1.8.2 or with 1.8.2 and the container. But with 1.8.0 everything is fine in an Ubuntu VM, with 1.8.2 it is duplicated. With 1.7.2 alsoso everything is ok. No idea what is going wrong here…

[Droolio]

Definitely seems related to either of these two changes:

@gchen

1.8.2 Changes:

• Fixed a bug in file/directory selection that prevents selected items from being unselected in the Restore page
• Properly show directories containing only empty files when browsing files in the Restore page

@gchen could this be a regression?

[gchen]

It is definitely the spaces – each directory appears twice, one with a leading space and the other without. I’m not sure how these spaces got there, but your backups are not affected at all – the bug is in the code parsing the output of the duplicacy list command.

What is the CLI version that your web GUI 1.8.2 shows? Go the Setting page and you should see the CLI version number there.

[items]

I’ve just checked it out and I’m sure that on the disk there are no spaces in the directory names. I’ve tried to configure a share with leading spaces and they are not created with that, they are always removed.
Also an ls -la in the comand line shows no spaces but they are shown with a backslash for folders which do have spaces. So I don’t think that there are any spaces on the disk as a reason. If the comand line version is the same in both docker instances like it is, maybe a Frontend Problem? I’ve also just checked three different browsers (Vivaldi, Firefox, Chrome) and they all show the right listing in 1.7.2 and the duplicated folders in 1.8.2.

If there is anything I can provide I’ll do of course.

CLI Version in 1.8.2 is 3.2.3
CLI Version in 1.7.2 is 3.2.3

[maccra]

I also have the same issue with duplicate folders and also duplicate BackupIDs.
WebGUI: 1.8.2
CLI: 3.2.3

[gchen]

This bug is caused by this change:

Properly show directories containing only empty files when browsing files in the Restore page

Basically when there is a file with a size of 0, the parser would incorrectly add leading spaces to the top level directory. This should not cause any issues, as you can simply ignore these directories with extra spaces.

I’ll prepare a new release to fix this bug.

[items]

Good to hear that, thanks @gchen!