Duplicacy Web Edition 0.2.10 Beta is now available

[themmm]

Hi!

I evaluate duplicacy web as an alternative for my duplicati setup. So far it seems to work without any issues, but i recognized one thing:

When i restore a folder the webinterface will not show me all the files that should be in the folder. Even in the restore log it tells me that only 1 File has been restored. When i check the restored files all files have been restored (see pics).

Log:

2018-12-28 21:57:56.659 INFO REPOSITORY_SET Repository set to /root
2018-12-28 21:57:56.659 INFO STORAGE_SET Storage set to gcd://server_tr
2018-12-28 21:58:09.312 INFO RESTORE_START Restoring /root to revision 1
2018-12-28 21:58:13.321 INFO DOWNLOAD_PROGRESS Downloaded chunk 1 size 3717522, 908KB/s 00:00:01 100.0%
2018-12-28 21:58:13.321 INFO DOWNLOAD_DONE Downloaded letsencrypt/live/my.domain.de/README (543)
2018-12-28 21:58:13.321 INFO RESTORE_END Restored /root to revision 1
2018-12-28 21:58:13.321 INFO RESTORE_STATS Files: 1 total, 543 bytes
2018-12-28 21:58:13.321 INFO RESTORE_STATS Downloaded 1 file, 543 bytes, 1 chunks
2018-12-28 21:58:13.321 INFO RESTORE_STATS Total running time: 00:00:05

Duplicacy Webinterface:

Server console of restored folder (restored to other location):

[Christoph]

So do I understand correctly that I need to start a separate instance of duplicacy_web for every repository?

I think duplicacy web should be able to detect presence of existing .duplicacy repository in the directory user is adding and offer to automatically import the configuration (storage, filters, keys).

Creation of backup job should allow use of CLI settings files, particularly filter files. OR, have an option to import job settings from CLI. In fact, if I pick a directory location to backup and there’s already a .duplicacy folder, the GUI should use those settings by default. Even if its only as a one time import. (when I say settings, I’m really interested in filters more than anything because of the time spent tweaking them)

I very much agree with this request. Is this on the roadmap, @gchen?

please report it in this thread.

Wouldn’t it make life easier for everyone if we followed a different procedure with future beta-releases, namely that issues should be reported in separate topics (possibly with a #beta-tag or so)?

[gchen]

@themmm Could that be that files under live/c.... were already up-to-date? The restore command will skip files with unchanged timestamps and sizes unless the -hash option is specified.

[gchen]

So do I understand correctly that I need to start a separate instance of duplicacy_web for every repository?

No, you can run one instance and add multiple repositories.

I very much agree with this request. Is this on the roadmap, @gchen?

I agree this can be very useful, but as this is for one-time use only so I may not implement it until all other more important features are done.

Wouldn’t it make life easier for everyone if we followed a different procedure with future beta-releases, namely that issues should be reported in separate topics (possibly with a #beta-tag or so)?

This makes sense now as the beta has become more stable. Previously it would have created too many topics.

[themmm]

I did not restore the files to the original, but to a different folder. Does duplicacy use the files from the original folder in this case instead of the files in the backup?

Here is the part of the Backup-Log where those files have been uploaded. Even here it shows only 1 file for this folder:

(i blacked out the file path because it would show the address of my server)

As i said before all files get restored without any issue, they just won`t show up in the backup log, the restore log and the backup file browser.

THX
themmm

[ronin-design]

1. When running two “check” jobs in parallel (grouped together in a list of about 6 jobs), clicking either “Completed” link shows same log entries.

I.e. I have two checks, one for local disk backup destination and another sftp storage destination.
I check both, with “parallel” checkbox selected, and after they complete, clicking either “Completed” link opens a new tab showing the identical log entries (same storage location, timestamps, etc.)

EDIT: Additionally, the checks do show correct log entries when “parallel” has not been checked at all.

2. Turns out the endpoint wanted host:port rather than URL, after removing “http://”, storage worked fine.
   Although, it was still a bit confusing the fields not changing to Minio “terms”, still, this is aesthetic. Functionally, it works great!

Previous confused attempt at configuration

S3-compatible storage configuration fields do not change when selecting different “Service Provider” options.
Specifically, I’m trying to configure Minio. When I select from Service Providers list and try to connect, it appears Duplicacy still tries to use “Amazon S3” protocols (e.g. requires region, etc) and I get error (used “test” for region, as an example):
“Failed to check the storage at minio://test@http://255.255.255.255:9000/duplicacy-test-bucket: RequestError: send request failed”
However, it’s somewhat strange, as I can click the “browse” icon to the right of “Bucket”, and it actually connects to my Minio server, showing my existing buckets to select from. (this is if I set Access ID -> AccessKey, and Access Key -> SecretKey)
Additionally, it seems “region” is not optional for Minio as without entering a value, I get:
“Failed to check the storage at minio://http://255.255.255.255:9000/duplicacy-test-bucket: MissingRegion: could not find region configuration”
Note: Also seems like I’m getting the same result from the native Windows GUI as well (failure on optional region, then “send request failed”.)

3. As noted below by mathome1 as well, email notifications are failing to send, blocked by some SMTP / MTA authorities because they are malformed. In one such error, I was told the notification email was RFC 5.7.1 non-compliant as it included no “FROM” header.

[mathome1]

Well done all the work on the web gui, and it all seems to be making good progress. I have a few issues with the latest Beta 0.2.7 running on Windows 10 (some could be user errors). I will break into multiple posts.

First issue is 2 issues with restores which I am not sure if they are user issues, bugs, or just the way it is :-

1. I have a storage on sftp. But when I select this storage, I can then select the revision, but then I try to select the files from the revision I get “Listing Session expired”. This problem does not seem to effect Wasabi storages.

2. When trying to do a restore from a Wasabi storage location (I can’t even try a restore on SFTP due to the issues above), to a new temporary directory on the local server, I only seem to be able to restore a single file in the root directory?? I hoped to be able to select the whole directory for restore, but this results in a restore of o files and 0 bytes and nothing restored. Even if I select a single file outside the root directions, it does not restore anything. Is this a bug, or am a missing something important about the way this is meant to work? Am I meant to be able to restore whole directory structures in 1 go?

Let me know if you need further detail in considering these issues.

[mathome1]

Some feedback on email alerts :-

Issues with email alerts :-

1. I could not get it to work with local exchange server as the mail server. I get a “Failed to send the email: 504 5.7.4 Unrecognized authentication type” error message.

2. It would be very useful if the email alert would add “success” of “failure” to subject of the email alerts. In a lot of cases, people will just want to see “success” or “failure” in the subject and it is tedious and unwieldy to have to open and email and scroll to the bottom to understand if the backup has been successful or not.

3. duplicacy-util is an example of backup email message that is nice and usable. Subject is something like “duplicacy-util: Backup results for configuration duplicacy-urgent-e (success)”. If it fails, it is “duplicacy-util: Backup results for configuration duplicacy-urgent-e (FAILURE)” so it is immediately obvious that there is a backup issue without opening the email. Then the body of the email has detail in an easily readable format that summaries the key things at the top. Example follows :-

Statistics for configuration: duplicacy-urgent-e
Backup Summary:
Storage	Duration	Total Chunks	Total Used	New Files	New File Size	New Chunks	New Uploaded
gui3 	49 seconds 	33211 	166,582M 	0 	0 	1 	3,190K 



Log Text:

05:00:00 Using global config: C:\Users\xxxx\.duplicacy-util\duplicacy-util.yaml
05:00:00 Using config file:   C:\Users\xxxx\.duplicacy-util\duplicacy-urgent-e.yaml
05:00:00 
05:00:00 Backup Information:
05:00:00   Num Storage             Threads
05:00:00    1 gui3                   12
05:00:00 
05:00:00 Prune Information:
05:00:00    1: Storage gui3       Keep: -keep 0:365 -keep 30:180 -keep 7:30 -keep 1:7
05:00:00 
05:00:00 Check Information:
05:00:00   Num Storage             All Snapshots
05:00:00    1 gui3                    true
05:00:00 
05:00:00 duplicacy-util starting, version: 1.5, Git Hash: f2e7147
05:00:00 Rotating log files
05:00:00 Beginning backup on 11-07-2018 05:00:00
05:00:06 Backing up to storage gui3 -vss with 12 threads
05:00:55   Files: 114012 total, 166,552M bytes; 0 new, 0 bytes
05:00:55   All chunks: 33211 total, 166,582M bytes; 1 new, 7,966K bytes, 3,190K bytes uploaded
05:00:55   Duration: 49 seconds
05:00:55 Operations completed in 55 seconds 

Note : duplicacy-util email is even nicer than this as the table above is in nice boxes and things bolded so it is easy to read.

4. one nice things about the duplicacy-util subject email format, you can create a run that checks and warns you if a backup email does not arrive in a certain time, which is nice, because it is a positive alert when duplicacy has stopped working for some reason or has failures which are harder to miss than routine email that might suddenly stop one day if the duplicacy-web process crashes. See GitHub - jeffaco/duplicacy-util: Utility to schedule and run duplicacy backup via the command line for details on what I am talking above.

5. It might be nice if backup reports for multiple jobs could optionally summaries in a single email once a day. Ideally I would implement 2 separate mechanisms so you can have either or both the summary email of all the jobs as well as the individual jobs.

Again, thanks for the great work.

[mathome1]

A very minor issues, but thought it worth mentioning in case you think it worth adjusting :-

1. in restores the the directory order seems to list files/directories for capital letters 1st and lower case letters later. Suspect that most OS and apps will sort ignoring case and make it easier for people to find their files if you do the same.

[gchen]

It doesn’t look like these .pem files were backed up at all. Can you check the include/exclude patterns to see if they were excluded?

[gchen]

1. When running two “check” jobs in parallel (grouped together in a list of about 6 jobs), clicking either “Completed” link shows same log entries.

I can’t reproduce this. Can you inspect the elements to see if they point to the correct log files. The link should be like this:

<a onclick="showLog('check-20181231-204344.log', false)" class="status-success" href="#"> Checked </a>

As noted below by mathome1 as well, email notifications are failing to send, blocked by some SMTP / MTA authorities because they are malformed. In one such error, I was told the notification email was RFC 5.7.1 non-compliant as it included no “FROM” header.

I’ll fix this in 0.2.8.

[gchen]

I have a storage on sftp. But when I select this storage, I can then select the revision, but then I try to select the files from the revision I get “Listing Session expired”. This problem does not seem to effect Wasabi storages.

This will be fixed in 0.2.8.

When trying to do a restore from a Wasabi storage location (I can’t even try a restore on SFTP due to the issues above), to a new temporary directory on the local server, I only seem to be able to restore a single file in the root directory??

You can select the root node (which reads revision xxxx) and the entire directory will be restored.

[gchen]

I could not get it to work with local exchange server as the mail server. I get a “Failed to send the email: 504 5.7.4 Unrecognized authentication type” error message.

I’ll fix this in 0.2.8.

It would be very useful if the email alert would add “success” of “failure” to subject of the email alerts. In a lot of cases, people will just want to see “success” or “failure” in the subject and it is tedious and unwieldy to have to open and email and scroll to the bottom to understand if the backup has been successful or not.

You can enter {{status}} in the subject which will be replaced by the job status (either success or error). This is a feature to be documented.

3,4, and 5 will be considered after the first official release.

[saspus]

Minor UI layout bug. To reproduce - change the width of the window slowly. (macOS/Safari)

[pdaemon]

Hi,

I’m running 0.2.7 Beta and I can’t see the “Storage” page anymore and getting this error:

019/01/01 19:01:49 192.168.1.13:53168 GET /storage
2019/01/01 19:01:49 An error has occurred: runtime error: index out of range
2019/01/01 19:01:49 goroutine 129 [running]:
runtime/debug.Stack(0xcc59a1, 0x19, 0xc4203f6c58)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
main.catchGoroutineError()
	/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy-web/dwe/dwe_utils.go:26 +0x94
panic(0xbab6e0, 0x11cc620)
	/usr/local/go/src/runtime/panic.go:502 +0x229
main.(*Date).Decrement(...)
	/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy-web/dwe/dwe_date.go:141
main.storagesHandler(0xe09820, 0xc4202ced20, 0xc420603000)
	/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy-web/dwe/dwe_storage.go:459 +0x3016
net/http.HandlerFunc.ServeHTTP(0xd7ce90, 0xe09820, 0xc4202ced20, 0xc420603000)
	/usr/local/go/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0x11e01c0, 0xe09820, 0xc4202ced20, 0xc420603000)
	/usr/local/go/src/net/http/server.go:2340 +0x130
main.logHTTPRequest.func1(0xe09820, 0xc4202ced20, 0xc420603000)
	/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy-web/dwe/dwe_main.go:542 +0x1c6
net/http.HandlerFunc.ServeHTTP(0xc42040e3e0, 0xe09820, 0xc4202ced20, 0xc420603000)
	/usr/local/go/src/net/http/server.go:1947 +0x44
net/http.serverHandler.ServeHTTP(0xc4201f52b0, 0xe09820, 0xc4202ced20, 0xc420603000)
	/usr/local/go/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc4200ae6e0, 0xe0a1e0, 0xc42055c040)
	/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2798 +0x27b

Not sure when to look for fix this problem. All other pages work fine except for the “Storage” page and all other functions are fine like backups etc etc…

Thanks Alot

Cheers pdaemon

[pdaemon]

Hi,

Also I was thinking there should be an option on how long to keep log files in the GUI instead of running a cronjob to clean them up on Linux…this is how many log files there are currently in the logs dir:

root@earth:~/.duplicacy-web/logs# ls -altr | wc -l
4332
root@earth:~/.duplicacy-web/logs# du -hs
1.9G .

Also, is there a way change the directory the software is running from to somewhere like:
/usr/local or /opt for example instead of /root/.duplicacy?

Thanks!

Cheers pdaemon

[Christoph]

I very much agree with this request. Is this on the roadmap, @gchen?

I agree this can be very useful, but as this is for one-time use only so I may not implement it until all other more important features are done.

Okay, so what is your recommendation for migrating to the web ui from CLI? Is there any workaround or do we have to manually add each repository and configure it anew? Where does Duplicacy Web store the config?

[gchen]

I just uploaded 0.2.8 that fixes this bug.

[leerspace]

There are a few issues I’ve run across in 0.2.7 which still appear to be present in 0.2.8 (posted while I was writing this up).

1. The time validation for schedules appears to be insufficient, allowing for apparently invalid times to be entered. Take for example this daily schedule meant to run at “02:00 am” every day (without the space between “0” and “am” it works as desired)

2. All of the dropdown controls jump a bit after clicking on them to select an entry. It looks like there’s a default style being applied before the intended styling eventually gets applied – which results in a jerky appearance.

3. The restore page file listing allows the rows to scroll under and overlap with the column headers (shown below). The Revision # row also scrolls over the header row (not pictured).

[pdaemon]

Hi gchen,

Thanks very much for fixing this bug so quickly.
I’ll check out the new version.

Cheers pdaemon