Customize log output

How-to
, ,
#1

I’m now running the Copy command on a nightly basis to upload more local storage to Wasabi and sending the console output to a log file.

With 200K+ chunks in the storage, the console output from the Copy is topping 20MB of entries in the log file. The majority of those entries are reporting “skipped” chunks.

Is there a way to lower the verbosity setting for the log entries?

If not, I would like to submit that as a future mod/enhancement request. And if there’s a better or more appropriate place to enter that request, just let me know where to do it.

Best,
David

#2

You can filter out the “skipped” lines this way:

duplicacy backup -stats | grep -v "Skipped chunk" > output.log
#3

Nah, that’s too simple. :wink:

Thanks!

#4

Here’s my solution that shows the log to the command line AND appends it to daily log files in logs/ dir (mkdir logs/ first though) all while adding timestamps to every duplicacy output line:

duplicacy backup -stats -threads 2 | ts '[%Y-%m-%d %H:%M:%S %Z]' | tee -a logs/duplicacy-backup.$( date +"%Y%m%d" ).log

Sample output:

duplicacy backup -stats -threads 2 | ts '[%Y-%m-%d %H:%M:%S %Z]' | tee -a logs/duplicacy-backup.$( date +"%Y%m%d" ).log
[2019-08-08 01:15:39 PDT] Repository set to ***
[2019-08-08 01:15:39 PDT] Storage set to gcd://duplicacy/***
[2019-08-08 01:15:43 PDT] Last backup at revision 1 found
[2019-08-08 01:15:43 PDT] Indexing ***
[2019-08-08 01:15:43 PDT] Parsing filter file preferences/filters
[2019-08-08 01:15:43 PDT] Loaded 6 include/exclude pattern(s)

Now you can cat logs/duplicacy-backup.20190808.log to see the same output inside.

I think it’s pretty neat.

1 Like
Duplicacy demolishes duplicati and rclone for large backups
How to specify log file names?
Cache and log location
#5

In my Windows scripts I use:

duplicacy backup -log -verbose ......... 1> !LOG-FILE! 2>&1

(LOG-FILE is a variable for naming the log file with date-time)

Which generates the log in this format:

2019-08-05 20:11:18.825 INFO REPOSITORY_SET Repository set to .....................
2019-08-05 20:11:18.851 INFO STORAGE_SET Storage set to b2://....................
2019-08-05 20:11:22.577 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2019-08-05 20:11:22.804 TRACE SNAPSHOT_DOWNLOAD_LATEST Downloading latest revision for snapshot .............
2019-08-05 20:11:22.804 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ...............
2019-08-05 20:11:35.097 INFO BACKUP_START Last backup at revision 168 found
2019-08-05 20:11:35.098 INFO BACKUP_INDEXING Indexing .................
2019-08-05 20:11:35.098 INFO SNAPSHOT_FILTER Parsing filter file ................................
2019-08-05 20:11:35.123 INFO SNAPSHOT_FILTER Parsing filter file ................................
2019-08-05 20:11:35.123 INFO SNAPSHOT_FILTER Loaded 11 include/exclude pattern(s)
...
2019-08-05 20:12:46.232 TRACE PACK_START Packing ................................
2019-08-05 20:12:46.232 INFO PACK_END Packed ................................ (813)
2019-08-05 20:12:46.233 TRACE PACK_START ................................
2019-08-05 20:12:46.233 INFO PACK_END ................................ (154)
...
2019-08-05 20:13:05.379 INFO UPLOAD_PROGRESS Uploaded chunk 21 size 379853, 905KB/s 00:00:08 72.9%
2019-08-05 20:13:06.971 INFO UPLOAD_PROGRESS Uploaded chunk 16 size 668315, 893KB/s 00:00:07 75.7%
2019-08-05 20:13:08.684 INFO UPLOAD_PROGRESS Uploaded chunk 17 size 846334, 849KB/s 00:00:06 79.2%
2019-08-05 20:13:09.259 INFO UPLOAD_PROGRESS Uploaded chunk 23 size 1101717, 859KB/s 00:00:05 83.8%
...
1 Like