Please describe what you are doing to trigger the bug:
I run Duplicacy-web backup hourly across a number of repositories. Some of those backup jobs run in parallel. I also run less-frequent check jobs which run in parallel.
Please describe what you expect to happen (but doesn’t):
Log files older than one month are normally deleted by Duplicacy-web when the user connects to the web UI.
Please describe what actually happens (the wrong behaviour):
My Duplicacy logs directory (~/.duplicacy-web/logs) is becoming filled with tens of thousands of old log files, dating back some years.
Most log files are cleaned up, it just leaves those produced by parallel runs (be they backups, checks or whatever…).
Log records of the following form are written to “duplicacy_web.log”:
2021/05/03 08:17:01 The log file backup-20210403-051610.log has been deleted
2021/05/03 08:17:01 Failed to delete the log file : remove /Users/brian/.duplicacy-web/logs: directory not empty
2021/05/03 08:17:01 The log file backup-20210403-061603.log has been deleted
2021/05/03 08:17:01 Failed to delete the log file : remove /Users/brian/.duplicacy-web/logs: directory not empty
The “has been deleted” records relate to successful file deletions, the “Failed to delete” records relate to two or more files not deleted.
Probable cause:
Looking at the combination of the log file and the various “*.stat” files in the “~/.duplicacy-web/stats/schedules” directory, it looks to me as if the cleanup process is based on the contents of those “*.stat” files and that the cleanup algorithm is not recognising, in those files, jobs of type “parallel”.
Within the relevant “.stat” file, each job is recorded as an element in a large JSON array. Each job element is an object with a number of “members” (name/value pairs - known in JSON syntax as “members”).
Jobs that are not run in parallel produce a JSON element that looks like:
{
"name": "Backup-routine",
"type": "backup",
"id": "MacBook-3-Brian@MacBook-Duplicacy-4T",
"status": "success",
"log": "backup-20210505-081603.log",
"start": "2021-05-05T08:16:03.969126+10:00",
"end": "2021-05-05T08:25:04.462217+10:00",
"jobs": null
},
Note that the value of the “jobs” member is NULL and that of the “log” member contains a filename.
Whereas parallel jobs produce an element like:
{
"name": "Backup-routine",
"type": "parallel",
"id": "",
"status": "success",
"log": "",
"start": "2021-05-05T08:15:01.248375+10:00",
"end": "2021-05-05T08:16:03.940826+10:00",
"jobs": [
{
"type": "backup",
"id": "MacBook-3-LaCie-2T@MacBook-Duplicacy-4T",
"log": "backup-20210505-081502.log"
},
{
"type": "backup",
"id": "MacBook-3-G-DriveMobile@MacBook-Duplicacy-4T",
"log": "backup-20210505-081501.log"
}
]
},
In this case, the value of the “log” member is NULL and that of the “jobs” member is an array , with multiple elements each including a “log” member whose value is a valid filename.
So it looks like the code isn’t recognising the special nature of the “parallel” job type.
It attempts to delete a file with the name it finds in the “log” attribute. (It does it using a file pathname consisting of that filename appended to the pathname of the “logs” directory.) This works successfully in the non-parallel case.
But in the parallel case it picks up the NULL “log” attribute, appends it to the pathname and attempts to delete a “file” whose full pathname is just the directory name.
- That directory isn’t empty, so the attempt fails, leading to the “Failed to delete” log entry.
- There’s only one “failed” record because the algorithm doesn’t recognise that the “parallel” element refers to multiple log files, so it only tries once to delete the log(s).
Regards and thanks for the great program.
Brian
Duplicacy Web Edition 1.5.0, Command Line Version 2.7.2, MacOS Big Sur 11.3