Ok, this seems to now happen every time I run prune.
Here is the log:
Running prune command from /var/services/homes/duplicacy-web/.duplicacy-web/repositories/localhost/all
Options: [-log prune -storage cumulonimbus -keep 31:360 -keep 7:90 -keep 1:14 -a]
2020-05-06 20:35:10.026 INFO STORAGE_SET Storage set to sftp://tuchka@cumulonimbus/NetBackup/Tuchka-Duplicacy
2020-05-06 20:35:11.059 INFO RETENTION_POLICY Keep 1 snapshot every 31 day(s) if older than 360 day(s)
2020-05-06 20:35:11.059 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 90 day(s)
2020-05-06 20:35:11.059 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 14 day(s)
2020-05-06 20:35:24.031 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 3
2020-05-06 20:35:25.372 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 4
2020-05-06 20:35:26.664 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 5
2020-05-06 20:35:27.907 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 6
2020-05-06 20:35:29.152 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 7
2020-05-06 20:35:30.375 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 8
2020-05-06 20:35:31.614 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 9
2020-05-06 20:35:32.850 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 10
2020-05-06 20:35:34.056 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 11
2020-05-06 20:35:35.259 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 12
2020-05-06 20:35:36.511 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 14
2020-05-06 20:35:37.741 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 15
2020-05-06 20:35:38.959 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 16
2020-05-06 20:35:40.194 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 17
2020-05-06 20:35:41.437 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 18
2020-05-06 20:35:42.659 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 19
2020-05-06 20:35:43.885 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 20
2020-05-06 20:35:45.117 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 21
2020-05-06 20:35:46.379 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 22
2020-05-06 20:35:47.619 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 23
2020-05-06 20:35:48.831 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 24
2020-05-06 20:35:50.073 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 25
2020-05-06 20:35:51.302 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 26
2020-05-06 20:35:52.526 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 27
2020-05-06 20:35:53.754 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 28
2020-05-06 20:35:54.990 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 29
2020-05-06 20:35:56.254 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 30
2020-05-06 20:35:57.469 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 31
2020-05-06 20:35:58.700 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 32
2020-05-06 20:35:59.914 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 34
2020-05-06 20:36:01.225 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 35
2020-05-06 20:36:02.529 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 36
2020-05-06 20:36:03.813 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 37
2020-05-06 20:36:05.108 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 38
2020-05-06 20:36:06.400 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 39
2020-05-06 20:36:07.696 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 40
2020-05-06 20:36:09.005 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 41
2020-05-06 20:36:10.312 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 42
2020-05-06 20:36:11.597 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 43
2020-05-06 20:36:12.879 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 44
2020-05-06 20:36:14.173 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 45
2020-05-06 20:36:15.461 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 46
2020-05-06 20:36:16.755 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 47
2020-05-06 20:36:18.044 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 49
2020-05-06 20:36:19.349 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 50
2020-05-06 20:36:20.649 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 51
2020-05-06 20:36:21.933 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 52
2020-05-06 20:36:23.222 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 53
2020-05-06 20:36:24.509 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 54
2020-05-06 20:36:25.775 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 55
2020-05-06 20:36:27.084 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 56
2020-05-06 20:36:28.404 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 57
2020-05-06 20:36:29.713 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 58
2020-05-06 20:36:31.028 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 59
2020-05-06 20:40:26.552 WARN SFTP_RETRY Encountered an error (EOF); retry after 1 second(s)
2020-05-06 20:40:28.326 WARN SFTP_RETRY Encountered an error (EOF); retry after 2 second(s)
2020-05-06 20:40:31.070 WARN SFTP_RETRY Encountered an error (EOF); retry after 4 second(s)
Current time 23:06
Processes:
ash-4.3# ps -ax | grep duplicacy | grep -v grep
17746 ? Sl 110:18 /var/services/homes/duplicacy-web/.duplicacy-web/bin/duplicacy_linux_x64_2.5.1 -log backup -storage cumulonimbus -stats
24164 ? Ssl 11:43 /var/services/homes/duplicacy-web/duplicacy_web_linux_x64_1.3.0
31577 ? Sl 5:58 /var/services/homes/duplicacy-web/.duplicacy-web/bin/duplicacy_linux_x64_2.5.1 -log prune -storage cumulonimbus -keep 31:360 -keep 7:90 -keep 1:14 -a
Never mind the version referring to 2.5.1. it is in fact renamed 2.5.2
ash-4.3# ls -alt /var/services/homes/duplicacy-web/.duplicacy-web/bin/
total 172516
drwx------+ 1 duplicacy-web users 142 May 6 21:02 ..
-rwx------+ 1 duplicacy-web users 29861372 May 6 20:06 duplicacy_linux_x64_2.5.1
-rwx------+ 1 alex users 31650873 May 6 20:06 duplicacy_linux_x64_2.5.1.old
drwx------+ 1 duplicacy-web users 308 May 6 20:06 .
-rwxrwxrwx 1 alex users 29861372 May 6 20:00 duplicacy_linux_x64_2.5.2
-rwx------+ 1 duplicacy-web users 31654969 Apr 14 12:45 duplicacy_linux_x64_2.5.0
-rwx------+ 1 duplicacy-web users 26806861 Apr 2 11:26 duplicacy_linux_x64_2.4.1
-rwx------+ 1 duplicacy-web users 26806950 Mar 1 00:24 duplicacy_linux_x64_2.3.0
Stacks of 31577:
(gdb) thread apply all bt
Thread 10 (LWP 31913):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0xc0003e8148, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0xc0003e8148) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043ab80 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1828
#4 0x000000000043c19d in runtime.findrunnable (gp=0xc000043000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2360
#5 0x000000000043ccdc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2520
#6 0x000000000043d25d in runtime.park_m (gp=0xc000001800) at /usr/local/go/src/runtime/proc.go:2690
#7 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#8 0x000000c000470001 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 9 (LWP 31594):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0xc000520148, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0xc000520148) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043ab80 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1828
#4 0x000000000043c19d in runtime.findrunnable (gp=0xc000040800, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2360
#5 0x000000000043ccdc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2520
#6 0x000000000043d25d in runtime.park_m (gp=0xc000130300) at /usr/local/go/src/runtime/proc.go:2690
#7 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#8 0x0000000000000000 in ?? ()
Thread 8 (LWP 31585):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0xc0000804c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0xc0000804c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043ab80 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1828
#4 0x000000000043c19d in runtime.findrunnable (gp=0xc00003e000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2360
#5 0x000000000043ccdc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2520
#6 0x000000000043d25d in runtime.park_m (gp=0xc000130300) at /usr/local/go/src/runtime/proc.go:2690
#7 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#8 0x0000000000000000 in ?? ()
Thread 7 (LWP 31584):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0xc000050bc8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0xc000050bc8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043ab80 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1828
#4 0x000000000043c19d in runtime.findrunnable (gp=0xc000043000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2360
#5 0x000000000043ccdc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2520
#6 0x000000000043d456 in runtime.goschedImpl (gp=0xc000000a80) at /usr/local/go/src/runtime/proc.go:2705
#7 0x000000000043d614 in runtime.gosched_m (gp=0xc000000a80) at /usr/local/go/src/runtime/proc.go:2713
#8 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#9 0x0000000000000000 in ?? ()
Thread 6 (LWP 31583):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0x1930f38 <runtime.newmHandoff+24>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0x1930f38 <runtime.newmHandoff+24>) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043aaa2 in runtime.templateThread () at /usr/local/go/src/runtime/proc.go:1806
#4 0x0000000000439753 in runtime.mstart1 () at /usr/local/go/src/runtime/proc.go:1097
#5 0x000000000043966e in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1062
#6 0x0000000000469323 in runtime.clone () at /usr/local/go/src/runtime/sys_linux_amd64.s:619
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 31582):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0xc000080148, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0xc000080148) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043ab80 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1828
#4 0x000000000043b5c4 in runtime.gcstopm () at /usr/local/go/src/runtime/proc.go:2028
#5 0x000000000043bfbb in runtime.findrunnable (gp=0xc000043000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2085
#6 0x000000000043ccdc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2520
#7 0x000000000043d25d in runtime.park_m (gp=0xc000001800) at /usr/local/go/src/runtime/proc.go:2690
#8 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#9 0x0000000000000000 in ?? ()
Thread 4 (LWP 31581):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0x1931040 <runtime.sig>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b146 in runtime.notetsleep_internal (n=0x1931040 <runtime.sig>, ns=-1, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:174
#3 0x000000000040b34c in runtime.notetsleepg (n=0x1931040 <runtime.sig>, ns=-1, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4 0x000000000044c0bc in os/signal.signal_recv (~r0=<optimized out>) at /usr/local/go/src/runtime/sigqueue.go:147
#5 0x0000000000730f42 in os/signal.loop () at /usr/local/go/src/os/signal/signal_unix.go:23
#6 0x0000000000467131 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1373
---Type <return> to continue, or q <return> to quit---
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 31579):
#0 runtime.epollwait () at /usr/local/go/src/runtime/sys_linux_amd64.s:705
#1 0x0000000000430932 in runtime.netpoll (delay=9999866741, ~r1=...) at /usr/local/go/src/runtime/netpoll_epoll.go:119
#2 0x000000000043bebb in runtime.findrunnable (gp=0xc000040800, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2323
#3 0x000000000043ccdc in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2520
#4 0x000000000043d25d in runtime.park_m (gp=0xc000130300) at /usr/local/go/src/runtime/proc.go:2690
#5 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#6 0x0000000000000000 in ?? ()
Thread 2 (LWP 31578):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430cd4 in runtime.futexsleep (addr=0x19061b8 <runtime.sched+280>, val=0, ns=9999853807) at /usr/local/go/src/runtime/os_linux.go:51
#2 0x000000000040b1ee in runtime.notetsleep_internal (n=0x19061b8 <runtime.sched+280>, ns=9999853807, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3 0x000000000040b2c1 in runtime.notetsleep (n=0x19061b8 <runtime.sched+280>, ns=9999853807, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:216
#4 0x0000000000441dd2 in runtime.sysmon () at /usr/local/go/src/runtime/proc.go:4492
#5 0x0000000000439753 in runtime.mstart1 () at /usr/local/go/src/runtime/proc.go:1097
#6 0x000000000043966e in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1062
#7 0x0000000000469323 in runtime.clone () at /usr/local/go/src/runtime/sys_linux_amd64.s:619
#8 0x0000000000000000 in ?? ()
Thread 1 (LWP 31577):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:568
#1 0x0000000000430c56 in runtime.futexsleep (addr=0x1906c68 <runtime.m0+328>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040b06f in runtime.notesleep (n=0x1906c68 <runtime.m0+328>) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043b218 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:1971
#4 0x000000000043ce86 in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2454
#5 0x000000000043d25d in runtime.park_m (gp=0xc000130900) at /usr/local/go/src/runtime/proc.go:2690
#6 0x000000000046518b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:318
#7 0x00000000004650a4 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:220
#8 0x0000000000000000 in ?? ()
(gdb)