Thousands of .davfs.tmp files

Thanks to rclone not hiding hidden files, I just noticed that I have thousands of .davfs.tmp files in my webdav backend (in the chunks folder). I’m assuming that these are the remains of failed upload attempts and I wonder if there is anything duplicacy could do to clean those up?

Wouldn’t a prune -exhaustive do that?

Duplicacy does create .tmp files, but they are renamed to the chunk files after upload and don’t have filesystem prefix. Furthermore, if upload failed, next time it would upload the same file and rename it again, so I don’t see how could accumulation of those files occur, unless the rename operation is broken on the remote in a way that behaves like copy.

But I don’t think even this is the case here: these files don’t look like anything that duplicacy would create — Duplicacy does not append .davfs to the file name; these files look like the artifact of davfs mounter. Do I remember correctly that you are backing up to locally mounted webdav folder using crooked pcloud mounter, as an attempt to workaround their webdav api issues?

Would not this serve as a last straw to finally move to another storage provider :wink:

1 Like

Good point. But does prune -exhaustive delete all files except for relevant chunks or does it merely delete all non-relevant chunks? Note that these tmp files are hidden (i.e. starting with a “.”…

Yes, these files most likely were created by rclone mount which I tried to work around pclouds unreliable webDAV interface.

I’ll rephrase that: rclone serve is my last straw before I’ll consider alternatives. And so far it looks promising:

I know that is what you (@saspus) recommend long ago, but I couldn’t get rcloud serve to work on my Mac, but it wasn’t a big deal on my Debian server.

Disclaimer: by promising I currently mean that my first prune command since month is now running for over 50 hours and it is still running. I don’t know whether it taking so long should worry me, but for now it’s positive that a job runs for so long without error.

These .davfs.tmp were not created by Duplicacy. The .tmp files created by Duplicacy have 8 random characters. But, prune -exhaustive will delete any file with the .tmp suffix so it should delete them.

Unfortunately, those files don’t end with .tmp:

Something else that’s strange is that the files’ last modified date goes back as far as 2018 even though I first tried rclone in 2022. So either these files for some reason carry the original date of the files they were meant to be named after or rclone is not the creator of these files.

Maybe the rclone delete command can solve your problem, with a proper filter.

Run with --dry-run flag to test… :wink:

1 Like

Yes, that should do the trick! Sometimes solutions can be so simple :slight_smile:

I’ll try it once my current pruning job is through. It looks like it’ll take a few days. According to nethogs rclone is using about 4-500 kB/s bandwith for the prune job. Not sure if that is a limitation imposed by the pcloud api, but I do wonder why pruning fossilizing chunks takes so long, i.e. between 1 and 10 seconds for a single chunk:

2022-04-07 08:10:10.630 TRACE CHUNK_FOSSILIZE The chunk 88256ce8cdc36831ef7c9b9ad96b3651c03b596e552bd42a3b2ba3cb5eb2b7be has been marked as a fossil
2022-04-07 08:10:20.513 WARN CHUNK_FOSSILIZE Chunk c7a0f270ce329ffa25ba752cc8502a057cef8bf944199570715c4e6cace34dbb is already a fossil
2022-04-07 08:10:23.955 TRACE CHUNK_FOSSILIZE The chunk 5241c4183f3aac2ed622ea49c2cbdb6b4f68fa0cb0676768efd0578f1942f65b has been marked as a fossil
2022-04-07 08:10:31.526 TRACE CHUNK_FOSSILIZE The chunk 89dc56bcac2f8b33d665ad5820203fc17e3e70b123b06e6c4241fd1f18bc3a67 has been marked as a fossil
2022-04-07 08:10:35.026 TRACE CHUNK_FOSSILIZE The chunk 16c0abe078464fc75fed4ccd6514879086286745f4e91b3197fc6d009f5080c4 has been marked as a fossil
2022-04-07 08:10:38.810 WARN CHUNK_FOSSILIZE Chunk 3fc3a83100da9bbc6737ace61187ab2b57a968774010f31fa60854636e228733 is already a fossil
2022-04-07 08:10:45.583 WARN CHUNK_FOSSILIZE Chunk 85746e943a1106fc17082e969aa413450580e6a26beec17c696768dc4ff6bf36 is already a fossil
2022-04-07 08:10:52.723 TRACE CHUNK_FOSSILIZE The chunk 17a5ced7905567d3f23c31655897646f6095cc88b82bcd492daf331fd037dd23 has been marked as a fossil
2022-04-07 08:10:57.210 TRACE CHUNK_FOSSILIZE The chunk 089d2760f105448502137d6f09910c0f36b2717f3962cb1a28999e79b3a7f432 has been marked as a fossil
2022-04-07 08:11:01.226 TRACE CHUNK_FOSSILIZE The chunk f0f87b5d7404c3c33743dc2630ab15a174a2f63edf5faabe3ceafcd18cb45aab has been marked as a fossil

And I don’t quite understand why this would involve so much bandwith either. (But I’m happy, because at least it’s running without errors).

That’s a band-aid, though, the issue is still there. But I guess it’s indeed easier to just keep deleting all that periodically than fighting the losing battle in getting it triaged and fixed.

It’s probably rate limit – the bandwidth use is low because there is no data transfer as such, all is sent are commands. They take very little bandwidth, but you can’t send them too fast. I have seen prune take 2+days on 4TB backup to google drive.

WRT rclone specifically, even if you mount the remote with local cache, deleting files still takes forever as rclone does it in sync mode, deleting stuff from the remote one by one, and would not return until all is done. I would prefer it to delete locally instantly (i.e., build the list of stuff to delete and then sync deletions to remote in the background, but they don’t do that; just like you can copy a bunch of files instantly to the local cache and have it upload later in the background). I’ts one of many reasons I’ve stopped using rclone mount altogether. (one of the other is reliance on kernel extension, which is exceedingly discouraged and deprecated on all modern OSes that already provide special API to implement third party cloud storage natively. There are some third party tools (e.g., Expan’s SugarSync) that implement this API for many cloud storage providers. Some cloud providers, e.g., Box.com, have already rewritten their native desktop client to use that API, instantly making it infinitely more attractive and stable. Some other providers went the other direction – Google Drive switched away from using osxfuse (good!) but instead of using FileProvider API they chose to run a local SMB server to serve data (what!?). As a result, stability plummeted.

There are talks about supporting it on rclone forums for the past 3 years at least, but all threads end with “somebody needs to implement that”, so I don’t hold my breath…)

2 Likes

I finally did this and it worked perfectly using rclone -v --max-size 0 delete pcloud: --include "*davfs*"

BTW: The reason it took me so long was that my first prune command took two weeks:

And the second one (to actually get those old chunks deleted) another week:

I probably should be so annoyed at the slow speed, but I’m not. This is the first time in years that pcloud works for me without any flaws (except for the speed, if that’s a flaw in this case). I’m so relieved. (I still need to see, though, how the actual storage performs. I hope that all the many problems I’ve had with missing and zero size chunks were due to webDAV so that those will disappear now. But who knows…)

So, to anyone with a lifetime plan struggling with pcloud: forget about webdav and use the pcloud api via rclone (if you’re not on a lifetime plan, you should follow @saspus advice and switch to a better provider).

Well, I guess I’m the champion now (I also have 4 TB).

Thanks for explaining. This is good to know. (In the case of pruning, most operations were renaming files, but I guess the same principle applies there).

2 Likes