What are tmp chunks?

Support
#1

I’m running a prune and noticed that it is trying to delete temporary (?) chunk files:

2021-03-10 09:07:06.971 INFO CHUNK_DELETE The chunk 5af0d44caf5281d4e6470c543fd87b28ae3efa0c128293c4f55d868a1949a844 has been permanently removed
2021-03-10 09:07:06.986 WARN CHUNK_DELETE Failed to remove the file 31/9ff1cc1ad97612c816e9192eb4da05714d499856fab5b8cff57448d5cbac15.xkkcvzck.tmp: Path does not exist
2021-03-10 09:07:07.025 INFO CHUNK_DELETE The chunk 5ad57a816dd336a255ab3c0614d6957a7ed6f6b997a20c10b563bdcb7704def8 has been permanently removed
2021-03-10 09:07:07.119 WARN CHUNK_DELETE Failed to remove the file 30/64b370ddbab28329c39e4d59398c8acb185f048ef0c9d68470724165cd9c29.pdqauqia.tmp: Path does not exist
2021-03-10 09:07:07.127 WARN CHUNK_DELETE Failed to remove the file 6f/5a8e40e3b21e7c40628a93ef0902a6a5372015ccd6c27224232091dd625693.vsptpjyq.tmp: Path does not exist
2021-03-10 09:07:07.167 WARN CHUNK_DELETE Failed to remove the file 3e/d49b5cebfe58e566957cb017054b63b2f264925ec36642eb4e1dc8200b0d3d.tvyhcprc.tmp: Path does not exist
2021-03-10 09:07:07.273 WARN CHUNK_DELETE Failed to remove the file 93/d5dd4d424e7d5197004ac399f972ee572c9c1867a3873d86f3558d900226cb.upscwedm.tmp: Path does not exist
2021-03-10 09:07:07.274 WARN CHUNK_DELETE Failed to remove the file 41/e23ae8b0eacbb499b83e5e191f86285f8476ecb45e172bde6edf1cd67c836d.igajnjft.tmp: Path does not exist
2021-03-10 09:07:07.322 WARN CHUNK_DELETE Failed to remove the file 2d/a7b558bdb8b02b3bde0b614eafe2e5b041223e6a933f127f097366ec48cb6d.fevgjvnf.tmp: Path does not exist
2021-03-10 09:07:07.423 WARN CHUNK_DELETE Failed to remove the file 68/b86905006c94582c746dd1a97813297192a7ab7be16fbc1774e10563cd1507.zvfbqnxh.tmp: Path does not exist
2021-03-10 09:07:07.425 WARN CHUNK_DELETE Failed to remove the file 4c/9b92853b7675fa0154dbb9c218440154b499705efdeb3925ac68a073976f2b.ijtlwooa.tmp: Path does not exist
2021-03-10 09:07:07.471 WARN CHUNK_DELETE Failed to remove the file be/6584f7035dc34eb15dec2c8bb569b0d9cdbbe5711b248f77d7e98d24162834.bhjqcknt.tmp: Path does not exist
2021-03-10 09:07:07.583 WARN CHUNK_DELETE Failed to remove the file 69/30e61d020b5189096efcb8c302b47ed3f1ec1ecc5f2ece8817e4190a81aab5.hxyrhhey.tmp: Path does not exist
2021-03-10 09:07:07.598 WARN CHUNK_DELETE Failed to remove the file c6/f901e733d184c6ff190083715a7fbabeab67a2772087a9067f592287bd526b.arzmyamj.tmp: Path does not exist
2021-03-10 09:07:07.620 WARN CHUNK_DELETE Failed to remove the file 98/f0c1fe338a39fb684537c25eb2625652817f7ba7ca0089bd080e890bb4aaff.hkutevrl.tmp: Path does not exist
2021-03-10 09:07:07.726 WARN CHUNK_DELETE Failed to remove the file 82/1af5d17479c4ee098ad68ca96dbd5aa9a9c2bb246fbb521cc44a82f0ac32f5.eptmsbxo.tmp: Path does not exist
2021-03-10 09:07:07.751 WARN CHUNK_DELETE Failed to remove the file 20/36f782bb7178c836cdc80a1cf87f457bd2c53d95e860a963f42784a5c07688.twuyxvtp.tmp: Path does not exist
2021-03-10 09:07:07.774 WARN CHUNK_DELETE Failed to remove the file 57/5f816f03addaa841bd11c4a10580a6616209af0624899a0f713185cb15dba7.gkmvffkc.tmp: Path does not exist
2021-03-10 09:07:07.998 INFO CHUNK_DELETE The chunk 163034084606b9f18e2ad1cf9b88920dcf323681f343f518239d0c56334c1b20 has been permanently removed

This seems strange to me so I’d like to check whether this is expected behaviour.

I understand that tmp files are created and then renamed once fully uploaded so I can see how the odd tmp file might be left if renaming fails for some reason (restart, connectivity or whatever). So I can see why such files might exist in the storage. But if I understand the prune command correctly, it deletes snapshots (i.e. chunks belonging to snapshots), so if it is trying to delete a tmp file, that would mean that that tmp file is listed as part of some snapshot, right? But that shouldn’t be possible, right?

Thinking about this a bit more, I realize that I’m running prune with the -exhaustive option. So I’m guessing that this is how it picks up files that are not in a snapshot. But in that case I would assume that in order to pick those files up, they would have to exist so that I don’t understand how it can fail deleting them because they don’t exist.

There is no other duplicacy command running in parallel with the prune process that might be fiddling with those files at the same time…

There is a previous issue related to tmp files, but I don’t think it’s related as it was on google storage and my storage is pcloud.

#2

These temporary chunks were collected during last prune operation and could be a result of a backup in progress (especially if you use many upload threads).

If you’re sure that no backup was running at that time, these chunks could have been deleted by an earlier prune operation that also picked them up.

#3

But if they were deleted, how can duplicity know about them?

#4

Remember the two-step prune procedure. The same set of temporary chunks can be picked up by multiple fossil collections if the condition to delete them isn’t met. When they are finally eligible for permanent deletion, the first prune operation will delete them successfully but subsequent ones will not find the files and thus generate those warnings.

1 Like