Restore speed with Duplicacy is quite slow, can that be improved?

If you only use it with a service account and share your Google drive with that service account you don’t need to compile it. It’s only needed for now to let service account to write to domain users’s drive without user expressly granting them permission.

1 Like

Ok, can you tell me how I can change my current existing Duplicacy backup config from using the shared google project credentials to using the service account I configured? Your guide only mentions how to do it when setting up a new backup, but I want to use my existing one of course, just with that different way of accessing the files.

I have never tried doing that (backup using sharing to service account) — but I will now and see how it goes — and will summarize it to you. I’m also curious.

1 Like

BTW - are you using consumer google account or part of Google Workspace/G-Suite? in case there would be difference in configuration…

1 Like

I am using Google Workspace currently, the Enterprise plan for $20 a month with unlimited storage.

1 Like

Hmm. I could not make it work. I found similar thread GCD Using Own Credentials without definitive answer.

Basically, when I share folder with another google account (be that service account or normal user account) that other account cannot enumerate shared folder – neither with duplicacy nor rclone (which explicitly supports that scenario and even has rclone ls --drive-shared-with-me mygoogledrive: syntax to filter shared folders. Not sure what’s the deal there.

Perhaps it will be easier for you to build it on windows and do the rest according to that my post – it would be more straightforward anyway – especially since it’s unclear when will that PR make it to the release.

I’ve just tried to build on windows. It’s fairly easy:

  1. download and install git: Git - Downloads (Next, Next, Next, aggree to all by default)
  2. download and install go: Downloads - The Go Programming Language
  3. Open CMD and run go get github.com/gilbertchen/duplicacy/duplicacy. It will take a while and download about 1.5GB of data.
  4. go to the source folder: cd %GOPATH%\src\github.com\gilbertchen\duplicacy.
  5. apply patch: curl https://blog.arrogantrabbit.com/assets/duplicacy_gcd_subject_scope.patch | "C:\Program Files\Git\usr\bin\patch.exe" -p1
  6. To see what has been patched in run git diff. k/j – up/down, q – exit viewer.
  7. build: go install github.com/gilbertchen/duplicacy/duplicacy
  8. pick the binary from C:\Users\me\go\bin\duplicacy.exe and use it instead of what is shipped.

Then you can properly impersonate the user with the service account and it is working well.

I’ll probably update the blogpost with windows instructions… apparently some people still use that OS :slight_smile:

2 Likes

Thanks, but in your post you only describe how to make it work when setting up a new backup. I want to keep using the backup I already have on Google Drive (which I have in gcd://Duplicacy). And currently my Dupliacy is configured to use that shared project thing to access Google Drive. So I’d somehow have to remove the current shared project access token from Duplicacy, and replace it with the new way explained in your post. How to do that?

Also, doesn’t that only build Duplicacy CLI? I am using Duplicacy Web GUI, not the CLI. Is it possible to build the GUI myself with that PR added?

You would just replace the reference to json file that you now use with with reference to the new one modified according to the article in the duplicacy.json file. I think the CLI preference files are generated based on that json file on every backup run.

You don’t need to / can’t build GUI. It simply downloads and executes the CLI version to do actual backup. You would rename the duplicacy.exe you built to whatever current CLI executable Web GUI is using (likely duplicacy_win_x64_2.7.2.exe) and replace the current one (it will be in the bin folder under .duplicacy_web folder, next to duplicacy.json); and maybe even fix the version in the settings so that it does not overwrite it once the new one is released)

Ah nice, that makes sense.

But where would I do that? Where is the reference to the json file? There is no reference to the json file in duplicacy.json. In duplicacy.json, there is a “gcs_token” and a “password”. And that would need to be replaced with something else I assume?

this refers to the json file that you downloaded before, no? or is the token inserted there in its entirety?

The “gcd_token” is a long string of numbers and letters. Not a reference to any file.

Ok. Maybe it’s base64 encoded entire token. Not sure.

You can add another backup target, this time using the new token json file, and then edit duplicacy.json to replace the value of gcd_token in the old backup target with that of the new one and then delete the new backup target.

2 Likes

gcd_token in duplicacy.json is the encrypted path of the token file, so you can’t just manually edit it there.

The easiest way is to delete the old storage and then create a new one with the same name using the new token file. This should not affect the existing backups.

1 Like

Related question: why is a path stored and not the content of the (conveniently json) token file itself?

Not only encrypting path to file has no value ( encrypting contents of the file would — but I would totally expect the file (that ends up in Downloads folder btw — transient location) to be safe to delete once it has been ingested by the app.

Same goes about the duplicacy CLI — the content of the token json should be incorporated into the preferences file or keychain, not path, because path to external files is unobvious and prone to data loss when user inevitably deletes that external file because there is no indication that only link is saved

I agree this is awkward. The fix I have in mind is to make the CLI take either the path or the content – if the input looks like a path and the file does actually exist, then treat it as a path; otherwise accept the input as the content of the token.

It would add one more step for the user - since it is received as a file: open json, remove line feeds (otherwise it becomes awkward to paste to CLI — end with empty line? What if there is empty line by accident?), copy and paste.

I think accepting the file, as today, but storing its content as opposed to path in the preferences file and duplicacy.json would be more streamlined and seamless change.

1 Like

@gchen can you say something about that? will I have to compile it myself, or do you maybe plan to merge that PR in the next few days anyways? Is anything blocking it that prevents you from merging it? Seems a bit surprising to me that you did not merge your own PR that’s already 1.5 months old now.

This is a feature that I expect will not be used by many people so I don’t know if I want to merge it.

what is the potential downside to merging it? does it change anything in any negative way in any circumstance? I thought it just makes use of some fields in that json file that would otherwise be ignored?

And in general, how likely do you find it that my slow restore speeds are caused by the things mentioned in this thread, and potentially fixed by changing that account setup that I need that PR for?