Google Drive fails

[PaulMoore]

I’m juggling with the idea of migrating to Duplicacy, but it appears the Google Drive implementation is buggy to the point of complete failure.

I’ve added my creds to a JSON file, made sure Duplicacy can see the file and created several backup jobs.

The smaller jobs (sub 1GB) work perfectly. The larger jobs always fail…

2020-04-08 10:51:30.273 INFO STORAGE_SET Storage set to gcd://DuplicacyBackup
2020-04-08 10:51:35.194 INFO BACKUP_START No previous backup found
2020-04-08 10:51:35.194 INFO BACKUP_INDEXING Indexing /backuproot/movies
2020-04-08 10:51:35.194 INFO SNAPSHOT_FILTER Parsing filter file /cache/localhost/0/.duplicacy/filters
2020-04-08 10:51:35.194 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2020-04-08 10:51:35.243 INFO INCOMPLETE_LOAD Incomplete snapshot loaded from /cache/localhost/0/.duplicacy/incomplete
2020-04-08 10:51:35.243 INFO BACKUP_LIST Listing all chunks
2020-04-08 10:51:37.049 INFO FILE_SKIP Skipped 0 files from previous incomplete backup
2020-04-08 12:04:01.548 INFO GCD_RETRY [0] Maximum number of retries reached (backoff: 64, attempts: 15)
2020-04-08 12:14:12.558 INFO GCD_RETRY [0] Maximum number of retries reached (backoff: 64, attempts: 15)
2020-04-08 12:14:12.558 ERROR UPLOAD_CHUNK Failed to upload the chunk fbc067227173f79cebac844437fa50d8eeb04523c5911b84f1404983d510009a: googleapi: Error 401: Invalid Credentials, authError
2020-04-08 12:14:12.563 INFO INCOMPLETE_SAVE Incomplete snapshot saved to /cache/localhost/0/.duplicacy/incomplete
Failed to upload the chunk fbc067227173f79cebac844437fa50d8eeb04523c5911b84f1404983d510009a: googleapi: Error 401: Invalid Credentials, authError

It obviously has valid permissions, as it’s able to read/write to the directories… but neither Duplicacy nor Google’s logs show any reason why it’s failing.

Any ideas please?

[gchen]

Duplicacy was likely refreshing the token when the authError was returned. It looks like a server issue to me.

I’ll push a new CLI release tonight which can access Google Drive with a service account. Maybe it will do better with a service account.

[gchen]

I’m still running the tests for the new CLI version. Will try to release it tomorrow.

[gchen]

CLI 2.5.0 has been released. If you’re running the web gui, restart the web gui and the new CLI will be downloaded automatically.

You can download the service account credential file from https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts.

[PaulMoore]

Thanks for the update.

If you could provide some instructions on how to use it, that’d be great.

[sean.branam]

I’m testing out the web gui and was wondering if there are plans to do AuthID to G Drive? I’ve seen other applications use that and it seems pretty slick.
thanks for your work!

[sean.branam]

It’s here, under Google Cloud Storage.

Supported storage backends

Duplicacy currently supports local file storage, SFTP, WebDav and many cloud storage providers. Local disk Storage URL: /path/to/storage (on Linux or Mac OS X) C:\path\to\storage (on Windows) SFTP Storage URL: sftp://username@server/path/to/storage (path relative to the home directory) sftp://username@server//path/to/storage (absolute path) Login methods include password authentication and public key authentication. You can set up SSH agent forwarding which is also supported by Duplicacy. …

[gchen]

The easiest way to use the service account credential file is to rename it after downloading, to replace the current gcd-token file you’re using. Duplicacy will be able to detect that this file is a service account file and use that to connect to Google Drive (it looks to me that there will be no need to refresh the token with a service account so it may be more stable).

You can also create a new Google Drive storage and enter the path to the downloaded service account file. But you won’t be able to select the storage directory by clicking the directory browsing button – you’ll have to enter the path of the storage directory manually.

[gchen]

If I understand correctly, when you use an AuthID, it is their server who downloads and stores the Google Drive token file for you. The AuthID is just a url to access your own token file without your knowledge. This looks unsafe to me – what if their server got hacked?

[PaulMoore]

[Edit]

It looks to be working. I’ll leave a huge backup running and come back to you when it completes… due 14th. Thanks.

[mattmagelssengreen]

I’ve also had a similar issue -

2020-04-17 18:10:44.002 INFO FILE_SKIP Skipped 0 files from previous incomplete backup
2020-04-17 20:21:45.075 INFO GCD_RETRY [0] Maximum number of retries reached (backoff: 64, attempts: 15)
2020-04-17 20:31:08.659 INFO GCD_RETRY [0] Maximum number of retries reached (backoff: 64, attempts: 15)
2020-04-17 20:31:08.659 ERROR UPLOAD_CHUNK Failed to upload the chunk 7d2753308c08e3fc48d7d4a42ae340541aec1aa520939ff419c266c8bb1434a4: googleapi: Error 401: Invalid Credentials, authError

I went through the steps listed here - tried a service account token and got:
Failed to retrieve the id of the root: Get https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id%2Cname&prettyPrint=false: oauth2: token expired and refresh token is not set

Any help would be appreciated! I may have missed something, though I even went to deinstalling/reinstalling duplicacy to make sure I had the most recent CLI.

[PaulMoore]

I’ve finally had chance to review the logs and it’s failed again, this time with “403 quota exceeded” despite having unlimited storage.

I’ve spoken to Google Drive support (we have a suite account) and they’ve confirmed there are no issues on the account and to confirm, I uploaded a 4GB ISO manually and it worked first time.

Unfortunately, I have no faith in this backup app and will revert back to the competitor with a similar name. The UI might be awful, but it never put a foot wrong. Backup solutions must be dependable… and I’m afraid this (with Google Drive at least) simply isn’t.

[gchen]

I went through the steps listed here - tried a service account token and got:
Failed to retrieve the id of the root: Get https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id%2Cname&prettyPrint=false: oauth2: token expired and refresh token is not set

You may be still running the previous version of the web GUI. In web GUI 1.3.0 you should see 3 text boxes on the configuration page for Google Drive: Token, Shared Drive, and Directory. In previous versions you’ll only see Token and Directory.

[gchen]

This “403 quota exceeded” error was clearly returned by Google server so there is nothing Duplicacy can do other than retrying.

Did anyone else get this same error? Does it mean the daily upload of 750GB is exceeded?

[PaulMoore]

I’ve read the logs myself, there are no quota errors returned. Google’s support team couldn’t see any issues either, which prompted me to upload the 4GB ISO, which uploaded fine.

I’ve only transferred 138GB today, so well below the limit. To further test it, I’m now running two backups… one in Duplicacy, the other using another app. The other backup is 31% complete, no errors so far. Duplicacy failed at 1% with quota exceeded.

Same drive, same credentials, different path.

[gchen]

Can you post the exact log message, like what you did in the first post?

Are you using a gcd-token file or a service account credential file?

[gchen]

The authError has been fixed. See Google Drive invalid credentials error fixed.

The quota exceeded is a different issue though. Remember that each app has its own quotas. For Duplicacy, the queries per 100 seconds per users quota is 2000, and the queries per 100 seconds for all users quota is 20000. If you think you’re hitting the 2000 queries per 100 seconds quota please let me know and I can ask Google to increase the quota.