AWS S3 403 Forbidden error reported as not found

dennis-tra · 8 December 2020 09:45

Please describe what you are doing to trigger the bug:

We were running the duplicacy check command against an AWS S3 storage backend and noticed that the chunks could not be found. As it turned out we didn’t give the right permissions for the Amazon IAM role associated with duplicacy and therefore received a 403 from AWS. However duplicacy reported that the chunk was not found and this lead us to think in the wrong direction. I saw that this is an intended behaviour as it swallows 403 errors here:

github.com

gilbertchen/duplicacy/blob/41a8f657c4c8f57b04784bea429c5e06156ffbe1/src/duplicacy_s3storage.go#L184


// GetFileInfo returns the information about the file or directory at 'filePath'.
func (storage *S3Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {

	input := &s3.HeadObjectInput{
		Bucket: aws.String(storage.bucket),
		Key:    aws.String(storage.storageDir + filePath),
	}

	output, err := storage.client.HeadObject(input)
	if err != nil {
		if e, ok := err.(awserr.RequestFailure); ok && (e.StatusCode() == 403 || e.StatusCode() == 404) {
			return false, false, 0, nil
		} else {
			return false, false, 0, err
		}
	}

	if output == nil || output.ContentLength == nil {
		return false, false, 0, nil
	} else {
		return true, false, *output.ContentLength, nil

Please describe what you expect to happen (but doesn’t):

I would expect to see an error message that indicates insufficient permissions - I would return the error in the first if-branch above.

Please describe what actually happens (the wrong behaviour):

I saw the error message that a certain chunk could not be found.

As this is clearly an intended behavior and I’m presumably missing something here, so I’d be glad if you could elaborate why the 403 is caught that way.

Thanks for your work

gchen · 9 December 2020 00:51

It does look like it is unnecessary to trap the 403 error, but we need to make sure that no other corner cases are affected. I’ll do some testing to confirm it