Limiting file-size and Content using GCS SignedURL v4

2022-03-18

I was looking into GCS SignedURLs yesterday for ways to limit the amount of data a SignedURL could upload.

I wasn’t particularly optimistic since its I understood SignedURLs to just say “ok, if you have this URL, you are free to upload whatever you want to this file destination”. Meaning, if you give someone a signedURL, they can upload 5MB or 5GB or whatever.

While Signed Policy Documents do allow some ways to set the size and type of files to upload, that mechanism isn’t generally encouraged and (IMO) there just to support/parity with other storage providers for browser-form uploads.

The better way to is to use the well-known Signed URL.

I was mostly sort-of familiar with the v2 signing process and just couldn’t think of a way to limit the file that was actually uploaded thinking that was out of the control of the issuer.

I began trying stuff out and noticed the v4Signer in go had the ability to set Header values.

So..i thought these headers must get honored when you use a PUT request. If thats possible, then i can sort of control the size of the file.

From there a bit of fiddling and there it is…i found out you can generate a SignedURL, limit the total size…and this shows how to do that…

but wait, this’ll also show how to also limit the upload to a known content! yeah, i can create a signed url, set its limit or set the hash of the file that will ultimately get uploaded.

image from: https://github.com/nijikokun/file-size

So..while I started writing about this, i looked around a bit more and found out Nacho Coloma already did something around this in Limit the size of files uploaded with Signed URLs on Google Cloud Storage

note to self: very few thing you do is really ever that unique

However, I think there’s a minor bit to correct in that article above: you can actually sign a PUT url and set the Content-Length it will use. The article above states you can’t but..i happened to stumble on it earlier.

So, this article is about the same but with a spin on golang and addresses a couple of things:

  1. Generate a Signed URL that ONLY allows for 10bytes AND the file that has content MD5hash of eB5eJF1ptWaXm4bijSPyxw==

    Once the signedURL is created, upload it with a PUT. (and yes, i didn’t need to set the size limit if i could just set the hash..)

  2. Generate a Resumable Signed URL that only allows for 10bytes.

    Once the signedURL is created, upload the file to the Location URL that is provided.

for other background reading for signedURLs

Anyway,

Setup

Create a bucket, service account:

export PROJECT_ID=`gcloud config get-value core/project`
export PROJECT_NUMBER=`gcloud projects describe $PROJECT_ID --format='value(projectNumber)'`

gcloud iam service-accounts create urlsigner --display-name="GCS URL Signer" --project=${PROJECT_ID}
gcloud iam service-accounts keys  create service_account.json --iam-account=urlsigner@${PROJECT_ID}.iam.gserviceaccount.com
gsutil mb gs://$PROJECT_ID-urlsigner

gsutil iam ch  serviceAccount:urlsigner@${PROJECT_NAME}.iam.gserviceaccount.com:roles/storage.admin gs://$PROJECT_ID-urlsigner

Now, what the demo shows is:

  • given file contents denoted by their byte count and md5hash

    const (
    fiveBytes        = "01234"   // md5hash QQDE1E2pF3JH5EpfwVRneA==
    tenBytes         = "0123456789"  // md5hash eB5eJF1ptWaXm4bijSPyxw==
    tenBytesReversed = "9876543210"  // md5hash 44jBxd9JM/oB9tqfkllViQ==
    twentyBytes      = "01234567890123456789"  // md5hash vkl8IWjjdPQUo1HEk3nAGg==
)

  • generate a non-resumable SignedURL that allows PUT for a file with Content-Length: 10 AND whose md5=eB5eJF1ptWaXm4bijSPyxw==

    which means only the tenBytes file can be uploaded

  • attempt uploading each of the files and see only tenBytes succeed

  • then generate a resumable SignedURL that allows only files of Content-Length: 10

  • exchange the signedURL for the upload Location.

  • attempting to use that signedURL to upload a file of any other content size will fail to issue the Location

  • attempt uploaded each of the files and see only tenBytes and tenBytesReversed succeed.

To see the program run:


export GOOGLE_APPLICATION_CREDENTIALS=`pwd`/service_account.json

go run main.go --bucketName $PROJECT_ID-urlsigner

______________________ upload _________________
SignedURL= https://storage.googleapis.com/fabled-ray-104117-urlsigner/file.txt?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=urlsigner%40fabled-ray-104117.iam.gserviceaccount.com%2F20220318%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220318T032746Z&X-Goog-Expires=9&X-Goog-Signature=4f38dcdf8551c6789f290eb8fbb9a735802fc808c1aca258624e0a438b6b3825c09c80740e15b16eac6215fbe6e6b5d6553141690b09cefb8b74c2fc6f7de5290e99da50db3e6ad3ff212235574ad7a68cd7986283c75eaa8eaf36f315306d2c9ba6f579efd19e7b391d237acae67996cdba4a70e09c5607190000e1a823c09c202ccf3379987c51ed4fe81236ab4da9fca768133873d323cd03be579a2304d5cb391ac3f54e9ee551480e190d0df45ef58ce7babb7d087e7c75824281423015b2ea07af3df2bbee932d01c58e64243b9cb706301d4dd7e3d86f812482936c8aaf2949c0013ec8facd40616cdf4e094abcdef0f51e7d2b75cddea7995383bb6265&X-Goog-SignedHeaders=content-length%3Bcontent-md5%3Bcontent-type%3Bhost
UploadStatus for file of size [5] with hash [QQDE1E2pF3JH5EpfwVRneA==]  403 Forbidden
UploadStatus for file of size [10] with hash [eB5eJF1ptWaXm4bijSPyxw==]  200 OK
UploadStatus for file of size [10] with hash [44jBxd9JM/oB9tqfkllViQ==]  403 Forbidden
UploadStatus for file of size [20] with hash [vkl8IWjjdPQUo1HEk3nAGg==]  403 Forbidden


______________________ resumable _________________
SignedURL= https://storage.googleapis.com/fabled-ray-104117-urlsigner/file.txt?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=urlsigner%40fabled-ray-104117.iam.gserviceaccount.com%2F20220318%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220318T032746Z&X-Goog-Expires=9&X-Goog-Signature=3c22421aa81a4b6692aead06a1c2e8815fc59640f09e70139e475fd13a9da0674f92d1c30c53b9b50007e21d68a18bf5f8d55669eab0bef499756500bd14e44fd2dd6bc18c0e9b0ec3e17d07588b93ef73f5a8826ba299a9c75d0432c873c02661659779e09d7f64e955de3987f3d38e74a16abfcaaac09870f6d00ab84c8a51327fd68c667e5881a6d0f4250b0a79ccfe93447332b42574ae4621e78107c631250feb214c924a3f1cc37e36884fe88da004ee35cf6bb0306a1510eac7b3a60ba18f550e0f4a02d5b1d90adf885434126e1219338fcc640e299ed495bcd11232f29cb073293fad74dabcdefghijklmnop250065e37a966b1c391297527dd5f61&X-Goog-SignedHeaders=content-type%3Bhost%3Bx-goog-resumable%3Bx-upload-content-length
Could not generate resumable location for size [5]  403 Forbidden
UploadStatus for file of size [10]  200 OK
UploadStatus for file of size [10]  200 OK
Could not generate resumable location for size [20]  403 Forbidden

What the above shows is files on the upload only tenBytes succeeded (since the file size and hash need to match)

The resumable section shows both tenBytes and tenBytesReversed succeeded since they both share the same size

So…i just happened to notice this capability and I do know customers time to time ask about how to limit sizes during uploads with SignedURL

This site supports webmentions. Send me a mention via this form.