Google Cloud Storage SignedURL with Cloud Run, Cloud Functions and GCE VMs

2021-12-15

Code snippet to create a GCS Signed URL in Cloud Run, Cloud Functions and GCE VMs

  • Why am i writing this repo?
    because it isn’t clear that in those environment that with some languages you can “just use” the default credentials (node, java, go) if and only if you enabled service account impersonation.

  • Whats wrong with the Documented samples for signedURL?
    They use service account keys.. don’t do that!

  • Why Impersonation?
    Well, Cloud Run, Cloud Functions and GCE environments do not have anyway to sign anything (and no, do NOT embed a service account key file anywhere!). Since those environments can’t sign by themselves, they need to use an API to sign on behalf of itself. That API is is listed above

  • Why are they different in different languages?
    java, node and go (now) automatically detects that its running in a Cloud Run|GCE and “knows” it can’t sign by itself and instead attempts to use impersonation automatically. In python, all the examples i saw around incorrectly uses the wrong Credential type to sign.

Impersonated signing uses (iamcredentials.serviceAccounts.signBlob.

The following examples uses the ambient service account to sign (eg, the service account cloud run uses is what signs the URL). It would be a couple more steps to make cloud run sign on behalf of another service account (and significantly more steps for java and node that made some assumptions on your behalf already)

Finally, if you must use a key, try to embed it into hardware, if possible.

you’ll find the source here

Setup

export PROJECT_ID=`gcloud config get-value core/project`
export PROJECT_NUMBER=`gcloud projects describe $PROJECT_ID --format="value(projectNumber)"`
export BUCKET_NAME=crdemo-$PROJECT_NUMBER
export SA_EMAIL=$PROJECT_NUMBER-compute@developer.gserviceaccount.com

gsutil mb gs://$BUCKET_NAME

echo foo > file.txt
gsutil cp file.txt gs://$BUCKET_NAME

# allow cloud run's default service account access
gsutil acl ch -u $SA_EMAIL:R gs://$BUCKET_NAME/file.txt

gcloud iam service-accounts  add-iam-policy-binding   --role=roles/iam.serviceAccountTokenCreator  \
 --member=serviceAccount:$SA_EMAIL $SA_EMAIL

gcloud config set run/region us-central1

Then build and push the language your’e interested in below

docker build -t gcr.io/$PROJECT_ID/crsigner .
docker push gcr.io/$PROJECT_ID/crsigner

gcloud run deploy signertest --image gcr.io/$PROJECT_ID/crsigner --platform=managed --set-env-vars="BUCKET_NAME=$BUCKET_NAME,SA_EMAIL=$SA_EMAIL"

export CR_URL=`gcloud run services describe  signertest --format="value(status.url)"`

curl -s $CR_URL

curl -s `curl -s $CR_URL`

If you are running in interactive mode in a GCE VM and have gsutil handy, first enable impersonation on the service account and then on the vm, use the -i switch as shown here:

$ gsutil -i $PROJECT_NUMBER-compute@developer.gserviceaccount.com signurl \
   -d 10m -u  gs://your-bucket/foo.txt

google-auth python offers two signer interfaces you can use:

You might be wondering why an IDToken credentials has a signer? Well, thats a side effect of an incorrect initial implementation of the compute engine ID Token (see issue #344). The interface users should use is impersonated_credential


import os

from flask import Flask

import google.auth
from google.auth import impersonated_credentials
from google.auth import compute_engine

from datetime import datetime, timedelta
from google.cloud import storage

app = Flask(__name__)


@app.route("/")
def hello_world():
    bucket_name = os.environ.get("BUCKET_NAME")
    sa_email =  os.environ.get("SA_EMAIL")


    credentials, project = google.auth.default()   

    storage_client = storage.Client()
    data_bucket = storage_client.bucket(bucket_name)
    blob = data_bucket.get_blob("file.txt")
    expires_at_ms = datetime.now() + timedelta(minutes=30)
    signing_credentials = impersonated_credentials.Credentials(
        source_credentials=credentials,
        target_principal=sa_email,
        target_scopes = 'https://www.googleapis.com/auth/devstorage.read_only',
        lifetime=2)

    # using the default credentials in the blob.generate_signed_url() will not work
    # signed_url = blob.generate_signed_url(expires_at_ms, credentials=credentials)
    signed_url = blob.generate_signed_url(expires_at_ms, credentials=signing_credentials)

    return signed_url, 200, {'Content-Type': 'text/plain'}


if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

In golang, we’re using the IAMCredentials api to sign the bytes.

After PR 4604 was merged, this is done automatically if you are using

	storageClient, _ := storage.NewClient(ctx)
	s, _ := storageClient.Bucket(bucketName).SignedURL(objectName, &storage.SignedURLOptions{
		Method:  http.MethodGet,
		Expires: expires,
	})

but if you use

See issues/1495#issuecomment-915514120

The full source

  • main.go
	ctx := context.Background()
	rootTokenSource, _ := google.DefaultTokenSource(ctx, "https://www.googleapis.com/auth/iam")
	delegates := []string{}
	s, _ := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
		Scheme:         storage.SigningSchemeV4,
		GoogleAccessID: serviceAccountName,
		SignBytes: func(b []byte) ([]byte, error) {
			client := oauth2.NewClient(context.TODO(), rootTokenSource)
			service, err := iamcredentials.New(client)
			if err != nil {
				return nil, fmt.Errorf("storage: Error creating IAMCredentials: %v", err)
			}
			signRequest := &iamcredentials.SignBlobRequest{
				Payload:   base64.StdEncoding.EncodeToString(b),
				Delegates: delegates,
			}
			name := fmt.Sprintf("projects/-/serviceAccounts/%s", serviceAccountName)
			at, err := service.Projects.ServiceAccounts.SignBlob(name, signRequest).Do()
			if err != nil {
				return nil, fmt.Errorf("storage: Error calling iamcredentials.SignBlob: %v", err)
			}
			sDec, err := base64.StdEncoding.DecodeString(at.SignedBlob)
			if err != nil {
				return nil, fmt.Errorf("storage: Error decoding iamcredentials.SignBlob response: %v", err)
			}
			return sDec, nil
		},
		Method:  http.MethodGet,
		Expires: expires,
	})

for java, first build

mvn clean install

then build the docker image, push to gcr then deploy to cloud run

package com.test;

import java.io.IOException;
import java.net.URL;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;

import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.ImpersonatedCredentials;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.HttpMethod;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.Storage.SignUrlOption;
import com.google.cloud.storage.StorageOptions;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;


@SpringBootApplication
public class TestApp {

  @Controller
  class HelloworldController {
    @RequestMapping(value="/", produces = MediaType.TEXT_PLAIN_VALUE)
	@ResponseBody
    String hello() {

		String bucketName = System.getenv("BUCKET_NAME");
		String saEmail = System.getenv("SA_EMAIL");
		String objectName = "file.txt";


		String signedURLString = "";
		GoogleCredentials sourceCredentials;
		try {
			sourceCredentials =  GoogleCredentials.getApplicationDefault();

			// you don't need to explicitly use the impersonatedcredentials class in java; the standard defaults detects the env and uses it automatically
			//ImpersonatedCredentials targetCredentials = ImpersonatedCredentials.create(sourceCredentials, saEmail, null,Arrays.asList("https://www.googleapis.com/auth/devstorage.read_only"), 2);
			//Storage storage_service = StorageOptions.newBuilder().setCredentials(targetCredentials).build().getService();    

			
			Storage storage_service = StorageOptions.newBuilder().setCredentials(sourceCredentials).build().getService();      
			
			BlobId blobId = BlobId.of(bucketName, objectName);
			BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
			URL signedUrl = storage_service.signUrl(blobInfo, 600,  TimeUnit.SECONDS, 	SignUrlOption.httpMethod(HttpMethod.GET));
			signedURLString = signedUrl.toExternalForm();
	
		} catch (IOException ioex) {
			return "Error " + ioex.getMessage();
		}


      return signedURLString;
    }
  }

  public static void main(String[] args){
    SpringApplication.run(TestApp.class, args);
  }
}

Nodejs its pretty easy since by default, the library automatically tries to use IAMCredentials API in these environments;

see bucket.getSignedUrl()

In Google Cloud Platform environments, such as Cloud Functions and App Engine, 
you usually don't provide a keyFilename or credentials during instantiation. In those environments, we call the signBlob API

However, my preference would’ve been to make it explicit applied which would also allow you to set a different account to sign with. For example, like in the sample below using the IAM google-auth-library-nodejs#impersonated-credentials-client

see #1443 Impersonated credentials should implement sign() capability

const { GoogleAuth, Impersonated } = require('google-auth-library');
const { Storage } = require('@google-cloud/storage');

async function main() {
  
  const scopes = 'https://www.googleapis.com/auth/cloud-platform'
  // get source credentials
  const auth = new GoogleAuth({
    scopes: scopes
  });
  const client = await auth.getClient();

  // First impersonate
  let targetPrincipal = 'target-serviceaccount@fabled-ray-104117.iam.gserviceaccount.com'
  let targetClient = new Impersonated({
    sourceClient: client,
    targetPrincipal: targetPrincipal,
    lifetime: 30,
    delegates: [],
    targetScopes: [scopes]
  });

  // test sign
  //console.log(await targetClient.sign("foo"));

  let projectId = 'fabled-ray-104117'
  let bucketName = 'fabled-ray-104117-test'

  // use the impersonated creds to bootstrap a storage client
  const storageOptions = {
    projectId,
    authClient: targetClient,
  };
  const storage = new Storage(storageOptions);

  // ####### GCS SignedURL
  const signOptions = {
    version: 'v4',
    action: 'read',
    expires: Date.now() + 15 * 60 * 1000, // 15 minutes
  };

  const  su = await storage
    .bucket(bucketName)
    .file('foo.txt')
    .getSignedUrl(signOptions);
  console.log(su);
}
main().catch(console.error);

Unfortunately, google-cloud node js auth storage library does not support signing using impersonation with signer

(contributions welcome)

Use Workload Identity

export PROJECT_ID=`gcloud config get-value core/project`
export PROJECT_NUMBER=`gcloud projects describe $PROJECT_ID --format='value(projectNumber)'`

gcloud iam service-accounts create ocsp-svc
gcloud  container clusters create cluster-1 --workload-pool=$PROJECT_ID.svc.id.goog

kubectl create namespace ns1
kubectl create serviceaccount  --namespace ns1  ksa-1

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.serviceAccountTokenCreator \
    --member "serviceAccount:$PROJECT_ID.svc.id.goog[ns1/ksa-1]" \
    ocsp-svc@$PROJECT_ID.iam.gserviceaccount.com


gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.serviceAccountTokenCreator \
    --member "serviceAccount:ocsp-svc@$PROJECT_ID.iam.gserviceaccount.com" \
    ocsp-svc@$PROJECT_ID.iam.gserviceaccount.com

gsutil mb  gs://$PROJECT_ID-cab1/
echo -n foo >/tmp/foo.txt
gsutil cp /tmp/foo.txt gs://$PROJECT_ID-cab1/

gsutil iam ch  serviceAccount:ocsp-svc@$PROJECT_ID.iam.gserviceaccount.com:objectViewer      gs://$PROJECT_ID-cab1/


kubectl annotate serviceaccount --namespace ns1 ksa-1 iam.gke.io/gcp-service-account=ocsp-svc@$PROJECT_ID.iam.gserviceaccount.com


cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: ns1
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: ksa-1
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: "true"
EOF



kubectl get po -n ns1
  NAME                     READY   STATUS    RESTARTS   AGE
  workload-identity-test   1/1     Running   0          3m28s


kubectl exec -it workload-identity-test \
    --namespace ns1 \
    -- gsutil -i ocsp-svc@$PROJECT_ID.iam.gserviceaccount.com signurl -r us -d 10m -u  gs://$PROJECT_ID-cab1/foo.txt

This site supports webmentions. Send me a mention via this form.