Cross Project Service Accounts on Google Cloud

2021-12-16

While i was chatting with a co-worker yesterday, he mentioned customers and even internal teams often run into issues when using Cross Project Service Accounts. I’m familiar Google Cloud IAM but this one important feature was lacking.

What does this do? Well until recently, you could not create a GCP resource (like a VM) that that used a service account from another project.

Why would you want to do that? Maybe your cloud organization wants to manage all service accounts in one central project and tightly control which resources can assume any given service account.

Lets show how this works…conceptually, VM is easiest to deal with.

Here’s the layout

You have

  • Project A has a VM which needs to run as Service Account B that ‘resides’ in Project B.

The critical think to know here is that this capability is gated by a project-level Organization policy called constraints/iam.disableCrossProjectServiceAccountUsage that needs to get set to Not Enforced (yes, i always get confused with negatives and double negatives)

The first step is to not enforce this Policy on ProjectB

images/cross_project_policy.png

Once thats done, the owner of ProjectB needs to grant the SERVICE AGENT in Project A permissions to impersonate Service Account B.

Wait…whats a service agent? Well, think of it as a default identity intrinsic and unique to each project and service that internally manages any credentials a resource can acquire.

You can find a list of all the service here

It took me sometime to interpret that…the actual agent is formatted as

  • service-{PROJECT_NUMBER_A}@DOMAIN

where the domain is service specific.

Compute Engine

For compute its @compute-system.iam.gserviceaccount.com. For cloud run its @serverless-robot-prod.iam.gserviceaccount.com

In our case we need to assign the Compute Engine Service Agent in Project A permissions to issue the credentials for Service Account B (i.,e ProjectA’s Compute Engine Service Agent needs the roles/iam.serviceAccountTokenCreator granted on Service Account B)

Finally, you can create a VM in project A and assign it to use service account B.

The iam.serviceAccountTokenCreator role includes concentric set of permission (see Concentric IAMCredentials Permissions: The secret life of signBlob. If you just need to generate an access_token or an id_token, consider generating a custom role with that specific capability.

For more information, see Enabling service account impersonation across projects

The full flow is shown here

images/cross_project_sa.png

As a concrete example

First just try to create a VM with a cross project SA:

gcloud compute instances create cross-project  \
  --zone=us-central1-a --machine-type=e2-micro  \
  --service-account=SVC_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com  \
  --scopes=email,openid,https://www.googleapis.com/auth/cloud-platform \
  --image=debian-11-bullseye-v20211105 --image-project=debian-cloud --project PROJECT_A

If you ssh into the VM and attempt to get a token, you’ll see

$ curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token"
    
      "Service account is invalid/disabled or not allowed to be used on this VM due to project or organization policies"

So…we need to take some steps

Project_A’s service agent for compute engine per Compute Engine Service Agent is in the form

  • service-{PROJECT_NUMBER_FOR_A}@compute-system.iam.gserviceaccount.com

Then in projectB, disable the policy constraints/iam.disableCrossProjectServiceAccountUsage

Assume the service account in B to impersonate is svcAccountB@{PROJECT_B}.iam.gserviceaccount.com

Grant the service Agent from A permissions on svcB:

gcloud iam service-accounts add-iam-policy-binding \
   SVC_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com  \
    --member=serviceAccount:service-PROJECT_NUMBER_FOR_A@compute-system.iam.gserviceaccount.com \
    --role=roles/iam.serviceAccountTokenCreator --project PROJECT_B

now, ssh in and see who the vm runs as

$ curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"

SVC_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com

Cloud Run

$ gcloud run deploy cross-project --image  gcr.io/PROJECT_A/iaprun --project PROJECT_A --service-account SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com 

# if you initially see                                                  
ERROR: (gcloud.run.deploy) User [user@domain.com] does not have permission to access namespaces instance [projectA] (or it may not exist): Permission 'iam.serviceaccounts.actAs' denied on service account SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com (or it may not exist).

# grant the deployer user@domain.com the  `Service Account User` role to 

## Then if you deploy and see,

ERROR: (gcloud.run.deploy) User [user@domain.com] does not have permission to access namespaces instance [PROJECT_A] (or it may not exist): Google Cloud Run Service Agent does not have permission to get access tokens for the service account SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com. Please give service-PROJECT_A_NUMBER@serverless-robot-prod.iam.gserviceaccount.com permission iam.serviceAccounts.getAccessToken on the service account. Alternatively, if the service account is unspecified or in the same project you are deploying in, ensure that the Service Agent is assigned the Google Cloud Run Service Agent role roles/run.serviceAgent.

# You need to grant PROJECT_A's Serverless Service Agent the iam.TokenCreator role to SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com

## The serverless agent for Cloud RUn is:
service-PROJECT_A_NUMBER@serverless-robot-prod.iam.gserviceaccount.com

$ gcloud run services describe cross-project --project PROJECT_A

Last updated on 2021-12-17T01:12:11.551963Z by user@domain.com:
  Revision cross-project-00001-gih
  Image:           gcr.io/PROJECT_A/iaprun
  Port:            8080
  Memory:          512Mi
  CPU:             1000m
  Service account: SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com
  Concurrency:     80
  Max Instances:   100
  Timeout:         300s

GKE

For GKE, you seem to need some unique permission sets unlike what was done for Cloud Run and GCE:

First enable org policy

Using SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com

grant iam.serviceAccountUser TO

  • service-PROJECT_A_NUMBER@container-engine-robot.iam.gserviceaccount.com
  • PROJECT_A_NUMBER@cloudservices.gserviceaccount.com

grant iam.serviceAccountTokenCreator TO

  • service-PROJECT_A_NUMBER@compute-system.iam.gserviceaccount.com

then

gcloud beta container \
  --project "PROJECT_A" clusters create "cluster-1" \
  --service-account "SERVICE_ACCOUNT_B@PROJECT_B.iam.gserviceaccount.com"

Troubleshooting

Note, you may see the following error even if you have the org policy set but not the impersonation!

"Service account is invalid/disabled or not allowed to be used on this VM due to project or organization policies"

This site supports webmentions. Send me a mention via this form.