just sal's blog > Articles > 2021 > Google Cloud VPC-SC basic ingress and egress rules

gcp

Google Cloud VPC-SC basic ingress and egress rules

2021-12-22

Simple example of using GCP VPC-SC perimeter setup with Ingress and Egress Rules

The ingress/egress terms frequently cause confusion.

It has nothing to do with direction of data flow traffic (i.,e which way the bytes flow when copying a file in a gcs bucket) but rather context about which project is inside and outside and who is calling who:

ingress = Project outside VPC-SC accessing project inside
egress = Project inside VPC-SC accessing outside project

TOC:

Setup
Extend VPCSC perimeter to IP address
Include project in VPCSC perimeter
Ingress: Access data FROM outside perimeter to inside
Egress: Access data FROM inside perimeter to outside

This tutorial is a very simple setup that uses GCP VPC-SC to transfer files in a GCS bucket to and from a Project that is placed in a VPC-SC Perimeter.

While the documentation for this feature is sufficient, this tutorial is the basic “hello world” of VPC-SC and ingress/egress rules.

This is what the tutorial shows

A) Create four projects:

A: project to hold the GCS files secured inside VPC-SC
B: project to hold a VM which we will place inside the VPC-SC
C: project outside the VPC SC which we will transfer a file FROM a bucket to project A (i.e, ingress rule)
D: project outside the VPC SC which we will transfer a file TO project A (i.e., egress rule)

(you ofcourse don’t need to create projects to follow this, just read the procedure and steps, thats all)

The specific projects and projects numbers used in the example below are:

A) gcs-project-332821: 247987397047
B) in-perimeter-gcs: 248336178977
C) ingress-vpcsc: 715952372567
D) egress-vpcsc: 1099154108957

We will also demonstrate extending the VPC-SC to a single external IP (in this case, my home’s wifi’s external IP; my laptop, essentially)

For that my fake home perimeter IP address 111.83.67.174/32

Note, in my test domain, admin@domain.com is the Org Admin and user4@domain.com is a regular user. We will use user4’s identity as a qualifier in the VPC-SC rule

To use this tutorial you ofcourse need to be an org admin to define the VPC-SC rules but even if you are not, you can inspect how this all works

Setup

## create environment variables
### ofcourse in your case use your own projects
export VPC_SC_PROJECT=gcs-project-332821
export VPC_SC_PROJECT_IN_PERIMETER=in-perimeter-gcs 
export INGRESS_PROJECT=ingress-vpcsc
export EGRESS_PROJECT=egress-vpc

## enable services
gcloud services enable storage.googleapis.com --project $VPC_SC_PROJECT 
gcloud services enable storage.googleapis.com compute.googleapis.com --project $VPC_SC_PROJECT_IN_PERIMETER
gcloud services enable storage.googleapis.com compute.googleapis.com --project $INGRESS_PROJECT
gcloud services enable storage.googleapis.com compute.googleapis.com --project $EGRESS_PROJECT

## Get project numbers
export VPC_SC_PROJECT_NUMBER=`gcloud projects describe $VPC_SC_PROJECT --format='value(projectNumber)'`
export VPC_SC_IN_PERIMETER_PROJECT_NUMBER=`gcloud projects describe $VPC_SC_PROJECT_IN_PERIMETER --format='value(projectNumber)'`
export INGRESS_PROJECT_NUMBER=`gcloud projects describe $INGRESS_PROJECT --format='value(projectNumber)'`
export EGRESS_PROJECT_NUMBER=`gcloud projects describe $EGRESS_PROJECT --format='value(projectNumber)'`

## create test buckets
gsutil mb -p $VPC_SC_PROJECT gs://$VPC_SC_PROJECT-bucket 
gsutil mb -p $INGRESS_PROJECT gs://$INGRESS_PROJECT-bucket 
gsutil mb -p $EGRESS_PROJECT gs://$EGRESS_PROJECT-bucket 

## assign permissions to read/write to appropriate buckets
## What these steps allows is "allow user4@ permissions to read from the VPC-SC bucket 
gsutil iam ch user:"user4@esodemoapp2.com":objectViewer   gs://$VPC_SC_PROJECT-bucket

# this allows for the VM that will eventually be inside the perimeter to Egress to a bucket outside
gsutil iam ch serviceAccount:"$VPC_SC_IN_PERIMETER_PROJECT_NUMBER-compute@developer.gserviceaccount.com":objectAdmin  gs://$EGRESS_PROJECT-bucket


## Ingress
# this allows the VM in the Ingress project to read a gcs object within the ingress project
gsutil iam ch serviceAccount:"$INGRESS_PROJECT_NUMBER-compute@developer.gserviceaccount.com":objectViewer  gs://$INGRESS_PROJECT-bucket

# this allows the vm in perimeter (allows vm added into the perimeter access to write to the vpc-sc protected bucket)
gsutil iam ch serviceAccount:"$VPC_SC_IN_PERIMETER_PROJECT_NUMBER-compute@developer.gserviceaccount.com":objectViewer   gs://$VPC_SC_PROJECT-bucket

# ingress rules permissions on buckets.  This allows a VM in the 'ingress' project to write a file within the vpc-sc  project's bucket (used for ingress vm=>bucket transfer)
gsutil iam ch serviceAccount:"$INGRESS_PROJECT_NUMBER-compute@developer.gserviceaccount.com":objectAdmin  gs://$VPC_SC_PROJECT-bucket

## Egress
# egress rules permissions on buckets to read from VPC_SC_PROJECT and write to EGRESS_PROJECT_NUMBER
gsutil iam ch serviceAccount:"$EGRESS_PROJECT_NUMBER-compute@developer.gserviceaccount.com":objectViewer  gs://$VPC_SC_PROJECT-bucket
gsutil iam ch serviceAccount:"$EGRESS_PROJECT_NUMBER-compute@developer.gserviceaccount.com":objectAdmin gs://$EGRESS_PROJECT-bucket


## upload test files
echo -n foo > foo.txt
echo -n bar > bar.txt

gsutil cp foo.txt  gs://$VPC_SC_PROJECT-bucket/foo.txt
gsutil cp bar.txt  gs://$INGRESS_PROJECT-bucket/bar.txt

## set IAM policy to create a VM inside the ingress project
gcloud iam service-accounts add-iam-policy-binding  \
    $INGRESS_PROJECT_NUMBER-compute@developer.gserviceaccount.com  \
    --member=user:`gcloud config get-value core/account` \
    --role='roles/iam.serviceAccountUser' --project $INGRESS_PROJECT

gcloud compute instances create ingress \
   --zone=us-central1-a --machine-type=e2-medium \
   --service-account=$INGRESS_PROJECT_NUMBER-compute@developer.gserviceaccount.com \
   --scopes=https://www.googleapis.com/auth/cloud-platform \
   --image=debian-11-bullseye-v20211105 --image-project=debian-cloud --project $INGRESS_PROJECT

Extend VPCSC perimeter to IP address

This will extend the VPCSC boundary to a single IP address (eg, the ip of my home network)

images/vpcsc_1.png

Given the initial policy

## Create policy.  
### first get the orgID
$ gcloud organizations list
DISPLAY_NAME               ID  DIRECTORY_CUSTOMER_ID
domain.com  673208786098              C023zw3x8

gcloud access-context-manager policies create --organization 673208786098 --title ip_trust_policy

$ gcloud access-context-manager policies list --organization 673208786098
NAME          ORGANIZATION  TITLE            ETAG
157300459203  673208786098  ip_trust_policy  f45c2de903c228d7

## create a basic policy with a fake IP address (so that we can see it fail)
gcloud access-context-manager levels create \
   accessPolicies/157300459203/accessLevels/gcs_policy --title vpc_gcs_allow --basic-level-spec=basic_level.yaml


$  gcloud access-context-manager levels describe gcs_policy
basic:
  conditions:
  - ipSubnetworks:
    - 1.2.3.4/32
name: accessPolicies/157300459203/accessLevels/gcs_policy
title: vpc_gcs_allow


$ gcloud access-context-manager levels list
NAME               TITLE              LEVEL_TYPE
gcs_policy         vpc_gcs_allow      Basic

images/fake_policy.png

Finally, create a perimeter using that policy

$ gcloud access-context-manager perimeters describe restrictGCS
    name: accessPolicies/157300459203/servicePerimeters/restrictGCS
    status:
      accessLevels:
      - accessPolicies/157300459203/accessLevels/gcs_policy
      resources:
      - projects/247987397047
      restrictedServices:
      - storage.googleapis.com
      vpcAccessibleServices:
        enableRestriction: true
    title: restrictGCS

images/basic_policy.png

Now, on your laptop at home, try to copy a file as user4:

$ gcloud config set account user4@domain.com

This will fail since access_context=gcs_policy contains a fake IP ip=1.2.3.4/32

$ gsutil cp  gs://$VPC_SC_PROJECT-bucket/foo.txt .
    AccessDeniedException: 403 SecurityPolicyViolated
    >Request is prohibited by organizations policy. vpcServiceControlsUniqueIdentifier: RTSWq5KzpbOOQHU5RUmqH4ipQtRtmsuQDDT2eqIxqMWIQivq0hejHA</Details></Error>

Now update access_contenxt=gcs_policy to contain your actual IP address. In my case, my home IP was ip=111.83.67.174/32

$ gcloud access-context-manager levels describe gcs_policy
basic:
  conditions:
  - ipSubnetworks:
    - 111.83.67.174/32
name: accessPolicies/157300459203/accessLevels/gcs_policy
title: vpc_gcs_allow

images/vpc_sc_home.png

Now your laptop is part of the VPC-SC perimeter. Try copying a file

$ gsutil cp  gs://$VPC_SC_PROJECT-bucket/foo.txt .
  Copying gs://gcs-project-332821-bucket/foo.txt...
  / [1 files][    3.0 B/    3.0 B]                                                
  Operation completed over 1 objects/3.0 B.

Include project in VPCSC perimeter

This will add an entire project inside the perimeter and perform copy of objects within it

images/vpcsc_2.png

Now try creating a VM in project in-perimeter-gcs. Note this project is actually outside outside the perimeter (since we haven’t done anyting yet!)

# as admin
$ gcloud compute instances create in-perimeter \
   --zone=us-central1-a --machine-type=e2-medium \
   --service-account=$VPC_SC_IN_PERIMETER_PROJECT_NUMBER-compute@developer.gserviceaccount.com \
   --scopes=https://www.googleapis.com/auth/cloud-platform \
   --image=debian-11-bullseye-v20211105 --image-project=debian-cloud --project $VPC_SC_PROJECT_IN_PERIMETER

Remember, we already gave IAM permissions to this service account access to read a file in the VPC-SC bucket

SSH into that VM

gcloud compute ssh in-perimeter --project $VPC_SC_PROJECT_IN_PERIMETER

$ gcloud config list
  [core]
  account = 248336178977-compute@developer.gserviceaccount.com
  project = in-perimeter-gcs



## Try to access the bucket
## this will fail because the project is not in perimeter yet
gsutil cp  gs://$VPC_SC_PROJECT-bucket/foo.txt .
  AccessDeniedException: 403 Request is prohibited by organizations policy. vpcServiceControlsUniqueIdentifier: 9xVUHOqTLcENf1qtY7wUSJ1VlrQlCtGbDMZUbQbmFJL71WuvhxGYCw

So Now add the project into the VPC-Perimeter AND allow GCS as a within vpc accessible service:

NOTE that we removed the access_context rule from above just for clarity

gcloud access-context-manager perimeters describe restrictGCS --policy 157300459203

name: accessPolicies/157300459203/servicePerimeters/restrictGCS
status:
  resources:
  - projects/247987397047
  - projects/248336178977
  restrictedServices:
  - storage.googleapis.com
  vpcAccessibleServices:
    allowedServices:
    - storage.googleapis.com
    enableRestriction: true
title: restrictGCS

(note the entry for vpcAccessibleServices)

images/in_perimeter_projects.png

(if you did not include GCS as an vpcAccessibleServices, you would see SERVICE_NOT_ALLOWED_FROM_VPC, error)

images/error_vpc_1.png

Now inside the VM, try copying.. This now works because Project B is now inside the perimeter and GCS API is allowed there (so is the home IP address shown below)

gsutil cp  gs://$VPC_SC_PROJECT-bucket/foo.txt .
  Copying gs://gcs-project-332821-bucket/foo.txt...
  / [1 files][    3.0 B/    3.0 B]                                                
  Operation completed over 1 objects/3.0 B.

Egress- Access data FROM inside perimeter to outside

Remember, egress rules defines a rule for a Project inside VPC-SC accessing outside project

Your a VM that we just added to the VPC-SC (the VM ), try to copy a file between buckets.

images/vpcsc_3.png

This will fails since although your laptop is inside the perimeter, the destination is not

gcloud compute ssh in-perimeter --project $VPC_SC_PROJECT_IN_PERIMETER

# then in the VM
$ gcloud config list
[core]
account = 248336178977-compute@developer.gserviceaccount.com
disable_usage_reporting = True
project = in-perimeter-gcs


$ gsutil cp  gs://$VPC_SC_PROJECT-bucket/foo.txt gs://$EGRESS_PROJECT-bucket

   AccessDeniedException: 403 Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: JPPkuXIwWrZIJcOL_F9WfAecuT9mJ8ONU71j09uTf9jJ-KlA25nN4A

The error shows RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER. Which means the we need to be able to copy an object FROM inside the perimeter (remember, the VM is in perimeter) to outside

images/error_vpc_2.png

So create Egress rule

gcloud access-context-manager perimeters describe restrictGCS

  name: accessPolicies/157300459203/servicePerimeters/restrictGCS
  status:
    egressPolicies:
    - egressFrom:
        identities:
        - serviceAccount:248336178977-compute@developer.gserviceaccount.com
      egressTo:
        operations:
        - methodSelectors:
          - method: '*'
          serviceName: storage.googleapis.com
        resources:
        - projects/1099154108957
    resources:
    - projects/247987397047
    - projects/248336178977
    restrictedServices:
    - storage.googleapis.com
    vpcAccessibleServices:
      allowedServices:
      - storage.googleapis.com
      enableRestriction: true
  title: restrictGCS

images/egress.png

That stipulates tht “serviceAccount:248336178977-compute@developer.gserviceaccount.com” (which is the VM running inside the perimeter) is allowed to egress a file from a client inside the perimeter (i.,e VPC_SC_PROJECT) from a protected bucket TO a project projects/247987397047 (in $EGRESS_PROJECT) that sits outside the perimeter)

So now try running the same command

$ gsutil cp  gs://$VPC_SC_PROJECT-bucket/foo.txt gs://$EGRESS_PROJECT-bucket
    Copying gs://gcs-project-332821-bucket/foo.txt [Content-Type=text/plain]...
    / [1 files][    3.0 B/    3.0 B]                                                
    Operation completed over 1 objects/3.0 B.

Ingress- Access data FROM outside perimeter to inside

Finally, we will try Ingress rules. These rules allow you to copy a file from somewhere outside the VPC-SC into the secure bucket.

remember, rules define Project outside VPC-SC accessing project inside

We will attempt two variations:

A. on a VM outside the perimeter, we will copy a file from a VM to a bucket within the VPC-SC by defining an ingress rule B. on a VM outside the perimeter, we will copy a file from a bucket within that project to a bucket within the VPC-SC

VM->Bucket

This will copy an object on a VM from a project outside the perimeter to inside

images/vpcsc_4.png

## first enter the VM inside the ingress project
$ gcloud compute ssh ingress --project $INGRESS_PROJECT

$ gcloud config list
  [core]
  account = 715952372567-compute@developer.gserviceaccount.com
  disable_usage_reporting = True
  project = ingress-vpcsc

## Try to copy
### this will fail since we dont' have a policy that exempts this project from the VPC-SC stipulations

$ echo -n "bar" > bar.txt
$ gsutil cp  bar.txt gs://$VPC_SC_PROJECT-bucket/
  AccessDeniedException: 403 Request is prohibited by organizations policy. vpcServiceControlsUniqueIdentifier: Aup5qJp7HU9va4bmDxazWo7FOGMMtPFhT4HI-Zjmtqh8PzSgmmjemw

The specific error here is NETWORK_NOT_IN_SAME_SERVICE_PERIMETER,

images/error_vpc_3.png

So add the VPC-SC rule for ingress that allows (please note i’ve omitted the previous configurations settings from the output below)

$ gcloud access-context-manager perimeters describe restrictGCS
    name: accessPolicies/157300459203/servicePerimeters/restrictGCS
    status:
      ingressPolicies:
      - ingressFrom:
          identities:
          - serviceAccount:715952372567-compute@developer.gserviceaccount.com
          sources:
          - resource: projects/715952372567
        ingressTo:
          operations:
          - methodSelectors:
            - method: '*'
            serviceName: storage.googleapis.com
          resources:
          - projects/247987397047
      resources:
      - projects/247987397047
      - projects/248336178977
      restrictedServices:
      - storage.googleapis.com
      vpcAccessibleServices:
        allowedServices:
        - storage.googleapis.com
        enableRestriction: true
    title: restrictGCS

images/ingress1.png

## now the copy works
gsutil cp  bar.txt gs://$VPC_SC_PROJECT-bucket/
    Copying file://bar.txt [Content-Type=text/plain]...
    / [1 files][    3.0 B/    3.0 B]                                                
    Operation completed over 1 objects/3.0 B.

Bucket->Bucket

For the second variation, attempt bucket-to-bucket transfer, from inside the egress project VM (outside the perimeter)

images/vpcsc_5.png

gsutil cp gs://$INGRESS_PROJECT-bucket/bar.txt gs://$VPC_SC_PROJECT-bucket/

    AccessDeniedException: 403 Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: fDcSFrzy6x63WZLBBz3C2wFXZRpPAngxasoNjJ9KKKSver8E36rMpw

hmm…i’m not sure why it did’t work…in looking at the log trace, doesn’t really say

images/error_vpc_4.png

the json is:

{
  "insertId": "10x36xne3ss7a",
  "logName": "projects/gcs-project-332821/logs/cloudaudit.googleapis.com%2Fpolicy",
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 7,
      "message": "Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: fDcSFrzy6x63WZLBBz3C2wFXZRpPAngxasoNjJ9KKKSver8E36rMpw",
      "details": [
        {
          "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
          "violations": [
            {
              "type": "VPC_SERVICE_CONTROLS",
              "description": "fDcSFrzy6x63WZLBBz3C2wFXZRpPAngxasoNjJ9KKKSver8E36rMpw"
            }
          ]
        }
      ]
    },
    "authenticationInfo": {
      "principalEmail": "715952372567-compute@developer.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "firstPartyPrincipal": {
            "principalEmail": "service-715952372567@compute-system.iam.gserviceaccount.com"
          }
        }
      ]
    },
    "requestMetadata": {
      "callerIp": "35.222.50.80",
      "callerNetwork": "//compute.googleapis.com/projects/ingress-vpcsc/global/networks/__unknown__",
      "requestAttributes": {},
      "destinationAttributes": {}
    },
    "serviceName": "storage.googleapis.com",
    "methodName": "google.storage.objects.create",
    "resourceName": "projects/247987397047",
    "metadata": {
      "@type": "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata",
      "resourceNames": [
        "projects/715952372567/buckets/ingress-vpcsc-bucket/objects/bar.txt",
        "projects/247987397047/buckets/gcs-project-332821-bucket/objects/bar.txt"
      ],
      "securityPolicyInfo": {
        "organizationId": "673208786098",
        "servicePerimeterName": "accessPolicies/157300459203/servicePerimeters/restrictGCS"
      },
      "violationReason": "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER",
      "vpcServiceControlsUniqueId": "fDcSFrzy6x63WZLBBz3C2wFXZRpPAngxasoNjJ9KKKSver8E36rMpw",
      "egressViolations": [
        {
          "targetResource": "projects/715952372567",
          "servicePerimeter": "accessPolicies/157300459203/servicePerimeters/restrictGCS",
          "source": "projects/247987397047",
          "sourceType": "Resource"
        }
      ]
    }
  },
  "receiveTimestamp": "2021-12-26T14:25:31.471317791Z",
  "resource": {
    "labels": {
      "service": "storage.googleapis.com",
      "project_id": "gcs-project-332821",
      "method": "google.storage.objects.create"
    },
    "type": "audited_resource"
  },
  "severity": "ERROR",
  "timestamp": "2021-12-26T14:25:30.748687635Z"
}

one thing i noticed is this line

      "violationReason": "RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER",
      "vpcServiceControlsUniqueId": "fDcSFrzy6x63WZLBBz3C2wFXZRpPAngxasoNjJ9KKKSver8E36rMpw",
      "egressViolations": [
        {
          "targetResource": "projects/715952372567",
          "servicePerimeter": "accessPolicies/157300459203/servicePerimeters/restrictGCS",
          "source": "projects/247987397047",
          "sourceType": "Resource"
        }
      ]

it says the source project is projects/247987397047 ($VPC_SC_PROJECT) and the target resource is in projects/715952372567 ($INGRESS_PROJECT)…

but thats backward…the GCS bucket i’m trying to copy TO is ($VPC_SC_PROJECT) and FROM a bucket in ($INGRESS_PROJECT)

$ gcloud projects list
PROJECT_ID          NAME              PROJECT_NUMBER

egress-vpc          egress-vpcsc      1099154108957
gcs-project-332821  gcs-project       247987397047
in-perimeter-gcs    in-perimeter-gcs  248336178977
ingress-vpcsc       ingress-vpcsc     715952372567


export VPC_SC_PROJECT=gcs-project-332821
export VPC_SC_PROJECT_IN_PERIMETER=in-perimeter-gcs 
export INGRESS_PROJECT=ingress-vpcsc
export EGRESS_PROJECT=egress-vpc

this is really backwards…i’m not certain why

…anyway, if i comply with the violation and setup an Egress rule that’d allow this

$ gcloud access-context-manager perimeters describe restrictGCS
    name: accessPolicies/157300459203/servicePerimeters/restrictGCS
    status:
      accessLevels:
      - accessPolicies/157300459203/accessLevels/gcs_policy
      egressPolicies:
      - egressFrom:
          identities:
          - serviceAccount:715952372567-compute@developer.gserviceaccount.com
        egressTo:
          operations:
          - methodSelectors:
            - method: '*'
            serviceName: storage.googleapis.com
          resources:
          - projects/715952372567
      ingressPolicies:
      - ingressFrom:
          identities:
          - serviceAccount:715952372567-compute@developer.gserviceaccount.com
          sources:
          - resource: projects/715952372567
        ingressTo:
          operations:
          - methodSelectors:
            - method: '*'
            serviceName: storage.googleapis.com
          resources:
          - projects/247987397047
      resources:
      - projects/247987397047
      restrictedServices:
      - storage.googleapis.com
      vpcAccessibleServices:
        enableRestriction: true
    title: restrictGCS

images/ingress2.png

Once that last step is done, the file copy should work.

echo -n "foo" > bar.txt
gsutil cp gs://$INGRESS_PROJECT-bucket/bar.txt gs://$VPC_SC_PROJECT-bucket/
    Copying file://bar.txt [Content-Type=text/plain]...
    / [1 files][    3.0 B/    3.0 B]                                                
    Operation completed over 1 objects/3.0 B.

again, i’m not sure why…if i find out, i’ll update this

This site supports webmentions. Send me a mention via this form.