just sal's blog > Articles > 2018 > Istio Hello World my way

Istio Hello World my way

2018-02-06

What is this repo?

This is a really simple application I wrote over holidays a year ago (12/17) that details my experiences and feedback with istio. To be clear, its a really basic NodeJS application that i used here but more importantly, it covers the main sections of Istio that i was seeking to understand better (if even just as a helloworld).

I do know isito has the “bookinfo” application but the best way i understand something is to rewrite sections and only those sections from the ground up.

You can find the source here

Istio Hello World my way

Istio version used

03/19/21: Istio 1.9.1
12/21/20: Istio 1.8.0
09/22/20: Istio 1.7.2
04/28/20: Istio 1.5.2
10/12/19: Istio 1.3.2
03/10/19: Istio 1.1.0
01/09/19: Istio 1.0.5
Prior Istio Versions

What i tested

Basic istio Installation on Google Kubernetes Engine
Grafana, Prometheus, Kiali, Jaeger
Route Control
Canary Deployments with VirtualService
Destination Rules
Egress Rules
Egress Gateway
WebAssembly
LUA HttpFilter
Authorization
JWT Authentication an Authorization
Service to Service Authentication Policy
Internal LoadBalancer (GCP)
Mixer Out of Process Authorization Adapter
Access GCE MetadataServer

You can also find info about istio+external authorization server here:

Istio External Authorization Server

What is the app you used?

NodeJS in a Dockerfile…something really minimal. You can find the entire source under the ’nodeapp’ folder in this repo.

The endpoints on this app are as such:

/: Does nothing; (source)
/varz: Returns all the environment variables on the current Pod (source)
/version: Returns just the “process.env.VER” variable that was set on the Deployment (source)
/backend: Return the nodename, pod name. Designed to only get called as if the applciation running is a backend (source)
/hostz: Does a DNS SRV lookup for the backend and makes an http call to its /backend, endpoint (source)
/requestz: Makes an HTTP fetch for several external URLs (used to show egress rules) (source)
/headerz: Displays inbound headers (source)
/metadata: Access the GCP MetadataServer using hostname and link-local IP address (source)
/remote: Access /backend while deployed in a remote istio cluster (source)

I build and uploaded this app to dockerhub at

docker.io/salrashid123/istioinit:1
docker.io/salrashid123/istioinit:2

(basically, they’re both the same application but each has an environment variable that signifies which ‘verison; they represent. The version information for each image is returned by the /version endpoint)

You’re also free to build and push these images directly:

docker build  --build-arg VER=1 -t your_dockerhub_id/istioinit:1 .
docker build  --build-arg VER=2 -t your_dockerhub_id/istioinit:2 .

docker push your_dockerhub_id/istioinit:1
docker push your_dockerhub_id/istioinit:2

To give you a sense of the differences between a regular GKE specification yaml vs. one modified for istio, you can compare:

all-istio.yaml vs all-gke.yaml (review Ingress config, etc)

Lets get started

Create a 1.18+ GKE Cluster and Bootstrap Istio

Note, the following cluster is setup with a aliasIPs (--enable-ip-alias )

We will be installing istio with istioctl

gcloud container  clusters create cluster-1 --machine-type "n1-standard-2" --zone us-central1-a  --num-nodes 4 --enable-ip-alias \
  --cluster-version "1.18" -q

gcloud container clusters get-credentials cluster-1 --zone us-central1-a

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)

kubectl create ns istio-system

export ISTIO_VERSION=1.9.1

wget -O /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-linux-amd64.tar.gz
tar xvf /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz  -C /tmp/

export PATH=/tmp/istio-$ISTIO_VERSION/bin:$PATH


istioctl install --set profile=demo \
 --set meshConfig.enableAutoMtls=true  \
 --set values.gateways.istio-ingressgateway.runAsRoot=true \
 --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY  \
 -f overlay-istio-gateway.yaml

$ istioctl profile dump --config-path components.ingressGateways demo
$ istioctl profile dump --config-path values.gateways.istio-ingressgateway demo

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/jaeger.yaml
sleep 10
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/kiali.yaml

kubectl label namespace default istio-injection=enabled

Wait maybe 2 to 3 minutes and make sure all the Deployments are live:

For reference, here are the Istio operator installation options

Make sure the Istio installation is ready

Verify this step by making sure all the Deployments are Available.

$ kubectl get no,po,rc,svc,ing,deployment -n istio-system 
NAME                                            STATUS   ROLES    AGE   VERSION
node/gke-cluster-1-default-pool-7907ec5d-4rnc   Ready    <none>   33m   v1.18.6-gke.4801
node/gke-cluster-1-default-pool-7907ec5d-7qsz   Ready    <none>   33m   v1.18.6-gke.4801
node/gke-cluster-1-default-pool-7907ec5d-jrwq   Ready    <none>   33m   v1.18.6-gke.4801
node/gke-cluster-1-default-pool-7907ec5d-nxvx   Ready    <none>   33m   v1.18.6-gke.4801

NAME                                        READY   STATUS    RESTARTS   AGE
pod/grafana-75b5cddb4d-bvwj7                1/1     Running   0          52s
pod/istio-egressgateway-7b49cdb77f-4d2gh    1/1     Running   0          6m21s
pod/istio-ilbgateway-789d466f67-tkqrp       1/1     Running   0          6m21s
pod/istio-ingressgateway-78c9bf4887-4bp2g   1/1     Running   0          6m20s
pod/istiod-95bffc969-qbf8b                  1/1     Running   0          6m36s
pod/jaeger-5795c4cf99-2h9sl                 1/1     Running   0          50s
pod/kiali-6c49c7d566-bcrdw                  1/1     Running   0          48s
pod/prometheus-9d5676d95-t7h7g              2/2     Running   0          54s

NAME                           TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                                         AGE
service/grafana                ClusterIP      10.4.12.115   <none>          3000/TCP                                        52s
service/istio-egressgateway    ClusterIP      10.4.14.212   <none>          80/TCP,443/TCP,15443/TCP                        6m20s
service/istio-ilbgateway       LoadBalancer   10.4.11.160   10.128.0.122    443:31947/TCP                                   6m20s
service/istio-ingressgateway   LoadBalancer   10.4.2.68     34.123.13.130   443:31538/TCP                                   6m20s
service/istiod                 ClusterIP      10.4.9.101    <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP   6m36s
service/kiali                  ClusterIP      10.4.5.232    <none>          20001/TCP,9090/TCP                              48s
service/prometheus             ClusterIP      10.4.7.125    <none>          9090/TCP                                        54s
service/tracing                ClusterIP      10.4.12.189   <none>          80/TCP                                          50s
service/zipkin                 ClusterIP      10.4.0.225    <none>          9411/TCP                                        50s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana                1/1     1            1           52s
deployment.apps/istio-egressgateway    1/1     1            1           6m21s
deployment.apps/istio-ilbgateway       1/1     1            1           6m21s
deployment.apps/istio-ingressgateway   1/1     1            1           6m21s
deployment.apps/istiod                 1/1     1            1           6m37s
deployment.apps/jaeger                 1/1     1            1           50s
deployment.apps/kiali                  1/1     1            1           48s
deployment.apps/prometheus             1/1     1            1           54s

Make sure the Istio an IP for the `LoadBalancer` is assigned:

Run

$ kubectl get svc istio-ingressgateway -n istio-system

export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IP

Setup some tunnels to each of the services

Open up several new shell windows and type in one line into each:

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000

kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001

Open up a browser (4 tabs) and go to:

Kiali http://localhost:20001/kiali (username: admin, password: admin)
Grafana http://localhost:3000/dashboard/db/istio-mesh-dashboard
Jaeger http://localhost:16686

Deploy the sample application

The default all-istio.yaml runs:

Ingress with SSL
Deployments:
- myapp-v1: 1 replica
- myapp-v2: 1 replica
- be-v1: 1 replicas
- be-v2: 1 replicas

basically, a default frontend-backend scheme with one replicas for each v1 and v2 versions.

Note: the default yaml pulls and run my dockerhub image- feel free to change this if you want.

kubectl apply -f all-istio.yaml
kubectl apply -f istio-lb-certs.yaml

Now enable the ingress gateway for both external and internal loadbalancer traffic on only port :443:

kubectl apply -f istio-ingress-gateway.yaml -f istio-ingress-ilbgateway.yaml 

kubectl apply -f istio-fev1-bev1.yaml

Wait until the deployments complete:


$ kubectl get po,deployments,svc,ing
NAME                            READY   STATUS    RESTARTS   AGE
pod/be-v1-58855cf854-t6rz9      2/2     Running   0          68s
pod/be-v2-8475cfb896-wxt4z      2/2     Running   0          68s
pod/myapp-v1-7fd949f8fb-cbkgp   2/2     Running   0          68s
pod/myapp-v2-65648669b-8q6ng    2/2     Running   0          68s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/be-v1      1/1     1            1           68s
deployment.apps/be-v2      1/1     1            1           68s
deployment.apps/myapp-v1   1/1     1            1           68s
deployment.apps/myapp-v2   1/1     1            1           68s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/be           ClusterIP   10.4.0.236   <none>        8080/TCP   68s
service/kubernetes   ClusterIP   10.4.0.1     <none>        443/TCP    37m
service/myapp        ClusterIP   10.4.10.14   <none>        8080/TCP   69s

Notice that each pod has two containers: one is from isto, the other is the applicaiton itself (this is because we have automatic sidecar injection enabled on the default namespace).

Also note that in all-istio.yaml we did not define an Ingress object though we’ve defined a TLS secret with a very specific metadata name: istio-ingressgateway-certs. That is a special name for a secret that is used by Istio to setup its own ingress gateway:

Ingress Gateway Secret in 1.0.0+

Note the istio-ingress-gateway secret specifies the Ingress cert to use (the specific metadata name is special and is required)

apiVersion: v1
data:
  tls.crt: _redacted_
  tls.key: _redacted_
kind: Secret
metadata:
  name: istio-ingressgateway-certs
  namespace: istio-system
type: kubernetes.io/tls

Remember we’ve acquired the $GATEWAY_IP earlier:

export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IP

Send Traffic

This section shows basic user->frontend traffic and see the topology and telemetry in the Kiali and Grafana consoles:

Frontend only

So…lets send traffic with the ip to the /versions on the frontend

for i in {1..1000}; do curl -k  https://$GATEWAY_IP/version; sleep 1; done

you may need to restart the ingress pod if the certs they used didn’t pickup

INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-system

You should see a sequence of 1’s indicating the version of the frontend you just hit

111111111111111111111111111111111

(source: /version endpoint)

You should also see on kiali just traffic from ingress -> fe:v1

alt text

and in grafana:

alt text

Frontend and Backend

Now the next step in th exercise:

to send requests to user-->frontend--> backend; we’ll use the /hostz endpoint to do that. Remember, the /hostz endpoint takes a frontend request, sends it to the backend which inturn echos back the podName the backend runs as. The entire response is then returned to the user. This is just a way to show the which backend host processed the requests.

(note i’m using jq utility to parse JSON)

for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done

you should see output indicating traffic from the v1 backend verison: be-v1-*. Thats what we expect since our original rule sets defines only fe:v1 and be:v1 as valid targets.

$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done

"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"

Note both Kiali and Grafana shows both frontend and backend service telemetry and traffic to be:v1

alt text

Route Control

This section details how to selectively send traffic to specific service versions and control traffic routing.

Selective Traffic

In this sequence, we will setup a routecontrol to:

Send all traffic to myapp:v1.
traffic from myapp:v1 can only go to be:v2

Basically, this is a convoluted way to send traffic from fe:v1-> be:v2 even if all services and versions are running.

The yaml on istio-fev1-bev2.yaml would direct inbound traffic for myapp:v1 to go to be:v2 based on the sourceLabels:. The snippet for this config is:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: be-virtualservice
spec:
  gateways:
  - mesh
  hosts:
  - be
  http:
  - match:
    - sourceLabels:
        app: myapp
        version: v1
    route:
    - destination:
        host: be
        subset: v2
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: be-destination
spec:
  host: be
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

So lets apply the config with kubectl:

kubectl replace -f istio-fev1-bev2.yaml

After sending traffic, check which backend system was called by invoking /hostz endpoint on the frontend.

What the /hostz endpoint does is takes a users request to fe-* and targets any be-* that is valid. Since we only have fe-v1 instances running and the fact we setup a rule such that only traffic from fe:v1 can go to be:v2, all the traffic outbound for be-* must terminate at a be-v2:

$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done

"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"

and on the frontend version is always one.

for i in {1..100}; do curl -k https://$GATEWAY_IP/version; sleep 1; done
11111111111111111111111111111

Note the traffic to be-v1 is 0 while there is a non-zero traffic to be-v2 from fe-v1:

alt text

Look at the incoming requests by source graph:

alt text

If we now overlay rules that direct traffic allow interleaved fe(v1|v2) -> be(v1|v2) we expect to see requests to both frontend v1 and backend

kubectl replace -f istio-fev1v2-bev1v2.yaml

then frontend is both v1 and v2:

for i in {1..1000}; do curl -k  https://$GATEWAY_IP/version;  sleep 1; done
111211112122211121212211122211

and backend is responses comes from both be-v1 and be-v2

$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done

"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"

alt text

Route Path

Now lets setup a more selective route based on a specific path in the URI:

The rule we’re defining is: “First route requests to myapp where path=/version to only go to the v1 set”…if there is no match, fall back to the default routes where you send 20% traffic to v1 and 80% traffic to v2

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myapp-virtualservice
spec:
  hosts:
  - "*"
  gateways:
  - my-gateway
  http:
  - match:
    - uri:
        exact: /version
    route:
    - destination:
        host: myapp
        subset: v1
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 20
    - destination:
        host: myapp
        subset: v2
      weight: 80

kubectl replace -f istio-route-version-fev1-bev1v2.yaml

So check all requests to /version are fe:v1

for i in {1..1000}; do curl -k  https://$GATEWAY_IP/version; sleep 1; done
1111111111111111111

You may have noted how the route to any other endpoint other than /version destination is weighted split and not delcared round robin (eg:)

  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 20
    - destination:
        host: myapp
        subset: v2
      weight: 80

Anyway, now lets edit rule to and change the prefix match to /xversion so the match doesn’t apply. What we expect is a request to http://gateway_ip/version will go to v1 and v2 (since the path rule did not match and the split is the fallback rule.

kubectl replace -f istio-route-version-fev1-bev1v2.yaml

Observe the version of the frontend you’re hitting:

for i in {1..1000}; do curl -k  https://$GATEWAY_IP/version; sleep 1; done
2121212222222222222221122212211222222222222222

What you’re seeing is myapp-v1 now getting about 20% of the traffic while myapp-v2 gets 80% because the previous rule doens’t match.

Undo that change /xversion –> /version and reapply to baseline:

kubectl replace -f istio-route-version-fev1-bev1v2.yaml

Canary Deployments with VirtualService

You can use this traffic distribuion mechanism to run canary deployments between released versions. For example, a rule like the following will split the traffic between v1|v2 at 80/20 which you can use to gradually roll traffic over to v2 by applying new percentage weights.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myapp-virtualservice
spec:
  hosts:
  - "*"
  gateways:
  - my-gateway
  - my-gateway-ilb
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 80
    - destination:
        host: myapp
        subset: v2
      weight: 20

Destination Rules

Lets configure Destination rules such that all traffic from myapp-v1 round-robins to both version of the backend.

First lets force all gateway requests to go to v1 only:

on istio-fev1-bev1v2.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myapp-virtualservice
spec:
  hosts:
  - "*"
  gateways:
  - my-gateway
  - my-gateway-ilb
  http:
  - route:
    - destination:
        host: myapp
        subset: v1

And where the backend trffic is split between be-v1 and be-v2 with a ROUND_ROBIN

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: be-destination
spec:
  host: be
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

After you apply the rule,

kubectl replace -f istio-fev1-bev1v2.yaml

you’ll see frontend request all going to fe-v1

for i in {1..1000}; do curl -k  https://$GATEWAY_IP/version; sleep 1; done
11111111111111

with backend requests coming from pretty much round robin

$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done

"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"

Now change the istio-fev1-bev1v2.yaml to RANDOM and see response is from v1 and v2 random:

$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done

"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9]    node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z]    node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"

Internal LoadBalancer

The configuration here sets up an internal loadbalancer on GCP to access an exposed istio service.

The config settings that enabled this during istio setup was done by an operator and annotation:

Specifically, we created a new ingressGateway and set its annotation to cloud.google.com/load-balancer-type: "internal"

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            ports:
            - name: https
              port: 443
              protocol: TCP
      - name: istio-ilbgateway
        enabled: true
        k8s:
          serviceAnnotations:
            cloud.google.com/load-balancer-type: "internal"
          service:
            ports:
            - port: 443
              name: https
              protocol: TCP

We did that duirng setup and later on, attached a Gateway to it which also exposed only :443

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway-ilb
spec:
  selector:
    istio: istio-ilbgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "*"    
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ilbgateway-certs/tls.crt
      privateKey: /etc/istio/ilbgateway-certs/tls.key

We also specified a VirtualService which selected these inbound gateways to the myapp service: This configuration was defined when we applied istio-fev1-bev1.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myapp-virtualservice
spec:
  hosts:
  - "*"
  gateways:
  - my-gateway
  - my-gateway-ilb  
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 100

Note the gateways: entry in the VirtualService includes my-gateway-ilb which is what defines host:myapp, subset:v1 as a target for the ILB

  gateways:
  - my-gateway
  - my-gateway-ilb

As mentioned above, we had to manually specify the port the ILB will listen on for traffic inbound to this service. \

Finally, the certficates Secret mounted at /etc/istio/ilbgateway-certs/ was specified this in the initial all-istio.yaml file:

apiVersion: v1
data:
  tls.crt: LS0tLS1CR...
  tls.key: LS0tLS1CR...
kind: Secret
metadata:
  name: istio-ilbgateway-certs
  namespace: istio-system
type: kubernetes.io/tls

Now that the service is setup, acquire the ILB IP allocated

export ILB_GATEWAY_IP=$(kubectl -n istio-system get service istio-ilbgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $ILB_GATEWAY_IP

images/ilb.png

Then from a GCE VM in the same VPC, send some traffic over on the internal address

$ curl -vk https://10.128.0.122/

* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; O=Google; OU=Enterprise; CN=gke.default.svc.cluster.local
*  start date: Dec 24 18:17:46 2017 GMT
*  expire date: Jun 11 18:17:46 2020 GMT
*  issuer: C=US; ST=California; L=Mountain View; O=Google; OU=Enterprise; CN=TestCAforESO
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

> GET / HTTP/2
> Host: 10.128.0.122
> User-Agent: curl/7.64.0
> Accept: */*

< HTTP/2 200 
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 19
< etag: W/"13-tsbq4e7agwVV6r9iE+Lb/lLwlzw"
< date: Tue, 22 Sep 2020 16:24:55 GMT
< x-envoy-upstream-service-time: 3
< server: istio-envoy


Hello from Express!

The Kiali console should show traffic from both gateways (if you recently sent traffic in externally and internally):

images/ilb_traffic.png

Egress Rules

By default, istio blocks the cluster from making outbound requests. There are several options to allow your service to connect externally:

Egress Rules
Egress Gateway
Setting global.proxy.includeIPRanges

Egress rules prevent outbound calls from the server except with whiteliste addresses.

For example:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: bbc-ext
spec:
  hosts:
  - www.bbc.com
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: google-ext
spec:
  hosts:
  - www.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: google-ext
spec:
  hosts:
  - www.google.com
  tls:
  - match:
    - port: 443
      sni_hosts:
      - www.google.com
    route:
    - destination:
        host: www.google.com
        port:
          number: 443
      weight: 100

Allows only http://www.bbc.com/* and https://www.google.com/*

To test the default policies, the /requestz endpoint tries to fetch the following URLs:

    var urls = [
                'https://www.google.com/robots.txt',
                'http://www.bbc.com/robots.txt',
                'http://www.google.com:443/robots.txt',
                'https://www.cornell.edu/robots.txt',
                'https://www.uwo.ca/robots.txt',
                'http://www.yahoo.com/robots.txt'
    ]

First make sure there is an inbound rule already running:

kubectl replace -f istio-fev1-bev1.yaml

And that you’re using REGISTRY_ONLY:

kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: REGISTRY_ONLY"

Without egress rule, requests will fail:

curl -k -s  https://$GATEWAY_IP/requestz | jq  '.'

gives

[
  {
    "url": "https://www.google.com/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
  },
  {
    "url": "http://www.google.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "http://www.bbc.com/robots.txt",
    "body": "",
    "statusCode": 502
  },
  {
    "url": "https://www.cornell.edu/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "https://www.uwo.ca/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
  },
  {
    "url": "https://www.yahoo.com/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
  },
  {
    "url": "http://www.yahoo.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
]

Note: the 502 response for the bbc.com entry is the actual denial rule from the istio-proxy (502->Bad Gateway)

then apply the egress policy which allows www.bbc.com:80 and www.google.com:443

kubectl apply -f istio-egress-rule.yaml

gives

curl -s -k https://$GATEWAY_IP/requestz | jq  '.'

[
  {
    "url": "https://www.google.com/robots.txt",
    "statusCode": 200
  },
  {
    "url": "http://www.google.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "http://www.bbc.com/robots.txt",
    "statusCode": 200
  },
  {
    "url": "https://www.cornell.edu/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "https://www.uwo.ca/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "https://www.yahoo.com/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "http://www.yahoo.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
]

Notice that only one of the hosts worked over SSL worked

Egress Gateway

THe egress rule above initiates the proxied connection from each sidecar….but why not initiate the SSL connection from a set of bastion/egress gateways we already setup? THis is where the Egress Gateway configurations come up but inorder to use this: The following configuration will allow egress traffic for www.yahoo.com via the gateway. See HTTPS Egress Gateway

So.. lets revert the config we setup above

kubectl delete -f istio-egress-rule.yaml

then lets apply the rule for the gateway:

kubectl apply -f istio-egress-gateway.yaml

Notice the gateway TLS mode is PASSTHROUGH ("Note the PASSTHROUGH TLS mode which instructs the gateway to pass the ingress traffic AS IS, without terminating TLS.")

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: tls
      protocol: TLS
    hosts:
    - www.yahoo.com
    tls:
      mode: PASSTHROUGH

curl -s -k https://$GATEWAY_IP/requestz | jq  '.'

[
  {
    "url": "https://www.google.com/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "http://www.google.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "http://www.bbc.com/robots.txt",
    "body": "",
    "statusCode": 502
  },
  {
    "url": "https://www.cornell.edu/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "https://www.uwo.ca/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
  },
  {
    "url": "https://www.yahoo.com/robots.txt",
    "statusCode": 200                  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
  },
  {
    "url": "http://www.yahoo.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
    }
  }
]

You can also tail the egress gateway logs:

$  kubectl logs -f --tail=0  -l istio=egressgateway -n istio-system
[2020-04-29T15:23:39.949Z] "- - -" 0 - "-" "-" 829 5706 144 - "-" "-" "-" "-" "72.30.35.10:443" outbound|443||www.yahoo.com 10.12.1.4:57332 10.12.1.4:443 10.12.2.10:41592 www.yahoo.com -
[2020-04-29T15:23:48.195Z] "- - -" 0 - "-" "-" 829 5722 138 - "-" "-" "-" "-" "98.138.219.231:443" outbound|443||www.yahoo.com 10.12.1.4:40632 10.12.1.4:443 10.12.2.10:41658 www.yahoo.com -

TLS Origination for Egress Traffic

In this mode, traffic exits the pod unencrypted but gets proxied via the gateway for an https destination. For this to work, traffic must originate from the pod unencrypted but specify the port as an SSL port. In current case, if you want to send traffic for https://www.yahoo.com/robots.txt, emit the request from the pod as http://www.yahoo.com:443/robots.txt. Note the traffic is http:// and the port is specified: :443

Ok, lets try it out, apply:

kubectl apply -f istio-egress-gateway-tls-origin.yaml

Then notice just the last, unencrypted traffic to yahoo succeeds

 curl -s -k https://$GATEWAY_IP/requestz | jq  '.'
[
  {
    "url": "https://www.google.com/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
    }
  },
  {
    "url": "http://www.google.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
    }
  },
  {
    "url": "http://www.bbc.com/robots.txt",
    "body": "",
    "statusCode": 502
  },
  {
    "url": "https://www.cornell.edu/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
  },
  {
    "url": "https://www.uwo.ca/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
    }
  },
  {
    "url": "http://www.yahoo.com/robots.txt",
    "statusCode": 200
  },
  {
    "url": "https://www.yahoo.com/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: Client network socket disconnected before secure TLS connection was established",
    }
  },
  {
    "url": "http://www.yahoo.com:443/robots.txt",
    "statusCode": {
      "name": "RequestError",
      "message": "Error: read ECONNRESET",
    }
  }
]

Bypass Envoy entirely

You can also configure the global.proxy.includeIPRanges= variable to completely bypass the IP ranges for certain serivces. This setting is described under Calling external services directly and details the ranges that should get covered by the proxy. For GKE, you need to cover the subnets included and allocated:

Access GCE MetadataServer

The /metadata endpoint access the GCE metadata server and returns the current projectID. This endpoint makes three separate requests using the three formats I’ve see GCP client libraries use. (note: the hostnames are supposed to resolve to the link local IP address shown below)

app.get('/metadata', (request, response) => {

  var resp_promises = []
  var urls = [
              'http://metadata.google.internal/computeMetadata/v1/project/project-id',
              'http://metadata/computeMetadata/v1/project/project-id',
              'http://169.254.169.254/computeMetadata/v1/project/project-id'
  ]

So if you make an inital request, you’ll see 404 errors from Envoy since we did not setup any rules.

[
  {
    "url": "http://metadata.google.internal/computeMetadata/v1/project/project-id",
    "body": "",
    "statusCode": 502
  },
  {
    "url": "http://metadata/computeMetadata/v1/project/project-id",
    "body": "",
    "statusCode": 502
  },
  {
    "url": "http://169.254.169.254/computeMetadata/v1/project/project-id",
    "body": "",
    "statusCode": 502
  }
]

So lets do just that:

  kubectl apply -f istio-egress-rule-metadata.yaml

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: metadata-ext
spec:
  addresses: 
  - 169.254.169.254    
  hosts:
  - metadata.google.internal
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: STATIC
  location: MESH_EXTERNAL
  endpoints:
  - address: 169.254.169.254

Try it again and you should see

[
  {
    "url": "http://metadata.google.internal/computeMetadata/v1/project/project-id",
    "statusCode": 200
  },
  {
    "url": "http://metadata/computeMetadata/v1/project/project-id",
    "statusCode": 502
  },
  {
    "url": "http://169.254.169.254/computeMetadata/v1/project/project-id",
    "statusCode": 200
  }
]

WebAssembly

Ref: Redefining extensibility in proxies - introducing WebAssembly to Envoy and Istio

The following steps will deploy a trivial wasm module to the cluster that returns hello world back as a header.

We are going to compile and use the main upstream istio envoy example for wasm. The expected output is a new response header and replaced location attribute

see envoy_filter_http_wasm_example.cc

To use this, you need bazel installed and ready to compile a c++ app from scratch

git clone --single-branch --branch  release-1.7 https://github.com/istio/envoy.git
bazel build examples/wasm:envoy_filter_http_wasm_example.wasm

Once you have that, upload the binary as a config map called example-filter

kubectl create cm -n default example-filter  --from-file=`pwd`/bazel-bin/examples/wasm/envoy_filter_http_wasm_example.wasm

Then direct inte sidecar to use a mounted wasm filter

kubectl apply -f fe-v1-wasm-inject.yaml
kubectl replace -f fe-v1-wasm.yaml

Once thats done, invoke the frontend:

$ curl -vk  https://$GATEWAY_IP/version

> GET /version HTTP/2
> Host: 34.123.13.130
> user-agent: curl/7.72.0
> accept: */*

< HTTP/2 200 
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 1
< etag: W/"1-NWoZK3kTsExUV00Ywo1G5jlUKKs"
< date: Tue, 22 Sep 2020 18:08:35 GMT
< x-envoy-upstream-service-time: 38
< newheader: newheadervalue
< location: envoy-wasm                    <<<<<<<<<<<<<<<<<
< server: istio-envoy                     <<<<<<<<<<<<<<<<<
<

You’ll see the two new headers.

TODO: use wasme cli

LUA HTTPFilter

The following will setup a simple Request/Response LUA EnvoyFilter for the frontent myapp:

The settings below injects headers in both the request and response streams:

kubectl apply -f istio-fev1-httpfilter-lua.yaml

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ui-examplefilter
  namespace: default
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      proxy:
        proxyVersion: '1\.9.*'      
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: envoy.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            function envoy_on_request(request_handle)
              request_handle:headers():add("foo", "bar")
            end
            function envoy_on_response(response_handle)
              response_handle:headers():add("foo2", "bar2")
            end            
  workloadSelector:
    labels:
      app: myapp
      version: v1

Note the response headers back to the caller (foo2:bar2) and the echo of the headers as received by the service from envoy (foo:bar)

$ curl -vk  https://$GATEWAY_IP/headerz 

> GET /headerz HTTP/2
> Host: 35.184.101.110
> User-Agent: curl/7.60.0
> Accept: */*

< HTTP/2 200 
< x-powered-by: Express
< content-type: application/json; charset=utf-8
< contLUAent-length: 626
< etag: W/"272-vkps3sJOT8NW67CxK6gzGw"
< date: Fri, 22 Mar 2019 00:40:36 GMT
< x-envoy-upstream-service-time: 7
< foo2: bar2                                                    <<<<<<<<<
< server: istio-envoy

{
  "host": "34.123.13.130",
  "user-agent": "curl/7.72.0",
  "accept": "*/*",
  "x-forwarded-for": "10.128.0.121",
  "x-forwarded-proto": "https",
  "x-request-id": "52fb996a-2f7a-9352-be5e-025c4328bcae",
  "x-envoy-attempt-count": "1",
  "content-length": "0",
  "x-envoy-internal": "true",
  "x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/myapp-sa;Hash=3cb34c33d650cdc2b9c62ad7d617be39dd778ec34895e03a1a405137a0fcf3f0;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
  "foo": "bar",                                                      <<<<<<<<<<<<<<<<
  "x-b3-traceid": "ec14fdbbe9ffd9d0499bc785667ed21f",
  "x-b3-spanid": "63d7f435af04f296",
  "x-b3-parentspanid": "499bc785667ed21f",
  "x-b3-sampled": "1"
}

You can also see the backend request header by running an echo back of those headers

curl -v -k https://$GATEWAY_IP/hostz

< HTTP/2 200 
< x-powered-by: Express
< content-type: application/json; charset=utf-8
< content-length: 168
< etag: W/"a8-+rQK5xf1qR07k9sBV9qawQ"
< date: Fri, 22 Mar 2019 00:44:30 GMT
< x-envoy-upstream-service-time: 33
< foo2: bar2   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
< server: istio-envoy

Authorization and Authorization

The following steps is basically another walkthrough of the RequestAuthentication and AuthorizationPolicy

JWT Authentication and RBAC Authorization

In this section, we will extend the sample to implement JWT authentication from the client and also use claims within the JWT payload for an enhanced Service Specific Policy.

Specifically, this section will add perimeter Authentication that validates a JWT token at ingress gateway and then RBAC policies at the Service level will further restrict requests.

There are two users: Alice, Bob and two services svc1, svc2. Alice should be allowed to access only svc1, Bob should only access svc2. Both users must present a JWT issued by the same issuer. In this case, a Self Signed JWT certificate issued by Google. You can also use Fireabase/Cloud Identity or any other JWT that provides a JWK URL)

This section involves several steps…first delete any configurations that may still be active. We need to do this because we will create two new services on the frontend svc1, svc2

kubectl delete -f istio-fev1-httpfilter-lua.yaml
kubectl delete -f istio-fev1-httpfilter-ext_authz.yaml 
kubectl delete -f istio-fev1-bev1v2.yaml	
kubectl delete -f all-istio.yaml


kubectl apply -f istio-lb-certs.yaml
kubectl apply -f istio-ingress-gateway.yaml
kubectl apply -f istio-ingress-ilbgateway.yaml

You can verify the configuration that are active by running:

$ kubectl get svc,deployments,po,serviceaccounts,serviceentry,VirtualService,DestinationRule,Secret,Gateway

Since the authentication mode described here involes a JWT, we will setup a Google Cloud Service Account the JWT provider. You are ofcourse free to use any identity provide or even Firebase/Cloud Identity

First redeploy an application that has two frontend services svc1, svc2 accessible using the Host: headervalues (svc1.domain.com and svc2.domain.com)

cd auth_rbac_policy/

kubectl apply -f auth-deployment.yaml -f istio-fe-svc1-fe-svc2.yaml

Check the application still works (it should; we didn’t apply policies yet yet)

 curl -k -H "Host: svc1.example.com" -w "\n" https://$GATEWAY_IP/version

Apply the authentication policy that checks for a JWT signed by the service account and audience match on the service. THe following policy will allow all three audience values through the ingress gateway but only those JWTs that match the audience for the service through at the service level:

To bootstrap all this, first we need some JWTS. In this case, we will use GCP serice accounts

To bootstrap the sample client, go to the Google Cloud Console and download a service account JSON file as described [here](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).  Copy the service account to the `auth_rbac_policy/jwt_cli` folder and save the JSON file as `svc_account.json`.

First get the name of the serice account that will sign the JWT:

```bash
cd auth_rbac_policy/jwt_cli/

export PROJECT_ID=`gcloud config get-value core/project`
gcloud iam service-accounts create sa-istio --display-name "JWT issuer for Istio helloworld"
export SA_EMAIL=sa-istio@$PROJECT_ID.iam.gserviceaccount.com
gcloud iam service-accounts keys create svc_account.json --iam-account=$SA_EMAIL
echo $SA_EMAIL

Edit auth-policy.yaml file and replace the values where the service account email $SA_EMAIL is specified

 envsubst < "auth-policy.yaml.tmpl" > "auth-policy.yaml"

After you apply the policy

kubectl apply -f auth-policy.yaml

Note that by default we have mTLS and deny by default

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: deny-all-authz-ns
spec:
  {} 
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default-peerauth
  namespace: default
spec:
  mtls:
    mode: STRICT

make an api call with a malformed authentication header:

$  curl -k -H "Host: svc1.example.com" -H "Authorization: Bearer foo" -w "\n" https://$GATEWAY_IP/version
   Jwt is not in the form of Header.Payload.Signature

now try without a header entirely:

$  curl -k -H "Host: svc1.example.com"  -w "\n" https://$GATEWAY_IP/version
   RBAC: access denied

The error indicates we did not send in the required header. In the next setp, we will use a small sample client library to acquire a JWT. You can also use google OIDC tokens or any other provider (Firebase, Auth0)

The policy above looks for a specific issuer and audience value. THe jwksUri field maps to the public certificate set for our service account. THe well known url for the service account for Google Cloud is:

https://www.googleapis.com/service_accounts/v1/jwk/<serviceAccountEmail>

virtualenv env
source env/bin/activate
pip3 install -r requirements.txt
python3 main.py

The command line utility will generate two tokens with different specifications. For Alice, .

{
  "iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
  "iat": 1571185182,
  "sub": "alice",
  "exp": 1571188782,
  "aud": "https://svc1.example.com"
}

And Bob

{
  "groups": [
    "group1",
    "group2"
  ],
  "sub": "bob",
  "exp": 1571188782,
  "iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
  "iat": 1571185182,
  "aud": "https://svc2.example.com"
}

Bob, no groups

{
  "iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
  "iat": 1571185734,
  "sub": "bob",
  "exp": 1571189334,
  "aud": "https://svc2.example.com"
}

Export these values as environment variables

export TOKEN_ALICE=<tokenvaluealice>
export TOKEN_BOB=<tokenvaluebob>
export TOKEN_BOB_NO_GROUPS=<tokenvaluebob2>

WARNING the sample code to generate the jwt at the client side uses a service account JWT where the client itself is minting the JWT specifications (meaning it can setup any claimsets it wants, any sub field.). In reality, you wouild want to use some other mechanism to acquire a token (Auth0, Firebase Custom Claims, etc).

Now inject the token into the Authorization: Bearer header and try to access the protected service:

for i in {1..1000}; do curl -k -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "\n" https://$GATEWAY_IP/version; sleep 1; done

The request should now pass validation and you’re in. What we just did is have one policy that globally to the ingress-gateway. Note, we also applied per-service policies in auth-policy.yaml that checks for the aud: value in the inbound token.

What that means is if you use Alice’s token to access svc2, you’ll see an authentication validation error because that token doesn’t have "https://svc2.example.com" in the audience

$ curl -k -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "\n" https://$GATEWAY_IP/version
   Audiences in Jwt are not allowed

In our example, we had a self-signed JWT locally meaning if the end-user had a service account capable of singing, they coudl setup any audience value (i.,e Alice could create a JWT token with the audience of svc). We need to back up and apply addtional controls through RBAC.

Authorization using JWT Claims

The other way is to push the allow/deny decision down from Authentication to Authorization and then using claims on the Authz polic

In auth-policy.yaml, uncomment the AuthorizationPolicy stanza which does the first check for correct audience value in the inbound token for svc2 (the one bob uses):

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: svc2-az
spec:
 action: ALLOW  
 selector:
   matchLabels:
     app: svc2
 rules:
 - to:
   - operation:
       methods: ["GET"]  
   when:
   - key: request.auth.claims[iss]
     values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
   - key: request.auth.claims[aud]
     values: ["https://svc2.example.com"]
  #  - key: request.auth.claims[groups]
  #    values: ["group1", "group2"]
   - key: request.auth.claims[sub]
     values: ["bob"]

kubectl apply -f auth-policy.yaml


Consider we have two JWT tokens for `Bob`:

One with groups
```json
{
  "groups": [
    "group1",
    "group2"
  ],
  "sub": "bob",
  "exp": 1571188782,
  "iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
  "iat": 1571185182,
  "aud": "https://svc2.example.com"
}

And one without

{
  "iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
  "iat": 1571185734,
  "sub": "bob",
  "exp": 1571189334,
  "aud": "https://svc2.example.com"
}

Both Tokens allow access through to the serivce because they pass authentication (the audience and subject):

$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -o /dev/null -w "%{http_code}\n"  https://$GATEWAY_IP/version
  200

$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB_NO_GROUPS" -o /dev/null -w "%{http_code}\n"  https://$GATEWAY_IP/version
  200

But what we want to do is deny a request if the token does not include the group header (i know, if Bob had the service account file, he could “just set it”…anyway)

For now, edit auth-policy.yaml and modify the authorization policy for the backend service to make sure the groups are specified and the groups claims are set

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: svc2-az
spec:
 selector:
   matchLabels:
     app: svc2
 rules:
 - to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.auth.claims[iss]
     values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
   - key: request.auth.claims[aud]
     values: ["https://svc2.example.com"]
   - key: request.auth.claims[groups]
     values: ["group1", "group2"]
   - key: request.auth.claims[sub]
     values: ["bob"]

Apply again,

kubectl apply -f auth-policy.yaml

Wait maybe 30seconds (it takes time for the policy to propagte)

Once you set that, only Alice should be able to access svc1 and only Bob access svc2 except when no group info is provided in the JWT

$ curl -sk -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_ALICE"  -w "%{http_code}\n" https://$GATEWAY_IP/version
  200

$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_ALICE"   -w "%{http_code}\n" https://$GATEWAY_IP/version
  403
  Audiences in Jwt are not allowed

$ curl -sk -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_BOB"   -w "%{http_code}\n" https://$GATEWAY_IP/version
  403
  Audiences in Jwt are not allowed

$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -o /dev/null -w "%{http_code}\n" https://$GATEWAY_IP/version
  200

$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB_NO_GROUPS" -o /dev/null --w "%{http_code}\n"  https://$GATEWAY_IP/version
  403
  RBAC: access denied

Notice that bob was only allowed in when the token carried group info.

Service to Service and Authentication Policy

In this section, we extend the working set to allow Alice and Bob to access frontend services and ALSO setup an RBAC policy that allows svcA to access svcB.

When we deployed the application, we associated a service account with each workoad

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: svc1-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: svc2-sa
---

We can use this service acount to say: ‘only allow requests from svc1-sa to access svc2’. We do this by placing another AuthorizationPolicy policy rule in for svc2

 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"]

That is,

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: svc2-az
spec:
 selector:
   matchLabels:
     app: svc2
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"] 
   to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.auth.claims[iss]
     values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
   - key: request.auth.claims[aud]
     values: ["https://svc2.example.com"]
   - key: request.auth.claims[groups]
     values: ["group1", "group2"]
   - key: request.auth.claims[sub]
     values: ["bob"]

Apply again,

kubectl apply -f auth-policy.yaml

Now, if bob tries to access svc2 externally even with a correct token, he will see

$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB"  -w "%{http_code}\n" https://$GATEWAY_IP/version
  RBAC: access denied

Let try to exc into a pod where svc1 is running and access svc2:

$ kubectl get po
NAME                    READY   STATUS    RESTARTS   AGE
svc1-77b6bd69cc-bldf8   2/2     Running   0          29m
svc2-bcf6cbd55-hrp25    2/2     Running   0          29m



$ kubectl exec -ti svc1-77b6bd69cc-bldf8 -- /bin/bash

First try to access the backend service:

curl -s -w "%{http_code}\n"  http://svc2.default.svc.cluster.local:8080/version
403
RBAC: access denied

You’ll see a 403 because although the request was inbound from svc1 which is using PEER authentication, we did not add Bob’s JWT token. So set an env-var and execute the request again:

root@svc1-7489fbf8d4-8tffm:/# export TOKEN_BOB=eyJhbGciOi...

root@svc1-7489fbf8d4-8tffm:/# curl -s -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN_BOB" http://svc2.default.svc.cluster.local:8080/version
200

You are now in!

This is a bit silly since we needed to use the JWT token for bob for just service to serice traffic.

You dont’ ofcourse need to do that: just edit the AuthorizationPolicy for svc2 and comment out

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: svc2-az
spec:
 action: ALLOW  
 selector:
   matchLabels:
     app: svc2
 rules:
 - to:
   - operation:
       methods: ["GET"]  
   from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"] 
  #  when:
  #  - key: request.auth.claims[iss]
  #    values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
  #  - key: request.auth.claims[aud]
  #    values: ["https://svc2.example.com"]
  #  - key: request.auth.claims[groups]
  #    values: ["group1", "group2"]
  #  - key: request.auth.claims[sub]
  #    values: ["bob"]

Bob can’t access svc2 from the outside but svc1 can access svc2

# apply

$ kubectl apply -f auth-policy.yaml 

# from external -> svc1
curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB"  -w "%{http_code}\n" https://$GATEWAY_IP/version
403
RBAC: access denied

# from svc1->svc
curl -s -w "%{http_code}\n"  http://svc2.default.svc.cluster.local:8080/version
200

External Authorization HTTPFilter

You can also setup envoy.ext_authz Filter in this cluster. When using the ext_authz filter on the frontend service, any request for app: myapp, version: v1 will undergo an external authorization check by a serivce you run elsewhere. The external serivice will only allow a request through if it carries Authorizaton: Bearer foo in the header.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter
spec:
  workloadSelector:
    labels:
      app: myapp
      version: v1
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        proxy:
          proxyVersion: ^1\.7.*      
        context: SIDECAR_INBOUND
        listener:
          portNumber: 8080
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        value:
         name: envoy.filters.http.ext_authz
         typed_config:
           "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
           grpc_service:
            google_grpc:
              target_uri: "your_grpc_server_ip:50051"
              stat_prefix: "ext_authz"

To use this type of authorization check, you will need to run a serivce somewhere (either within istio (strongly preferred for latency) or external to istio). The following runs the serivce external to istio:

First spin up a GCP VM that has an external IP, install golang there and startup the authz server in the git repo provided. You’ll also need to open up port :50051 to that VM.

After that, add in the ip address of yoru vm to the yaml file and apply the envoy filter:

kubectl apply -f  istio-fev1-httpfilter-ext_authz.yaml

Once you do that, every request to the fronend service will fail unless the specific header is sent through.

Note you’ll ofcourse not want to run this serivce anywhere thas externally accessible!…this is just for a demo!!

THis is what an inbound request from istio to the authorization server may look like:

$ go run grpc_server.go
2020/02/20 20:41:40 Starting gRPC Server at :50051
2020/02/20 20:42:41 >>> Authorization called check()
2020/02/20 20:42:41 Inbound Headers:
2020/02/20 20:42:41 {
  ":authority": "35.238.81.95",
  ":method": "GET",
  ":path": "/version",
  "accept": "*/*",
  "authorization": "Bearer foo",
  "content-length": "0",
  "user-agent": "curl/7.66.0",
  "x-b3-sampled": "0",
  "x-b3-spanid": "b228f9e2179794c5",
  "x-b3-traceid": "725c448565d59423b228f9e2179794c5",
  "x-envoy-internal": "true",
  "x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/myapp-sa;Hash=ae6b57b6ce2932c74b54c40c5e1a7a13daf2828edeb688f9d273a6ea54f38dbf;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
  "x-forwarded-for": "10.128.0.61",
  "x-forwarded-proto": "https",
  "x-istio-attributes": "CiMKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIHEgVteWFwcAoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0Ck4KCnNvdXJjZS51aWQSQBI+a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTk3ZGNkN2Y4Ny02MnpjZC5pc3Rpby1zeXN0ZW0KPQoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EiESH215YXBwLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKOwoXZGVzdGluYXRpb24uc2VydmljZS51aWQSIBIeaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL215YXBw",
  "x-request-id": "e7932678-b3a2-40b8-bc49-6b645448ae28"
}

Cleanup

The easiest way to clean up what you did here is to delete the GKE cluster!

gcloud container clusters delete cluster-1

Conclusion

The steps i outlined above is just a small set of what Istio has in store. I’ll keep updating this as it move towards 1.0 and subsequent releases.

If you find any are for improvements, please submit a comment or git issue in this repo,.

This site supports webmentions. Send me a mention via this form.

Istio Hello World my way

What is this repo?

Istio version used

What i tested

What is the app you used?

Lets get started

Create a 1.18+ GKE Cluster and Bootstrap Istio

Make sure the Istio installation is ready

Make sure the Istio an IP for the LoadBalancer is assigned:

Setup some tunnels to each of the services

Deploy the sample application

Ingress Gateway Secret in 1.0.0+

Send Traffic

Frontend only

Frontend and Backend

Route Control

Selective Traffic

Route Path

Canary Deployments with VirtualService

Destination Rules

Internal LoadBalancer

Egress Rules

Egress Gateway

TLS Origination for Egress Traffic

Bypass Envoy entirely

Access GCE MetadataServer

WebAssembly

LUA HTTPFilter

Authorization and Authorization

JWT Authentication and RBAC Authorization

Authorization using JWT Claims

Service to Service and Authentication Policy

External Authorization HTTPFilter

Cleanup

Conclusion

If you find any are for improvements, please submit a comment or git issue in this repo,.

Make sure the Istio an IP for the `LoadBalancer` is assigned: