just sal's blog > Articles > 2022 > Decoding gRPC messages over TLS using eBPF

gcp

Decoding gRPC messages over TLS using eBPF

2022-01-23

This is simple (really, really basic) demo showing how to intercept TLS gRPC traffic using eBPF.

Basically, we will dynamically inject an eBPF program similar to sslsniff.py

but in our case, we will add hooks to openssl’s TLS library to first surface the raw payload. With the raw payload, we will decode the http2 Frames and then finally for each DATA frame, attempt to statically unmarshall as a given protobuf messages.

Yes, its statically decoded as this is just a demo/helloworld type app.

there are ofcourse other ways to do decode grpc traffic on the wire but these involve some other type userland proxies and even an application ssl keylogging feature:

but on the otherhand…this is a kernel capability (alteast the interception…).

One of the biggest drawbacks to what is curently in this repo is i have not yet figured out how to attach to boringssls symbols. The repo below uses openssl and iovisor/bcc easy-to-use tracing hooks.

The demo below will show all the steps to set it up including a sample grpc client-server and how you can override python grpcio library to use openssl

Currently this tutorial will only trace clients using openssl as underlying library, eg

curl

python grpcio compiled with openssl

golang is not supported as it does not use openssl.. TODO: (Tracing Go with eBPF ))

Setup

Install ubuntu 20.04.3

I used qemu-system-x86_64 here with a KVM but feel free to use whatever you have

wget https://releases.ubuntu.com/20.04/ubuntu-20.04.3-live-server-amd64.iso

qemu-img create -f qcow2 boot.img 15G

qemu-system-x86_64 -hda boot.img -net nic -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::50051-:50051 -cpu host -smp `nproc` \
    -cdrom  ubuntu-20.04.3-live-server-amd64.iso --enable-kvm -m 2048 -boot menu=on   -machine q35,smm=on  --vga vmware
# (enable openssh-server)

# if you reboot, you don't need to specify the boot iso
qemu-system-x86_64 -hda boot.img -net nic -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::50051-:50051 \
   -cpu host -smp `nproc`   --enable-kvm -m 2048 -boot menu=on   -machine q35,smm=on  --vga vmware


# now in a new window, ssh in
ssh -o UserKnownHostsFile=/dev/null     -o CheckHostIP=no -o StrictHostKeyChecking=no sal@127.0.0.1 -p 10022

Verify:

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"

# uname -r
  5.4.0-96-generic

Install iovisor

Install per ubuntu instructions

apt-get update

apt install -y bison build-essential cmake \
  flex git libllvm7 llvm-7-dev libclang-7-dev libedit-dev python3-pip \
  python zlib1g-dev libelf-dev libfl-dev python3-distutils
# TODO a way to use default llvm-10 but i got lazy


git clone https://github.com/iovisor/bcc.git
mkdir bcc/build
cd bcc/build
cmake ..

make  -j`nproc`

make install

cmake -DPYTHON_CMD=python3 ..
pushd src/python/
make
sudo make install
popd

Test iovisor

At this point, lets run a simple test to decode plain http request (yeah, no ssl, just a way to monitor some other process http requests through an interface tap)

HTTP

# find interface name (for me its enp0s2)

ip link show
cd cd /root/bcc/examples/networking/http_filter
python3 http-parse-simple.py -i enp0s2

Now ssh into the VM from a new window and run

$ curl -s http://httpbin.org/get

what you should see is the http request-response headers (yeah, its basic)

# python3 http-parse-simple.py -i enp0s2
binding socket to 'enp0s2'
GET /get HTTP/1.1
HTTP/1.1 200 OK

SSL

Now lets do something a bit more like decrypt TLS traffic if using openssl or gnutls or nss (i haven’t gotten it to work with boringssl yet…but more on that later)

Fortunately, libbpf already has a tool to do just that all for you sslsniff.py

What we are going to do is decode all TLS traffic using openssl

So, open up a new window and ssh in. If you created a new user in ubuntu, the uid should be 1000 (its the default)

So as root, run

cd /root/bcc/tools
 python3 sslsniff.py -u 1000

Then as user 1000 try to access a site using https:

curl -s --http1.1 https://httpbin.org/get

What you should see is the decrypted request-response stream (we used http1.1 since we just want to demo non-binary data)

$ python3 sslsniff.py -u 1000
FUNC         TIME(s)            COMM             PID     LEN   
WRITE/SEND   0.000000000        curl             1157    78    
----- DATA -----
GET /get HTTP/1.1
Host: httpbin.org
User-Agent: curl/7.68.0
Accept: */*


----- END DATA -----


READ/RECV    0.006161189        curl             1157    484   
----- DATA -----
HTTP/1.1 200 OK
Date: Tue, 18 Jan 2022 23:49:21 GMT
Content-Type: application/json
Content-Length: 254
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.68.0", 
    "X-Amzn-Trace-Id": "Root=1-61e75201-32feddad46df4d4b13ba7f7e"
  }, 
  "origin": "72.83.67.123", 
  "url": "https://httpbin.org/get"
}

----- END DATA -----

If you want, you can also just use openssl to test the same

wget -O server_crt.pem https://raw.githubusercontent.com/salrashid123/squid_proxy/master/server_crt.pem
wget -O server_key.pem https://raw.githubusercontent.com/salrashid123/squid_proxy/master/server_key.pem
wget -O tls-ca.crt https://raw.githubusercontent.com/salrashid123/squid_proxy/master/tls-ca.crt 

echo foo > index.html
# as root, run server
openssl s_server   \
      -cert server_crt.pem \
      -key server_key.pem \
      -port 8443 \
      -CAfile tls-ca.crt \
      -tlsextdebug \
      -tls1_2  \
      -WWW

# as uid=100, run various clients
# with openssl, transmit on connect  GET /index.html HTTP/1.1
openssl  s_client  -cipher 'ECDHE-RSA-AES256-GCM-SHA384' -tls1_2  --connect localhost:8443 -debug
curl -vk https://localhost:8443/index.html

gRPC

We’re now ready to test with gRPC. For this test, we’ll use a small app i wrote up years ago while trying to learn the wire protocol grpc with curl.

As root, start the server:

cd
git clone https://github.com/salrashid123/grpc_curl
cd grpc_curl/src

pip3 install grpcio-tools grpcio hexdump hyperframe

python3 server.py

As uid=1000, attempt to access the server locally

# note, as i'm using TLS and SNI, set as root /etc/hosts 127.0.0.1 server.domain.com
cd /tmp/
git clone https://github.com/salrashid123/grpc_curl
cd grpc_curl/src
python3 client.py server.domain.com 50051
   message: "Hello, john doe!"

$ curl -sk  --raw -X POST --http2  \
    --cacert CA_crt.pem     --resolve  server.domain.com:50051:127.0.0.1 \
    -H "Content-Type: application/grpc"     -H "TE: trailers" \
    --data-binary @frame.bin  \
    https://server.domain.com:50051/echo.EchoServer/SayHello | xxd -p

00000000120a1048656c6c6f2c206a6f686e20646f6521

The output in hex matches the unary response wire-protocol bytes here

Just for your reference on the wire frame seuence a unary request like the above uses, see

images/capture.png

(yeah, i used a remote server for that capture, not localhost…)

gRPC TLS Decoding

Now w’ere really ready to intercept the traffic above using eBPF (finally!)

In a new window, ssh to the VM and as root run the interceptor:

cd /root/grpc_curl/src
# create a file called grpcsniff.py with the contents below
# we need to be in this folder since it contains the compiled `echo_pb.py` files that we need
#  if i wasn't so lazy, i'd improve on this...

#!/usr/bin/python

# this is a modification of
#  https://github.com/iovisor/bcc/blob/master/tools/sslsniff.py
#  https://gist.github.com/summerwind/21c853e5b19593e548dc908a8db82073

from __future__ import print_function
from bcc import BPF
import argparse
import binascii
import textwrap

import echo_pb2
import hexdump, binascii


from hyperframe.frame import (
    Frame, Flags, DataFrame, PriorityFrame, RstStreamFrame, SettingsFrame,
    PushPromiseFrame, PingFrame, GoAwayFrame, WindowUpdateFrame, HeadersFrame,
    ContinuationFrame, AltSvcFrame, ExtensionFrame
)
from hyperframe.exceptions import (
    UnknownFrameError, InvalidPaddingError, InvalidFrameError, InvalidDataError
)

import ctypes as ct

FRAMES = [
    "DATA",
    "HEADERS",
    "PRIORITY",
    "RST_STREAM",
    "SETTINGS",
    "PUSH_PROMISE",
    "PING",
    "GOAWAY",
    "WINDOW_UPDATE",
    "CONTINUATION",
    "ALT_SVC",   
]



# arguments
examples = """examples:
    ./sslsniff              # sniff OpenSSL and GnuTLS functions
    ./sslsniff -p 181       # sniff PID 181 only
    ./sslsniff -u 1000      # sniff only UID 1000
    ./sslsniff -c curl      # sniff curl command only
    ./sslsniff --no-openssl # don't show OpenSSL calls
    ./sslsniff --no-gnutls  # don't show GnuTLS calls
    ./sslsniff --no-nss     # don't show NSS calls
    ./sslsniff --hexdump    # show data as hex instead of trying to decode it as UTF-8
    ./sslsniff -x           # show process UID and TID
"""
parser = argparse.ArgumentParser(
    description="Sniff SSL data",
    formatter_class=argparse.RawDescriptionHelpFormatter,
    epilog=examples)
parser.add_argument("-p", "--pid", type=int, help="sniff this PID only.")
parser.add_argument("-u", "--uid", type=int, default=None,
                    help="sniff this UID only.")
parser.add_argument("-x", "--extra", action="store_true",
                    help="show extra fields (UID, TID)")
parser.add_argument("-c", "--comm",
                    help="sniff only commands matching string.")
parser.add_argument("-o", "--no-openssl", action="store_false", dest="openssl",
                    help="do not show OpenSSL calls.")
parser.add_argument("-g", "--no-gnutls", action="store_false", dest="gnutls",
                    help="do not show GnuTLS calls.")
parser.add_argument("-n", "--no-nss", action="store_false", dest="nss",
                    help="do not show NSS calls.")
parser.add_argument('-d', '--debug', dest='debug', action='count', default=0,
                    help='debug mode.')
parser.add_argument("--ebpf", action="store_true",
                    help=argparse.SUPPRESS)
parser.add_argument("--hexdump", action="store_true", dest="hexdump",
                    help="show data as hexdump instead of trying to decode it as UTF-8")
parser.add_argument('--max-buffer-size', type=int, default=8192,
                    help='Size of captured buffer')
args = parser.parse_args()


prog = """
#include <linux/ptrace.h>
#include <linux/sched.h>        /* For TASK_COMM_LEN */

#define MAX_BUF_SIZE __MAX_BUF_SIZE__

struct probe_SSL_data_t {
        u64 timestamp_ns;
        u32 pid;
        u32 tid;
        u32 uid;
        u32 len;
        int buf_filled;
        char comm[TASK_COMM_LEN];
        u8 buf[MAX_BUF_SIZE];
};

#define BASE_EVENT_SIZE ((size_t)(&((struct probe_SSL_data_t*)0)->buf))
#define EVENT_SIZE(X) (BASE_EVENT_SIZE + ((size_t)(X)))


BPF_PERCPU_ARRAY(ssl_data, struct probe_SSL_data_t, 1);
BPF_PERF_OUTPUT(perf_SSL_write);

int probe_SSL_write(struct pt_regs *ctx, void *ssl, void *buf, int num) {
        int ret;
        u32 zero = 0;
        u64 pid_tgid = bpf_get_current_pid_tgid();
        u32 pid = pid_tgid >> 32;
        u32 tid = pid_tgid;
        u32 uid = bpf_get_current_uid_gid();

        PID_FILTER
        UID_FILTER
        struct probe_SSL_data_t *data = ssl_data.lookup(&zero);
        if (!data)
                return 0;

        data->timestamp_ns = bpf_ktime_get_ns();
        data->pid = pid;
        data->tid = tid;
        data->uid = uid;
        data->len = num;
        data->buf_filled = 0;
        bpf_get_current_comm(&data->comm, sizeof(data->comm));
        u32 buf_copy_size = min((size_t)MAX_BUF_SIZE, (size_t)num);

        if (buf != 0)
                ret = bpf_probe_read_user(data->buf, buf_copy_size, buf);

        if (!ret)
                data->buf_filled = 1;
        else
                buf_copy_size = 0;

        perf_SSL_write.perf_submit(ctx, data, EVENT_SIZE(buf_copy_size));
        return 0;
}

BPF_PERF_OUTPUT(perf_SSL_read);

BPF_HASH(bufs, u32, u64);

int probe_SSL_read_enter(struct pt_regs *ctx, void *ssl, void *buf, int num) {
        u64 pid_tgid = bpf_get_current_pid_tgid();
        u32 pid = pid_tgid >> 32;
        u32 tid = (u32)pid_tgid;
        u32 uid = bpf_get_current_uid_gid();

        PID_FILTER
        UID_FILTER

        bufs.update(&tid, (u64*)&buf);
        return 0;
}

int probe_SSL_read_exit(struct pt_regs *ctx, void *ssl, void *buf, int num) {
        u32 zero = 0;
        u64 pid_tgid = bpf_get_current_pid_tgid();
        u32 pid = pid_tgid >> 32;
        u32 tid = (u32)pid_tgid;
        u32 uid = bpf_get_current_uid_gid();
        int ret;

        PID_FILTER
        UID_FILTER

        u64 *bufp = bufs.lookup(&tid);
        if (bufp == 0)
                return 0;

        int len = PT_REGS_RC(ctx);
        if (len <= 0) // read failed
                return 0;

        struct probe_SSL_data_t *data = ssl_data.lookup(&zero);
        if (!data)
                return 0;

        data->timestamp_ns = bpf_ktime_get_ns();
        data->pid = pid;
        data->tid = tid;
        data->uid = uid;
        data->len = (u32)len;
        data->buf_filled = 0;
        u32 buf_copy_size = min((size_t)MAX_BUF_SIZE, (size_t)len);

        bpf_get_current_comm(&data->comm, sizeof(data->comm));

        if (bufp != 0)
                ret = bpf_probe_read_user(&data->buf, buf_copy_size, (char *)*bufp);

        bufs.delete(&tid);

        if (!ret)
                data->buf_filled = 1;
        else
                buf_copy_size = 0;

        perf_SSL_read.perf_submit(ctx, data, EVENT_SIZE(buf_copy_size));
        return 0;
}
"""

if args.pid:
    prog = prog.replace('PID_FILTER', 'if (pid != %d) { return 0; }' % args.pid)
else:
    prog = prog.replace('PID_FILTER', '')

if args.uid is not None:
    prog = prog.replace('UID_FILTER', 'if (uid != %d) { return 0; }' % args.uid)
else:
    prog = prog.replace('UID_FILTER', '')

prog = prog.replace('__MAX_BUF_SIZE__', str(args.max_buffer_size))

if args.debug or args.ebpf:
    print(prog)
    if args.ebpf:
        exit()


b = BPF(text=prog)

# It looks like SSL_read's arguments aren't available in a return probe so you
# need to stash the buffer address in a map on the function entry and read it
# on its exit (Mark Drayton)
#
if args.openssl:
    b.attach_uprobe(name="ssl", sym="SSL_write", fn_name="probe_SSL_write",
                    pid=args.pid or -1)
    b.attach_uprobe(name="ssl", sym="SSL_read", fn_name="probe_SSL_read_enter",
                    pid=args.pid or -1)
    b.attach_uretprobe(name="ssl", sym="SSL_read",
                       fn_name="probe_SSL_read_exit", pid=args.pid or -1)

if args.gnutls:
    b.attach_uprobe(name="gnutls", sym="gnutls_record_send",
                    fn_name="probe_SSL_write", pid=args.pid or -1)
    b.attach_uprobe(name="gnutls", sym="gnutls_record_recv",
                    fn_name="probe_SSL_read_enter", pid=args.pid or -1)
    b.attach_uretprobe(name="gnutls", sym="gnutls_record_recv",
                       fn_name="probe_SSL_read_exit", pid=args.pid or -1)

if args.nss:
    b.attach_uprobe(name="nspr4", sym="PR_Write", fn_name="probe_SSL_write",
                    pid=args.pid or -1)
    b.attach_uprobe(name="nspr4", sym="PR_Send", fn_name="probe_SSL_write",
                    pid=args.pid or -1)
    b.attach_uprobe(name="nspr4", sym="PR_Read", fn_name="probe_SSL_read_enter",
                    pid=args.pid or -1)
    b.attach_uretprobe(name="nspr4", sym="PR_Read",
                       fn_name="probe_SSL_read_exit", pid=args.pid or -1)
    b.attach_uprobe(name="nspr4", sym="PR_Recv", fn_name="probe_SSL_read_enter",
                    pid=args.pid or -1)
    b.attach_uretprobe(name="nspr4", sym="PR_Recv",
                       fn_name="probe_SSL_read_exit", pid=args.pid or -1)

# define output data structure in Python


# header
header = "%-12s %-18s %-16s %-7s %-6s" % ("FUNC", "TIME(s)", "COMM", "PID", "LEN")

if args.extra:
    header += " %-7s %-7s" % ("UID", "TID")

print(header)
# process event
start = 0


def print_event_write(cpu, data, size):
    print_event(cpu, data, size, "WRITE/SEND", "perf_SSL_write")


def print_event_read(cpu, data, size):
    print_event(cpu, data, size, "READ/RECV", "perf_SSL_read")


def print_event(cpu, data, size, rw, evt):
    global start
    event = b[evt].event(data)
    if event.len <= args.max_buffer_size:
        buf_size = event.len
    else:
        buf_size = args.max_buffer_size

    if event.buf_filled == 1:
        buf = bytearray(event.buf[:buf_size])
    else:
        buf_size = 0
        buf = b""

    # Filter events by command
    if args.comm:
        if not args.comm == event.comm.decode('utf-8', 'replace'):
            return

    if start == 0:
        start = event.timestamp_ns
    time_s = (float(event.timestamp_ns - start)) / 1000000000

    s_mark = "-" * 5 + " STREAM " + "-" * 5
    print(s_mark + "\n")
    e_mark = "-" * 5 + " END STREAM " + "-" * 5

    truncated_bytes = event.len - buf_size
    if truncated_bytes > 0:
        e_mark = "-" * 5 + " END STREAM (TRUNCATED, " + str(truncated_bytes) + \
                " bytes lost) " + "-" * 5

    base_fmt = "%(func)-12s %(time)-18.9f %(comm)-16s %(pid)-7d %(len)-6d"

    if args.extra:
        base_fmt += " %(uid)-7d %(tid)-7d"

    fmt = ''.join([base_fmt, ""])

    data = bytearray(buf)
    while (len(data)>9):
        f, frame_len  = Frame.parse_frame_header(data[:9])
        # https://github.com/salrashid123/grpc_curl#decode-the-response
        f.parse_body(memoryview(data[9:9 + frame_len]))
        print(rw + " " + str(f))
        try:
            if (FRAMES[f.type] == "DATA"):
                wire_msg = binascii.b2a_hex(f.data)
                print('DATA wire_message: ' + wire_msg.decode())

                while len(wire_msg) > 0:
                    message_length = wire_msg[2:10]
                    msg = wire_msg[10:10+int(message_length, 16)*2]
                    if rw == "READ/RECV":
                        r = echo_pb2.EchoReply()
                        r.ParseFromString(binascii.a2b_hex(msg))
                        print('     Proto READ/RECV Decode: ' + r.message)
                    if rw == "WRITE/SEND":
                        r = echo_pb2.EchoRequest()
                        r.ParseFromString(binascii.a2b_hex(msg))
                        print('     Proto WRITE/SEND Decode: ' + r.firstname + " " + r.lastname)
                    wire_msg = wire_msg[10+int(message_length, 16)*2:]
                # print(textwrap.fill(unwrapped_data.decode('utf-8', 'replace'), width=32))
        except IndexError:
            print("Unknown Frame Type " + str(f.type))

        last = 9 + frame_len 
        data = data[last:]

    if args.hexdump:
        unwrapped_data = binascii.hexlify(buf)
        data = textwrap.fill(unwrapped_data.decode('utf-8', 'replace'), width=32)
    else:
        data = buf.decode('utf-8', 'replace')

    fmt_data = {
        'func': rw,
        'time': time_s,
        'comm': event.comm.decode('utf-8', 'replace'),
        'pid': event.pid,
        'tid': event.tid,
        'uid': event.uid,
        'len': event.len,
        'data': data
    }


    print(fmt % fmt_data)
    print(e_mark + "\n")


b["perf_SSL_write"].open_perf_buffer(print_event_write)
b["perf_SSL_read"].open_perf_buffer(print_event_read)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit()

Now as uid=1000, run the curl command:

$ curl -sk  --raw -X POST --http2    \
  --cacert CA_crt.pem     --resolve  server.domain.com:50051:127.0.0.1  \
  -H "Content-Type: application/grpc"     -H "TE: trailers"   \
  --data-binary @frame.bin   \
  https://server.domain.com:50051/echo.EchoServer/SayHello | xxd -p

then grpcsniff.py output should show you each frame, each stream and if the stream is of type=DATA, it will try to decode it as a specific protobuf type (yes, i know, i can generalize this..)

python3 grpcsniff.py

FUNC         TIME(s)            COMM             PID     LEN   
----- STREAM -----
WRITE/SEND ExtensionFrame(stream_id=541611092, flags=[]): type=32, flag_byte=42, body=<hex:502f322e300d0a0d0a53...>
Unknown Frame Type 32
WRITE/SEND   0.000000000        curl             11742   24    
----- END STREAM -----

----- STREAM -----
WRITE/SEND SettingsFrame(stream_id=0, flags=[]): settings={3: 100, 4: 1073741824, 2: 0}
WRITE/SEND   0.000089965        curl             11742   27    
----- END STREAM -----

----- STREAM -----
WRITE/SEND WindowUpdateFrame(stream_id=0, flags=[]): window_increment=1073676289
WRITE/SEND   0.000139100        curl             11742   13    
----- END STREAM -----

----- STREAM -----
WRITE/SEND HeadersFrame(stream_id=1, flags=['END_HEADERS']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:83049360a49cebe024e7...>
WRITE/SEND   0.000214074        curl             11742   95    
----- END STREAM -----

----- STREAM -----
WRITE/SEND   0.000678252        curl             11742   9     
----- END STREAM -----

----- STREAM -----
WRITE/SEND DataFrame(stream_id=1, flags=['END_STREAM']): <hex:000000000b0a046a6f68...>
DATA wire_message: 000000000b0a046a6f686e1203646f65
     Proto WRITE/SEND Decode: john doe                                   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
WRITE/SEND   0.000735952        curl             11742   25    
----- END STREAM -----

----- STREAM -----
WRITE/SEND PingFrame(stream_id=0, flags=['ACK']): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
WRITE/SEND   0.001253866        curl             11742   17    
----- END STREAM -----

----- STREAM -----
READ/RECV SettingsFrame(stream_id=0, flags=[]): settings={4: 4194303, 5: 4194303, 6: 8192, 65027: 1}
READ/RECV WindowUpdateFrame(stream_id=0, flags=[]): window_increment=4128768
READ/RECV    0.000662461        curl             11742   46    
----- END STREAM -----

----- STREAM -----
READ/RECV SettingsFrame(stream_id=0, flags=['ACK']): settings={}
READ/RECV WindowUpdateFrame(stream_id=1, flags=[]): window_increment=21
READ/RECV WindowUpdateFrame(stream_id=0, flags=[]): window_increment=21
READ/RECV PingFrame(stream_id=0, flags=[]): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
READ/RECV    0.001239806        curl             11742   52    
----- END STREAM -----

----- STREAM -----
READ/RECV HeadersFrame(stream_id=1, flags=['END_HEADERS']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:88400c636f6e74656e74...>
READ/RECV DataFrame(stream_id=1, flags=[]): <hex:00000000120a1048656c...>
DATA wire_message: 00000000120a1048656c6c6f2c206a6f686e20646f6521
     Proto READ/RECV Decode: Hello, john doe!                           <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
READ/RECV HeadersFrame(stream_id=1, flags=['END_HEADERS', 'END_STREAM']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:400b677270632d737461...>
READ/RECV    0.001341266        curl             11742   172   
----- END STREAM -----

Notice the lines pointed out by the chevrons….it shows actual decoded data from the grpc Request and Response messages

You can also run server-side streaming responses by invoking echo.EchoServer/SayHelloStream

curl -sk  --raw -X POST --http2    \
  --cacert CA_crt.pem     --resolve  server.domain.com:50051:127.0.0.1  \
  -H "Content-Type: application/grpc"     -H "TE: trailers"   \
  --data-binary @frame.bin   \
  https://server.domain.com:50051/echo.EchoServer/SayHelloStream | xxd -p

Ok, thats great!…we can use curl …but nobody uses raw curl with grpc…people use actual grpc clients

if we now try to run that

$ python3 client.py server.domain.com 50051
message: "Hello, john doe!"

we don’t see anything captured….

thats because grpc is compiled with boringssl and the script tries to attach to symbols for openssl

apt-get install libssl-dev
pip3 uninstall grpcio-tools grpcio
git clone  https://github.com/grpc/grpc
cd grpc
git submodule update --init
pip3 install -rrequirements.txt
## i'm using (from grpcio==1.45.0.dev0) (1.14.0)
GRPC_PYTHON_BUILD_WITH_CYTHON=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 pip install .

Once you do that, even a python grpc client would show the trace:

$ python3 grpcsniff.py 

FUNC         TIME(s)            COMM             PID     LEN   
----- STREAM -----

WRITE/SEND ExtensionFrame(stream_id=541611092, flags=[]): type=32, flag_byte=42, body=<hex:502f322e300d0a0d0a53...>
Unknown Frame Type 32
WRITE/SEND   0.000000000        python3          20905   82    
----- END STREAM -----

----- STREAM -----

WRITE/SEND SettingsFrame(stream_id=0, flags=[]): settings={4: 4194303, 5: 4194303, 6: 8192, 65027: 1}
WRITE/SEND WindowUpdateFrame(stream_id=0, flags=[]): window_increment=4128768
WRITE/SEND   0.000105221        python3          20814   46    
----- END STREAM -----

----- STREAM -----

READ/RECV ExtensionFrame(stream_id=541611092, flags=[]): type=32, flag_byte=42, body=<hex:502f322e300d0a0d0a53...>
Unknown Frame Type 32
READ/RECV    0.000151662        python3          20814   82    
----- END STREAM -----

----- STREAM -----

READ/RECV SettingsFrame(stream_id=0, flags=[]): settings={4: 4194303, 5: 4194303, 6: 8192, 65027: 1}
READ/RECV WindowUpdateFrame(stream_id=0, flags=[]): window_increment=4128768
READ/RECV    0.000839627        python3          20905   46    
----- END STREAM -----

----- STREAM -----

READ/RECV SettingsFrame(stream_id=0, flags=['ACK']): settings={}
READ/RECV HeadersFrame(stream_id=1, flags=['END_HEADERS']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:40053a70617468192f65...>
READ/RECV WindowUpdateFrame(stream_id=1, flags=[]): window_increment=5
READ/RECV DataFrame(stream_id=1, flags=['END_STREAM']): <hex:000000000b0a046a6f68...>
DATA wire_message: 000000000b0a046a6f686e1203646f65
     Proto READ/RECV Decode: john
READ/RECV WindowUpdateFrame(stream_id=0, flags=[]): window_increment=5
READ/RECV    0.001025115        python3          20814   315   
----- END STREAM -----

----- STREAM -----

WRITE/SEND SettingsFrame(stream_id=0, flags=['ACK']): settings={}
WRITE/SEND HeadersFrame(stream_id=1, flags=['END_HEADERS']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:40053a70617468192f65...>
WRITE/SEND WindowUpdateFrame(stream_id=1, flags=[]): window_increment=5
WRITE/SEND DataFrame(stream_id=1, flags=['END_STREAM']): <hex:000000000b0a046a6f68...>
DATA wire_message: 000000000b0a046a6f686e1203646f65
     Proto WRITE/SEND Decode: john doe
WRITE/SEND WindowUpdateFrame(stream_id=0, flags=[]): window_increment=5
WRITE/SEND   0.000949536        python3          20905   315   
----- END STREAM -----

----- STREAM -----

WRITE/SEND SettingsFrame(stream_id=0, flags=['ACK']): settings={}
WRITE/SEND WindowUpdateFrame(stream_id=0, flags=[]): window_increment=16
WRITE/SEND PingFrame(stream_id=0, flags=[]): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
WRITE/SEND   0.001105273        python3          20814   39    
----- END STREAM -----

----- STREAM -----

READ/RECV SettingsFrame(stream_id=0, flags=['ACK']): settings={}
READ/RECV WindowUpdateFrame(stream_id=0, flags=[]): window_increment=16
READ/RECV PingFrame(stream_id=0, flags=[]): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
READ/RECV    0.001642416        python3          20905   39    
----- END STREAM -----

----- STREAM -----

WRITE/SEND PingFrame(stream_id=0, flags=['ACK']): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
WRITE/SEND   0.001670095        python3          20905   17    
----- END STREAM -----

----- STREAM -----

READ/RECV PingFrame(stream_id=0, flags=['ACK']): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
READ/RECV    0.001729416        python3          20814   17    
----- END STREAM -----

----- STREAM -----

WRITE/SEND HeadersFrame(stream_id=1, flags=['END_HEADERS']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:88400c636f6e74656e74...>
WRITE/SEND WindowUpdateFrame(stream_id=1, flags=[]): window_increment=21
WRITE/SEND DataFrame(stream_id=1, flags=[]): <hex:00000000120a1048656c...>
DATA wire_message: 00000000120a1048656c6c6f2c206a6f686e20646f6521
     Proto WRITE/SEND Decode: Hello, john doe! 
WRITE/SEND HeadersFrame(stream_id=1, flags=['END_HEADERS', 'END_STREAM']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:400b677270632d737461...>
WRITE/SEND WindowUpdateFrame(stream_id=0, flags=[]): window_increment=5
WRITE/SEND   0.002214334        python3          20814   169   
----- END STREAM -----

----- STREAM -----

WRITE/SEND PingFrame(stream_id=0, flags=[]): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
WRITE/SEND   0.002478823        python3          20905   17    
----- END STREAM -----

----- STREAM -----

READ/RECV HeadersFrame(stream_id=1, flags=['END_HEADERS']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:88400c636f6e74656e74...>
READ/RECV WindowUpdateFrame(stream_id=1, flags=[]): window_increment=21
READ/RECV DataFrame(stream_id=1, flags=[]): <hex:00000000120a1048656c...>
DATA wire_message: 00000000120a1048656c6c6f2c206a6f686e20646f6521
     Proto READ/RECV Decode: Hello, john doe!
READ/RECV HeadersFrame(stream_id=1, flags=['END_HEADERS', 'END_STREAM']): exclusive=False, depends_on=0, stream_weight=0, data=<hex:400b677270632d737461...>
READ/RECV WindowUpdateFrame(stream_id=0, flags=[]): window_increment=5
READ/RECV    0.002426836        python3          20905   169   
----- END STREAM -----

----- STREAM -----

READ/RECV PingFrame(stream_id=0, flags=[]): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
READ/RECV    0.002551958        python3          20814   17    
----- END STREAM -----

----- STREAM -----

WRITE/SEND PingFrame(stream_id=0, flags=['ACK']): opaque_data=b'\x00\x00\x00\x00\x00\x00\x00\x00'
WRITE/SEND   0.002580908        python3          20814   17    
----- END STREAM -----

Ofcourse its far better to attach to boringssls symbols…i just know how to do that yet…its a TODO for this repo an an update someday..

Thats really it…this is my effort in trying to decode grpc with ebpf…i’m sure theres gonna be a built in decoder/sniffer for this thats professionally done…

Appendix

References

SSLSniff of goBPF The problem with the gist cited is it only displays outbound traffic.
To process gRPC you need several more steps:
1. Find a go library which will dissect http2 streams
2. For Data streams parse the out from the wireformat (see psanford/lencode, issue#5)
3. Unmarshall each message
http2sniff.py I used this to help decode http2
Use KernelTLS LPC2018 - Combining kTLS and BPF for Introspection and Policy Enforcement
Tracing with BPF

golang

You should be able to port any of the samples above to golang. or reference, see:

(optional) Install libbpf

In this step, we will optionally install libbpf (github.com/libbpf/libbpf).

We will not be using this for the tests but I’ve left the installer here for your reference. Note, libbpf comes built in with linux-tools default packages. I’m installing it here since i ran into an issue/bug with the default package

cd
apt-get install clang -y
git clone https://github.com/libbpf/libbpf.git
cd libbpf/src
make && make install

## verify by compiling samples
git clone https://github.com/libbpf/libbpf-bootstrap.git
cd libbpf-bootstrap
git submodule update --init --recursive
cd examples/c/
export LLVM_STRIP=`which llvm-strip-10`
make minimal

Thats really it for libbpf sicne w’ere not going to use it..

This site supports webmentions. Send me a mention via this form.