just sal's blog > Articles > 2021 > QUIC HTTP/3 with nginx, envoy and curl

QUIC HTTP/3 with nginx, envoy and curl

2021-12-26

A basic set of Dockerfile definitions which demonstrate HTTP3+QUIC with nginx, envoy and curl.

Why enclose this in a dockerfile? well, its not yet (as of writing) in a main release channel yet for either nginx or curl (envoy is a different story)

What this does is basically allows you to run nginx (or envoy) and curl together over http3+quic. You can capture, alter or change settings on nginx to test out the various features described in the RFC’s below that are supported in nginx and curl.

For background information on quic, see

Also see Decrypting TLS, HTTP/2 and QUIC with Wireshark

TOC:

Build Nginx
Build curl
Nginx HTTP3 request (nginx)
Envoy request (envoy)
Dotnet request (dotnet)
Decoding QUIC

You can find the source docker images and wireshark traces here QUIC HTTP/3 with nginx, envoy and curl

Build nginx

First build the docker images on your own or just use the provided image here (docker.io/salrashid123/nginx-http3)

docker build -t salrashid123/nginx-http3 -f Dockerfile.nginx .

(TODO: make the image smaller…but this is fine for a demo)

Build curl

Now build curl with http3 support or use the provided image here (docker.io/salrashid123/curl-http3)

docker build -t salrashid123/curl-http3 -f Dockerfile.curl .

nginx HTTP3 request

Now start the nginx container:

docker run -v `pwd`/logs:/apps/http3/nginx/logs --net=host -p 8443:8443 -t salrashid123/nginx-http3

Note, nginx access logs will be written in the appropriately named logs/ folder

Run curl:

# optionally capture the keys
mkdir -p /tmp/keylog

docker run --net=host -v`pwd`/config/:/certs -v /tmp/keylog/:/tmp/keylog -e SSLKEYLOGFILE=/tmp/keylog/sslkeylog.log \
    -t salrashid123/curl-http3 \
   -vvv --cacert certs/tls-ca-chain.pem  \
   --resolve  localhost.esodemoapp2.com:8443:127.0.0.1 --http3 https://localhost.esodemoapp2.com:8443/


* Added localhost.esodemoapp2.com:8443:127.0.0.1 to DNS cache
* Hostname localhost.esodemoapp2.com was found in DNS cache
*   Trying 127.0.0.1:8443...
* Connect socket 6 over QUIC to 127.0.0.1:8443
* Connected to localhost.esodemoapp2.com () port 8443 (#0)
* Using HTTP/3 Stream ID: 0 (easy handle 0x564489214f60)
> GET / HTTP/3
> Host: localhost.esodemoapp2.com:8443
> user-agent: curl/7.81.0-DEV
> accept: */*
> 
* ngh3_stream_recv returns 0 bytes and EAGAIN
< HTTP/3 200 
< server: nginx/1.21.5
< date: Mon, 27 Dec 2021 13:59:16 GMT
< content-type: text/html
< content-length: 615
< last-modified: Sun, 26 Dec 2021 21:26:33 GMT
< etag: "61c8de09-267"
< alt-svc: h3=":8443"; ma=86400
< accept-ranges: bytes
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host localhost.esodemoapp2.com left intact

Notice that curl is using quic

* Connect socket 6 over QUIC to 127.0.0.1:8443
* Connected to localhost.esodemoapp2.com () port 8443 (#0)
* Using HTTP/3 Stream ID: 0 (easy handle 0x564489214f60)
> GET / HTTP/3
> Host: localhost.esodemoapp2.com:8443
> user-agent: curl/7.81.0-DEV
> accept: */*

Envoy HTTP3 request

Not surprisingly, Envoy already supports quic:

QUIC listener config

Configuring a listener was pretty challenging to get the specific protobuf envoy yaml expects. As with just about everything, someone else (lkpdn@) already the the heavy lifting…i just used that sample.

Anyway, first get an envoy binary. I usually just extract it locally on linux

docker cp `docker create envoyproxy/envoy-dev:latest`:/usr/local/bin/envoy .

Then run envoy in debug mode (remember to stop nginx container, if its still running)

/tmp/envoy -c envoy.yaml -l debug

This will also listen on udp :8443 so you can use the same curl command:

$  docker run --net=host -v`pwd`/config/:/certs -v /tmp/keylog/:/tmp/keylog \
   -e SSLKEYLOGFILE=/tmp/keylog/sslkeylog.log     -t salrashid123/curl-http3   \
   -vvv --cacert certs/tls-ca-chain.pem   \
   --resolve  localhost.esodemoapp2.com:8443:127.0.0.1 \
   --http3 https://localhost.esodemoapp2.com:8443/

* Added localhost.esodemoapp2.com:8443:127.0.0.1 to DNS cache
* Hostname localhost.esodemoapp2.com was found in DNS cache
*   Trying 127.0.0.1:8443...
* Connect socket 6 over QUIC to 127.0.0.1:8443
* Connected to localhost.esodemoapp2.com () port 8443 (#0)
* Using HTTP/3 Stream ID: 0 (easy handle 0x556f134d0f60)
> GET / HTTP/3
> Host: localhost.esodemoapp2.com:8443
> user-agent: curl/7.81.0-DEV
> accept: */*
> 
* ngh3_stream_recv returns 0 bytes and EAGAIN
< HTTP/3 200 
< content-length: 2
< content-type: text/plain
< date: Mon, 27 Dec 2021 14:55:29 GMT
< server: envoy
< 
* Connection #0 to host localhost.esodemoapp2.com left intact

ok

The envoy logs will show the full capture

21-12-27 09:55:26.099][674972][debug][config] [source/server/listener_impl.cc:743] add active listener: name=listener_udp, hash=12775932757126651534, address=0.0.0.0:8443


[2021-12-27 09:55:29.718][674979][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:160] [C5477787520329841858][S0] Received headers: { :method=GET, :path=/, :scheme=https, :authority=localhost.esodemoapp2.com:8443, user-agent=curl/7.81.0-DEV, accept=*/*, }.
[2021-12-27 09:55:29.718][674979][debug][http] [source/common/http/conn_manager_impl.cc:867] [C5477787520329841858][S8968662946347409001] request headers complete (end_stream=false):
':method', 'GET'
':path', '/'
':scheme', 'https'
':authority', 'localhost.esodemoapp2.com:8443'
'user-agent', 'curl/7.81.0-DEV'
'accept', '*/*'

[2021-12-27 09:55:29.718][674979][debug][http] [source/common/http/filter_manager.cc:947] [C5477787520329841858][S8968662946347409001] Sending local reply with details direct_response
[2021-12-27 09:55:29.718][674979][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C5477787520329841858][S8968662946347409001] encoding headers via codec (end_stream=false):
':status', '200'
'content-length', '2'
'content-type', 'text/plain'
'date', 'Mon, 27 Dec 2021 14:55:29 GMT'
'server', 'envoy'

[2021-12-27 09:55:29.718][674979][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:59] [C5477787520329841858][S0] encodeHeaders (end_stream=false) ':status', '200'
'content-length', '2'
'content-type', 'text/plain'
'date', 'Mon, 27 Dec 2021 14:55:29 GMT'
'server', 'envoy'
.
[2021-12-27 09:55:29.718][674979][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:71] [C5477787520329841858][S0] encodeData (end_stream=true) of 2 bytes.
[2021-12-27 09:55:29.719][674979][debug][http] [source/common/http/conn_manager_impl.cc:204] [C5477787520329841858][S8968662946347409001] doEndStream() resetting stream
[2021-12-27 09:55:29.719][674979][debug][http] [source/common/http/conn_manager_impl.cc:1517] [C5477787520329841858][S8968662946347409001] stream reset

Dotnet HTTP3 request

see Use HTTP/3 with the ASP.NET Core Kestrel web server

Build

docker build -t salrashid123/dotnet-http3 .

Run

docker run --net=host -p 8443:8443 -t salrashid123/dotnet-http3 dotnet run

call client

$ docker run --net=host -v`pwd`/config/:/certs -v /tmp/keylog/:/tmp/keylog -e SSLKEYLOGFILE=/tmp/keylog/sslkeylog.log     -t salrashid123/curl-http3    -vvvvvv --cacert certs/tls-ca-chain.pem     --resolve  localhost.esodemoapp2.com:8443:127.0.0.1 --http3 https://localhost.esodemoapp2.com:8443/

* Added localhost.esodemoapp2.com:8443:127.0.0.1 to DNS cache
* Hostname localhost.esodemoapp2.com was found in DNS cache
*   Trying 127.0.0.1:8443...
* Connect socket 6 over QUIC to 127.0.0.1:8443
* Connected to localhost.esodemoapp2.com () port 8443 (#0)
* Using HTTP/3 Stream ID: 0 (easy handle 0x55d43ace6f60)
> GET / HTTP/3
> Host: localhost.esodemoapp2.com:8443
> user-agent: curl/7.81.0-DEV
> accept: */*
> 
* ngh3_stream_recv returns 0 bytes and EAGAIN
* ngh3_stream_recv returns 0 bytes and EAGAIN
* ngh3_stream_recv returns 0 bytes and EAGAIN
< HTTP/3 200 
< content-type: text/plain; charset=utf-8
< date: Mon, 27 Dec 2021 19:41:49 GMT
< server: Kestrel
< 
* Connection #0 to host localhost.esodemoapp2.com left intact
Hello World!

gives server output confirming http3

dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[1]
      Connection id "0HME9E39Q5RR3" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39]
      Connection id "0HME9E39Q5RR3" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1]
      Connection id "0HME9E39Q5RR3" started.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[3]
      Stream id "0HME9E39Q5RR3:00000003" type Unidirectional connected.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[2]
      Stream id "0HME9E39Q5RR3:00000002" type Unidirectional accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[2]
      Stream id "0HME9E39Q5RR3:00000006" type Unidirectional accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[2]
      Stream id "0HME9E39Q5RR3:0000000A" type Unidirectional accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[2]
      Stream id "0HME9E39Q5RR3:00000000" type Bidirectional accepted.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/3 GET https://localhost.esodemoapp2.com:8443/ - -
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
      1 candidate(s) found for the request path '/'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
      Request matched endpoint 'HTTP: GET /'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'HTTP: GET /'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'HTTP: GET /'
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[10]
      Stream id "0HME9E39Q5RR3:00000000" shutting down writes because: "The QUIC transport's send loop completed gracefully.".
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/3 GET https://localhost.esodemoapp2.com:8443/ - - - 200 - text/plain;+charset=utf-8 6.7929ms
dbug: Microsoft.AspNetCore.Server.Kestrel[25]
      Connection id "0HME9E39Q5RR3", Request id "0HME9E39Q5RR3:00000000": started reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel[26]
      Connection id "0HME9E39Q5RR3", Request id "0HME9E39Q5RR3:00000000": done reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel.BadRequests[28]
      Connection id "0HME9E39Q5RR3", Request id "0HME9E39Q5RR3:00000000": the connection was closed because the response was not read by the client at the specified minimum data rate.
dbug: Microsoft.AspNetCore.Server.Kestrel.Http3[53]
      Connection id "0HME9E39Q5RR3": GOAWAY stream ID 4.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[6]
      Connection id "0HME9E39Q5RR3" aborted by application with error code 258 because: "The connection was timed out by the server because the response was not read by the client at the specified minimum data rate.".
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Quic[10]
      Stream id "0HME9E39Q5RR3:00000003" shutting down writes because: "Operation aborted.".
dbug: Microsoft.AspNetCore.Server.Kestrel.BadRequests[20]
      Connection id "0HME9E39Q5RR3" request processing ended abnormally.
      Microsoft.AspNetCore.Connections.ConnectionAbortedException: The connection was timed out by the server because the response was not read by the client at the specified minimum data rate.
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.AcceptAsync(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
dbug: Microsoft.AspNetCore.Server.Kestrel.Http3[44]
      Connection id "0HME9E39Q5RR3" is closed. The last processed stream ID was 0.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2]
      Connection id "0HME9E39Q5RR3" stopped.

Decoding QUIC

The example/ folder contains a sample curl-to-nginx trace capture with sslkeylogging` enabled. You can use it within wireshark to decode the http3/quic traffic and understand the layers.

To use this, first set the keylog file first to the folder

images/keylog.png

Then copy the keylog to the folder or wireshark to use:

$ mkdir /tmp/keylog
$ cp example/sslkeylog.log /tmp/keylog/sslkeylog.log 
$ wireshark example/wireshark.pcapng

then after loading the pcap file, you should see the full trace images/response.png

TODO

Use qpack decoder in go to see the actual protocol

References

More references:

This site supports webmentions. Send me a mention via this form.