GPG Stream Encryption for Google Storage in golang by chaining Pipes

2020-03-02

Sometime last year I wroteup a small article about how to use Cloud Run and Cloud Functions to encrypt data ‘on the fly

In that article a source file in a Google Cloud Storage bucket is read in by a Cloud Function/Run, encrypted “on the fly” and saved in another GCS Bucket. All this happens in a way where the data is encrypted as a stream so as to minimize memory footprint.

As a followup, this is just a standalone app that demonstrates the same technology except the source file is just a named pipe in linux.

The reason i’m writing this is because it chains golang’s io.Pipe TWICE because we’re going to chain one reader into GPG’s reader/writer stream subsystem which inturn writes to GCS!

basically:

  1. on your laptop, create a pipe
  2. write some data to the pipe
  3. this app will then read the inbound data as a stream, encrypt it with GPG and write it as a stream to GCS

Basically, we’re going to chain io.Pipes and stream together…the complication that GPG gives is you need to do this twice and have a lot of indirection

for more info, please see this excellent article here:

So..start with a plain io pipe on the filesystem where we will send the cleartext data

mkfifo /tmp/testpipe

Create a reader for whatever comes into it

fileSrcReader := bufio.NewReader("/tmp/testpipe")

Create temp Pipe (we need this to help broker data between the input, GPG and GCS (we will use this shortly)

pr, pw := io.Pipe()

In a go routine, use GPG to encrypt and encode data using the ‘writer’ end of the temp pipe as the output pw (again, which we haven’t used this yet)

wm, err := armor.Encode(pw, "PGP MESSAGE", nil)

armor.Encode

// Encode returns a WriteCloser which will encode the data written to it in OpenPGP armor.

func Encode(out io.Writer, blockType string, headers map[string]string) (w io.WriteCloser, err error)

This will encode the data provided into the io.WriteCloser (wm) and output that to an io.Writer, pw

The Encode function itself receives the output of the actual encryption from opengpg.SymmetricallyEncrypt call:


//SymmetricallyEncrypt acts like gpg -c: it encrypts a file with a passphrase. The resulting WriteCloser must be closed after the contents of the file have been written. If config is nil, sensible defaults will be used.

pt, err := openpgp.SymmetricallyEncrypt(wm, password, nil, packetConfig) 
func SymmetricallyEncrypt(ciphertext io.Writer, passphrase []byte, hints *FileHints, config *packet.Config) (plaintext io.WriteCloser, err error)

This does the same writecloser-writer flow we did in the encode step…

So how do we get the input data to this function kick everything off?

Copy the file input to of GPG’s writeCloser for SymmetricallyEncrypt (yes, you need to send the plaintext data into pt…once you do that, the encrypted data come out as wm variable which will again be used by the Encode function…

The Encode function will output one half of the original temp pipe writer, pw

…er..ok, so remember that writer pw was declared as one end of a pipe to another reader pr.

remember from earlier:

pr, pw := io.Pipe()

Finally take the reader pr and write it to GCS’s writer:

io.Copy(gcsDstWriter, pr)

Basically, you’ve chained IO pipes, readers such that the original input is read in as GPG’s encryption reader is written out as a writer again to a final reader which writes to GCS! right….right, (i hope i got that right…you should really read the article cited in the opening paragraph…thats the real deal…i’m just pretending)

anyway, here is the code and usage..

Usage:

  1. create the pipe
mkfifo /tmp/testpipe
  1. run the app
go run main.go
  1. Add some data to the source pipe
echo “foooooo” > /tmp/testpipe

$ go run main.go
		open a named pipe file for read.
		2020/02/23 08:24:45 Encrypter: 231 bytes are received.

$ gsutil cp gs://yourbucket/streamedFile.gpg .
		Copying gs://yourbucket/streamedFile.gpg...
		/ [1 files][  231.0 B/  231.0 B]
		Operation completed over 1 objects/231.0 B.
    
$ cat streamedFile.gpg
		-----BEGIN PGP MESSAGE-----
		wy4ECQMIN+AE+bxjmjdgG3pN4XsnGrljVNBwxCeHnqnLINN7TwrsHms0v9Ipq2VM
		0uAB5OlMCdLKvVqOYqB3E61vbbThA23gd+B64QpZ4LziLdF7oOA+4+ZE+DUw/SQ/
		4L3kXZhXU6BUyyevuQMvRvXBMOI7XiHa4TvkAA==
		=9zmb
		-----END PGP MESSAGE-----
    
$  gpg  --decrypt streamedFile.gpg
		gpg: AES256 encrypted session key
		gpg: encrypted with 1 passphrase
		foooooo
  • main.go:

This site supports webmentions. Send me a mention via this form.