Sometime last year I wroteup a small article about how to use Cloud Run and Cloud Functions to encrypt data ‘on the fly
In that article a source file in a Google Cloud Storage bucket is read in by a Cloud Function/Run, encrypted “on the fly” and saved in another GCS Bucket. All this happens in a way where the data is encrypted as a stream so as to minimize memory footprint.
As a followup, this is just a standalone app that demonstrates the same technology except the source file is just a named pipe in linux.
The reason i’m writing this is because it chains golang’s
io.Pipe TWICE because we’re going to chain one reader into GPG’s reader/writer stream subsystem which inturn writes to GCS!
Basically, we’re going to chain io.Pipes and stream together…the complication that GPG gives is you need to do this twice and have a lot of indirection
for more info, please see this excellent article here:
So..start with a plain io pipe on the filesystem where we will send the cleartext data
Create a reader for whatever comes into it
fileSrcReader := bufio.NewReader("/tmp/testpipe")
Create temp Pipe (we need this to help broker data between the input, GPG and GCS (we will use this shortly)
pr, pw := io.Pipe()
In a go routine, use GPG to encrypt and encode data using the ‘writer’ end of the temp pipe as the output
pw (again, which we haven’t used this yet)
wm, err := armor.Encode(pw, "PGP MESSAGE", nil)
// Encode returns a WriteCloser which will encode the data written to it in OpenPGP armor. func Encode(out io.Writer, blockType string, headers map[string]string) (w io.WriteCloser, err error)
This will encode the data provided into the
wm) and output that to an
The Encode function itself receives the output of the actual encryption from opengpg.SymmetricallyEncrypt call:
//SymmetricallyEncrypt acts like gpg -c: it encrypts a file with a passphrase. The resulting WriteCloser must be closed after the contents of the file have been written. If config is nil, sensible defaults will be used. pt, err := openpgp.SymmetricallyEncrypt(wm, password, nil, packetConfig) func SymmetricallyEncrypt(ciphertext io.Writer, passphrase byte, hints *FileHints, config *packet.Config) (plaintext io.WriteCloser, err error)
This does the same writecloser-writer flow we did in the encode step…
So how do we get the input data to this function kick everything off?
Copy the file input to of GPG’s writeCloser for
SymmetricallyEncrypt (yes, you need to send the plaintext data into
pt…once you do that, the encrypted data come out as
wm variable which will again be used by the Encode function…
The Encode function will output one half of the original temp pipe writer,
…er..ok, so remember that writer
pw was declared as one end of a pipe to another reader
remember from earlier:
pr, pw := io.Pipe()
Finally take the reader pr and write it to GCS’s writer:
Basically, you’ve chained IO pipes, readers such that the original input is read in as GPG’s encryption reader is written out as a writer again to a final reader which writes to GCS! right….right, (i hope i got that right…you should really read the article cited in the opening paragraph…thats the real deal…i’m just pretending)
anyway, here is the code and usage..
go run main.go
echo “foooooo” > /tmp/testpipe
$ go run main.go open a named pipe file for read. 2020/02/23 08:24:45 Encrypter: 231 bytes are received. $ gsutil cp gs://yourbucket/streamedFile.gpg . Copying gs://yourbucket/streamedFile.gpg... / [1 files][ 231.0 B/ 231.0 B] Operation completed over 1 objects/231.0 B. $ cat streamedFile.gpg -----BEGIN PGP MESSAGE----- wy4ECQMIN+AE+bxjmjdgG3pN4XsnGrljVNBwxCeHnqnLINN7TwrsHms0v9Ipq2VM 0uAB5OlMCdLKvVqOYqB3E61vbbThA23gd+B64QpZ4LziLdF7oOA+4+ZE+DUw/SQ/ 4L3kXZhXU6BUyyevuQMvRvXBMOI7XiHa4TvkAA== =9zmb -----END PGP MESSAGE----- $ gpg --decrypt streamedFile.gpg gpg: AES256 encrypted session key gpg: encrypted with 1 passphrase foooooo