gpg

GPG Stream Encryption in golang by chaining Pipes

2020-02-21

Sometime last year i wroteup a small article about how to use Cloud Run and Cloud Functions to encrypt data ‘on the fly’:

In that article a source file in a Google Cloud Storage bucket is read in by a Cloud Function/Run, encrypted “on the fly” and saved in another GCS Bucket. All this happens in a way where the data is encrypted as a stream so as to minimize memory footprint.

As a followup, tHis is just a standalone app that demonstrates the same technology except the source file is just a named pipe in linux.

The reason i’m writing this is because it chains golang’s io.Pipe TWICE because we’re going to chain one reader into GPG’s reader/writer stream subsystem which inturn writes to GCS!

for more info, please see this excellent article here:


So..start with a plain io pipe on the filesystem:

mkfifo /tmp/testpipe

Create a reader for whatever comes into it

fileSrcReader := bufio.NewReader("/tmp/testpipe")

Create temp Pipe (we need this to help broker data between the input, GPG and GCS (we will use this shortly)

pr, pw := io.Pipe()

In a go routine, use GPG to encrypt and encode data using the ‘writer’ end of the temp pipe as the output pw (again, which we haven’t used this yet)

wm, err := armor.Encode(pw, "PGP MESSAGE", nil)	

armor.Encode

func Encode(out io.Writer, blockType string, headers map[string]string) (w io.WriteCloser, err error)

Encode returns a WriteCloser which will encode the data written to it in OpenPGP armor.

This will encode the data provided into the io.WriteCloser (wm) and output that to an io.Writer, pw

The Encode function itself receives the output of the actual encryption from opengpg.SymmetricallyEncrypt call:

pt, err := openpgp.SymmetricallyEncrypt(wm, password, nil, packetConfig)	
func SymmetricallyEncrypt(ciphertext io.Writer, passphrase []byte, hints *FileHints, config *packet.Config) (plaintext io.WriteCloser, err error)

SymmetricallyEncrypt acts like gpg -c: it encrypts a file with a passphrase. The resulting WriteCloser must be closed after the contents of the file have been written. If config is nil, sensible defaults will be used.

This does the same writecloser-writer flow we did in the encode step…

So how do we get the input data to this function kick everything off?

Copy the file input to of GPG’s writeCloser for SymmetricallyEncrypt (yes, you need to send the plaintext data into pt…once you do that, the encrypted data come out as wm variable which will again be used by the Encode function…

The Encode function will output one half of the original temp pipe writer, pw

…er..ok, so remember that writer pw was declared as one end of a pipe to another reader pr.

remember from earlier:

pr, pw := io.Pipe()

Finally take the reader pr and write it to GCS’s writer:

io.Copy(gcsDstWriter, pr)

Basically, you’ve chained IO pipes, readers such that the original input is read in as GPG’s encryption reader is written out as a writer again to a final reader which writes to GCS! right….right, (i hope i got that right…you should really read the article cited in the opening paragraph…thats the real deal…i’m just pretending)

anyway, here is the code and usage..

Usage:

  1. create the pipe mkfifo /tmp/testpipe

  2. run the app go run main.go

  3. Add some data to the source pipe echo “foooooo” > /tmp/testpipe

$ go run main.go
		open a named pipe file for read.
		2020/02/23 08:24:45 Encrypter: 231 bytes are received.

$ gsutil cp gs://yourbucket/streamedFile.gpg .
		Copying gs://yourbucket/streamedFile.gpg...
		/ [1 files][  231.0 B/  231.0 B]
		Operation completed over 1 objects/231.0 B.

$ cat streamedFile.gpg
		-----BEGIN PGP MESSAGE-----

		wy4ECQMIN+AE+bxjmjdgG3pN4XsnGrljVNBwxCeHnqnLINN7TwrsHms0v9Ipq2VM
		0uAB5OlMCdLKvVqOYqB3E61vbbThA23gd+B64QpZ4LziLdF7oOA+4+ZE+DUw/SQ/
		4L3kXZhXU6BUyyevuQMvRvXBMOI7XiHa4TvkAA==
		=9zmb
		-----END PGP MESSAGE-----

$  gpg  --decrypt streamedFile.gpg
		gpg: AES256 encrypted session key
		gpg: encrypted with 1 passphrase

		foooooo

Source:

package main

/*

  "on the fly" stream encryption with Google Cloud Storage and GPG

  Snippet below reads data from a named pipe, encrypts it with a
  symmetric GPG key, then uploads it to GCS all within a stream.

  ref: https://github.com/salrashid123/gpg_gcf
*/

import (
	"fmt"
	"log"

	"bufio"
	"context"
	"io"
	"os"

	"cloud.google.com/go/storage"
	"golang.org/x/crypto/openpgp"
	"golang.org/x/crypto/openpgp/armor"
	"golang.org/x/crypto/openpgp/packet"
)

var (
	bucketName = "your-bucket"
	fileName   = "streamedFile.gpg"
	pipeFile   = "/tmp/testpipe"

	password = []byte("helloworld")

	packetConfig = &packet.Config{
		DefaultCipher: packet.CipherAES256,
	}
)

func main() {

	ctx := context.Background()
	gcsClient, err := storage.NewClient(ctx)
	if err != nil {
		log.Fatalf("Error creating gcs client %v", err)
	}
	defer gcsClient.Close()

	dstBucket := gcsClient.Bucket(bucketName)
	gcsDstObject := dstBucket.Object(fileName)
	gcsDstWriter := gcsDstObject.NewWriter(ctx)

	fmt.Println("open a named pipe file for read.")
	file, err := os.OpenFile(pipeFile, os.O_RDONLY, os.ModeNamedPipe)
	if err != nil {
		log.Fatal("Open named pipe file error:", err)
	}

	fileSrcReader := bufio.NewReader(file)

	pr, pw := io.Pipe()

	go func() {
		defer pw.Close()

		wm, err := armor.Encode(pw, "PGP MESSAGE", nil)
		if err != nil {
			log.Fatalf("[Encrypter] armor.Encode(pw,, nil): (%s) ", err)
		}
		pt, err := openpgp.SymmetricallyEncrypt(wm, password, nil, packetConfig)
		if err != nil {
			log.Fatalf("[Encrypter] openpgp.SymmetricallyEncrypt: (%s) ", err)
		}

		if _, err := io.Copy(pt, fileSrcReader); err != nil {
			log.Fatalf("[Encrypter] Error io.Copy(pt, r.Body): (%s) ", err)
		}
		pt.Close()
		wm.Close()
	}()

	n, err := io.Copy(gcsDstWriter, pr)
	if err != nil {
		log.Fatalf("[Encrypter] Error io.Copy(gcsDstWriter, encbuf): (%s) ", err)
	}

	err = gcsDstWriter.Close()
	if err != nil {
		log.Fatalf("[Encrypter] Error gcsDstWriter.Close: (%s) ", err)
	}
	log.Printf("Encrypter: %d bytes are received.\n", n)
}

This site supports webmentions. Send me a mention via this form.