Embedding AWS_SECRET_ACCESS_KEY into Trusted Platform Modules, PKCS-11 devices, Hashicorp Vault and KMS wrapped TINK Keyset

2021-09-17

Sample procedure to encrypt AWS Access Secret Access Key using GCP Tink and a way to embed the the Key into an HSM device supporting PKCS #11, Trusted Platform Module and Hashicorp Vault

AWS secret key and ID can be thought of as a username/password and should be carefully managed, rotated, secured as described in Best practices for managing AWS access keys. However, if you need to invoke AWS from remote systems which do not provide ambient federation (eg on GCP using OIDC tokens), then you must either utilize an AWS credentials file or set them dynamically as an environment variable.

This repo provides four ways to protect the aws secret in various ways:

  1. Wrap the secret using KMS and access it via TINK.
  2. Embed the secret into an HSM an access it via PKCS11
  3. Embed the secret into an TPM an access it via go-tpm
  4. Embed the secret into an Hashicorp Vault an access it via Vault APIs

usecases 2,3,4 are far more interesting (esp 3)

Basically for 2,3,4 above we will encode the key into hardware/sealed devices and then ‘ask’ the device to create HMAC signature compatible with AWS.

You maybe wondering: why can’t i embed an HMAC key into Google Cloud KMS? Well, for some odd reason, HMAC-SHA256 keys on GCP KMS must have a length of 32 bytes…and our hmac key for aws is larger than that (on keysize for hmac, see hmac keysize testing). Why that weird limit is there; no clue

You can find the source here

In (1) you are using KMS to encrypt the Secret and save it in encrypted format. When you need to access the Secret to make it generate an AWS v4 signing request, the raw Secret is automatically decrypted by TINK using KMS and made to HMAC sign. The user will never need to “see” the secret but it is true that the key gets decoded in memory. You will also need to have access to KMS in the first place so this example is a bit convoluted. Use this if you already have access to another cloud providers KMS (eg GCP KMS), then use that decrypt the Key and then make it sign:

The encrypted Key would look like the following:

{
	"encryptedKeyset": "CiUAmT+VVWAVKQZfFW6UheHPI1E3VmvTFlv2C4cspNaqpbxc8YvEEqUBACsKZVI1IW8U+86r2Yset0WOKwnggDitP0hi0oUapgOrF4W7Pklrbso93gfMoNDVw2QCWW4HwJwKzElQRi3zWHuL6NJP4t/t2VtIWORgWLz76zpH7+JWn6IrlqA/M4sammN0kAn+ZcgiG6kCvoMXzczUz3jzyk96Uz6U2LIuZb+bFaCasMYyka2fpSndMQ2SxpmHbVSe2AvhBVMLhM29LOcio41D",
	"keysetInfo": {
		"primaryKeyId": 2596996162,
		"keyInfo": [
			{
				"typeUrl": "type.googleapis.com/google.crypto.tink.HmacKey",
				"status": "ENABLED",
				"keyId": 2596996162,
				"outputPrefixType": "RAW"
			}
		]
	}
}

In (2),(3),(4), you are embedding the HMAC key INTO an HSM. When you then need to access the secret, you ask the HSM to generate an HMAC for the AWS v4 signing process. At no time does the client ever see the secret after it is embedded: the actual HMAC is done within the HSM.

Once the key is embedded into an HSM, you can access it to sign, verify, etc but the key is ever exposed

$ pkcs11-tool --module /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so  --list-objects --pin mynewpin
	Secret Key Object; unknown key algorithm 43
	label:      HMACKey
	ID:         0100
	Usage:      verify
	Access:     sensitive

The big advantage of these modes is clear: the HSM owns the key and is not exportable: nobody will see the raw key once thats done but yet you can use it to create an AWS s4 sign.

AWS v4 Signing Protocol

The AWS S4 Signing protocol described shows something interesting: the raw AWS Secret is used in a chain to create any AWS compatible HMAC secret. On careful inspection, the only time the key is actually used is in the first step where a small prefix AWS4 is prefixed to the key and hashed with the current calendar date. The output of that key is chained through the rest of the signing.

  • images/sign_protocol.png

This gives us an opportunity to seal the prefixed AWS Secret into hardware and then invoke the hardware’s HMAC system to kick off the processs.

Thats what we’ll do here

The implementations for the golang aws signer is here

This repo provides an AWS Credential and Signerv4 which is intended to be used standalone

References

Some references for TINK and PKCS11:

TINK

To use TINK, we will assume you are a GCP customer with access to GCP KMS and want to access AWS via v4 Signing

First create a KMS keychain,key for Symmetric Encryption:

export PROJECT_ID=`gcloud config get-value core/project`
export PROJECT_NUMBER=`gcloud projects describe $PROJECT_ID --format='value(projectNumber)'`
export LOCATION=us-central1
export USER=`gcloud config get-value core/account`

# create keyring
gcloud kms keyrings create mykeyring --location $LOCATION

# create key
gcloud kms keys create key1 --keyring=mykeyring --purpose=encryption --location=$LOCATION

gcloud kms keys add-iam-policy-binding key1 \
    --keyring mykeyring \
    --location $LOCATION \
    --member user:$USER \
    --role roles/cloudkms.cryptoKeyDecrypter

Now create an EncryptedKeySet with Tink, then read in that KeySet and make Tink generate an HMAC signature:

export AWS_ACCESS_KEY_ID=AKIAUH3H6EGKERNFQLHJ
export AWS_SECRET_ACCESS_KEY=YRJ86SK5qTOZQzZTI1u-redacted

$ go run main.go --mode=tink \
  --keyURI "projects/$PROJECT_ID/locations/$LOCATION/keyRings/mykeyring/cryptoKeys/key1" \
  --awsRegion=us-east-2 -accessKeyID $AWS_ACCESS_KEY_ID \
  -secretAccessKey $AWS_SECRET_ACCESS_KEY

		2021/06/02 17:50:44    Create tink subtle.HMAC using secret
		2021/06/02 17:50:44    Tink Keyset:
		{
			"encryptedKeyset": "CiUAmT+VVfqKvEJIXDR1j+kuMjx1fatYYcFmxmrPjtPMhD+p+/E8EqUBACsKZVKCnrihcBHGUoD2ql1CqLsMVzM2MnZkYrKalNdhxB7vUs3y3CScnnsdH+80cTSiVr8ybugaG7c4LKMDw4dB06ox8TS1YbB/hL5+W3IX2yOWPqyFN/t/RVe2QyjGQr7rPqQGM0gOJHDEyTdMX8a9YzG6D7sVM15QQmQtLwKkXNYWr2c0O8iRNRiBcV0ekFwouenMj7+VseyeN0m/dyHNM610",
			"keysetInfo": {
				"primaryKeyId": 2596996162,
				"keyInfo": [
					{
						"typeUrl": "type.googleapis.com/google.crypto.tink.HmacKey",
						"status": "ENABLED",
						"keyId": 2596996162,
						"outputPrefixType": "RAW"
					}
				]
			}
		}
		2021/06/02 17:50:44     Constructing MAC with TINK
		2021/06/02 17:50:44     Signed RequestURI: 
		2021/06/02 17:50:44     STS Response:  
		<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
		<GetCallerIdentityResult>
			<Arn>arn:aws:iam::291738886548:user/svcacct1</Arn>
			<UserId>AIDAUH3H6EGKDO36JYJH3</UserId>
			<Account>291738886548</Account>
		</GetCallerIdentityResult>
		<ResponseMetadata>
			<RequestId>985a21aa-42b3-47af-ab12-9602de5f4a88</RequestId>
		</ResponseMetadata>
		</GetCallerIdentityResponse>

The relevant code that read in TINK is:


import (

	"github.com/golang/protobuf/proto"
	"github.com/google/tink/go/keyset"
	"github.com/google/tink/go/mac/subtle"
	common_go_proto "github.com/google/tink/go/proto/common_go_proto"
	tinkpb "github.com/google/tink/go/proto/tink_go_proto"
	"github.com/google/tink/go/tink"
	hmacpb "github.com/google/tink/go/proto/hmac_go_proto"
	"github.com/google/tink/go/core/registry"
	"github.com/google/tink/go/integration/gcpkms"

	hmacsigner "github.com/salrashid123/aws_hmac/aws"
	hmaccred "github.com/salrashid123/aws_hmac/aws/credentials"
)

	// register the backend KMS, in this case its GCP
		gcpClient, err := gcpkms.NewClient("gcp-kms://")
		registry.RegisterKMSClient(gcpClient)

		backend, err := gcpClient.GetAEAD("gcp-kms://" + *keyURI)

	// read the KeySetJSON as 
		var prettyJSON bytes.Buffer
		error := json.Indent(&prettyJSON, buf.Bytes(), "", "\t")

	// create credentials
		cc, err = hmaccred.NewHMACCredential(&hmaccred.HMACCredentialConfig{
			TinkConfig: hmaccred.TinkConfig{
				KmsBackend: backend,
				JSONBytes:  prettyJSON.Bytes(),
			},
			AccessKeyID: *accessKeyID,
		})

	hs := hmacsigner.NewSigner()

	body := strings.NewReader("")
	sreq, err := http.NewRequest(http.MethodPost, "https://sts.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15", body)

	payloadHash := "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

	// Sign the request
	hs.SignHTTP(ctx, *cc, sreq, payloadHash, "sts", "us-east-1", time.Now())

    // use signedURL
	sres, err := http.DefaultClient.Do(sreq)

PKCS11

For PKCS, we will use SoftHSM which supports the PKCS mechanism to use HMAC. You should be able to use other HSM like yubikey, etc but unfortunately, TPM’s CLi i used for PKCS does not support HMAC imports: see Support TPM HMAC Import.

Anyway, first install SoftHSM and confirm you have the library linked properly (i used linux)

$ ldd /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so
	linux-vdso.so.1 (0x00007fff91bd7000)
	libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f56ad1a1000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f56acfd4000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56ace0f000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f56acdf5000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f56acdef000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f56acdcd000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f56acc87000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f56ad589000)

Now initialize the SoftHSM device:

edit aws_hmac/softhsm/softhsm.conf
# set directories.tokendir = /path/to/aws_hmac/softhsm/tokens

export SOFTHSM2_CONF=`pwd`/softhsm/softhsm.conf

# initialize softHSM and list supported mechanisms
rm -rf /tmp/tokens
mkdir /tmp/tokens
pkcs11-tool --module /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so --slot-index=0 --init-token --label="token1" --so-pin="123456"
pkcs11-tool --module /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so  --label="token1" --init-pin --so-pin "123456" --pin mynewpin
pkcs11-tool --module /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so  --list-objects --pin mynewpin

At this point, we are ready to run the sample application which will

  1. Connect to the HSM
  2. Embed the AccessKey into the HSM
  3. Use the embedded HMAC key to create AWS V4 signature
  4. Access AWS API

note, after step2, the AWS secret is embedded inside the HSM and can only be used to make HMAC signatures.

export AWS_ACCESS_KEY_ID=AKIAUH3H6EGKERNFQLHJ
export AWS_SECRET_ACCESS_KEY=YRJ86SK5qTOZQzZTI1u-redacted

go run main.go --mode=pkcs \
  --hsmLibrary /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so  \
  --awsRegion=us-east-2 -accessKeyID $AWS_ACCESS_KEY_ID \
  -secretAccessKey $AWS_SECRET_ACCESS_KEY

The output of this will run STS.GetCallerIdentity

$ go run main.go --mode=pkcs --hsmLibrary /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so \
    --awsRegion=us-east-2 -accessKeyID $AWS_ACCESS_KEY_ID \
	-secretAccessKey $AWS_SECRET_ACCESS_KEY
	
		Using slot with index 0 (0x0)
		Token successfully initialized
		Using slot 0 with a present token (0x42f8fd2b)
		User PIN successfully initialized
		bash: pcs11-tool: command not found
		CryptokiVersion.Major 2
		2021/06/02 20:19:01 Created HMAC Key: 2
		2021/06/02 20:19:01     Signed RequestURI: 
		2021/06/02 20:19:01     STS Response:  
		<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
		<GetCallerIdentityResult>
			<Arn>arn:aws:iam::291738886548:user/svcacct1</Arn>
			<UserId>AIDAUH3H6EGKDO36JYJH3</UserId>
			<Account>291738886548</Account>
		</GetCallerIdentityResult>
		<ResponseMetadata>
			<RequestId>b273ed3f-279f-43f8-af07-e97a04d0a1ef</RequestId>
		</ResponseMetadata>
		</GetCallerIdentityResponse>

The this code does is the magic:

    // crate a credential which will use the HSM embedded AWS Secret:
		cc, err = hmaccred.NewHMACCredential(&hmaccred.HMACCredentialConfig{
			PKCSConfig: hmaccred.PKCSConfig{
				Library: "/usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so",
				Slot:    0,
				Label:   "HMACKey",
				PIN:     "mynewpin",
				Id:      id,
			},
			AccessKeyID: *accessKeyID,
		})

    // create a signer
	ctx := context.Background()
	hs := hmacsigner.NewSigner()

    // create the request
	body := strings.NewReader("")
	sreq, err := http.NewRequest(http.MethodPost, "https://sts.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15", body)

	// sign the request
	// $ touch empty.txt
    // $ sha256sum empty.txt 
    //   e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  empty.txt	
	payloadHash := "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
	hs.SignHTTP(ctx, *cc, sreq, payloadHash, "sts", "us-east-1", time.Now())

    // make the api call and print the result
	sres, err := http.DefaultClient.Do(sreq)
	sb, err := ioutil.ReadAll(sres.Body)

TPM

In this variation, you will embed the AWS HMAC key into a Trusted Platform Module (TPM). One embedded, the key will never leave the device can only be accessed to “sign” similar to the PKCS example above. To note, the TPM itself has a PKCS interface but at the moment, it does not support HMAC operations like import. See Issue #688. On the other end, go-tpm does not support pretty much any hmac operations: Issue 249

Usage:

One you installed go, just git clone this repo, export the env-vars and run

go run main.go -mode=tpm \
    --awsRegion=us-east-2 -accessKeyID $AWS_ACCESS_KEY_ID \
	-secretAccessKey $AWS_SECRET_ACCESS_KEY

What the script above does is

  1. opens tpm
  2. creates a primary tpm contxet
  3. creates a tpm public and private sections for HMAC
  4. set the ‘sensitive’ part of the private key to the raw AWS secret
  5. imports the public private key to the tpm
  6. writes the handle to that hmac object to a file or to a persistent handle.
  7. initialize the hmaccredentials object in this repo
  8. Use just the file handle and aws KeyID to access an AWS API.

Note, this example does an import of a key every time…you can ofcorse stop after step 6 and just run step 7,8 separately.

this bit bootstraps the tpm and the file handle:

		// pHandle := tpmutil.Handle(0x81010002)
		// err = tpm2.EvictControl(rwc, emptyPassword, tpm2.HandleOwner, newHandle, pHandle)
		// if err != nil {
		// 	fmt.Fprintf(os.Stderr,"Error  persisting hash key  %v\n", err)
		// 	os.Exit(1)
		// }
		// defer tpm2.FlushContext(rwc, pHandle)

		cc, err = hmaccred.NewHMACCredential(&hmaccred.HMACCredentialConfig{
			TPMConfig: hmaccred.TPMConfig{
				TpmDevice:     *tpmPath,
				TpmHandleFile: *hmacKeyHandle,
				//TpmHandle: 0x81010002,
			},
			AccessKeyID: *accessKeyID,
		})

$ go run main.go -mode=tpm     --awsRegion=us-east-2 -accessKeyID $AWS_ACCESS_KEY_ID -secretAccessKey $AWS_SECRET_ACCESS_KEY
		2021/06/10 13:55:51 Using  Standard AWS v4Signer
		2021/06/10 13:55:52    Response using AWS STS NewStaticCredentials and Standard v4.Singer 
		<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
		<GetCallerIdentityResult>
			<Arn>arn:aws:iam::291738886548:user/svcacct1</Arn>
			<UserId>AIDAUH3H6EGKDO36JYJH3</UserId>
			<Account>291738886548</Account>
		</GetCallerIdentityResult>
		<ResponseMetadata>
			<RequestId>8bab0855-1536-4689-854a-10fe2fdd9500</RequestId>
		</ResponseMetadata>
		</GetCallerIdentityResponse>
		Handle 0x80000000 flushed
		======= ContextSave (newHandle) ========
		2021/06/10 13:55:52     Signed RequestURI: 
		2021/06/10 13:55:52     STS Response:  
		<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
		<GetCallerIdentityResult>
			<Arn>arn:aws:iam::291738886548:user/svcacct1</Arn>
			<UserId>AIDAUH3H6EGKDO36JYJH3</UserId>
			<Account>291738886548</Account>
		</GetCallerIdentityResult>
		<ResponseMetadata>
			<RequestId>fe67e9dd-1d58-4ace-9678-3021fd6e90eb</RequestId>
		</ResponseMetadata>
		</GetCallerIdentityResponse>
  • Note, the key we generate has the following attributes
# tpm2_readpublic -c 0x81010002

name: 000b8fb81fa736a1cc6aae780f7ff5d32d6efc3b495ce1854af8aac9dc56dec287ee
qualified name: 000b23eba35005caaacdfa30defedc83a9c4986ed43be8728d544cbb93223d5b8045
name-alg:
  value: sha256
  raw: 0xb
attributes:
  value: fixedtpm|fixedparent|userwithauth|sign
  raw: 0x40052
type:
  value: keyedhash
  raw: 0x8
algorithm: 
  value: hmac
  raw: 0x5
hash-alg:
  value: sha256
  raw: 0xb
keyedhash: 3d2733a76ef6723e5ddb7e1ab88eab0c5e9b2728606756334cc9639570b26cea

Hashicorp Vault

In this variation, you will embed the AWS HMAC key into a Vault’s Transit Engine

Vault already has a secrets engine for AWS which returns temp AWS Access keys to you.

However, in this we are doing something different: we are going to embed an AWS Secret INTO vault and use Vault’s own transit hmac to sign the AWS request

Usage:

# add to /etc/hosts
#    127.0.0.1 vault.domain.com

# start vault
cd vault_resources
vault server -config=server.conf 


# new window
export VAULT_ADDR='https://vault.domain.com:8200'
export VAULT_CACERT=/full/path/to/aws_hmac/vault_resources/ca.pem
vault  operator init
# note down the UNSEAL_KEYS and the INITIAL_ROOT_TOKEN

## unseal using the various unsealing keys
vault  operator unseal [$UNSEAL_KEYS]

export VAULT_TOKEN=$INITIAL_ROOT_TOKEN
vault secrets enable transit

## Create a temp transit key


export AWS_ACCESS_KEY_ID=AKIAUH3H6EGKF4ZY5GGQ
export AWS_SECRET_ACCESS_KEY=HMrL6cNwJCNQzRX8oN-redacted

## this is critical...prfix "AWS4" to the secret_access_key and use that as the hmac key to import
export IMPORT_AWS_SECRET_ACCESS_KEY=`echo -n "AWS4$AWS_SECRET_ACCESS_KEY" | base64`
echo $IMPORT_AWS_SECRET_ACCESS_KEY

Edit key_backup.json and specify the IMPORT_AWS_SECRET_ACCESS_KEY as the hmac_key value

"hmac_key": "QVdTNEhNckw2Y053SkNOUXpSWDhvTjNtbm8vamlLT-redacted",

Now import the key into vault as a backup (thats the only way to import a key, it seems)

export BACKUP_B64="$(cat key_backup.json | base64)"
vault write transit/restore/aws-key-1 backup="${BACKUP_B64}"
vault read transit/keys/aws-key-1

# check hmac works
echo -n "foo" | base64 | vault write transit/hmac/aws-key-1/sha2-256  input=-

Create a vault policy to test the signer (token_policy.hcl, secrets_policy.hcl)

vault policy write token-policy  token_policy.hcl
vault policy write secrets-policy  secrets_policy.hcl

# create a token with those policies (VAULT_TOKEN_FROM_POLICY)
vault token create -policy=token-policy -policy=secrets-policy

Now run the vault client application

# go run main.go -mode=vault    --awsRegion=us-east-1 \
   -accessKeyID $AWS_ACCESS_KEY_ID \
   -secretAccessKey $AWS_SECRET_ACCESS_KEY \
   -vaultToken=$VAULT_TOKEN_FROM_POLICY


		2021/06/10 13:55:51 Using  Standard AWS v4Signer
		2021/06/10 13:55:52    Response using AWS STS NewStaticCredentials and Standard v4.Singer 
		<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
		<GetCallerIdentityResult>
			<Arn>arn:aws:iam::291738886548:user/svcacct1</Arn>
			<UserId>AKIAUH3H6EGKF4ZY5GGQ</UserId>
			<Account>291738886548</Account>
		</GetCallerIdentityResult>
		<ResponseMetadata>
			<RequestId>8bab0855-1536-4689-854a-10fe2fdd9500</RequestId>
		</ResponseMetadata>
		</GetCallerIdentityResponse>


		2021/06/10 13:55:52     Signed RequestURI: 
		2021/06/10 13:55:52     STS Response:  
		<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
		<GetCallerIdentityResult>
			<Arn>arn:aws:iam::291738886548:user/svcacct1</Arn>
			<UserId>AKIAUH3H6EGKF4ZY5GGQ</UserId>
			<Account>291738886548</Account>
		</GetCallerIdentityResult>
		<ResponseMetadata>
			<RequestId>fe67e9dd-1d58-4ace-9678-3021fd6e90eb</RequestId>
		</ResponseMetadata>
		</GetCallerIdentityResponse>

What that just did is used Vaults transit engine to sign aws’s auentication request.

In this flow, the raw hmac was never exposed…only vault knew the key an used it for aws’s signature protocol…

Conclusion

THis is just a POC, caveat emptor

This site supports webmentions. Send me a mention via this form.