Implementing TLS For An Etcd Cluster (1/N)

So one of the things that I deliberatly punted on in configuring an etcd cluster was any sort of security. Because security is hard and PKI is hard and on and on and...

etcd wants a variety of certificates to secure its operations, so I decided to circle 'round and see what's available these days in the way of open source PKI. One system which immediately caught my eye is Hashicorp Vault, in large part because it has a built-in PKI engine which reduces a lot of the complications associated with managing certificates. So I figured I'd give it a whirl and see whether I could get it to work. Usual caveats apply, not vetted for production etc.

So, first thing which needs to be done is to put together a minimal configuration file. Here's what I picked to start:

storage "file" {
  path = "/var/lib/vault/bootstrap_data"
}
listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

This is about as stripped down as you can make it. It uses local disk for storage and listens on localhost. Note also the tls_disable directive; we can't turn on TLS because we don't (yet) have any certificates. This configuration is suitable for using a single machine to establish a root of trust. The perceptive reader will also note that I named the data directory "bootstrap_data"; later on I'm going to try to use this initial configuration to bootstrap into a full-blown, HA Vault config.

Name this file vault_boostrap_config.hcl and put it in your Vagrant directory, then update bootstrap.sh to install Vault:

yum install -y unzip
wget -nv https://releases.hashicorp.com/vault/1.1.0/vault_1.1.0_linux_amd64.zip
unzip vault_1.1.0_linux_amd64.zip
mv vault /usr/local/bin/
mkdir /var/lib/vault/
mkdir /etc/vault/
mv /vagrant/vault_bootstrap_config.hcl /etc/vault/bootstrap_config.hcl
rm vault_1.1.0_linux_amd64.zip

vagrant up gives you a system with Vault installed but not yet running. I've held off on creating a service description for the moment since we're bootstrapping and will be doing a bunch of manual work.

Alright then, let's fire up the vault:

[root@turtle1 ~]# vault server -config /etc/vault/bootstrap_config.hcl

This won't fork, so open up another connection to do the remainder of the work.

Tell the CLI how to find the server. Note the use of HTTP since TLS is turned off:

[vagrant@turtle1 ~]$ export VAULT_ADDR=http://127.0.0.1:8200

Confirm that everything is operating correctly by checking the status of the server:

[vagrant@turtle1 ~]$ vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         false

Note a couple of items:

• The vault isn't initialized; we'll need to do that before anything else.
• The vault is "sealed", which means that it's running but none of the information is decrypted. The vault will need to be "unsealed" before any other operations can take place.

Alright then, let's initialize the vault:

[vagrant@turtle1 ~]$ vault operator init
Unseal Key 1: BAqaQJiVsYE1ufAD6hraNoKL6aprDWqbZKtj5Kslig3B
Unseal Key 2: G2AqRzuUKlmLefb98YT8xv2ei2SQApaUvPa09jiXb8Uh
Unseal Key 3: I70F/pMpjHIoSb2Gp66Y7M5AwxwO2LJ6Jx9hHrhfuHnz
Unseal Key 4: IY6cD7ymCNQyiGaMWsa6xQ37mqwVFU05GQ+9xts6TYBS
Unseal Key 5: TPR2p9WA4P0G2Q6/g+kEp9SWKlwl83d7EXpl8Hosnevp

Initial Root Token: s.C0mtdFlTNUUjK6LFHPwuxOYV

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

The keys and token above are required to unseal and authenticat to the vault, and there's no way to get them back if you lose them. If you were doing this in a production environment you'd want to print the above out and stick it in a safe or some such.

Note that the initialization process produced 5 keys. Vault implement's Shamir's Secret Sharing, which is basically a way to share out a secret in pieces such that it can be reconstructed by a quorum of piece-holders. By default Vault produces 5 keys, any 3 of which can be used to condocuct the unsealing process. So let's get to it:

[vagrant@turtle1 ~]$ vault operator unseal
Unseal Key (will be hidden): 
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       ae24a0e1-ec84-6793-e077-0fca1947a6d7
Version            1.1.0
HA Enabled         false
[vagrant@turtle1 ~]$ vault operator unseal
Unseal Key (will be hidden): 
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       ae24a0e1-ec84-6793-e077-0fca1947a6d7
Version            1.1.0
HA Enabled         false
[vagrant@turtle1 ~]$ vault operator unseal
Unseal Key (will be hidden): 
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.1.0
Cluster Name    vault-cluster-4698ed15
Cluster ID      77f26182-854d-ffe8-c2ef-d194f93f13e3
HA Enabled      false

And there you have it, one fresh, unsealed, fully operational vault instance. We'll pick up next time with using the PKI engine to generate certificates for etcd.