Implementing TLS for An Etcd Cluster (2/N)

Ok, so we've got our Vault instance up an running. Let's go make some certs!

The etcd docs include a good guide on security. Per Example 3 in the doc, we'll need the following materials:

• A CA certificate
• A certificate and key for each cluster member, signed by the CA.

First things first, we need to do a little bit of setup to make use of the Vault PKI engine. So, let's authenticate to the server and enable PKI:

[vagrant@turtle1 ~]$ vault login
Token (will be hidden): 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.KG6rIkpFyovI6JWdoorTR5mv
token_accessor       Es6proDGPrzvhOZsiPdrGRNB
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
[vagrant@turtle1 ~]$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

The docs on the PKI engine also suggest tweaking the maximum lifetime for certificates. Here we set it for 1 year:

[vagrant@turtle1 ~]$ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/

So far, so good. Now let's move on to generation of the root cert:

[vagrant@turtle1 ~]$ vault write pki/root/generate/internal common_name=localdomain ttl=8760h
Key              Value
---              -----
certificate      -----BEGIN CERTIFICATE-----
MIIDNTCCAh2gAwIBAgIUCROAk3hiLsmnpCwvBhakfmAeaUswDQYJKoZIhvcNAQEL
BQAwFjEUMBIGA1UEAxMLbG9jYWxkb21haW4wHhcNMTkwNDA0MTg1MTQzWhcNMjAw
...

We've just generated a self-signed CA cert, which is good enough for experimentation but isn't recommended for production setups.

Now let's move on to the actual setup needed to issue certificates signed by our CA. First, configure the location of our issuing CA and associated CRL, which will be embedded in the certificates that we generate:

[vagrant@turtle1 ~]$ vault write pki/config/urls \
>     issuing_certificates="http://127.0.0.1:8200/v1/pki/ca" \
>     crl_distribution_points="http://127.0.0.1:8200/v1/pki/crl"
Success! Data written to: pki/config/urls

Note that I've used 127.0.0.1 for the present, because that's the only only place we're listening. That will need to change in the future as continue bootstrapping our Vault cluster.

A useful feature of Vault's PKI engine is that it lets you define "roles", basically different flavors or classes of certificates. Each role can have different default values, constraints on the type of certificates that can be issued, etc. When you ask Vault to issue a certificate you do so w.r.t. an existing role, so you gotta set up a role first. Here, we set up a role for certificates for our etcd cluster:

[vagrant@turtle1 ~]$ vault write pki/roles/etcd-cluster allowed_domains=localdomain allow_subdomains=true max_ttl=8760h
Success! Data written to: pki/roles/etcd-cluster

Hey Vault, gimme one o' them etcd-cluster certs for mah machine:

[vagrant@turtle1 ~]$ vault write -format json pki/issue/etcd-cluster common_name=turtle1.localdomain > turtle1.localdomain.json

turtle1.localdomain.json now contains the cert, its associated private key, and some other metadata. It's very important that you capture the output of the command; that's the only time the private key for the cert is ever available. I don't strictly understand why they chose to only retain the cert and not the private key, since the whole point of Vault is to store sensitive data, but it is what it is.

Now I'd like to stop and just marvel at how well that all worked. The last time I tried to do something like this it involved making a bunch of different directories and running a bunch of scripts which invoked openssl and all that jazz. Fragile and complicated. This is much, much cleaner.

Alright, let's install some certs! Grab the CA cert:

[root@turtle1 ~]# curl http://127.0.0.1:8200/v1/pki/ca/pem > /etc/pki/ca-trust/source/anchors/vault_root.pem
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1171  100  1171    0     0   421k      0 --:--:-- --:--:-- --:--:-- 1143k

and trust it

[root@turtle1 ~]# update-ca-trust

At least, that's how you do it on CentOS; YMMV if you're using something else. One thing that hasn't improved (much) since the last time I ventured this way is that there's still not much of a standard for where you're supposed to put your certificates. Just take a look in /etc/pki and tell me that it's self-explanatory...

Move the host's certificate and private key into place:

[vagrant@turtle1 ~]$ jq -r .data.certificate turtle1.localdomain.json > turtle1.localdomain.crt
[vagrant@turtle1 ~]$ jq -r .data.private_key turtle1.localdomain.json > turtle1.localdomain.key
[vagrant@turtle1 ~]$ sudo mv turtle1.localdomain.crt /etc/pki/tls/certs/
[vagrant@turtle1 ~]$ sudo mv turtle1.localdomain.key /etc/pki/tls/private/

And that's that! turtle1.localdomain now 1) trusts the Vault CA and 2) has its own cert. We now have enough cert material in place to enable TLS on our Vault instance and let it talk to the outside world. Here's the new config:

storage "file" {
  path = "/var/lib/vault/bootstrap_data"
}
listener "tcp" {
  address     = "turtle1.localdomain:8200"
  tls_cert_file = "/etc/pki/tls/certs/turtle1.localdomain.crt"
  tls_key_file = "/etc/pki/tls/private/turtle1.localdomain.key"
  tls_disable_client_certs = 1
}

In contrast to the previous config the server is now listening on a public IP and using TLS. Vault will request client certs by default, but we're not set up for that at present, so client cert checking is disabled.

Kill the Vault server and restart it using the new config:

[root@turtle1 ~]# vault server -config /etc/vault/single_host_config.hcl

and there you go, chatting with Vault using TLS:

[vagrant@turtle1 ~]$ export VAULT_ADDR=https://turtle1.localdomain:8200
[vagrant@turtle1 ~]$ vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.1.0
HA Enabled         false

That's enough for now. When we pick up we'll generate a couple more certs and then reconfigure the etcd cluster.