Monitoring The Bottom Turtle (4/N)

When we last left off we'd just configured Prometheus to monitor our etcd cluster. Next up, visualizing all that lovely data via Grafana.

Making a dashboard for etcd is a two-step process:

1. Configure Grafana to talk to Prometheus.
2. Create a dashboard using the Prometheus data.

Grafana supports Prometheus as a data source out-of-the-box, so step 1 is easy. Here's the appropriate config file:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090

This defines a data source named "Prometheus" of type "prometheus" (clever, huh?) running on the same machine as Grafana. Name this grafana_prometheus.yml and drop it in your Vagrant project directy.

Putting together a dashboard is a little more complex. Graphana is highly, highly flexible, providing lots of different ways to slice and visualize data. Thankfully, the CoreOS people have an etcd Grafana dashboard which we can take and modify slightly. Save that file as grafana_etcd.json and put it in your Vagrant directory.

That's all you need, configuration-wise. Here's the bootstrap automation to put it all together:

yum install -y https://dl.grafana.com/oss/release/grafana-6.0.2-1.x86_64.rpm
mkdir -p -m655 /var/lib/grafana/dashboards
mv /vagrant/grafana_prometheus.yml /etc/grafana/provisioning/datasources/
mv /vagrant/grafana_etcd.json /var/lib/grafana/dashboards/etcd.json
sed -i '
/title/ s/test-etcd/Etcd/
/datasource/ s/test-etcd/Prometheus/
' /var/lib/grafana/dashboards/etcd.json
sed -i -e 's/^#//' /etc/grafana/provisioning/dashboards/sample.yaml
service grafana-server start

Note that if you were to vagrant up at this point you'd end up with Grafana servers running on port 3000 on each of the cluster nodes. However, you wouldn't be able to conveniently get to any of them. Instead, configure port forwarding like we did for Prometheus; add a line to forward port 3000 on the host system to port 3000 on turtle1.

  config.vm.define "turtle1" do |turtle1|
    turtle1.vm.hostname = "turtle1"
    turtle1.vm.network "private_network", ip: "10.0.0.2",
      virtualbox__intnet: true
    turtle1.vm.network "forwarded_port", guest: 9090, host: 9090
    turtle1.vm.network "forwarded_port", guest: 3000, host: 3000
    turtle1.vm.provision :shell, path: "bootstrap.sh", args: "10.0.0.2 turtle1"
  end

Provided that everything has been bolted together correctly, when you vagrant up and start the etcd cluster you should be able to connect to port 3000 on your local machine and see something like this:

What's neat about the whole Prometheus/Grafana setup is that it pretty much just does the right thing. For example, it automatically provides metrics for all three hosts where appropriate.

In my first post on the subject, I said that I wanted to set up a cluster which would monitor itself. We've pretty much done that here, but in a production setup with 5 nodes it would be overkill to have that much monitoring infrastructure. Having put all the pieces together I think it would make more sense to only install the monitoring on a subset of the cluster nodes. 2 Prometheus/Grafana servers, set up to monitor each other, should provide a sufficient degree of reliability in most cases.