I love data visualisation…

When I wrote about the experience of producing my first video, I mentioned that I wasn’t happy with the way I produced the slides. I had bigger ideas for the visualisations but wasn’t able to pull them together, so resorted back to Keynote.

I’ve really wanted to have a live visualisation of what the cluster is doing, such as showing the nodes, indices, and shards in the cluster, which node each shard lives on (in the context of its index), differentiate between primary and replica shards, indicate when shards are initialising, and showing the source and destination nodes for relocating shards. This would be a wonderful teaching aid and help people get a visual grip on how the cluster is laid out.

I’ve been writing a video on how to automate routine Elasticsearch cluster maintenance tasks like performing rolling restarts, and upgrading nodes. The video would really benefit from the sort of visualisation I had been planning, so I set about building it.

D3 is a widely-used tool for building visualisations like this. I’ve built with it before, but I needed to re-learn a lot of it and come up to speed with the latest version. I used Peter Cook’s excellent D3 in Depth, and his D3 Start to Finish course. They are great resources and gave me some new insights on patterns to use with D3. It didn’t take long to become familiar with D3 again and start getting data on the page.

The most complicated task was the shard layout. I wanted every shard to be the same size, and picking the ‘right’ size was an interesting job.

This was a lot of fun to build. Shards that are being relocated are rendered on both nodes with an arrow between them showing the direction. Initialising shards pulse. Nodes are removed when they leave the cluster, and reappear when they rejoin. Even the first version does a good job of showing everything I wanted to, but there are still features to implement and improve, such as:

• Filtering indices
• Showing aliases
• Only dropping nodes when the node left timeout has expired
• Tightening up the animations
• Improving the shapes and layout
• Adding shading

The short video below shows the main features. The upcoming Ansible video will showcase some improvements!