The idea behind Incident Management is to be ready. Not ready for anything, as that can be an unrealistic expectation, but ready to respond when the unexpected inevitably happens. DevOps teams often create incident playbooks in order to ensure they are as ready as possible to handle situations as they arise. Luckily, there is some amazing documentation on how to do just that from our friends at PagerDuty.
Mattermost is a platform based on collaboration — not built simply for facilitating team and asynchronous communication, but built on the philosophy that having the ability to collaborate efficiently makes the world safer and more productive for everyone. This is true in many day-to-day situations in an organization, but it is especially true in the world of DevOps. When an emergency arises, information needs to be moved from person to person and team to team as quickly as possible.
One of the key benefits of using Kubernetes is that it’s very flexible and fault tolerant. However, that also means that it has quite a lot of complexity to deal with. A well-built operator abstracts that complexity away and helps manage updates and upgrades seamlessly. The Mattermost Kubernetes operator is basically like having a Mattermost Cloud Engineer running inside your Kubernetes cluster managing Mattermost for you.
Kubernetes is said to be the platform to build platforms on. In Freshworks’ case, this is very true. Once Kubernetes established itself as the de-facto container orchestration platform, we set out to create a platform around it with certain very specific capabilities in mind. Kubernetes is a complex beast and takes some effort to tame. Once that is done, there are tremendous advantages.
Lately we’ve been working on improving different parts of the Mattermost server, including our monitoring and observability capabilities. We’ve been using Prometheus and Grafana to monitor our cluster for a while now, and you can read this great post where my colleague Stylianos explains how we have them working for our multi-cluster environment.
Tanay is the Head of Developer Relations at n8n. He has published books on WebVR, virtual assistants on Raspberry Pi, and FirefoxOS. He has been listed in the about:credits of the Firefox web browser for his contributions to the different open source projects of the Mozilla Foundation. I’ve been involved in the DevOps world for a while and yet I finished reading The Phoenix Project only recently. The book piqued my interest in how teams execute their incident response playbooks.
On Monday, March 16, 2020, I had the privilege to (virtually) join Shota Gvinepadze and his students at the Free University of Tbilisi and speak about “Advanced Git @ Mattermost” for a portion of their class time. The following are my speaking notes from the session, slightly modified from the original slides for this format. Keep in mind that the command line examples are illustrative of my workflow, and not meant to be run in isolation.
