Last November at CloudNativeCon, we introduced the Operators pattern. It’s a way to extend Kubernetes’ self-healing features to the complex world of stateful apps.
For instance, consider what happens if a database node crashes. The ideal response may not be as simple as restarting the container. Checking what’s on disk and rejoining the cluster likely requires a bit more nuance. Operators combine third party resources + custom controllers to provide that app-specific, nuanced response.
We created the first set of Operators while working on the “self-driving infrastructure” that powers CoreOS Tectonic. When we saw their potential first hand, we were excited to share the concept with the broader Kubernetes community. That first look at Operators happened around CloudNativeCon last November. This week, as we celebrate CloudNativeCon in Berlin, we thought it might be fitting to revisit the idea.
Since then we’ve seen a vibrant Operator ecosystem emerge. Distributed databases, message queues, and distributed filesystems are a few of the domains where we’ve seen Operators sprout up. This post is an update and a look at some of their specific uses, today.
In our initial Operators post, we included examples for Prometheus and etcd. We’ve continued to work on each. The etcd Operator has even seen a Helm chart appear. That makes it easier than ever to install on your own cluster.
We noticed Operators also found a great fit in the byzantine world of highly available storage. Message queues saw representation in the form of the Kafka Operator. Elasticsearch also had it’s own Operator developed. And, Rook represents possibly one of the most challenging implementations, as an Operator for managing block and object storage.
We’re pleased to see the Operators ecosystem is continuing to grow, making complex applications easy to deploy on Kubernetes. Are you working on an Operator? We’d love to hear about it.
For this post, we reached out to some community leaders working with Operators. We asked them to share their experience as early adopters. Below are some of their comments.
Travis Nielson, creator of the Rook Operator: We were already working on bringing Rook storage to the Kubernetes world before operators were announced last fall, just with a different name and slightly different approach. We realized the operator pattern was a natural evolution of where we were headed to automate storage integration with Kubernetes.
Steve Sloka, creator of the Elasticsearch Operator: Operators help me automate my infrastructure. At it's simplest, in the case of the Elasticsearch operator, it will distribute the data nodes across availability zones without me having to figure out which zones are in the particular cluster I'm deploying to, as well as automatically backing up my indexes to AWS S3. These are all things that I would typically have to write into a utility app somehow (or perform manually), but now it's all bundled within. Consumers of the operator now get the same features for free, which takes away the operational burden from them, they can just assume it works and not have to think about it (just like Kubernetes!)
Jakob Karalus, creator of the Kafka Operator: I am writing this operator to make running a Kafka cluster in Kubernetes easier. While it’s easy to get a Kafka cluster up and running with stateful sets, it also lacks a lot of operational knowledge. The Operator pattern allows me to implement specific state transitions (which the default kubernetes "state machine" does not have). Upsizing with rebalancing is for example a use case where a custom state transition is needed. In combination with ThirdPartyResources the Operator allows me to encapsulate complex behavior behind a simple and familiar API which can be consumed by developers.
Travis Nielson, creator of the Rook Operator: The Operator pattern is actually quite straight-forward. You have a piece of software that automates management of your application. The complicated part is really in the application that the Operator is managing. It's a really hard problem to ensure that a stateful application is managed correctly so you are confident that your data is safe through upgrades, failover, etc. The advice I have for others writing Operators is to focus on areas of your stateful app that require manual intervention, and use the operator to automate the pain points. It's all about making the application work more reliably in the Kubernetes cluster.
Steve Sloka, creator of the Elasticsearch Operator: The hardest part I ran into was integrating into the Kube Api. I used client-go which has been going through some flux. I know how to build out deployments and replication controllers in yaml or json, but doing in Go was a new experience. Some pieces were easy but others weren't and I could see how it could be intimidating for folks.
Jakob Karalus, creator of the Kafka Operator: For new people writing operators it would be nice to have something like a hello world operator or a shared library. I looked at the Prometheus/etcd Operators in the beginning, but they are quite complex on their own. So, a simple Hello World example would be highly beneficial to get people started. I think that the Operator pattern is great for Kubernetes as it allows to add custom behavior into the system without the overhead of a full blown scheduler or framework. I also feel that the pattern isn't really known in the community (and outside) and could be promoted better.
First, a big thanks to Travis, Steve, and Jakob for sharing their thoughts. Second, the keen reader should note there are some big opportunities to contribute to Operators. We are thrilled to see the excitement and the resulting Operators ecosystem flourish.
Operators are still a young idea. Working with them, and sharing your experience is a fantastic way to help push the idea forward. As Operators become more commonplace, we’ll undoubtedly all be happier running our exceptionally robust clusters.
Hungry for more? Keep reading about Kubernetes Operators on our website. And while you’re at it, register to join us at CoreOS Fest on May 31 and June 1 for all things Kubernetes and CoreOS, and to learn even more about Kubernetes Operators.