Troubleshooting Kubernetes using Robusta on Kind

Introduction to Robusta
Robusta is an observation and remediation platform that provides valuable insights into Kubernetes clusters and integrates with various monitoring and alerting tools.

It’s an event processing and automation engine that listens to events from various sources e.g. Prometheus alerts (which are received by a webhook), logs, changes from API server in Kubernetes, and many other sources of events. Robusta uses these events to run automations which are defined in a YAML file. Below we will quickly look at Robusta architecture at a higher level and then dive into an example where we see it in action.

The Architecture of Robusta

Before setting up Robusta on your Kubernetes cluster, let’s understand how Robusta works.

It has two major components (deployed as k8s deployments):

Robusta-forwarder uses a Kubewatch image to monitor changes on your Kubernetes cluster using API server. It sends all the API server events to Robusta-runner.
Robusta-runner checks if there’s any playbook that should be triggered by that event. Whether to take an action on an event, is configured via playbooks (we will look at it in the example).
Besides these deployments, Robusta also provides Kubernetes devops with optional components such as

Prometheus stack (Prometheus Operator, Alert Manager, Grafana) which is pre-configured to work with Kubernetes. Robusta configures an alert manager receiver that sends the Prometheus alerts to Robusta.
Default Prometheus alerts.
Client-side Robusta CLI.
Cloud UI hosted by Robusta to view health and alerts of your cluster.
Next, we will see how to set up Robusta on a local cluster using kind.

Setting Robusta on Local Environment
Kind is a tool used by devops solutions providers for setting up a local Kubernetes cluster on your machine using Docker container “nodes”. It can be used for local development or CI, although it was originally created for testing Kubernetes itself. It supports Linux, macOS, and Windows.

Requirements
Before setting up your cluster, you need to install the following dependencies.

Helm (https://helm.sh/docs/intro/install/)
pip with python version 2 or 3 (https://pip.pypa.io/en/stable/installation/)
Kind CLI (https://kind.sigs.k8s.io/docs/user/quick-start/)
1. Create your Kubernetes Cluster
Create your cluster by running the following command.

2. Install Robusta – CLI
Run the following commands to install Helm charts and Robusta – CLI.

Figure: Installing Robusta
3. Create your own Slack workspace (Optional)
Create a Slack workspace to receive alerts from Robusta by going to https://slack.com/help/articles/206845317-Create-a-Slack-workspace.

Note: This step is optional if you already have a slack workspace on which you are an owner/admin/moderator.

4. Generated YAML file for Helm
After installing Robusta-CLI, you have to configure Robusta by running the command below, which will generate a file named “generated_values.yaml” for Helm installation of Robusta.

Note: When Robusta-CLI asks for your email address make sure you provide a Gmail/Microsoft address as the cloud UI of Robusta as of September 2022 only supports “Login with Google/Azure”.

There would be multiple questions when configuring Robusta make sure you configure it correctly by following the image below.

During the installation, Robusta-CLI will ask you to configure the Slack integration. After saying “Y” it will prompt you to connect Robusta to your Slack workspace.

Figure: Connecting Robusta to Slack
After clicking “Add to Slack”, select your Slack workspace from the top right corner and press “Allow”.

Figure: Giving Robusta permission to post on Slack
After authorization, Robusta will be connected to your Slack workspace, and you can continue with the remaining steps in your terminal.

Figure: Robusta connected to Slack
After completing the remaining steps, you will receive an alert from Robusta and a generated-values.yaml file will be saved in the current directory, which will be used during your helm installation of Robusta.

5. Install Robusta using Helm
Using the “generated_values.yaml” file, we can install Robusta on your kind cluster by running the following command, which will install your Robusta deployments in a separate namespace called “Robusta”.

Figure: Message by Robusta after installation
Run the following command and wait for both deployments to be in running state.

Figure: Robusta deployments
Example Robusta Usage (Crashing Pod)
After installing Robusta successfully on your cluster, it’s time to test it and see if we are receiving alerts on your Slack. We can do this by setting up a crashing pod on your cluster.

1. Deploy a crashing pod
Run the following command to set up a crashing pod.

2. Check if the pod is crashing
Run the following command and wait for the pod to crash two times, which will trigger Robusta to send an alert to your Slack.

Figure: Pod crashed after deployment
3. Check Slack for Robusta Alert
Open your slack workspace and check if you received a notification from Robusta about your crashing pod.

Figure: Slack notification by Robusta
4. Login to Robusta Cloud UI
Go to https://platform.robusta.dev/, which is Robusta’s cloud UI where you can watch events, alerts, and monitor the health of your cluster. You can also silence specific alerts from the cloud UI.

Login with Google by using the same Gmail address you provided to Robusta-CLI when generating your “generated_values.yaml” file.

Ref: https://platform.robusta.dev/
Click on the crashpod on the card, where you can see all the events under the issue tab of that pod. You can also see other apps also deployed on your cluster in other namespaces because the filters are set to “ALL”. You can set filters according to your choice to only see the apps you want.

Figure: Robusta Cloud-UI
After clicking the crashpod card, you can see all the issues with the pod. You can click on it to see all the details related to the pod.

Figure: Pod details of the cluster on Robusta Cloud – UI
In the image below, you can see the complete flow from the pod crashing to Robusta alerting you about it on your Slack.

Figure: Flowchart of a playbook in Robusta.
Conclusion
In this post, we saw how Robusta enables automation and contextualizes Kubernetes events from various data sources. Analyzing events and taking actions, is at the heart of Kubernetes observability and Robusta makes it easier for us to correlate events and take appropriate actions.

Please leave any comments or suggestions in the section below, and don’t forget to like it if you found this useful. Also, please let me know how useful you found Robusta. We will be releasing more tutorials in the future where we explore more about Robusta.

Next Blog
Our next blog post will guide you through the process of writing your own custom playbook action in Robusta. Learn how to create customized automation workflows by defining inputs, and outputs, and writing the code to execute during the playbook run. Stay tuned for more details on how to extend Robusta’s capabilities to meet your specific automation needs.