Shipping Kubernetes Cluster Metrics to Elasticsearch with Metricbeat

Kubernetes is a popular container orchestration and container management platform for automating deployment, scheduling, and update of your containerized workloads in distributed compute environments. It goes without saying that managing multiple nodes and applications in Kubernetes requires an efficient monitoring system. You need to have a real-time picture of events happening in your cluster to get actionable insights for optimization and improving performance.

Kubernetes ships with some default monitoring and metrics solutions like Heapster and Kubernetes Metrics Server. However, in order to apply analytics, do data discovery, and visualize metrics data flowing from your cluster, you’ll be better off using solutions designed specifically for such type of tasks. One popular option for log and metrics monitoring and analysis is the ELK stack (Elasticsearch, Logstash, Kibana) used in pair with Elastic Beats log shippers.

In this article, we introduce you to monitoring Kubernetes with ELK and Elastic Beats. In particular, we’ll show how to send Kubernetes metrics to Elasticsearch indexes using Metricbeat and access them in your Kibana dashboard for subsequent processing. Let’s get started!

What is Metricbeat?

Metricbeat is a lightweight data shipper that can periodically collect metrics from the OS, applications, and services running on the server(s). It can be configured to work with a variety of metric sources like OS, Apache servers, MongoDB, and Zookeeper and to send their metrics directly to Elasticsearch.

In this article, we are interested in Metricbeat’s support for Kubernetes metrics. Metricbeat ships with the Kubernetes module that allows fetching metrics from Kubernetes kubelet agent and kube-state-metrics service. The module contains a number of metrics sets with container, node, pod, system, and volume metrics enabled by default. These sets collect both cluster-level, node-level, and application-level metrics. The module also supports fetching metrics from kube-state-metric service. However, the service should be manually deployed before running Metricbeat on Kubernetes.

In this article, we describe a minimal Metricbeat configuration required for shipping system metrics and default Kubernetes metrics mentioned above.

Tutorial

To complete examples in this tutorial, you need:

• a running Kubernetes cluster. See Supergiant GitHub wiki for more information about deploying a Kubernetes cluster with Supergiant. As an alternative, you can install a single-node Kubernetes cluster on a local system using Minikube.
• a kubectl command line tool installed and configured to communicate with the cluster. See how to install kubectl here.

To use the Kubernetes module, we’ll first need to deploy Metricbeat on your Kubernetes cluster and connect it to the Kubernetes API server. Production clusters normally have more than one nodes spun up. If this is your case, you’ll need to deploy Metricbeat on each node so it can collect metrics from all nodes. The simplest way to do this in Kubernetes is to create a special kind of Deployment called DaemonSet. The DaemonSet controller will ensure that for every node running in your cluster you have a copy of the Metricbeat pod. The controller will also regularly check the number of nodes in the cluster and adjust the number of running Metricbeat pods if a new node is added or deleted. DaemonSet structure is especially suitable for monitoring solutions that need access to node-level metrics and logs.

Before deploying a Metricbeat DaemonSet, let’s first configure Metricbeat. We’ll do that with the help of Kubernetes ConfigMap API, which allows putting configuration in a separate resource and then referring to it in pods. We are going to define two ConfigMaps. The first one contains general configuration stored in the metricbeat.yml:

---
apiVersion: v1
kind: ConfigMap
metadata:
 name: metricbeat-config
 namespace: kube-system
 labels:
 k8s-app: metricbeat
data:
 metricbeat.yml: |-
 metricbeat.config.modules:
 # Mounted `metricbeat-daemonset-modules` configmap:
 path: ${path.config}/modules.d/*.yml
 # Reload module configs as they change:
 reload.enabled: false
 processors:
 - add_cloud_metadata:
 output.elasticsearch:
 hosts: ["YOUR_ELASTICSEARCH_HOST"]
 username: "YOUR_ELASTICSEARCH_USERNAME"
 password: "YOUR_ELASTICSEARCH_PASSWORD"
 setup.dashboards.enabled: true
 setup.kibana.host: "YOUR_KIBANA_HOST"
 setup.kibana.protocol: "https"
 setup.kibana.username: "YOUR_KIBANA_USERNAME"
 setup.kibana.password: "YOUR_KIBANA_PASSWORD"
 setup.template.settings:
 index.number_of_shards: 5
 index.number_of_replicas: 1
 index.number_of_routing_shards: 30
 

This ConfigMap defines the contents of metricbeat.yml file that provides a general configuration for Metricbeat. The ConfigMap will be saved in the kube-system namespace where Metricbeat should be installed. See the brief description of key configuration fields below:

• metricbeat.config.modules.path — this property specifies a Glob pattern to load external configuration files that are defined in the second ConfigMap.
• output.elasticsearch — specifies the output to send Metricbeat data to. We will be sending metrics to Elasticsearch. Therefore, you’ll need to specify your Elasticsearch host and username/password if credentials are needed.
• setup.dashboards.enabled — this Boolean property allows loading Kibana dashboards with pre-configured Metricbeat visualizations and searches.
• setup.kibana — loading Kibana dashboards into Kibana requires configuring the Kibana endpoint in Metricbeat. To configure the endpoint, you need to specify the Kibana host, protocol, and user credentials, if needed.
• setup.template.settings — this optional configuration allows specifying the template for Metricbeat index. In particular, you may specify the number of shards, replicas, and routing shards for your Metricbeat index. Note: you need to ensure that the number of shards and routing shards in the index is the factor of 30 (e.g., 6, 5, 3), or Metricbeat might complain and fail to create an event publisher.

Now, let’s save this file in the metricbeat-config.yaml and create the ConfigMap:

kubectl create -f metricbeat-config.yaml
configmap "metricbeat-config" created

Great! Let’s move on to the second ConfigMap that includes the configuration for our Kubernetes modules. The manifest looks like:

apiVersion: v1
kind: ConfigMap
metadata:
 name: metricbeat-modules
 namespace: kube-system
 labels:
 k8s-app: metricbeat
data:
 system.yml: |-
 - module: system
 period: 10s
 metricsets:
 - cpu
 - load
 - memory
 - network
 - process
 - process_summary
 processes: ['.*']
 process.include_top_n:
 by_cpu: 5 # include top 5 processes by CPU
 by_memory: 5 # include top 5 processes by memory
 kubernetes.yml: |-
 - module: kubernetes
 metricsets:
 - node
 - system
 - pod
 - container
 - volume
 enabled: true
 period: 10s
 hosts: ["192.168.99.100:8443"]

As you see, this ConfigMap defines two Metricbeat configuration files: system.yml and kubernetes.yml. The first file includes the configuration for the Metricbeat system module that collects common system statistics (e.g., running processes, CPU utilization, etc.), and the second file configures our Kubernetes module.

For the system module, we have specified six common metric sets such as CPU and load. We also configured the module to return the top 5 processes by CPU and RAM utilization. In its turn, the Kubernetes module is configured with five main metric sets (node, system, pod, container, and volume) collecting data every 10 seconds.

We have also specified a host to access the Kubernetes API. If you are using Minikube, you can check the IP of the host by running minikube ip. By default, the Kubernetes master you need to access is running on localhost:10255. In any case, verify your host’s IP before putting anything into the configuration.

Now, let’s save this spec in the metricbeat-modules.yaml and create the ConfigMap:

kubectl create -f metricbeat-modules.yaml
configmap "metricbeat-modules" created 

Boom! We’ve created and saved configuration for our Metricbeat DaemonSet. Let’s create a spec for it:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
 name: metricbeat
 namespace: kube-system
 labels:
 k8s-app: metricbeat
spec:
 template:
 metadata:
 labels:
 k8s-app: metricbeat
 spec:
 terminationGracePeriodSeconds: 30
 hostNetwork: true
 dnsPolicy: ClusterFirstWithHostNet
 containers:
 - name: metricbeat
 image: docker.elastic.co/beats/metricbeat:6.3.2
 args: [
 "-c", "/etc/metricbeat.yml",
 "-e",
 "-system.hostfs=/hostfs",
 ]
 env:
 - name: ELASTICSEARCH_HOST
 value: "YOUR_ELASTICSEARCH_HOST"
 - name: ELASTICSEARCH_PORT
 value: "YOUR_ELASTICSEARCH_PORT"
 - name: ELASTICSEARCH_USERNAME
 value: "YOUR_ELASTICSEARCH_USERNAME"
 - name: ELASTICSEARCH_PASSWORD
 value: "YOUR_ELASTICSEARCH_PASSWORD"
 - name: POD_NAMESPACE
 valueFrom:
 fieldRef:
 fieldPath: metadata.namespace
 securityContext:
 runAsUser: 0
 resources:
 limits:
 memory: 200Mi
 requests:
 cpu: 100m
 memory: 100Mi
 volumeMounts:
 - name: config
 mountPath: /etc/metricbeat.yml
 readOnly: true
 subPath: metricbeat.yml
 - name: modules
 mountPath: /usr/share/metricbeat/modules.d
 readOnly: true
 - name: dockersock
 mountPath: /var/run/docker.sock
 - name: proc
 mountPath: /hostfs/proc
 readOnly: true
 - name: cgroup
 mountPath: /hostfs/sys/fs/cgroup
 readOnly: true
 volumes:
 - name: proc
 hostPath:
 path: /proc
 - name: cgroup
 hostPath:
 path: /sys/fs/cgroup
 - name: dockersock
 hostPath:
 path: /var/run/docker.sock
 - name: config
 configMap:
 defaultMode: 0600
 name: metricbeat-config
 - name: modules
 configMap:
 defaultMode: 0600
 name: metricbeat-modules
 # We set an `emptyDir` here to ensure the manifest will deploy correctly.
 # It's recommended to change this to a `hostPath` folder, to ensure internal data
 # files survive pod changes (ie: version upgrade)
 - name: data
 emptyDir: {}

As you see, the spec came out a little bit lengthy. We will clarify everything for you in a moment.

• spec.template.spec.containers[].args — this field specifies commands to run the Metricbeat image with. These commands override the default Entrypoint and CMD of the image. In particular, we are going to run Metricbeat with the /etc/metricbeat.yml config and use -e flag set to -system.hostfs=/hostfs to allow Metricbeat to monitor the host machine from within the container.
• spec.template.spec.containers[].env — we specify several useful environmental variables that can be accessed inside the Metricbeat container. You can use them to store the name of your Elasticsearch host, port, username, password, and the namespace where the DaemonSet runs,
• spec.template.spec.containers[].securityContext.runAsUser — this field specifies a security context for the Metricbeat container. In particular, we are running the container as root (0) to enable access to the system metrics and statistics.
• spec.template.spec.containers[].resources — this object specifies resource request and limits for the Metricbeat container. See our recent tutorial for more information about assigning container resources in Kubernetes.
• spec.template.spec.volumes — we specified a set of the hostPath and configMap volumes for our DaemonSet. Some of them will store Docker and cgroup data required by Metricbeat to monitor the system and others will store Metricbeat configuration specified in the ConfigMaps. Each volume was mounted at the corresponding path in the Metricbeat container so that the program has access to all needed configuration to run its System and Kubernetes modules.

Now, save the spec above in the metricbeat-daemonset.yaml and create the DaemonSet with the following command:

kubectl create -f metricbeat-daemonset.yaml
daemonset "metricbeat" created

Almost immediately Metricbeat will start shipping your Kubernetes metrics to Elasticsearch.

Log in to your Kibana dashboard and set up Metricbeat index pattern following simple steps described there. If Kubernetes metrics was successfully saved in the Metricbeat index, you’ll be able to create that index pattern that looks like this:

You can also check data from different metric sets sent to Elasticsearch using Discover tab.

Now, as your Kubernetes metrics is saved in Elasticsearch, you can produce visualizations in Kibana or use Metricbeat sample visualizations that come with Kibana dashboards out of the box. You can learn how to create visualizations of Metricbeat metrics in Kibana in this tutorial.

Conclusion

That’s it! We hope you now have a better understanding of how to ship Kubernetes metrics and monitor your Kubernetes cluster using Metricbeat. The main benefit of using Metricbeat with Kubernetes is that this shipper is very lightweight and easy to deploy in Kubernetes and that it can directly ship cluster metrics to Elasticsearch. Once a metrics pipeline is set up to work with Elasticsearch, you can use Kibana native data analytics and visualization tools to infer insights from your cluster and applications’ data.

Metricbeat’s Kubernetes module includes a variety of useful metrics to monitor pods, containers, volumes, and other Kubernetes objects running in all nodes of your cluster. Also, you can optionally configure access to the metrics on the state and health of pods, nodes, deployments, or ReplicaSets by installing kube-state-metrics server. Once that’s done, you can easily retrieve state information and process it in Elasticsearch and Kibana.