
Shipping Kubernetes Cluster Metrics to Elasticsearch with Metricbeat
Posted by Kirill Goltsman August 12, 2018Kubernetes is a popular container orchestration and container management platform for automating deployment, scheduling, and update of your containerized workloads in distributed compute environments. It goes without saying that managing multiple nodes and applications in Kubernetes requires an efficient monitoring system. You need to have a real-time picture of events happening in your cluster to get actionable insights for optimization and improving performance.
Kubernetes ships with some default monitoring and metrics solutions like Heapster and Kubernetes Metrics Server. However, in order to apply analytics, do data discovery, and visualize metrics data flowing from your cluster, you’ll be better off using solutions designed specifically for such type of tasks. One popular option for log and metrics monitoring and analysis is the ELK stack (Elasticsearch, Logstash, Kibana) used in pair with Elastic Beats log shippers.
In this article, we introduce you to monitoring Kubernetes with ELK and Elastic Beats. In particular, we’ll show how to send Kubernetes metrics to Elasticsearch indexes using Metricbeat and access them in your Kibana dashboard for subsequent processing. Let’s get started!
What is Metricbeat?
Metricbeat is a lightweight data shipper that can periodically collect metrics from the OS, applications, and services running on the server(s). It can be configured to work with a variety of metric sources like OS, Apache servers, MongoDB, and Zookeeper and to send their metrics directly to Elasticsearch.
In this article, we are interested in Metricbeat’s support for Kubernetes metrics. Metricbeat ships with the Kubernetes module that allows fetching metrics from Kubernetes kubelet
agent and kube-state-metrics service. The module contains a number of metrics sets with container
, node
, pod
, system
, and volume
metrics enabled by default. These sets collect both cluster-level, node-level, and application-level metrics. The module also supports fetching metrics from kube-state-metric service. However, the service should be manually deployed before running Metricbeat on Kubernetes.
In this article, we describe a minimal Metricbeat configuration required for shipping system metrics and default Kubernetes metrics mentioned above.


Tutorial
To complete examples in this tutorial, you need:
- a running Kubernetes cluster. See Supergiant GitHub wiki for more information about deploying a Kubernetes cluster with Supergiant. As an alternative, you can install a single-node Kubernetes cluster on a local system using Minikube.
- a kubectl command line tool installed and configured to communicate with the cluster. See how to install kubectl here.
To use the Kubernetes module, we’ll first need to deploy Metricbeat on your Kubernetes cluster and connect it to the Kubernetes API server. Production clusters normally have more than one nodes spun up. If this is your case, you’ll need to deploy Metricbeat on each node so it can collect metrics from all nodes. The simplest way to do this in Kubernetes is to create a special kind of Deployment called DaemonSet. The DaemonSet
controller will ensure that for every node running in your cluster you have a copy of the Metricbeat pod. The controller will also regularly check the number of nodes in the cluster and adjust the number of running Metricbeat pods if a new node is added or deleted. DaemonSet
structure is especially suitable for monitoring solutions that need access to node-level metrics and logs.
Before deploying a Metricbeat DaemonSet
, let’s first configure Metricbeat. We’ll do that with the help of Kubernetes ConfigMap
API, which allows putting configuration in a separate resource and then referring to it in pods. We are going to define two ConfigMaps
. The first one contains general configuration stored in the metricbeat.yml
:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-config
namespace: kube-system
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
# Mounted `metricbeat-daemonset-modules` configmap:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
output.elasticsearch:
hosts: ["YOUR_ELASTICSEARCH_HOST"]
username: "YOUR_ELASTICSEARCH_USERNAME"
password: "YOUR_ELASTICSEARCH_PASSWORD"
setup.dashboards.enabled: true
setup.kibana.host: "YOUR_KIBANA_HOST"
setup.kibana.protocol: "https"
setup.kibana.username: "YOUR_KIBANA_USERNAME"
setup.kibana.password: "YOUR_KIBANA_PASSWORD"
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 1
index.number_of_routing_shards: 30
This ConfigMap
defines the contents of metricbeat.yml
file that provides a general configuration for Metricbeat. The ConfigMap
will be saved in the kube-system
namespace where Metricbeat should be installed. See the brief description of key configuration fields below:
metricbeat.config.modules.path
— this property specifies a Glob pattern to load external configuration files that are defined in the secondConfigMap
.output.elasticsearch
— specifies the output to send Metricbeat data to. We will be sending metrics to Elasticsearch. Therefore, you’ll need to specify your Elasticsearch host and username/password if credentials are needed.setup.dashboards.enabled
— this Boolean property allows loading Kibana dashboards with pre-configured Metricbeat visualizations and searches.setup.kibana
— loading Kibana dashboards into Kibana requires configuring the Kibana endpoint in Metricbeat. To configure the endpoint, you need to specify the Kibana host, protocol, and user credentials, if needed.setup.template.settings
— this optional configuration allows specifying the template for Metricbeat index. In particular, you may specify the number of shards, replicas, and routing shards for your Metricbeat index. Note: you need to ensure that the number of shards and routing shards in the index is the factor of 30 (e.g., 6, 5, 3), or Metricbeat might complain and fail to create an event publisher.
Now, let’s save this file in the metricbeat-config.yaml
and create the ConfigMap
:
kubectl create -f metricbeat-config.yaml
configmap "metricbeat-config" created
Great! Let’s move on to the second ConfigMap
that includes the configuration for our Kubernetes modules. The manifest looks like:
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-modules
namespace: kube-system
labels:
k8s-app: metricbeat
data:
system.yml: |-
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
kubernetes.yml: |-
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
enabled: true
period: 10s
hosts: ["192.168.99.100:8443"]
As you see, this ConfigMap
defines two Metricbeat configuration files: system.yml
and kubernetes.yml
. The first file includes the configuration for the Metricbeat system module that collects common system statistics (e.g., running processes, CPU utilization, etc.), and the second file configures our Kubernetes module.
For the system module, we have specified six common metric sets such as CPU and load. We also configured the module to return the top 5 processes by CPU and RAM utilization. In its turn, the Kubernetes module is configured with five main metric sets (node, system, pod, container, and volume) collecting data every 10 seconds.
We have also specified a host to access the Kubernetes API. If you are using Minikube, you can check the IP of the host by running minikube ip
. By default, the Kubernetes master you need to access is running on localhost:10255
. In any case, verify your host’s IP before putting anything into the configuration.
Now, let’s save this spec in the metricbeat-modules.yaml
and create the ConfigMap
:
kubectl create -f metricbeat-modules.yaml
configmap "metricbeat-modules" created
Boom! We’ve created and saved configuration for our Metricbeat DaemonSet
. Let’s create a spec for it:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
spec:
template:
metadata:
labels:
k8s-app: metricbeat
spec:
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:6.3.2
args: [
"-c", "/etc/metricbeat.yml",
"-e",
"-system.hostfs=/hostfs",
]
env:
- name: ELASTICSEARCH_HOST
value: "YOUR_ELASTICSEARCH_HOST"
- name: ELASTICSEARCH_PORT
value: "YOUR_ELASTICSEARCH_PORT"
- name: ELASTICSEARCH_USERNAME
value: "YOUR_ELASTICSEARCH_USERNAME"
- name: ELASTICSEARCH_PASSWORD
value: "YOUR_ELASTICSEARCH_PASSWORD"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
- name: dockersock
mountPath: /var/run/docker.sock
- name: proc
mountPath: /hostfs/proc
readOnly: true
- name: cgroup
mountPath: /hostfs/sys/fs/cgroup
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: config
configMap:
defaultMode: 0600
name: metricbeat-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-modules
# We set an `emptyDir` here to ensure the manifest will deploy correctly.
# It's recommended to change this to a `hostPath` folder, to ensure internal data
# files survive pod changes (ie: version upgrade)
- name: data
emptyDir: {}
As you see, the spec came out a little bit lengthy. We will clarify everything for you in a moment.
spec.template.spec.containers[].args
— this field specifies commands to run the Metricbeat image with. These commands override the default Entrypoint and CMD of the image. In particular, we are going to run Metricbeat with the/etc/metricbeat.yml
config and use-e
flag set to-system.hostfs=/hostfs
to allow Metricbeat to monitor the host machine from within the container.spec.template.spec.containers[].env
— we specify several useful environmental variables that can be accessed inside the Metricbeat container. You can use them to store the name of your Elasticsearch host, port, username, password, and the namespace where theDaemonSet
runs,spec.template.spec.containers[].securityContext.runAsUser
— this field specifies a security context for the Metricbeat container. In particular, we are running the container as root (0) to enable access to the system metrics and statistics.spec.template.spec.containers[].resources
— this object specifies resource request and limits for the Metricbeat container. See our recent tutorial for more information about assigning container resources in Kubernetes.spec.template.spec.volumes
— we specified a set of thehostPath
andconfigMap
volumes for ourDaemonSet
. Some of them will store Docker andcgroup
data required by Metricbeat to monitor the system and others will store Metricbeat configuration specified in theConfigMaps
. Each volume was mounted at the corresponding path in the Metricbeat container so that the program has access to all needed configuration to run its System and Kubernetes modules.
Now, save the spec above in the metricbeat-daemonset.yaml
and create the DaemonSet
with the following command:
kubectl create -f metricbeat-daemonset.yaml
daemonset "metricbeat" created
Almost immediately Metricbeat will start shipping your Kubernetes metrics to Elasticsearch.
Log in to your Kibana dashboard and set up Metricbeat index pattern following simple steps described there. If Kubernetes metrics was successfully saved in the Metricbeat index, you’ll be able to create that index pattern that looks like this:

You can also check data from different metric sets sent to Elasticsearch using Discover
tab.


Now, as your Kubernetes metrics is saved in Elasticsearch, you can produce visualizations in Kibana or use Metricbeat sample visualizations that come with Kibana dashboards out of the box. You can learn how to create visualizations of Metricbeat metrics in Kibana in this tutorial.
Conclusion
That’s it! We hope you now have a better understanding of how to ship Kubernetes metrics and monitor your Kubernetes cluster using Metricbeat. The main benefit of using Metricbeat with Kubernetes is that this shipper is very lightweight and easy to deploy in Kubernetes and that it can directly ship cluster metrics to Elasticsearch. Once a metrics pipeline is set up to work with Elasticsearch, you can use Kibana native data analytics and visualization tools to infer insights from your cluster and applications’ data.
Metricbeat’s Kubernetes module includes a variety of useful metrics to monitor pods, containers, volumes, and other Kubernetes objects running in all nodes of your cluster. Also, you can optionally configure access to the metrics on the state and health of pods, nodes, deployments, or ReplicaSets by installing kube-state-metrics
server. Once that’s done, you can easily retrieve state information and process it in Elasticsearch and Kibana.