Container environments are very dynamic and fluid. New containers are stopped and started all the time due to scaling, rescheduling to new nodes, and updating. Solutions for monitoring containers should thus be flexible, too. For example, we should be able to respond to container start/stop events by launching or stopping some monitoring services and modules. In this way, we can dynamically respond to the changing state of the container environment. 

Starting with version 6.1, Metricbeat introduced support for the Autodiscover feature that allows tracking Docker and Kubernetes APIs to respond to container start and stop events. For example, if a new Apache HTTP container is launched, Autodiscover would automatically enable the Apache module with specific metric sets and channel Apache events to the specified output. Without this feature, we would have to launch all Filebeat or Metricbeat modules manually before running the shipper or change a configuration when a container starts/stops. Autodiscover solves this problem well. Let's see how to set up it with Metricbeat and send Docker container metrics directly to Elasticsearch.

How Does Autodiscover Work?

Autodiscover includes a number of providers such as Docker and Kubernetes that listen to container API events. For example, the Docker provider listens for the Docker API container start/stop events. When a new event is captured, the Docker provider gets the event data and transforms it into the internal Autodiscover event.

Metricbeat Autodiscover Architecture

This is done by populating Docker event fields into Metricbeat internal variables that can be then referenced inside the Autodiscover configuration. The Docker provider sees the following fields of the Docker container events:

  • host -- the host on which the Docker container runs.
  • port -- the port on which the Docker container listens.
  • docker.container.id -- id of the Docker container
  • docker.container.image -- container image used to start the container
  • docker.container.name -- container's name.
  • docker.container.labels -- container's labels.

For example, when the Apache httpd starts, the Docker provider can create the following internal event object:

{
  "host": "13.4.15.8",
  "port": 80,
  "docker": {
    "container": {
      "id": "384584ecdb385cfd5d1f1a65f78921054c8511ae009655300ac28b4fc357ce51"
      "name": "httpd",
      "image": "httpd:2.4.34",
      "labels": {
        "io.kubernetes.pod.namespace": "default"
      }
    }
  }
}

To control how the provider responds to the event, you must define a condition and a config template for that provider. As a condition, for example, we can specify that the Autodiscover provider takes action if the docker.container.image value matches httpd (i.e., Apache HTTP server). The action taken can be specified in the config template. For example, we can tell Metricbeat to start the  Apache module with a specified list of metrics whenever the Docker httpd container is started. We expand on this in the tutorial below.

Tutorial Prerequisites

  • This tutorial was tested on Ubuntu 16.04 (Xenial Xerus).
  • We used Metricbeat 6.3.2 downloaded from the apt repository.
  • We used Docker 18.03.1-ce.

We assume that you already have a working Docker environment on your system. If not, see the official Docker installation guide and learn how to run some containers as daemons.

Install Metricbeat

Once you prepare your Docker environment, we can proceed with installing Metricbeat. As you might already know, Metricbeat is a lightweight data shipper that can periodically collect metrics from the OS, applications, and services running on the server(s). It can be configured to work with a variety of metric sources like OS, Redis, Zookeeper, and Apache and send their metrics directly to Elasticsearch.

To install Metricbeat, you'll first have to add Elastic's signin key needed to verify the downloaded package:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key 
add -

Next, add Elastic repository to your repository source list:

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo 
tee -a /etc/apt/sources.list.d/elastic-6.x.list

Finally, update the repos on your system and install Metricbeat using apt-get:

sudo apt-get update && sudo apt-get install metricbeat

Metricbeat General Configuration

To run Metricbeat, you should configure input(s) (metrics sources like Docker), output/s (remote service(s) or database(s) to send metrics to), and various modules if needed. This configuration is located in the metricbeat.yml inside the Metricbeat folder. Take a look at the edits we've made:

#==========================  Modules configuration ============================
metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: false
  # Period on which files under path should be checked for changes
  reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana.host: "YOUR_KIBANA_HOST"
setup.kibana.protocol: "https"
setup.kibana.username: "YOUR_KIBANA_USERNAME"
setup.kibana.password: "YOUR_KIBANA_PASSWORD"
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
      hosts: ["YOUR_ELASTICSEARCH_HOST"]
      username: "YOUR_ELASTICSEARCH_USERNAME"
      password: "YOUR_ELASTICSEARCH_PASSWORD"

Note: Make you sure to specify actual Elasticsearch and Kibana hosts and credentials if needed in your configuration.

As you see, we've specified minimal settings needed for the Metricbeat to work. However, we also need to configure our Autodiscover feature with the Docker provider. Let's add these settings to the metricbeat.yml and we'll explain them right now:

autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: nginx
          config:
            - module: nginx
              metricsets: ["stubstatus"]
              hosts: "${data.host}:${data.port}"

Let's describe key fields used in this configuration:

autodiscover.providers.type.docker -- We use the Docker autodiscover provider that watches for Docker container start/stop events.

autodiscover.providers.templates.condition.contains -- The condition under which the Docker autodiscover provider takes action. In our case, the condition is when the container image of the started container contains "nginx" in its name.

autodiscover.providers.templates.config -- The action to take if the condition is met. The action specified is to start Metricbeat Nginx module to ship metrics from the Nginx container.

Also, as you see, the configuration template contains ${data.host} and ${data.port} variables. Recall that the Autodiscover provider puts data from the Docker API event into the internal data object and exposes it to the configuration template. Therefore, ${data.host} variable contains the host of the Nginx container and ${data.port} variable stores the port number. Such variable expansion is very useful because Docker may assign IPs and ports dynamically, and we cannot safely hardcode the host value before the container is started.

Great! The Autodiscover is configured and we can start Metricbeat.

On Linux, you can run Metricbeat in the shell specifying the config file as a parameter (the -e flag tells Metricbeat to log to stdout, which is useful when we run it in shell).

sudo ./metricbeat -e -c metricbeat.yml

Alternatively, you can start Metricbeat as a service to run at startup as a daemon:

sudo service metricbeat start

If you specified some inputs and modules in your configuration, Metricbeat will begin sending metrics to Elasticsearch index almost immediately. However, if the Docker Nginx container has not been started yet, our Autodiscover feature won't launch the Nginx module.

Let's manually launch the Nginx container to trigger the Autodiscover event.

Assuming that you have a running Docker on your system, run the following command in your terminal:

docker run -d nginx 

This command will start the Docker nginx container in the background and Nginx metrics will be now shipped to Elasticsearch.

Let's create an index pattern for the Metricbeat index in Kibana and access metrics directly from the Kibana dashboard. Log in to your Kibana dashboard and follow simple steps for creating a new Index Pattern under Management -> Index Patterns -> Create Index Pattern. If everything is ok, you'll see the following index mapping:

Metricbeat data in Kibana


Once the Autodiscover captures the first Nginx event, it will start shipping metrics directly to Elasticsearch. Correspondingly, if the Nginx container is stopped for some reason, the Autodiscover container stop event will be detected, and Metricbeat will stop monitoring the Nginx container. You can check Nginx metrics under the Discover tab of your Kibana dashboard or using REST API tools like curl.

Conclusion

In this tutorial, we discussed a new Autodiscover feature introduced in Metricbeat 6.1 and also available in other Beats components such as Filebeat. Autodiscover is perfect for dynamic container environments where containers are frequently started and stopped. Being able to dynamically capture container start/stop events and respond to them by starting/stopping Metricbeat monitoring modules adds flexibility to your monitoring system. Autodiscover's simple declarative syntax allows configuring metric sets, hosts, ports before the container started avoiding the need to launch, and stop all modules manually. In this article, we have discussed only Docker Autodiscover, but Metricbeat also supports Kubernetes Autodiscover and a number of other features like hint-based Autodiscover. They will be discussed in the next tutorials.