In my last post ELK Stack Monitoring On Ubuntu 18.04 I attempted to get hands down on full ELK stack monitoring of your ubuntu machines. I ended up with a brief overview of Kibana, and how to install ElasticSearch and MetricBeats.

Today I want to make up for that and establish Logstash as part of the team.

Install Logstash

First, let us bring the os up to date. The last update is a week old. Then, in the next step, install Logstash from the repositories that we already setup last time.

$ apt update
$ apt upgrade
$ apt install logstash

Configuring Logstash

Logstash uses pipelines to receive, enrich, format, and send data to elasticsearch. So in order to have Logstash get to work, we have to setup such a pipeline. Let’s start with an easy one, to get an idea of it.

Let’s create a file called /etc/logstash/conf.d/system-metrics-pipeline.conf. Put the following text in it:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Restart the service with systemctl restart logstash.

Now head over to Kibana in your browser, reload the tab and start to panic. Your Dashboard should now not show any metrics anymore. To change this, we have to change MetricBeat, so it sends data to Logstash directly instead of ElasticSearch.

Configure MetricBeat

As already said, the default output of MetricBeat sends data to ElasticSearch. Change /etc/metricbeat/metricbeat.yml like so:

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch: <-- comment this line out
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"] # <-- comment this line out as well

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:  # <-- uncomment this line
  # The Logstash hosts
  hosts: ["localhost:5044"]  # <-- uncomment this line as well

Now systemctl restart metricbeat, refresh your browser and feel a little relief. Your metrics are working again. Another thing that Logstash is really great with are filters. Filters let you take values from any field of your input stream, format them, change them.. think “verything” them, then put them back into the stream. I’ll cover that in a later post.

Send to Logstash from multiple hosts

In order to send data to Logstash from more than just “localhost”, we need to set it up accordingly. From my previous post just follow the steps to:

  • install Java Runtime Environment
  • add the ElasticSearch 6.x Repository
  • install MetricBeat
  • configure MetricBeat to send data to our ELK Server (make sure to enter the correct hostname or IP in /etc/metricbeat/modules.d/system.yml and send it to the Logstash output, not the ElasticSearch output)
  • restart MetricBeat

Once these steps are complete, data from your other hosts will start pouring into Elasticsearch accordingly.

The [Metricbeat System] Overview dashboard should now show number of total hosts that are sending their system log to their master.

Kibana: System Overview

Summary

We do now know how to setup Logstash on our master server to receive data. Also we have MetricBeat sending data to the right output and can send data from other hosts as well.

From my first post in this series, there are still some things to do, and many more to explore.

Thank you for reading!

If you have anything to add or questions to ask, feel free to shoot me a message!