Elasticsearch: How to install and set up FileBeat on Ubuntu 20.04.

Simple, it’s just a few steps. Really!

Photo by Andrea Piacquadio from Pexels

In my previous articles, I’ve been talking about how to use the Elastic stack to consume and process logs from your systems and softwares and we’ve been using Logstash as the core technology for these processing pipelines.

However, Logstash can be pretty heavy when it comes to installing it on the machine you want to collect logs from. That’s where the beats come into play. The beats intend to let you ship logs and metrics from thousands of machines, systems, and softwares, without worrying about long setup and heavy software installations. Each beat is meant to ship different types of data from systems logs, network data to metrics logs, and so on.

FileBeat is one of the beats family members. It helps you collect data from security devices, the cloud, containers, and hosts.

In this tutorial, we’ll walk through the process of installing FileBeat on Ubuntu 20.04 and start collecting logs.

Let’s jump right in.

1- Install Filebeat on your system

Let’s first upgrade and update our system.

sudo apt-get update && sudo apt-get upgrade

Much simpler than you could think of, here is how the magic happens:

sudo apt-get install filebeat

And that it is, Filebeat is installed.

2- Configure Filebeat to send data to Elasticsearch

The next step of our setup is to tell Filebeat which Elasticsearch cluster it has to connect to in order to send the collected data.

Let’s head up to your filebeat.yml and open it.

cd /etc/filebeat
sudo nano filebeat.yml

Once opened, edit the output section with your Elasticsearch host data:

output.elasticsearch:
  hosts: ["ELASTICSEARCH-HOST-ADDRESS:9200"]
  username: "YOUR_ELASTICSEARCH_USERNAME"
  password: "YOUR_PASSWORD"

Chances are that you want to visualize and get insights from the collected logs on Kibana. If that’s the case, let’s configure Kibana access for filebeat.

3- Configure Filebeat to send data to Kibana

Skip this step if Kibana is running on the same host as Elasticsearch. Let's configure the Kibana endpoint:

setup.kibana:
    host: "KIBANA-HOST-ADDRESS:5601" 
    username: "YOUR_KIBANA_USERNAME"  
    password: "YOUR_PASSWORD"

4- Set the path to your logs

When we look into that Kibana config file, we notice that Filebeat reads logs base on some YAML files located at ${kibana-path}/modules.d.

Let’s navigate there and configure our first logs shipping YAML file.

For the sake of example, we’re going to configure Filebeat to ship apache logs.

cd /etc/filebeat/modules.d
ls -l

You can see there are tons of systems configuration files ready to use. Let’s copy the pre-made apache logs processing config file and edit it.

cp apache.yml.disabled apache.yml
sudo nano apache.yml

Then we can add where we want that our apache logs to get read. Let’s edit the apache section.

- module: apache
  # Access logs
  access:
    enabled: true
# Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: /var/log/apache2/access.log
# Error logs
  error:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: /var/log/apache2/error.log

And then save and close it. I know I know that this can be a little bit boring but hold on if you’re really interested in seeing the magic happen. Let’s start Filebeat.

sudo systemctl start filebeat

5- Set up Kibana dashboard for Filebeat

Filebeat offers this amazing feature. It comes with a kind of pre-built dashboard that can be set up on Kibana with the data gathered by Filebeat.

Let’s set up that dashboard.

cd /usr/share/filebeat/bin
sudo filebeat setup --dashboards

And then move to your Kibana instance host and restart Kibana.

6- Visualize the output on Kibana

Finally, we can look at the result of our long long setup. Navigte to your Kibana Home Page. On your local, the address may be 127.0.0.1:5601.

Then create an index pattern name filebeat-* and you’re good to go.

Now you can look at all of your apache server logs, get insights from them and get to know if your users are facing some issues accessing some of your website pages.

That is it folks. Till next time, take care!