Monitor HAProxy using ELK

Now when I have a pretty ElastiFlow dashboard to monitor my network I need to monitor more. So lets install filebeat on my FreeBSD server to monitor my reverse proxy, HAProxy, by sending the logs over to my ELK stack.

Installing Beats

Installing Beats was easy, it was already present in the FreeBSD repo. I’m using pre-built packages with pkg just because I don’t have enough walls to climb while I’m waiting for ports to build..

# pkg install beats7
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    beats7: 7.7.1
    
Number of packages to be installed: 1

The process will require 122 MiB more space.
28 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching beats7-7.7.1.txz: 100%   28 MiB   7.3MB/s    00:04    
Checking integrity... done (0 conflicting)
[1/1] Installing beats7-7.7.1...
[1/1] Extracting beats7-7.7.1: 100%

Beats configuration

These are the settings I have in /usr/local/etc/beats/filebeat.yml

filebeat.config.module:
  path: ${path.config}/filebeat.modules.d/*.yml
  reload.enabled: false
  
setup.dashboards.enable: true

setup.kibana:
  host: "192.168.1.17:5601"
  protocol: "http"
  ssl.enabled: false
  space.id: "Beats" # This will create a 'Beats' space in kibana for all beats dashboards
  
output.elasticsearch:
  hosts: ["192.168.1.17:9200"]  

Enable haproxy module

There configuration file is located at /usr/local/etc/beats/filebeats.modules.d/haproxy.yml.disable we need to make some changes

# Module: haproxy
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.7/filebeat-module-hapr$

- module: haproxy
  # All logs
  log:
    enabled: true

    # Set which input to use between syslog (default) or file.
    var.input: "file"

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/haproxy.log"]

My HAProxy install is configured to log to logfile and not syslog. If you are logging to syslog change to var.input: "syslog" or just leave it uncommented. No need to add any paths if you are using syslog.

Then we need to rename the file to enable it

cd /usr/local/etc/beats/filebeats.modules.d/
mv haproxy.yml.disable haproxy.yml

Lots of errors

Well when I finally started filebeat service filebeat start it did not work, it failed on uploading the dashboards to Kibana.

Upgrade Beats

First I installed Beats version matching my ELK stack, v7.8.1.

Installing build dependencies

need to install some packages to be able to build beats

cd /usr/local
fetch https://golang.org/dl/go1.15.1.freebsd-amd64.tar.gz
tar xfz go1.15.1.freebsd-amd64.tar.gz
rm go1.15.1.freebsd-amd64.tar.gz

Download latest go Add new path in .cshrc or whatever shell you are running

set path = (/usr/local/go/bin ...)

Install Git, bash and gmake from packages

pkg install -y git gmake bash

Download an install Beats 7.8.1 from source

cd && mkdir build
git clone https://github.com/elastic/beats.git
cd beats
git checkout v7.8.1

I’m only using filebeat so lets build that one

cd filebeat
gmake

Replace current version

mv /usr/local/bin/filebeat /usr/local/bin/filebeat.old
cp filebeat /usr/local/bin/filebeat

Verify version

# filebeat version
filebeat version 7.8.1 (amd64), libbeat 7.8.1 [unknown built unknown]

Still dashboard errors

After lots of testing and compiling, I still could not get the dashboards to upload to kibana.

The solution to fix the Dashboards

Finally I found some posts somewhere (I lost the link) saying there is errors in the dashboards in the FreeBSD package. To solve this issue we need to download the Linux package and copy the dashboards from there.

This is a pretty quick solution, I downloaded the Linux version that match my installed version

fetch https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-linux-x86_64.tar.gz
tar xfz filebeat-7.8.1-linux-x86_64.tar.gz
cp filebeat-7.8.1-linux-x86_64/kibana/7/dashboard/*.json /usr/local/share/beats/filebeat/kibana/7/dashboard/

And finally filebeat started and uploaded all the dashboards to Kibana.

I have not reverted my upgraded filebeat, so I don’t know if this works on 7.7.1 which was the version I got from FreeBSD packages. But I can’t see why it should not work.