Cool Solution - ElasticSearch

From Univention Wiki

Jump to: navigation, search
Produktlogo UCS Version 4.1

Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.

Also regard the legal notes at Terms of Service.
Note: This article is not yet reviewed.

Elastic Search together with Logstash allows the collection and processing of log files and metric data. ElasticSearch is the default backend for Kibana and can also be used with Grafana, both of which display the state of your environment.

Elastic Search in a VM

Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore it does not like to be suspended. While we did not experience any long-lasting issues, we had short term inconsistencies during the test, which can create a lot of false positive errors in the frontends.


Install Java

Download Java from [Oracle] and copy it to the server using SCP.

Create a directory under the optional directory

mkdir /opt/jdk

extract Java to the new directory

tar zxf <file name> -C /opt/jdk

Add Oracle Java to the update-alternatives

update-alternatives --install /usr/bin/java java /opt/jdk/<java version>/bin/java 100
update-alternatives --install /usr/bin/javac javac /opt/jdk/<java version>/bin/javac 100

Install sudo

Install sudo from the Univention Repository

univention-install sudo

Set the Repository

Add the Elastic GPG Key

wget -qO - | apt-key add -

Add the repository

echo "deb stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list

Install Elasticsearch

Install Elasticsearch using apt

apt-get update && apt-get install elasticsearch

Add elasticsearch to autostart

systemctl enable elasticsearch

Install Logstash

Logstash can be installed through the elastic repository

apt-get install logstash

Add elasticsearch to autostart

update-rc.d logstash defaults 95 10
service logstash start


Restart Elasticsearch

service elasticsearch restart

Configure Logstash

First Logstash needs multiple input and output plugins to process Data. Within this website, we will look at Beats and Syslog for input and output the Data to Elasticsearch.


Install the plugins for beats and syslog:

/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

Edit the configure file at


input {
  beats {
    port => 5044

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"

Logstash needs to have port 5044 open to accept Beats input. To open the port set the following UCR variable

ucr set security/packetfilter/tcp/5044/all=ACCEPT
ucr set security/packetfilter/udp/5044/all=ACCEPT
service univention-firewall restart


Install the plugins for beats and syslog:

/usr/share/logstash/bin/logstash-plugin install logstash-input-syslog
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

Edit the configure file at


input {
  syslog {

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][syslog]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"

By default logstash will be using the default syslog port 514. You need to open it in the UCS firewall with:

ucr set security/packetfilter/tcp/514/all=ACCEPT
ucr set security/packetfilter/udp/514/all=ACCEPT
service univention-firewall restart

Collect Data

Elasticsearch uses Beats to collect data. This article will look into Filebeat for Collecting Logfiles and Metricbeats for collecting data on the server. We will also show you how to enable the Syslog input.

If you are installing the Beat on a different server then the Elasticsearch server, you will need to add the repository and import the gpg key.

wget -qO - | apt-key add -
echo "deb stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list


The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also the FileBeat is better at shaping traffic, if you direct a lot of routine trafic but need to react fast to errors.

You can install the FileBeat from the repository

sudo apt-get update && sudo apt-get install filebeat

Autostart the FileBeat

update-rc.d filebeat defaults 95 10

To configure the FileBeat, open


and add the input and output section

- input_type: log
    - /var/log/*.log
    - /var/log/apache2/*.log
    - /var/log/apt/*.log
    - /var/log/samba/*.log
    - /var/log/univention/*.log
    - /var/log/syslog
  hosts: ["<Hostname Logstash Search Server>:5044"]

Upload the index file to elastic search

curl -XPUT 'http://<Hostname Elastic Search Server>:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Start File Beat

service filebeat start


Metricbeats collects statistics of the system and sends them to elastic search.

You can install the Metricbeats from the repository

sudo apt-get update && sudo apt-get install metricbeat

The MetricBeat can be automatically started on boot with

update-rc.d metricbeat defaults 95 10

To configure the Beat, open


and add the input and output section

- module: system
    - cpu
    - load
    - core
    - diskio
    - filesystem
    - fsstat
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
- module: apache
  metricsets: ["status"]
  enabled: true
  period: 1s
  hosts: [""]

name: "<server name>"
tags: ["UCS Server"]

  hosts: ["<Hostname Logstash Search Server>:5044"]

logging.level: error

Start the beat with

service metricbeat start
Personal tools