Cool Solution - ElasticSearch

From Univention Wiki

Revision as of 14:05, 6 April 2018 by Hpeter (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
Produktlogo UCS Version 4.2
Produktlogo UCS Version 4.3

Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.

Also regard the legal notes at Terms of Service.

Elastic Search allows the collection and processing of log files and metric data. ElasticSearch is the default backend for Kibana and can also be used with Grafana.

Elastic Search in a VM

Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.

Installation

Install Java

Elasticsearch requires Java 8 or later

univention-install openjdk-8-jre

Set the Repository

Add the Elastic GPG Key and add the repository

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list

Install Elasticsearch

Install Elasticsearch and enable it as an autostarting service

univention-install elasticsearch
systemctl enable elasticsearch.service
systemctl start elasticsearch.service

Elasticsearch needs to have port 9200 open to accept Beats input. To open the port set the following UCR variable

ucr set security/packetfilter/tcp/9200/all=ACCEPT \
security/packetfilter/tcp/9200/en="Elasticsearch" \
security/packetfilter/udp/9200/all=ACCEPT \
security/packetfilter/udp/9200/en="Elasticsearch"
service univention-firewall restart

Collect Data

Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server.

FileBeat

The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.

You can install the FileBeat from the repository

univention-install filebeat

To configure the FileBeat, open /etc/filebeat/filebeat.yml and adjust the filebeat.prospectors section by setting the following values:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/apache2/*.log
    - /var/log/apt/*.log
    - /var/log/samba/*.log
    - /var/log/univention/*.log
    - /var/log/syslog

Now confirm that the correct elasticsearch server is set:

output.elasticsearch:
  hosts: ["<Hostname Elasticsearch Server>:9200"]

Enable and start this beat as a service

systemctl enable filebeat.service
systemctl start filebeat.service

Metricbeats

Metricbeats collects statistics of the system and sends them to elastic search.

You can install the Metricbeats from the repository

univention-install metricbeat

To configure the Beat, open /etc/metricbeat/metricbeat.yml and set the following variables:

name: "<server name>"
tags: ["UCS Server"]
logging.level: error

Confirm that the correct elasticsearch server is set:

output.elasticsearch:
  hosts: ["<Hostname Elasticsearch Server>:9200"]

Uncomment the metricsets 'core' and 'diskio' in the module

/etc/metricbeat/modules.d/system.yml

Activate the module apache:

mv /etc/metricbeat/modules.d/apache.yml.disabled /etc/metricbeat/modules.d/apache.yml

Enable and start this beat as a service

systemctl enable metricbeat.service
systemctl start metricbeat.service

Visualization

ElasticSearch can be used as the backend for Kibana and Grafana. Both can display the state of your environment.

Personal tools