Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.
Elastic Search allows the collection and processing of log files and metric data. ElasticSearch is the default backend for Kibana and can also be used with Grafana.
Elastic Search in a VM
Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.
Installation
Install Java
Elasticsearch requires Java 8 or later
univention-install openjdk-8-jre
Set the Repository
Add the Elastic GPG Key and add the repository
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
Install Elasticsearch
Install Elasticsearch and enable it as an autostarting service
univention-install elasticsearch
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
Elasticsearch needs to have port 9200 open to accept Beats input. To open the port set the following UCR variable
ucr set security/packetfilter/tcp/9200/all=ACCEPT \
security/packetfilter/tcp/9200/en="Elasticsearch" \
security/packetfilter/udp/9200/all=ACCEPT \
security/packetfilter/udp/9200/en="Elasticsearch"
service univention-firewall restart
Collect Data
Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server.
FileBeat
The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.
You can install the FileBeat from the repository
univention-install filebeat
To configure the FileBeat, open /etc/filebeat/filebeat.yml and adjust the filebeat.prospectors section by setting the following values:
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/apache2/*.log
- /var/log/apt/*.log
- /var/log/samba/*.log
- /var/log/univention/*.log
- /var/log/syslog
Now confirm that the correct elasticsearch server is set:
output.elasticsearch:
hosts: ["<Hostname Elasticsearch Server>:9200"]
Enable and start this beat as a service
systemctl enable filebeat.service
systemctl start filebeat.service
Metricbeats
Metricbeats collects statistics of the system and sends them to elastic search.
You can install the Metricbeats from the repository
univention-install metricbeat
To configure the Beat, open /etc/metricbeat/metricbeat.yml and set the following variables:
name: "<server name>"
tags: ["UCS Server"]
logging.level: error
Confirm that the correct elasticsearch server is set:
output.elasticsearch:
hosts: ["<Hostname Elasticsearch Server>:9200"]
Uncomment the metricsets ‘core’ and ‘diskio’ in the module
/etc/metricbeat/modules.d/system.yml
Activate the module apache:
mv /etc/metricbeat/modules.d/apache.yml.disabled /etc/metricbeat/modules.d/apache.yml
Enable and start this beat as a service
systemctl enable metricbeat.service
systemctl start metricbeat.service
Visualization
ElasticSearch can be used as the backend for Kibana and Grafana. Both can display the state of your environment.