Difference between revisions of "Cool Solution - ElasticSearch"

From Univention Wiki

Jump to: navigation, search
 
Line 1: Line 1:
{{Version|UCS=4.2}}  
+
{{Version|UCS=4.2}}
 +
{{Version|UCS=4.3}}  
 
{{Cool Solutions Disclaimer}}
 
{{Cool Solutions Disclaimer}}
 
{{#seo:
 
{{#seo:
Line 5: Line 6:
 
<!--|description=-->
 
<!--|description=-->
 
}}
 
}}
{{Review-Status}}
 
  
Elastic Search together with Logstash allows the collection and processing of log files and metric data. ElasticSearch is the default backend for [[Kibana]] and can also be used with [[Grafana]], both of which display the state of your environment.
+
Elastic Search allows the collection and processing of log files and metric data. ElasticSearch is the default backend for [[Kibana]] and can also be used with [[Grafana]].
  
 
== Elastic Search in a VM ==
 
== Elastic Search in a VM ==
 
 
Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.
 
Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.
  
 
== Installation ==
 
== Installation ==
 
 
=== Install Java ===
 
=== Install Java ===
 
+
Elasticsearch requires Java 8 or later
Download Java JRE from [http://www.oracle.com/technetwork/java/javase/downloads/index.html Oracle] and copy it onto the server using SCP.
 
 
 
 
 
Create a directory under the optional directory
 
 
 
 
<pre>
 
<pre>
mkdir /opt/jdk
+
univention-install openjdk-8-jre
</pre>
 
 
 
extract Java to the new directory
 
 
 
<pre>
 
tar zxf <file name> -C /opt/jdk
 
</pre>
 
 
 
Add Oracle Java to the update-alternatives
 
 
 
<pre>
 
update-alternatives --install /usr/bin/java java /opt/jdk/<java version>/bin/java 100
 
update-alternatives --install /usr/bin/javac javac /opt/jdk/<java version>/bin/javac 100
 
</pre>
 
 
 
=== Install sudo ===
 
 
 
Install sudo from the Univention Repository
 
 
 
<pre>
 
univention-install sudo
 
 
</pre>
 
</pre>
  
 
=== Set the Repository===
 
=== Set the Repository===
 
+
Add the Elastic GPG Key and add the repository
Add the Elastic GPG Key
 
 
 
 
<pre>
 
<pre>
 
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
 
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
</pre>
 
 
Add the repository
 
 
<pre>
 
 
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
 
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
 
</pre>
 
</pre>
  
 
=== Install Elasticsearch ===
 
=== Install Elasticsearch ===
 
+
Install Elasticsearch and enable it as an autostarting service
Install Elasticsearch
 
 
 
 
<pre>
 
<pre>
 
univention-install elasticsearch
 
univention-install elasticsearch
 +
systemctl enable elasticsearch.service
 +
systemctl start elasticsearch.service
 
</pre>
 
</pre>
  
Add elasticsearch to autostart
+
Elasticsearch needs to have port 9200 open to accept Beats input. To open the port set the following UCR variable
 
 
<pre>
 
systemctl enable elasticsearch
 
systemctl start elasticsearch
 
</pre>
 
 
 
=== Install Logstash ===
 
 
 
Logstash can be installed through the elastic repository
 
 
 
<pre>
 
univention-install logstash
 
</pre>
 
 
 
Add elasticsearch to autostart
 
 
 
<pre>
 
systemctl enable logstash
 
systemctl start logstash
 
</pre>
 
 
 
=== Restart ===
 
 
 
Restart Elasticsearch
 
 
 
<pre>
 
systemctl restart elasticsearch
 
</pre>
 
 
 
== Configure Logstash ==
 
 
 
First Logstash needs multiple input and output plugins to process Data. Within this website, we will look at Beats and Syslog for input and output the Data to Elasticsearch.  
 
 
 
=== Beats ===
 
 
 
Install the plugins for beats and syslog:
 
 
 
<pre>
 
/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
 
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch
 
</pre>
 
 
 
Edit the config file at /etc/logstash/conf.d/beats.conf
 
 
 
 
<pre>
 
<pre>
input {
+
ucr set security/packetfilter/tcp/9200/all=ACCEPT \
  beats {
+
security/packetfilter/tcp/9200/en="Elasticsearch" \
    port => 5044
+
security/packetfilter/udp/9200/all=ACCEPT \
  }
+
security/packetfilter/udp/9200/en="Elasticsearch"
}
 
 
 
output {
 
  elasticsearch {
 
    hosts => "localhost:9200"
 
    manage_template => false
 
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
 
    document_type => "%{[@metadata][type]}"
 
  }
 
}
 
</pre>
 
 
 
Logstash needs to have port 5044 open to accept Beats input. To open the port set the following UCR variable
 
 
 
<pre>
 
ucr set security/packetfilter/tcp/5044/all=ACCEPT \
 
security/packetfilter/udp/5044/all=ACCEPT
 
service univention-firewall restart
 
</pre>
 
 
 
=== Syslog ===
 
 
 
Install the plugins for beats and syslog:
 
 
 
<pre>
 
/usr/share/logstash/bin/logstash-plugin install logstash-input-syslog
 
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch
 
</pre>
 
 
 
Edit the config file at /etc/logstash/conf.d/syslog.conf
 
 
 
<pre>
 
input {
 
  syslog {
 
  }
 
}
 
 
 
output {
 
  elasticsearch {
 
    hosts => "localhost:9200"
 
    manage_template => false
 
    index => "%{[@metadata][syslog]}-%{+YYYY.MM.dd}"
 
    document_type => "%{[@metadata][type]}"
 
  }
 
}
 
</pre>
 
 
 
By default logstash will be using the default syslog port 514. You need to open it in the UCS firewall with:
 
 
 
<pre>
 
ucr set security/packetfilter/tcp/514/all=ACCEPT \
 
security/packetfilter/udp/514/all=ACCEPT
 
 
service univention-firewall restart
 
service univention-firewall restart
 
</pre>
 
</pre>
  
 
== Collect Data ==
 
== Collect Data ==
 
+
Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server.  
Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server. We will also show you how to enable the Syslog input.
 
  
 
=== FileBeat ===
 
=== FileBeat ===
 
 
The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.
 
The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.
  
 
You can install the FileBeat from the repository
 
You can install the FileBeat from the repository
 
 
<pre>
 
<pre>
sudo univention-install filebeat
+
univention-install filebeat
 
</pre>
 
</pre>
  
Autostart the FileBeat
+
To configure the FileBeat, open ''/etc/filebeat/filebeat.yml'' and adjust the filebeat.prospectors section by setting the following values:
 
 
<pre>
 
update-rc.d filebeat defaults 95 10
 
</pre>
 
 
 
To configure the FileBeat, open /etc/filebeat/filebeat.yml and add the input and output section
 
 
 
 
<pre>
 
<pre>
 
filebeat.prospectors:
 
filebeat.prospectors:
- input_type: log
+
- type: log
 +
  enabled: true
 
   paths:
 
   paths:
 
     - /var/log/*.log
 
     - /var/log/*.log
Line 206: Line 66:
 
     - /var/log/univention/*.log
 
     - /var/log/univention/*.log
 
     - /var/log/syslog
 
     - /var/log/syslog
output.logstash:
 
  hosts: ["<Hostname Logstash Search Server>:5044"]
 
 
</pre>
 
</pre>
  
Upload the index file to elastic search
+
Now confirm that the correct elasticsearch server is set:
 
 
 
<pre>
 
<pre>
curl -XPUT 'http://<Hostname Elastic Search Server>:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json
+
output.elasticsearch:
 +
  hosts: ["<Hostname Elasticsearch Server>:9200"]
 
</pre>
 
</pre>
  
Start File Beat
+
Enable and start this beat as a service
 
 
 
<pre>
 
<pre>
service filebeat start
+
systemctl enable filebeat.service
 +
systemctl start filebeat.service
 
</pre>
 
</pre>
  
 
=== Metricbeats ===
 
=== Metricbeats ===
 
 
Metricbeats collects statistics of the system and sends them to elastic search.
 
Metricbeats collects statistics of the system and sends them to elastic search.
  
 
You can install the Metricbeats from the repository
 
You can install the Metricbeats from the repository
 +
<pre>
 +
univention-install metricbeat
 +
</pre>
  
 +
To configure the Beat, open ''/etc/metricbeat/metricbeat.yml'' and set the following variables:
 
<pre>
 
<pre>
sudo univention-install metricbeat
+
name: "<server name>"
 +
tags: ["UCS Server"]
 +
logging.level: error
 
</pre>
 
</pre>
  
The MetricBeat can be automatically started on boot with
+
Confirm that the correct elasticsearch server is set:
 
 
 
<pre>
 
<pre>
update-rc.d metricbeat defaults 95 10
+
output.elasticsearch:
 +
  hosts: ["<Hostname Elasticsearch Server>:9200"]
 
</pre>
 
</pre>
  
To configure the Beat, open <pre>/etc/metricbeat/metricbeat.yml</pre> and add the input and output section
+
Uncomment the metricsets 'core' and 'diskio' in the module
 +
<pre>/etc/metricbeat/modules.d/system.yml</pre>
  
 +
Activate the module apache:
 
<pre>
 
<pre>
metricbeat.modules:
+
mv /etc/metricbeat/modules.d/apache.yml.disabled /etc/metricbeat/modules.d/apache.yml
- module: system
+
</PRE>
  metricsets:
 
    - cpu
 
    - load
 
    - core
 
    - diskio
 
    - filesystem
 
    - fsstat
 
    - memory
 
    - network
 
    - process
 
  enabled: true
 
  period: 10s
 
  processes: ['.*']
 
- module: apache
 
  metricsets: ["status"]
 
  enabled: true
 
  period: 1s
 
  hosts: ["http://127.0.0.1"]
 
 
 
name: "<server name>"
 
tags: ["UCS Server"]
 
 
 
output.logstash:
 
  hosts: ["<Hostname Logstash Search Server>:5044"]
 
 
 
logging.level: error
 
</pre>
 
  
Start the beat with
+
Enable and start this beat as a service
 
<pre>
 
<pre>
service metricbeat start
+
systemctl enable metricbeat.service
 +
systemctl start metricbeat.service
 
</pre>
 
</pre>
  
[[Category:EN]]
+
== Visualization ==
 +
ElasticSearch can be used as the backend for [[Kibana]] and [[Grafana]]. Both can display the state of your environment.

Latest revision as of 14:05, 6 April 2018

Produktlogo UCS Version 4.2
Produktlogo UCS Version 4.3

Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.

Also regard the legal notes at Terms of Service.

Elastic Search allows the collection and processing of log files and metric data. ElasticSearch is the default backend for Kibana and can also be used with Grafana.

Elastic Search in a VM

Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.

Installation

Install Java

Elasticsearch requires Java 8 or later

univention-install openjdk-8-jre

Set the Repository

Add the Elastic GPG Key and add the repository

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list

Install Elasticsearch

Install Elasticsearch and enable it as an autostarting service

univention-install elasticsearch
systemctl enable elasticsearch.service
systemctl start elasticsearch.service

Elasticsearch needs to have port 9200 open to accept Beats input. To open the port set the following UCR variable

ucr set security/packetfilter/tcp/9200/all=ACCEPT \
security/packetfilter/tcp/9200/en="Elasticsearch" \
security/packetfilter/udp/9200/all=ACCEPT \
security/packetfilter/udp/9200/en="Elasticsearch"
service univention-firewall restart

Collect Data

Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server.

FileBeat

The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.

You can install the FileBeat from the repository

univention-install filebeat

To configure the FileBeat, open /etc/filebeat/filebeat.yml and adjust the filebeat.prospectors section by setting the following values:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/apache2/*.log
    - /var/log/apt/*.log
    - /var/log/samba/*.log
    - /var/log/univention/*.log
    - /var/log/syslog

Now confirm that the correct elasticsearch server is set:

output.elasticsearch:
  hosts: ["<Hostname Elasticsearch Server>:9200"]

Enable and start this beat as a service

systemctl enable filebeat.service
systemctl start filebeat.service

Metricbeats

Metricbeats collects statistics of the system and sends them to elastic search.

You can install the Metricbeats from the repository

univention-install metricbeat

To configure the Beat, open /etc/metricbeat/metricbeat.yml and set the following variables:

name: "<server name>"
tags: ["UCS Server"]
logging.level: error

Confirm that the correct elasticsearch server is set:

output.elasticsearch:
  hosts: ["<Hostname Elasticsearch Server>:9200"]

Uncomment the metricsets 'core' and 'diskio' in the module

/etc/metricbeat/modules.d/system.yml

Activate the module apache:

mv /etc/metricbeat/modules.d/apache.yml.disabled /etc/metricbeat/modules.d/apache.yml

Enable and start this beat as a service

systemctl enable metricbeat.service
systemctl start metricbeat.service

Visualization

ElasticSearch can be used as the backend for Kibana and Grafana. Both can display the state of your environment.

Personal tools