Difference between revisions of "Cool Solution - ElasticSearch"

Latest revision as of 14:05, 6 April 2018

Version 4.2

Version 4.3

Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.

Also regard the legal notes at Terms of Service.

Elastic Search allows the collection and processing of log files and metric data. ElasticSearch is the default backend for Kibana and can also be used with Grafana.

Elastic Search in a VM

Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.

Installation

Install Java

Elasticsearch requires Java 8 or later

univention-install openjdk-8-jre

Set the Repository

Add the Elastic GPG Key and add the repository

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list

Install Elasticsearch

Install Elasticsearch and enable it as an autostarting service

univention-install elasticsearch
systemctl enable elasticsearch.service
systemctl start elasticsearch.service

Elasticsearch needs to have port 9200 open to accept Beats input. To open the port set the following UCR variable

ucr set security/packetfilter/tcp/9200/all=ACCEPT \
security/packetfilter/tcp/9200/en="Elasticsearch" \
security/packetfilter/udp/9200/all=ACCEPT \
security/packetfilter/udp/9200/en="Elasticsearch"
service univention-firewall restart

Collect Data

Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server.

FileBeat

The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.

You can install the FileBeat from the repository

univention-install filebeat

To configure the FileBeat, open /etc/filebeat/filebeat.yml and adjust the filebeat.prospectors section by setting the following values:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/apache2/*.log
    - /var/log/apt/*.log
    - /var/log/samba/*.log
    - /var/log/univention/*.log
    - /var/log/syslog

Now confirm that the correct elasticsearch server is set:

output.elasticsearch:
  hosts: ["<Hostname Elasticsearch Server>:9200"]

Enable and start this beat as a service

systemctl enable filebeat.service
systemctl start filebeat.service

Metricbeats

Metricbeats collects statistics of the system and sends them to elastic search.

You can install the Metricbeats from the repository

univention-install metricbeat

To configure the Beat, open /etc/metricbeat/metricbeat.yml and set the following variables:

name: "<server name>"
tags: ["UCS Server"]
logging.level: error

Confirm that the correct elasticsearch server is set:

output.elasticsearch:
  hosts: ["<Hostname Elasticsearch Server>:9200"]

Uncomment the metricsets 'core' and 'diskio' in the module

/etc/metricbeat/modules.d/system.yml

Activate the module apache:

mv /etc/metricbeat/modules.d/apache.yml.disabled /etc/metricbeat/modules.d/apache.yml

Enable and start this beat as a service

systemctl enable metricbeat.service
systemctl start metricbeat.service

Visualization

ElasticSearch can be used as the backend for Kibana and Grafana. Both can display the state of your environment.

Difference between revisions of "Cool Solution - ElasticSearch"

From Univention Wiki

Latest revision as of 14:05, 6 April 2018

Contents

Elastic Search in a VM

Installation

Install Java

Set the Repository

Install Elasticsearch

Collect Data

FileBeat

Metricbeats

Visualization

Views

Personal tools

Search

Tools

@@ Line 1: / Line 1: @@
 {{Version|UCS=4.2}}
+{{Version|UCS=4.3}}
 {{Cool Solutions Disclaimer}}
 {{#seo:
@@ Line 5: / Line 6: @@
 <!--|description=-->
 }}
-{{Review-Status}}
-Elastic Search together with Logstash allows the collection and processing of log files and metric data. ElasticSearch is the default backend for [[Kibana]] and can also be used with [[Grafana]], both of which display the state of your environment.
+Elastic Search allows the collection and processing of log files and metric data. ElasticSearch is the default backend for [[Kibana]] and can also be used with [[Grafana]].
 == Elastic Search in a VM ==
+Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore, it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.
-Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.
 == Installation ==
 === Install Java ===
+Elasticsearch requires Java 8 or later
-Download Java from [http://www.oracle.com/technetwork/java/javase/downloads/jdk9-downloads-3848520.html Oracle] and copy it to the server using SCP.
-Create a directory under the optional directory
-<pre>
-mkdir /opt/jdk
-</pre>
-extract Java to the new directory
 <pre>
-tar zxf <file name> -C /opt/jdk
+univention-install openjdk-8-jre
-</pre>
-Add Oracle Java to the update-alternatives
-<pre>
-update-alternatives --install /usr/bin/java java /opt/jdk/<java version>/bin/java 100
-update-alternatives --install /usr/bin/javac javac /opt/jdk/<java version>/bin/javac 100
-</pre>
-=== Install sudo ===
-Install sudo from the Univention Repository
-<pre>
-univention-install sudo
 </pre>
 === Set the Repository===
+Add the Elastic GPG Key and add the repository
-Add the Elastic GPG Key
 <pre>
 wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
-</pre>
+echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
-Add the repository
-<pre>
-echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list
 </pre>
 === Install Elasticsearch ===
+Install Elasticsearch and enable it as an autostarting service
-Install Elasticsearch using apt
-<pre>
-apt-get update && apt-get install elasticsearch
-</pre>
-Add elasticsearch to autostart
-<pre>
-systemctl enable elasticsearch
-systemctl start elasticsearch
-</pre>
-=== Install Logstash ===
-Logstash can be installed through the elastic repository
-<pre>
-apt-get install logstash
-</pre>
-Add elasticsearch to autostart
-<pre>
-systemctl enable logstash
-systemctl start logstash
-</pre>
-=== Restart ===
-Restart Elasticsearch
-<pre>
-systemctl restart elasticsearch
-</pre>
-== Configure Logstash ==
-First Logstash needs multiple input and output plugins to process Data. Within this website, we will look at Beats and Syslog for input and output the Data to Elasticsearch.
-=== Beats ===
-Install the plugins for beats and syslog:
-<pre>
-/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
-/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch
-</pre>
-Edit the configure file at <pre>/etc/logstash/conf.d/beats.conf
 <pre>
-input {
+univention-install elasticsearch
-  beats {
+systemctl enable elasticsearch.service
-    port => 5044
+systemctl start elasticsearch.service
-  }
-}
-output {
-  elasticsearch {
-    hosts => "localhost:9200"
-    manage_template => false
-    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
-    document_type => "%{[@metadata][type]}"
-  }
-}
 </pre>
-Logstash needs to have port 5044 open to accept Beats input. To open the port set the following UCR variable
+Elasticsearch needs to have port 9200 open to accept Beats input. To open the port set the following UCR variable
 <pre>
-ucr set security/packetfilter/tcp/5044/all=ACCEPT \
+ucr set security/packetfilter/tcp/9200/all=ACCEPT \
-security/packetfilter/udp/5044/all=ACCEPT
+security/packetfilter/tcp/9200/en="Elasticsearch" \
+security/packetfilter/udp/9200/all=ACCEPT \
+security/packetfilter/udp/9200/en="Elasticsearch"
 service univention-firewall restart
 </pre>
-=== Syslog ===
-Install the plugins for beats and syslog:
-<pre>
-/usr/share/logstash/bin/logstash-plugin install logstash-input-syslog
-/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch
-</pre>
-Edit the configure file at <pre>/etc/logstash/conf.d/syslog.conf
-<pre>
-input {
-  syslog {
-  }
-}
-output {
-  elasticsearch {
-    hosts => "localhost:9200"
-    manage_template => false
-    index => "%{[@metadata][syslog]}-%{+YYYY.MM.dd}"
-    document_type => "%{[@metadata][type]}"
-  }
-}
-</pre>
-By default logstash will be using the default syslog port 514. You need to open it in the UCS firewall with:
-<pre>
-ucr set security/packetfilter/tcp/514/all=ACCEPT \
-security/packetfilter/udp/514/all=ACCEPT
-service univention-firewall restart
-</pre>
 == Collect Data ==
+Elasticsearch uses Beats to collect data. This article will look into Filebeat for collecting Logfiles and Metricbeats for collecting data on the server.
-Elasticsearch uses Beats to collect data. This article will look into Filebeat for Collecting Logfiles and Metricbeats for collecting data on the server. We will also show you how to enable the Syslog input.
-If you are installing the Beat on a different server then the Elasticsearch server, you will need to add the repository and import the gpg key.
-<pre>
-wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
-echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list
-</pre>
 === FileBeat ===
+The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also, the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.
-The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also the FileBeat is better at shaping traffic, if you direct a lot of routine trafic but need to react fast to errors.
 You can install the FileBeat from the repository
 <pre>
-sudo apt-get update && sudo apt-get install filebeat
+univention-install filebeat
 </pre>
-Autostart the FileBeat
+To configure the FileBeat, open ''/etc/filebeat/filebeat.yml'' and adjust the filebeat.prospectors section by setting the following values:
-<pre>
-update-rc.d filebeat defaults 95 10
-</pre>
-To configure the FileBeat, open <pre>/etc/filebeat/filebeat.yml</pre> and add the input and output section
 <pre>
 filebeat.prospectors:
-- input_type: log
+- type: log
+  enabled: true
    paths:
      - /var/log/*.log
@@ Line 214: / Line 66: @@
      - /var/log/univention/*.log
      - /var/log/syslog
-output.logstash:
-  hosts: ["<Hostname Logstash Search Server>:5044"]
 </pre>
-Upload the index file to elastic search
+Now confirm that the correct elasticsearch server is set:
 <pre>
-curl -XPUT 'http://<Hostname Elastic Search Server>:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json
+output.elasticsearch:
+  hosts: ["<Hostname Elasticsearch Server>:9200"]
 </pre>
-Start File Beat
+Enable and start this beat as a service
 <pre>
-service filebeat start
+systemctl enable filebeat.service
+systemctl start filebeat.service
 </pre>
 === Metricbeats ===
 Metricbeats collects statistics of the system and sends them to elastic search.
 You can install the Metricbeats from the repository
+<pre>
+univention-install metricbeat
+</pre>
+To configure the Beat, open ''/etc/metricbeat/metricbeat.yml'' and set the following variables:
 <pre>
-sudo apt-get update && sudo apt-get install metricbeat
+name: "<server name>"
+tags: ["UCS Server"]
+logging.level: error
 </pre>
-The MetricBeat can be automatically started on boot with
+Confirm that the correct elasticsearch server is set:
 <pre>
-update-rc.d metricbeat defaults 95 10
+output.elasticsearch:
+  hosts: ["<Hostname Elasticsearch Server>:9200"]
 </pre>
-To configure the Beat, open <pre>/etc/metricbeat/metricbeat.yml</pre> and add the input and output section
+Uncomment the metricsets 'core' and 'diskio' in the module
+<pre>/etc/metricbeat/modules.d/system.yml</pre>
+Activate the module apache:
 <pre>
-metricbeat.modules:
+mv /etc/metricbeat/modules.d/apache.yml.disabled /etc/metricbeat/modules.d/apache.yml
-- module: system
+</PRE>
-  metricsets:
-    - cpu
-    - load
-    - core
-    - diskio
-    - filesystem
-    - fsstat
-    - memory
-    - network
-    - process
-  enabled: true
-  period: 10s
-  processes: ['.*']
-- module: apache
-  metricsets: ["status"]
-  enabled: true
-  period: 1s
-  hosts: ["http://127.0.0.1"]
-name: "<server name>"
-tags: ["UCS Server"]
-output.logstash:
-  hosts: ["<Hostname Logstash Search Server>:5044"]
-logging.level: error
-</pre>
-Start the beat with
+Enable and start this beat as a service
 <pre>
-service metricbeat start
+systemctl enable metricbeat.service
+systemctl start metricbeat.service
 </pre>
-[[Category:EN]]
+== Visualization ==
+ElasticSearch can be used as the backend for [[Kibana]] and [[Grafana]]. Both can display the state of your environment.