Difference between revisions of "Cool Solution - ElasticSearch"

From Univention Wiki

Jump to: navigation, search
Line 63: Line 63:
 
=== Install Elasticsearch ===
 
=== Install Elasticsearch ===
  
Install Elasticsearch using apt
+
Install Elasticsearch
  
 
<pre>
 
<pre>
Line 81: Line 81:
  
 
<pre>
 
<pre>
apt-get install logstash
+
univention-install logstash
 
</pre>
 
</pre>
  
Line 188: Line 188:
 
=== FileBeat ===
 
=== FileBeat ===
  
The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also the FileBeat is better at shaping traffic, if you direct a lot of routine trafic but need to react fast to errors.
+
The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.
  
 
You can install the FileBeat from the repository
 
You can install the FileBeat from the repository
  
 
<pre>
 
<pre>
sudo apt-get update && sudo apt-get install filebeat
+
sudo univention-install filebeat
 
</pre>
 
</pre>
  
Line 237: Line 237:
  
 
<pre>
 
<pre>
sudo apt-get update && sudo apt-get install metricbeat
+
sudo univention-install metricbeat
 
</pre>
 
</pre>
  

Revision as of 13:53, 1 December 2017

Produktlogo UCS Version 4.2

Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.

Also regard the legal notes at Terms of Service.
Note: This article is not yet reviewed.


Elastic Search together with Logstash allows the collection and processing of log files and metric data. ElasticSearch is the default backend for Kibana and can also be used with Grafana, both of which display the state of your environment.

Elastic Search in a VM

Please note, ElasticSearch is a database that communicates with other aspects of your environment. Therefore it does not like to be suspended. While we did not experience any long-lasting issues, we had short-term inconsistencies during the test, which can create a lot of false positive errors in the frontends.

Installation

Install Java

Download Java JRE from Oracle and copy it onto the server using SCP.


Create a directory under the optional directory

mkdir /opt/jdk

extract Java to the new directory

tar zxf <file name> -C /opt/jdk

Add Oracle Java to the update-alternatives

update-alternatives --install /usr/bin/java java /opt/jdk/<java version>/bin/java 100
update-alternatives --install /usr/bin/javac javac /opt/jdk/<java version>/bin/javac 100

Install sudo

Install sudo from the Univention Repository

univention-install sudo

Set the Repository

Add the Elastic GPG Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

Add the repository

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list

Install Elasticsearch

Install Elasticsearch

univention-install elasticsearch

Add elasticsearch to autostart

systemctl enable elasticsearch
systemctl start elasticsearch

Install Logstash

Logstash can be installed through the elastic repository

univention-install logstash

Add elasticsearch to autostart

systemctl enable logstash
systemctl start logstash

Restart

Restart Elasticsearch

systemctl restart elasticsearch

Configure Logstash

First Logstash needs multiple input and output plugins to process Data. Within this website, we will look at Beats and Syslog for input and output the Data to Elasticsearch.

Beats

Install the plugins for beats and syslog:

/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

Edit the configure file at

/etc/logstash/conf.d/beats.conf

<pre>
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Logstash needs to have port 5044 open to accept Beats input. To open the port set the following UCR variable

ucr set security/packetfilter/tcp/5044/all=ACCEPT \
security/packetfilter/udp/5044/all=ACCEPT
service univention-firewall restart

Syslog

Install the plugins for beats and syslog:

/usr/share/logstash/bin/logstash-plugin install logstash-input-syslog
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

Edit the configure file at

/etc/logstash/conf.d/syslog.conf

<pre>
input {
  syslog {
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][syslog]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

By default logstash will be using the default syslog port 514. You need to open it in the UCS firewall with:

ucr set security/packetfilter/tcp/514/all=ACCEPT \
security/packetfilter/udp/514/all=ACCEPT
service univention-firewall restart


Collect Data

Elasticsearch uses Beats to collect data. This article will look into Filebeat for Collecting Logfiles and Metricbeats for collecting data on the server. We will also show you how to enable the Syslog input.

If you are installing the Beat on a different server then the Elasticsearch server, you will need to add the repository and import the gpg key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list

FileBeat

The FileBeat is used on every server to collect Log files. While traditionally Syslog has been employed for this task and forwarding data, there are some drawbacks in Syslog. The most important one is that some services do not use Syslog but merely write into a file. Also the FileBeat is better at shaping traffic, if you direct a lot of routine traffic but need to react fast to errors.

You can install the FileBeat from the repository

sudo univention-install filebeat

Autostart the FileBeat

update-rc.d filebeat defaults 95 10

To configure the FileBeat, open

/etc/filebeat/filebeat.yml

and add the input and output section

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
    - /var/log/apache2/*.log
    - /var/log/apt/*.log
    - /var/log/samba/*.log
    - /var/log/univention/*.log
    - /var/log/syslog
output.logstash:
  hosts: ["<Hostname Logstash Search Server>:5044"]

Upload the index file to elastic search

curl -XPUT 'http://<Hostname Elastic Search Server>:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Start File Beat

service filebeat start

Metricbeats

Metricbeats collects statistics of the system and sends them to elastic search.

You can install the Metricbeats from the repository

sudo univention-install metricbeat

The MetricBeat can be automatically started on boot with

update-rc.d metricbeat defaults 95 10

To configure the Beat, open

/etc/metricbeat/metricbeat.yml

and add the input and output section

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - load
    - core
    - diskio
    - filesystem
    - fsstat
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
- module: apache
  metricsets: ["status"]
  enabled: true
  period: 1s
  hosts: ["http://127.0.0.1"]

name: "<server name>"
tags: ["UCS Server"]

output.logstash:
  hosts: ["<Hostname Logstash Search Server>:5044"]

logging.level: error

Start the beat with

service metricbeat start
Personal tools