Difference between revisions of "Cool Solution - ElasticSearch"

From Univention Wiki

Jump to: navigation, search
(Created page with "{{Version|UCS=4.1}} {{Cool Solutions Disclaimer}} Elastic Search together with Logstash allows the collection and processing of log files and metric data. This can be used fo...")
 
Line 1: Line 1:
 
{{Version|UCS=4.1}} {{Cool Solutions Disclaimer}}
 
{{Version|UCS=4.1}} {{Cool Solutions Disclaimer}}
  
Elastic Search together with Logstash allows the collection and processing of log files and metric data. This can be used for [Kiba] and Grafana to display the state of your environment.
+
Elastic Search together with Logstash allows the collection and processing of log files and metric data. This can be used for [Kibana] and Grafana to display the state of your environment.
  
 
== Installation ==
 
== Installation ==

Revision as of 00:27, 13 December 2016

Produktlogo UCS Version 4.1

Note: Cool Solutions are articles documenting additional functionality based on Univention products. Not all of the shown steps in the article are covered by Univention Support. For questions about your support coverage contact your contact person at Univention before you want to implement one of the shown steps.

Also regard the legal notes at Terms of Service.


Elastic Search together with Logstash allows the collection and processing of log files and metric data. This can be used for [Kibana] and Grafana to display the state of your environment.

Installation

Install Java

Download Java from [Oracle] and copy it to the server using SCP.


Create a directory under the optional directory

mkdir /opt/jdk

extract Java to the new directory

tar zxf <file name> -C /opt/jdk

Add Oracle Java to the updtae alternatives

update-alternatives --install /usr/bin/java java /opt/jdk/<java version>/bin/java 100
update-alternatives --install /usr/bin/javac javac /opt/jdk/<java version>/bin/javac 100

Install sudo

Install sudo from the Univention Repository

univention-install sudo

Set the Repository

Add the Elastic GPG Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

Add the repository

echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list

Install Elasticsearch

Install Elasticsearch using apt

apt-get update && apt-get install elasticsearch

Add elasticsearch to autostart

update-rc.d elasticsearch defaults 95 10
service elasticsearch start

Install Logstash

Logstash can be installed through the elastic repository

apt-get install logstash

Add elasticsearch to autostart

update-rc.d logstash defaults 95 10
service logstash start

Restart

Restart Elasticsearch

service elasticsearch restart

Configure Logstash

First Logstash needs multiple input and output plugins to process Data. Within this website we will look at Beats and syslog for input and outputing the Data to Elasticsearch.

Beats

Install the plugins for beats and syslog:

/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

Edit the configure file at

/etc/logstash/conf.d/beats.conf

<pre>
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Logstash needs to have port 5044 open to accept Beats input. To open the port set the following UCR variable

ucr set security/packetfilter/tcp/5044/all=ACCEPT
ucr set security/packetfilter/udp/5044/all=ACCEPT
service univention-firewall restart

Syslog

Install the plugins for beats and syslog:

/usr/share/logstash/bin/logstash-plugin install logstash-input-syslog
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

Edit the configure file at

/etc/logstash/conf.d/syslog.conf

<pre>
input {
  syslog {
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][syslog]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

By default logstash will be using the default syslog port 514. You need to open it in the UCS firewall with:

ucr set security/packetfilter/tcp/514/all=ACCEPT
ucr set security/packetfilter/udp/514/all=ACCEPT
service univention-firewall restart


Collect Data

Elasticsearch uses Beats to collect data. This article will look into Filebeat for Collecting Logfiles and Metricbeats for collecting data on the server. We will also show you how to enable the syslog input.

If you are installing the Beat on a different server then the Elasticsearch server, you will need to add the repository and import the gpg key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list

FileBeat

The FileBeat is used on every server to collect Log files. While traditionally Syslog has been used for this task and for forwarding data, there are a number of drawbacks in Syslog. The two main is, that a number of services do not use syslog but simply write into a file.

You can install the FileBeat from the repository

sudo apt-get update && sudo apt-get install filebeat

Autostart the FileBeat

update-rc.d filebeat defaults 95 10

To configure the FileBeat, open

/etc/filebeat/filebeat.yml

and add the input and output section

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
    - /var/log/apache2/*.log
    - /var/log/apt/*.log
    - /var/log/samba/*.log
    - /var/log/univention/*.log
    - /var/log/syslog
output.logstash:
  hosts: ["<Hostname Logstash Search Server>:5044"]

Upload the index file to elastic search

curl -XPUT 'http://<Hostname Elastic Search Server>:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Start File Beat

service filebeat start

Metricbeats

Metricbeats collects statistics of the system and sends them to elastic search.

You can install the Metricbeats from the repository

sudo apt-get update && sudo apt-get install metricbeat

The MetricBeat can be automatically started on boot with

update-rc.d metricbeat defaults 95 10

To configure the Beat, open

/etc/metricbeat/metricbeat.yml

and add the input and output section

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - load
    - core
    - diskio
    - filesystem
    - fsstat
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
- module: apache
  metricsets: ["status"]
  enabled: true
  period: 1s
  hosts: ["http://127.0.0.1"]

name: "<server name>"
tags: ["UCS Server"]

output.logstash:
  hosts: ["<Hostname Logstash Search Server>:5044"]

logging.level: error

Start the beat with

service metricbeat start
Personal tools