[openingText]With the advancement of IT infrastructure, the organizations and professionals are using the cloud services. With the increase in cloud-based servers, the amount of log generation also increases. It is very important to analyze these logs for multiple reasons.[/openingText]
The elastic stack is an open source system which combines Elasticsearch, Logstash, and Kibana.
- Logstash – Passes the logs to the Elasticsearch
- Elasticsearch – A database which stores all the parsed logs
- Kibana – UI integrated with the elastic search to query the required fields.
Prerequisites
- [tool]OS: Ubuntu 18.04[/tool]
- [tool]RAM: 4GB[/tool]
- [tool]CPU: 2[/tool]
Step 1: [stepName]Perform a System Update[/stepName]
[step]
It is recommended to update the system before installing any packages. Open the terminal run the following commands to update the system.
$ sudo apt update $ sudo apt -y upgrade
Proceed to the next step after the update process is completed.
[/step]
Step 2: [stepName]Install Java[/stepName]
[step]
Installing Java is a must for the Elastic stack to work. In this tutorial, we will install Oracle Java.
[howToDirection]
To install Oracle Java on your Ubuntu system, you will need to add the Oracle Java PPA by running:
$ sudo add-apt-repository ppa:webupd8team/java
[/howToDirection]
[howToDirection]
Now update the repository information by running:
$ sudo apt update
[/howToDirection]
[howToDirection]
Now install the Java by using the following command.
$ sudo apt -y install oracle-java8-installer
[/howToDirection]
[howToDirection]
Accept the license agreement, and also check that java is successfully installed using the following command.
$ java -version
[/howToDirection]
[howToDirection]
You will see a message similar to this:
user@anyone:~$ java -version java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[/howToDirection]
[howToDirection]
You can also set the JAVA_HOME and other defaults by installing oracle-java8-set-default. Run:
$sudo apt -y install oracle-java8-set-default
[/howToDirection]
[howToDirection]
You can now verify if the JAVA_HOME variable is set by running:
$echo "$JAVA_HOME"
[/howToDirection]
[/step]
Step 3: [stepName]Install Elasticsearch[/stepName]
[step]
[howToDirection]
With the help of a package manager, install Elasticsearch using following command.
$wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
[/howToDirection]
[howToDirection]
If the prompt is hanging,
Create the Elasticsearch source list:
$echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
[/howToDirection]
[howToDirection]
Update your apt package database:
$sudo apt-get update
[/howToDirection]
[howToDirection]
Install Elasticsearch with this command:
$sudo apt-get -y install elasticsearch
[/howToDirection]
[howToDirection]
Elasticsearch is now installed. Edit it’s configurations now, using following commands.
$sudo vi /etc/elasticsearch/elasticsearch.yml
[/howToDirection]
[howToDirection]
You will want to restrict outside access to your Elasticsearch instance (port 9200), so outsiders can’t read your data or shutdown your Elasticsearch cluster through the HTTP API. Find the line that specifies network.host, uncomment it, and replace its value with “localhost” so it looks like this:
elasticsearch.yml excerpt (updated) network.host: localhost
Save and exit elasticsearch.yml.
[/howToDirection]
[howToDirection]
Now start Elasticsearch:
Then run the following command to start Elasticsearch on startup:
$sudo update-rc.d elasticsearch defaults 95 10
[/howToDirection]
[/step]
Step 4: [stepName]Install Kibana[/stepName]
[step]
Kibana can be installed with a package manager.
[howToDirection]
Create the Kibana source list:
$echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.5.x.list
[/howToDirection]
[howToDirection]
Update your apt package database:
$sudo apt-get update
[/howToDirection]
[howToDirection]
Install Kibana with this command:
$sudo apt-get -y install kibana
[/howToDirection]
Kibana is now installed.
[howToDirection]
Open the Kibana configuration file for editing:
$sudo vi /opt/kibana/config/kibana.yml
[/howToDirection]
[howToDirection]
In the Kibana configuration file, find the line that specifies server.host, and replace the IP address (“0.0.0.0” by default) with “localhost”:
kibana.yml excerpt (updated)
server.host: “localhost”
[/howToDirection]
[howToDirection]Save and exit.[/howToDirection]
[howToDirection]
Now enable the Kibana service, and start it:
$sudo update-rc.d kibana defaults 96 9 $sudo service kibana start
[/howToDirection]
[/step]
Step 5: [stepName]Install Logstash[/stepName]
[step]
[howToDirection]
The logstash is installed from the same repository. So now just create the Logstash source list:
$echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
[/howToDirection]
[howToDirection]
Update your apt package database:
$sudo apt-get update
[/howToDirection]
[howToDirection]
Install Logstash with this command:
$sudo apt-get install logstash
Logstash is installed now lets configure it.
[/howToDirection]
[/step]
Step 6 : [stepName]Configure Logstash[/stepName]
[step]
Logstash configuration files are in the JSON-format.
[howToDirection]
Create a configuration file called 02-beats-input.conf and set up our “filebeat” input:
$sudo vi /etc/logstash/conf.d/02-beats-input.conf
[/howToDirection]
[howToDirection]
Insert the following input> configuration:
02-beats-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}Save and quit.
[/howToDirection]
[howToDirection]
Create a configuration file called 10-syslog-filter.conf, where we will add a filter for syslog messages:
$sudo vi /etc/logstash/conf.d/10-syslog-filter.conf
[/howToDirection]
[howToDirection]
Insert the following syslog filter configuration:
10-syslog-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] }
}}Save and quit..
[/howToDirection]
[howToDirection]
Lastly, we will create a configuration file called 30-elasticsearch-output.conf:
$sudo vi /etc/logstash/conf.d/30-elasticsearch-output.conf
[/howToDirection]
[howToDirection]
Insert the following output configuration:
/etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}Save and exit.
[/howToDirection]
[howToDirection]
Test your Logstash configuration with this command:
$sudo service logstash configtest
[/howToDirection]
[howToDirection]It should display Configuration OK if there are no syntax errors. Otherwise, try and read the error output to see what’s wrong with your Logstash configuration.[/howToDirection]
[howToDirection]
Restart Logstash, and enable it, to put our configuration changes into effect:
$sudo service logstash restart $sudo update-rc.d logstash defaults 96 9
[/howToDirection]
[howToDirection]Next, we’ll load the sample Kibana dashboards.[/howToDirection]
[/step]
Step 7: [stepName]Load Kibana Dashboards[/stepName]
[step]
[howToDirection]
First, download the sample dashboards archive to your home directory:
$cd ~ $curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
[/howToDirection]
[howToDirection]
Install the unzip package with this command:
$sudo apt-get -y install unzip
[/howToDirection]
[howToDirection]
Next, extract the contents of the archive:
unzip beats-dashboards-*.zip
[/howToDirection]
[howToDirection]
And load the sample dashboards, visualizations and Beats index patterns into Elasticsearch with these commands:
$cd beats-dashboards-* $./load.sh
Elastic Stack is successfully installed and configured.
[/howToDirection]
[/step]
Check out these top 3 Linux hosting services
- If you want to get the best cloud hosting, click here and get the suitable one.
