Elastic Stack Guide Part — 1

6 min readSep 17, 2021

As nowadays lots of our servers are deployed on Cloud and many applications are running on these servers , it is impossible to monitor and analyze logs by going to each servers . Central Logging and Monitoring solution is a must in present time .

In this Bog Series , we will learn about usage of Elastic Stack aka ELK .

Overview :

Elastic Stack is a group of open source products from Elastic designed to help users take data from any type of source and in any format and search, analyze, and visualize that data in real time. The product group was formerly known as ELK Stack, in which the letters in the name stood for the products in the group: ElasticSearch, Logstash and Kibana. A fourth product, Beats, was subsequently added to the stack, rendering the potential acronym unpronounceable. Elastic Stack can be deployed on premises or made available as Software as a Service

Architechture :

For a small-sized development environment, the classic architecture will look as follows :

Here there are many different types of beats you can read them from https://www.elastic.co/beats/ . Each beat have different set of usecases .

In this blog we will learn about two beats MetricBeat and FileBeat .

Note — LogStash is an options part in the architecture and should not be needed in most of the cases . Read more about Logstash at https://www.elastic.co/logstash/

Usage Elastic Stack :

I am running experiments on CentOS7 machine and using rpm to setup the elastic stack .

Elastic Search Installation :

Commands to install Elastic Search :

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.14.0-x86_64.rpm
sudo rpm -i elasticsearch-7.14.0-x86_64.rpm 
sudo service elasticsearch start

How to check if Elastic Search is running :

[root@localhost elk]# curl http://127.0.0.1:9200 { "name" : "localhost.localdomain", "cluster_name" : "elasticsearch", "cluster_uuid" : "MxKYDoJAQRG9D6krdFThsQ", "version" : { "number" : "7.14.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1", "build_date" : "2021-07-29T20:49:32.864135063Z", "build_snapshot" : false, "lucene_version" : "8.9.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

If you are getting output like above , it means elastic search is installed successfully .

Note : To change listen address and port you can change in the following file : /etc/elasticsearch/elasticsearch.yml

Kibana :

Kibana is the Front end tool which communicates to Elastic search where anyone can monitor and analyze logs .

Commands to install kibana :

curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.14.0-linux-x86_64.tar.gz 
tar xzvf kibana-7.14.0-linux-x86_64.tar.gz 
cd kibana-7.14.0-linux-x86_64/ 
./bin/kibana

Access kibana from the url :

http://127.0.0.1:5601/app/home#/

Note : configure vim config/kibana.yml for port and ip addressed for listening settings .

Beats

These will be installed on all servers from where we want to collect information . they are like agents which will send data to Elastic Search .

Enabling Metric Beat :

Every Beats supports different modules , it is up to the use that which module one wnts to enable in each beats . if we talk about MetricBeat it has many modules like System,Postgres,Nginx and so on . In this Blog we will see usage of System Module of MetricBeat .

Commands to install MetricBeat :

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.14.0-x86_64.rpm 
sudo rpm -vi metricbeat-7.14.0-x86_64.rpm

Enabling System Module of Metric Beat :

sudo metricbeat modules enable system 
sudo metricbeat setup -e 
sudo service metricbeat start

Here we are only enabling system module of metri beats , there are many odule for basic monitoring of aplications like postgresql , nginx , tomcat etc .

Fo list of modules available in metric beats : command is

metricbeat modules list

Yeipeee Now we can Monitor System Data in kibana as follows .

Open [Metricbeat System] Host overview ECS in Dashboards in kibana UI . There you can apply filter of host of which one wants to see data .

Traditionally after accessing linux servers , we gather system information by using many different commands and tools which also takes time , specially when there is some running issue on production . Following is the list of information :

System Module MetricBeat Uses : What analysis can be Done by System module of MetricBeat :

All these type of information now can be seen in seconds for some particular host using kibana UI . Following are some screenshots :

Size information of all partitions
Read/Write Performance of Hardisk
InboundOutBound Traffic analysis per Ethernet Port
Load Avergae analysis of system
Top Proesses consuming High CPU and RAM

Enabling FileBeat

Whether you’re collecting from security devices, cloud, containers, hosts, or OT, Filebeat helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files.

Commands to install Filebeat :

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.14.0-x86_64.rpm 
rpm -ivh filebeat-7.14.0-x86_64.rpm

Note : For configuring filebeat that where to send data to elastic search or filebeat configue in /etc/filebeat/filebeat.yml , cureent as i have only one machine so no need to do an conf as defaut conf will work for me You can check the following lion : https://www.elastic.co/guide/en/beats/filebeat/7.14/configuring-howto-filebeat.html

enabling system logs module in filebeat :

filebeat modules enable system (for system logs if we want to set custom paths : edit the file /etc/filebeat/modules.d/system.yml) -- Generally no need to change these config in all cases filebeat setup -e 
sudo service filebeat start

Like Metric Beat , FileBeats also have list of modules like postgres,nginx , and it also supports logging of popular framework like spring and can collect logs of these applications and provides ways to analyze them easily .

To check modules list available for filebeat use following command :

[root@localhost elk]# filebeat modules list | less

System Module Filebeat Uses :

Now you can use Kibana UI to analyze system logs like messages etc .

Open [Filebeat System] Syslog dashboard ECS in Dashboard Tab in Kibana .

Following are some screen shots which one can see :

Configure filebeat for custom log files :

Now we may have situation where none of Modules and integration with framework logging work in filebeat for our custom application log then in that case you can configure your input manually to configure path of logs to read and analayse them in logs and stream section in kibana UI

Follow the following link to configure your log path : https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

you can watch logs by : http://127.0.0.1:5601/app/logs/stream 

Here you can search in logs by hostname , filepath and can also search in whole message which is fetched .

By default only message column is shown . One can configure then in settings tabs of logs tabs in kibana .

Following are some screenshot :

By Default logs lines are only one column , if for advance debugging we want to break log tine into columns then we need to use Logstash with Grok Filter .

In next blog we will see the usage of LogStash to break custom logs into columns for better understanding .

ThankYou all

Originally published at http://hello-worlds.in on September 17, 2021.