Content Introduction Prerequisites Virtual Host Redirect to HTTPS Conclusion Introduction To secure the data transfer redirecting the HTTP traffic to...
Read MoreELK stack is formerly known as Elastic Stack. It consists of Elasticsearch, Logstash, and Kibana. It helps you to store all your logs in one place and analyze the issues by correlating the events at a particular time.ELK allows you to search, analyze, and visualize logs generated from any source in any format. In this documentation, we will install the ELK Stack on CentOS 7—that is, Elasticsearch 7.1.x, Logstash 7.1.x, and Kibana 7.1.x
“Logstash is a tool that can collect and parse data or logs for analyzing it. Kibana is a web interface or UI that can be used to search and view the logs which were indexed by Logstash. Both of the tools (Logstash and Kibana) are based on Elasticsearch. Elasticsearch, Logstash, and Kibana, together is known as an ELK stack.“
The components of the ELK Stack are:
Where Beats are installed on the servers as agents known as “data shippers” which are used to send many kinds of operational data either directly to Elasticsearch or through Logstash. You can transfer the data within very little time and can modify or enhance the data.
Elasticsearch is a full-text search engine that is capable of serving multiple customers with its single instance, and it is distributed with an HTTP web interface and schema-free JSON documents. Elasticsearch is a Restful search engine an open-source which is built on Apache Lucene and released under an Apache license. It is based on Java which can search and index document files in different formats. Elasticsearch index documents to the repository and during this operation, Elasticsearch transforms raw data like logs or messages into internal documents and stores them in a data structure which is similar to a JSON object. You can do an HTTP POST that transfers your document as a JSON object. Elasticsearch service is also available on AWS and Google cloud platform.
Elasticsearch is a No-SQL database that indexes and stores information. You can also query structured data, and use Elasticsearch as an analytics platform. You can run queries which can aggregate data and use the results for making graphs, pie charts, line charts, etc.
Kibana is a web-Interface for searching and visualizing data. It is a plugin for Elasticsearch which provides visualization capabilities on the indexed patterns in the Elasticsearch cluster. Kibana is a User Interface for the stack in which you can create index patterns and analyze the data or logs, and you can query for data by using a query language and can generate visualizations/charts, etc.
Logsath is an open-source server-side pipeline for data processing. This tool used to manage real-time logs and events by collecting a variety of data from a different source. It used as an Elasticsearch pipeline. Logstash provides a variety of filters that make meaningful data by parsing and transforming it.
Install Java with the yum command given below:
$ sudo yum install java-1.8.0-openjdk-devel
$ java -version
Install and Configure Elasticsearch
Command to import the public GPG key of Elasticsearch to the rpm package manager
$ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
Create a file elasticsearch.repo in the /etc/yum.repos.d/ directory
Use any editor to create an elasticsearch.repo files than save and exit the file.
$ sudo vi /etc/yum.repos.d/elasticsearch.repo
Create a file elasticsearch.repo in the /etc/yum.repos.d/ directory
Use any editor to create an elasticsearch.repo file.
$ sudo vi /etc/yum.repos.d/elasticsearch.repo
Add the following lines to the file :
$ [elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Your repository is ready to use, now install Elasticsearch Package with the command:
$ sudo yum install elasticsearch
Elasticsearch is Installed now, open configuration directory, and edit the elasticsearch.yml file. elasticsearch.yml file contains various configuration settings.
$ sudo vi /etc/elasticsearch/elasticsearch.yml
In the elasticsearch.yml file find the line that specifies a network and uncomment it and replace its value with “localhost” and http port value with 9200.
Note: 9200 is the default port of the elasticsearch.
network.host: localhost
http.port: 9200
Start the Elasticsearch service with the systemctl command:
Run the enable elasticsearch command to enable Elasticsearch to start up every time your server boots:
$ sudo systemctl enable elasticsearch
Check the status of Elasticsearch and make sure it is Running.
$ curl -X GET "localhost:9200"
Installing and Configuring the Kibana Dashboard
According to the installation of ELK stack order in the official documentation, you should install Kibana after the setup of the Elasticsearch. After setting Kibana, we will be able to use its GUI to search for data and visualize the data that Elasticsearch stores. Because you already added the Elastic repository in the previous step, you can install the remaining components of the Elastic Stack using yum:
$ sudo yum install kibana
Enable and start the Kibana service:
$ sudo systemctl enable kibana
$ sudo systemctl start kibana
Open the kibana.yml file and edit it. Uncomment the lines for server.port, server.host and elasticsearch.host. Then Save and exit the file.
$ sudo vi /etc/kibana/kibana.yml
Note: Uncomment and edit server.host and change it value to “0.0.0.0 ” means allow all traffic to access kibana.
server.port: 5601
server.host: 0.0.0.0
elasticsearch.hosts: "http://localhost:9200"
$ sudo systemctl restart kibana
Kibana is now accessible with the public IP address of your CentOS7 server. You can check the Kibana server’s status by opening your browser and navigate to the following address:
http://your_server_ip:5601
Note: 5601 is the default port of Kibana.
Installing Logstash
Logstash is used to process the data first. Logstash is a tool which can collect and parse data or logs for analyzing it. This will collect data from different sources, transform it into a standard format, and export it to another database.
$ sudo yum install logstash
Enable the logstash and start the service of logstash and check the status of logstash.
$ sudo systemctl start logstash
$ sudo systemctl enable logstash
$ sudo systemctl status logstash
Content Introduction Prerequisites Virtual Host Redirect to HTTPS Conclusion Introduction To secure the data transfer redirecting the HTTP traffic to...
Read MoreContent Introduction Requirement Getting Started Conclusion Introduction Angular is an open-source web application framework. It is a TypeScript-based free and development...
Read More