Elastic Stack

Introduction

This manual describes the installation procedure for the Elastic Stack (ELK).

Preparation for Installation

This section covers the essential steps required for the installation.

All steps must be executed with root privileges on all nodes, unless otherwise indicated.

To install the ELK, you will need:

• Root permission on the server

• GBDS installed on the server

If you do not have the file, contact Griaule support team.

Then, follow the steps shown below.

1. Log in to the server as root.

2. Prepare the Repository.

3. Install and Configure Elasticsearch.

4. Install and Configure Kibana.

5. Install and Configure Logstash.

6. Configure the ELK with SmartSense.

Prepare the Repository

To install the ELK, first the repository must be added to the server.

To do this, import the GPG key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create the repository file:

vim /etc/yum.repos.d/elasticsearch.repo

Add the following content to the file and save it:

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Then, update the package manager cache. Start by cleaning the cache:

yum clean all

Finally, rebuild the package cache:

yum makecache

Installing the ELK

Installing and Configuring Elasticsearch

Install the Elasticsearch package:

yum install elasticsearch -y

Then, open the Elasticsearch configuration file:

vim /etc/elasticsearch/elasticsearch.yml

In the Networksection, look for the line that begins with #network.host:. Uncomment the line and change its value to:

Make sure to replace <host-ip> with the server's IP address.

network.host: <host-ip>
              ^^^^^^^^^

Next, turn off SSL by changing the following settings to false:

xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false

Then, start the Elasticsearch service:

sudo systemctl start elasticsearch

And enable the Elasticsearch service to start automatically on machine boot:

sudo systemctl enable elasticsearch

Finally, check if the Elasticsearch service is running:

Make sure to replace <host-ip> with the server's IP address.

curl -X GET "<host-ip>:9200"
             ^^^^^^^^^

The result should be similar to:

{
	"name": "QDexH8a",
	"cluster_name": "elasticsearch",
	"cluster_uuid": "gAAIqERvS_msO7Y1_759Ja",
	"version": {
		"number": "6.8.23",
		"build_flavor": "default",
		"build_type": "rpm",
		"build_hash": "4f67856",
		"build_date": "2022-01-06T21:30:50.087716Z",
		"build_snapshot": false,
		"lucene_version": "7.7.3",
		"minimum_wire_compatibility_version": "5.6.0",
		"minimum_index_compatibility_version": "5.0.0"
	},
	"tagline": "You Know, for Search"
}

Installing and Configuring Kibana

Install the Kibana package:

yum install kibana -y

Then, open the Kibana configuration file:

vim /etc/kibana/kibana.yml

Look for the line that begins with #server.host:. Uncomment the line and change its value to:

Make sure to replace <hostname> to the server hostname. Keep the double quotes.

server.host: "<hostname>"
              ^^^^^^^^^^

Next, look for the line that begins with #elasticsearch.hosts:. Uncomment the line and change its value to:

Make sure to replace <elasticsearch-host-ip> to the IP address configured in Elasticsearch. Keep the double quotes.

elasticsearch.hosts: ["http://<elasticsearch-host-ip>:9200"]
                              ^^^^^^^^^^^^^^^^^^^^^^^

Then, start the Kibana service:

sudo systemctl start kibana

And enable the Kibana service to start automatically on machine boot:

sudo systemctl enable kibana

Next, install and configure Nginx.

Installing and Configuring Nginx

Install the Nginx package:

yum install nginx -y

Then, create a file that will contain the authentication credentials for Kibana. To do this, run the following command and enter the desired password when prompted:

echo "kibanaadmin:`openssl passwd -apr1`" | tee -a /etc/nginx/htpasswd.users

Then, create a new configuration file for Nginx:

Make sure to replace <hostname> with the server hostname.

vim /etc/nginx/conf.d/<hostname>_kibana.conf
                      ^^^^^^^^^^

Add the following content to the file, making the appropriate changes in server_name and proxy_pass:

Make sure to replace <host-ip> to the server IP address and <kibana-host-ip> to the IP address of the server where Kibana is installed.

Below, the lines containing "^^^^^^^^^" are present only to highlight the changes that should be made. Remove them before saving the file.

server {
	listen 80;

	server_name <host-ip>;
	            ^^^^^^^^^

	auth_basic "Restricted Access";
	auth_basic_user_file /etc/nginx/htpasswd.users;

	location / {
		proxy_pass http://<kibana-host-ip>:5601;
		                  ^^^^^^^^^^^^^^^^
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_set_header Host $host;
		proxy_cache_bypass $http_upgrad;
	}
}

Test the Nginx configuration file:

nginx -t

Then, restart the Nginx service:

systemctl restart nginx

If necessary, configure the connection in SELinux:

setsebool httpd_can_network_connect 1 -P

Finally, verify that the Kibana service is running by accessing the following URL in a browser:

Make sure to replace <host-ip> with the server's IP address.

http://<host-ip>/status
       ^^^^^^^^^

The username is kibanaadmin and the password is the one created above.

Installing and Configuring Logstash

Install the Logstash package:

yum install logstash -y

Then, install the MySQL Connector/J package:

yum install mysql-connector-java -y

If it is not found, download it at: https://dev.mysql.com/downloads/connector/j/

Then, create the Logstash configuration file:

vim /etc/logstash/conf.d/smartsense.conf

Add the following content to the file, making the appropriate changes in jdbc_connection_string, jdbc_user, jdbc_password and hosts:

Make sure to replace <database-ip>, <database-username>, <database-password> and <elasticsearch-host-ip> with the appropriate values. Keep the double quotes.

Below, the lines containing "^^^^^^^^^" are present only to highlight the changes that should be made. Remove them before saving the file.

input {
	jdbc {
		jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
		jdbc_driver_class => "com.mysql.jdbc.Driver"
		jdbc_connection_string => "jdbc:mysql://<database-ip>:3306/"
		                                        ^^^^^^^^^^^^^
		jdbc_user => "<database-username>"
		              ^^^^^^^^^^^^^^^^^^^
		jdbc_password => "<database-password>"
		                  ^^^^^^^^^^^^^^^^^^^
		jdbc_validate_connection => true
		tracking_column => "id"
		use_column_value => true
		statement => "SELECT * FROM smartsense.load_balancing_count where id > :sql_last_value;"
		schedule => "*/2 * * * *"
		clean_run => false
	}
}
output {
	elasticsearch {
		hosts => ["<elasticsearch-host-ip>:9200"]
		           ^^^^^^^^^^^^^^^^^^^^^^^
		index => "smart_sense_index_pattern"
		document_id => "%{[id]}"
	}
	stdout {
		codec => rubydebug
	}
}

Next, the Logstash systemd file needs to be modified to ensure it is started using the configuration file created earlier. To do this, open the file:

vim /etc/systemd/system/logstash.service

The file may be located at /usr/lib/systemd/system/logstash.service.

Look for the line that begins with ExecStart=. Change its value from:

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

To:

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" "-f" "/etc/logstash/conf.d/smartsense.conf"

Then, apply the changes by reloading the systemd configuration:

systemctl daemon-reload

If you are installing on a new server that has an empty database, insert a dummy value into the table smartsense.load_balancing_count to avoid errors. To do this, run the following command and enter the database password:

Make sure to replace <database-username> and <mysql-database-ip> with the appropriate values.

#        vvvvvvvvvvvvvvvvvvv       vvvvvvvvvvvvvvvvvvv
mysql -u <database-username> -p -h <mysql-database-ip> \
      -e "USE smartsense; INSERT INTO load_balancing_count
         (id, hostname, load_time, api_id, transaction_type,
         latent, ul, load_count, extraction_time_avg, extraction_quality_avg,
         match_avg, total_avg, extraction_time_min, extraction_quality_min, match_min,
         total_min, extraction_time_max, extraction_quality_max, match_max, total_max)
         VALUES
         (1, 'hostname', '2023-08-31 21:25:40', '8829E30D-4994-4D09-99AF-B6F818473928',
         'IDENTIFY', 'false', 'false', 1, '541.0', '0.0', '48.0', '599.0',
         '541', '0', '48', '599', '541', '0', '48', '599');"

Then, enable the Logstash service to start automatically on machine boot:

sudo systemctl enable logstash

Then, start the Logstash service:

sudo systemctl start logstash

And follow the log:

tail -f /var/log/logstash/logstash-plain.log

If an error occurs indicating that Logstash cannot write to the directory /var/lib/logstash/{folder}, run the following command to change its owner:

chown -R logstash:logstash /var/lib/logstash

Finally, to verify that Logstash created the index in Elasticsearch, run the following command:

Make sure to replace <elasticsearch-host-ip> to the IP address of the server where Elasticsearch is installed.

curl -X GET "<elasticsearch-host-ip>:9200/_cat/indices?v"
             ^^^^^^^^^^^^^^^^^^^^^^^

The output should be similar to:

health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   smart_sense_index_pattern 6Ux_yM25SvG2zWGdGR0HQw   5   1          1            0      6.7kb          6.7kb
green  open   .kibana_1                 BBO89yLnTUC3F7nhqKwf9w   1   0          4            0       18kb           18kb
green  open   .kibana_task_manager      sIMoATiBRsS8bXiVBCscrA   1   0          2            0     12.5kb         12.5kb

Configuring the ELK with SmartSense

Configuring Kibana

Creating the Data View

Make sure to replace <kibana-host-ip> to the IP address of the server where Kibana is installed.

In a browser, go to: http://<kibana-host-ip>:5601. Then, open the options sidebar by clicking this icon, located in the top left corner of the screen:

Click Management (last section). Then, in the options on the left side, in the Datesection, click Index Management.

Or access the following URL directly:

http://<kibana-host-ip>:5601/app/management/data/index_management/indices
       ^^^^^^^^^^^^^^^^

Make sure that the index smart_sense_index_pattern appears in the list.

Then, in the Kibana section of the left side options, click Data Views.

Click the blue Create data view button and fill in the fields with the following information:

• Name: SS Pattern

• Index pattern: smart_sense_index_pattern

• Timestamp field: load_time

Confirm the creation of the Data View by clicking Save data view to Kibana.

Creating the Dashboards

Open the options sidebar again by clicking the icon in the top left corner of the screen. In the Analytics section, click Dashboards.

Or access the following URL directly:

http://<kibana-host-ip>:5601/app/dashboards
       ^^^^^^^^^^^^^^^^

Click the blue Create dashboard button. Then, click Create visualization. On the right side, configure the visualization with the following information:

• Visualization type: Vertical bar stacked

• Data view: SS Pattern

• Horizontal Axis:

  • Functions: Date histogram

  • Field: load_time

• Vertical Axis:

  • Functions: Sum

  • Field: load_count

Then, click the + symbol, located in the top left corner of the screen, to create a new filter. Configure the filter with the following information: transaction_type is ENROLL. Confirm by clicking Add filter.

Finally, save the dashboard by clicking Save to library, located in the top right corner of the screen, and entering the following information:

• Title: SS Enroll Dashboard

• Tags: smartsense-enroll

Click Save and return.

Repeat the above operations to create the following dashboards:

Adjust the names and tags as necessary.

• For VERIFY add the filter: transaction_type is VERIFY

• For UPDATE add the filter: transaction_type is UPDATE

• For IDENTIFY add the filter: transaction_type is IDENTIFY and latent is false

• For LATENT add the filter: transaction_type is IDENTIFY and latent is true

With the five dashboards created, enter each of them and set the time range to be displayed by clicking the calendar icon, located in the top right corner of the screen.

Then, click Share and Copy link. Save the link, as it will be used later.

Repeat the operation for the five dashboards.

At the end of each link, add the following information:

&hide-filter-bar=true&show-time-filter=true&embed=true

For example, the link:

http://172.16.0.185:5601/app/lens#/edit/a0a936d5-4e92-4015-b3e7-37810c2a114a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))

Will become:

http://172.16.0.185:5601/app/lens#/edit/a0a936d5-4e92-4015-b3e7-37810c2a114a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

Repeat the operation for the five obtained links.

Save the links, as they will be used in the next step.

Configuring the Dashboards in SmartSense

Open the SmartSense configuration file, config.properties, located in the folder /var/lib/tomcats/smart-sense/conf:

vim /var/lib/tomcats/smart-sense/conf/config.properties

Find the section # SMARTSENSE - ELK CONFIGURATION.

For each property (linkEnroll, linkIdentify, linkIdentifyLatent, linkUpdate, linkVerify), insert the link of the corresponding dashboard obtained earlier. For example:

linkEnroll=http://172.16.0.185:5601/app/lens#/edit/a0a936d5-4e92-4015-b3e7-37810c2a114a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkUpdate=http://172.16.0.185:5601/app/lens#/edit/25d53ee8-7adc-4b06-b05d-f38bfda39c66?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkVerify=http://172.16.0.185:5601/app/lens#/edit/8bfa1546-7990-4ed3-baae-86e421a60aef?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkIdentify=http://172.16.0.185:5601/app/lens#/edit/0d5edf08-ca78-40fc-ac5f-59ca91d07412?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkIdentifyLatent=http://172.16.0.185:5601/app/lens#/edit/e3f84cc5-68dd-4c76-a84e-d209da2e777a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

Save and close the file.

After completing all the steps of the Elastic Stack installation procedure, go back to the SmartSense Server Configuration manual to finish the configuration.