Elastic Stack

Introduction

This manual describes the installation procedures for the Elastic Stack (ELK).

Install Preparations

This section covers the essential steps required for the installation.

All steps must be performed with root privileges unless stated otherwise.

To install ELK you will need:

• Root permission on the server

• GBDS installed on the server

If you do not have the file, contact the Griaule Support Team at [email protected].

Then, follow the steps presented below.

1. Login to the server as root.

2. Prepare the Repository.

3. Install and Configure Elasticsearch.

4. Install and Configure Kibana.

5. Install and Configure Logstash.

6. Configure ELK with SmartSense.

Prepare the Repository

To install ELK, you must first add the repository to the server.

To do so, import the GPG key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create the repository file:

vim /etc/yum.repos.d/elasticsearch.repo

Add the following content to the file and save it:

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Then, update the cache of the packages manager. Start by cleaning the cache:

yum clean all

Finally, rebuild the package cache:

yum makecache

Installing ELK

Installing and Configuring Elasticsearch

Install the Elasticsearch package:

yum install elasticsearch -y

Then, open the Elasticsearch configuration file:

vim /etc/elasticsearch/elasticsearch.yml

In the Network section, look for the line that starts with #network.host:. Uncomment it and change its value to:

Make sure to replace <host-ip> with the IP address of the server.

network.host: <host-ip>
              ^^^^^^^^^

Next, disable SSL by changing the following settings to false:

xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false

Then, start the Elasticsearch service:

sudo systemctl start elasticsearch

And enable the Elasticsearch service to start automatically on boot:

sudo systemctl enable elasticsearch

Finally, check if the Elasticsearch service is running:

Make sure to replace <host-ip> with the IP address of the server.

curl -X GET "<host-ip>:9200"
             ^^^^^^^^^

The output should be similar to:

{
	"name": "QDexH8a",
	"cluster_name": "elasticsearch",
	"cluster_uuid": "gAAIqERvS_msO7Y1_759Ja",
	"version": {
		"number": "6.8.23",
		"build_flavor": "default",
		"build_type": "rpm",
		"build_hash": "4f67856",
		"build_date": "2022-01-06T21:30:50.087716Z",
		"build_snapshot": false,
		"lucene_version": "7.7.3",
		"minimum_wire_compatibility_version": "5.6.0",
		"minimum_index_compatibility_version": "5.0.0"
	},
	"tagline": "You Know, for Search"
}

Installing and Configuring Kibana

Install the Kibana package:

yum install kibana -y

Then, open the Kibana configuration file:

vim /etc/kibana/kibana.yml

Look for the line that starts with #server.host:. Uncomment it and change its value to:

Make sure to replace <hostname> with the hostname of the server. Keep the double quotes.

server.host: "<hostname>"
              ^^^^^^^^^^

Next, look for the line that starts with #elasticsearch.hosts:. Uncomment it and change its value to:

Make sure to replace <elasticsearch-host-ip> with the IP address configured on Elasticsearch. Keep the double quotes.

elasticsearch.hosts: ["http://<elasticsearch-host-ip>:9200"]
                              ^^^^^^^^^^^^^^^^^^^^^^^

Then, start the Kibana service:

sudo systemctl start kibana

And enable the Kibana service to start automatically on boot:

sudo systemctl enable kibana

Next, install and configure Nginx.

Installing and Configuring Nginx

Install the Nginx package:

yum install nginx -y

Next, create a file that will contain the authentication credentials for Kibana. To do so, run the following command and enter the desired password when prompted:

echo "kibanaadmin:`openssl passwd -apr1`" | tee -a /etc/nginx/htpasswd.users

Then, create a new configuration file for Nginx:

Make sure to replace <hostname> with the hostname of the server.

vim /etc/nginx/conf.d/<hostname>_kibana.conf
                      ^^^^^^^^^^

Add the following content to the file, making the appropriate changes in server_name and proxy_pass:

Make sure to replace <host-ip> with the IP address of the server and <kibana-host-ip> with the IP address of the server where Kibana is installed.

Below, the lines containing “^^^^^^^^^” are only present to highlight the changes that must be made. Remove them before saving the file.

server {
	listen 80;

	server_name <host-ip>;
				^^^^^^^^^

	auth_basic "Restricted Access";
	auth_basic_user_file /etc/nginx/htpasswd.users;

	location / {
		proxy_pass http://<kibana-host-ip>:5601;
						  ^^^^^^^^^^^^^^^^
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_set_header Host $host;
		proxy_cache_bypass $http_upgrad;
	}
}

Test the Nginx configuration file:

nginx -t

Then, restart the Nginx service:

systemctl restart nginx

If necessary, configure the connection in the SE:

setsebool httpd_can_network_connect 1 -P

Finally, check if the Kibana service is running, by accessing the following URL in a browser:

Make sure to replace <host-ip> with the IP address of the server.

http://<host-ip>/status
       ^^^^^^^^^

The username is kibanaadmin and the password is the one created above.

Installing and Configuring Logstash

Install the Logstash package:

yum install logstash -y

Next, install the MySQL Connector/J package:

yum install mysql-connector-java -y

If it is not found, download it at: https://dev.mysql.com/downloads/connector/j/

Then, create a new configuration file for Logstash:

vim /etc/logstash/conf.d/smartsense.conf

Add the following content to the file, making the appropriate changes in jdbc_connection_string, jdbc_user, jdbc_password, and hosts:

Make sure to replace <database-ip>, <database-username>, <database-password>, and <elasticsearch-host-ip> with the appropriate values. Keep the double quotes.

Below, the lines containing “^^^^^^^^^” are only present to highlight the changes that must be made. Remove them before saving the file.

input {
    jdbc {
        jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_connection_string => "jdbc:mysql://<database-ip>:3306/"
                                                ^^^^^^^^^^^^^
        jdbc_user => "<database-username>"
                      ^^^^^^^^^^^^^^^^^^^
        jdbc_password => "<database-password>"
                          ^^^^^^^^^^^^^^^^^^^
        jdbc_validate_connection => true
        tracking_column => "id"
        use_column_value => true
        statement => "SELECT * FROM smartsense.load_balancing_count where id > :sql_last_value;"
        schedule => "*/2 * * * *"
        clean_run => false
    }
}
output {
    elasticsearch {
        hosts => ["<elasticsearch-host-ip>:9200"]
                   ^^^^^^^^^^^^^^^^^^^^^^^
        index => "smart_sense_index_pattern"
        document_id => "%{[id]}"
    }
    stdout {
        codec => rubydebug
    }
}

Next, the systemd file for Logstash needs to be modified to ensure it initializes using the previously created configuration file. To do so, open the file:

vim /etc/systemd/system/logstash.service

The file might be located at /usr/lib/systemd/system/logstash.service.

Look for the line that starts with ExecStart=. Change its value from:

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

To:

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" "-f" "/etc/logstash/conf.d/smartsense.conf"

Then, apply changes by reloading the systemd manager configuration:

systemctl daemon-reload

If installing in a new server that has an empty database, insert a dummy value in the smartsense.load_balancing_count table to avoid errors. To do so, run the following command and enter the database password:

Make sure to replace <database-username> and <mysql-database-ip> with the appropriate values.

#        vvvvvvvvvvvvvvvvvvv       vvvvvvvvvvvvvvvvvvv
mysql -u <database-username> -p -h <mysql-database-ip> \
      -e "USE smartsense; INSERT INTO load_balancing_count
         (id, hostname, load_time, api_id, transaction_type,
         latent, ul, load_count, extraction_time_avg, extraction_quality_avg,
         match_avg, total_avg, extraction_time_min, extraction_quality_min, match_min,
         total_min, extraction_time_max, extraction_quality_max, match_max, total_max)
         VALUES
         (1, 'hostname', '2023-08-31 21:25:40', '8829E30D-4994-4D09-99AF-B6F818473928',
         'IDENTIFY', 'false', 'false', 1, '541.0', '0.0', '48.0', '599.0',
         '541', '0', '48', '599', '541', '0', '48', '599');"

Next, enable the Logstash service to start automatically on boot:

sudo systemctl enable logstash

Then, start the Logstash service:

sudo systemctl start logstash

And follow the log:

tail -f /var/log/logstash/logstash-plain.log

If an error occurs indicating that Logstash cannot write to the directory /var/lib/logstash/{folder}, execute the following command to change its owner:

chown -R logstash:logstash /var/lib/logstash

Finally, to check if Logstash created the index in Elasticsearch, run the following command:

Make sure to replace <elasticsearch-host-ip> with the IP address of the server where Elasticsearch is installed.

curl -X GET "<elasticsearch-host-ip>:9200/_cat/indices?v"
             ^^^^^^^^^^^^^^^^^^^^^^^

The output should be similar to:

health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   smart_sense_index_pattern 6Ux_yM25SvG2zWGdGR0HQw   5   1          1            0      6.7kb          6.7kb
green  open   .kibana_1                 BBO89yLnTUC3F7nhqKwf9w   1   0          4            0       18kb           18kb
green  open   .kibana_task_manager      sIMoATiBRsS8bXiVBCscrA   1   0          2            0     12.5kb         12.5kb

Configuring ELK with SmartSense

Configuring Kibana

Creating the Data View

Make sure to replace <kibana-host-ip> with the IP address of the server where Kibana is installed.

In a browser, go to: http://<kibana-host-ip>:5601. Then, open the sidebar by clicking on this icon, located in the upper left corner of the screen:

Click on Management (last section). Then, in the options on the left side, in the Data section, click on Index Management.

Or go directly to the following URL:

http://<kibana-host-ip>:5601/app/management/data/index_management/indices
       ^^^^^^^^^^^^^^^^

Make sure that the smart_sense_index_pattern index appears in the list.

Next, in the Kibana section of the options on the left side, click on Data Views.

Click the blue button Create data view and fill in the fields with the following information:

• Name: SS Pattern

• Index pattern: smart_sense_index_pattern

• Timestamp field: load_time

Confirm the creation of the Data View by clicking on Save data view to Kibana.

Creating the Dashboards

Open the sidebar again by clicking the icon in the upper left corner of the screen. In the Analytics section, click on Dashboards.

Or go directly to the following URL:

http://<kibana-host-ip>:5601/app/dashboards
       ^^^^^^^^^^^^^^^^

Click the blue button Create dashboard. Then, click on Create visualization. On the right side, configure the visualization with the following information:

• Visualization type: Bar vertical stacked

• Data view: SS Pattern

• Horizontal Axis:

  • Functions: Date histogram

  • Field: load_time

• Vertical Axis:

  • Functions: Sum

  • Field: load_count

Then, click on the + symbol, located in the upper left corner of the screen, to create a new filter. Configure the filter with the following information: transaction_type is ENROLL. Confirm by clicking on Add filter.

Finally, save the dashboard by clicking on Save to library, located in the upper right corner of the screen, and entering the following information:

• Title: SS Enroll Dashboard

• Tags: smartsense-enroll

Click on Save and return.

Repeat the above operations to create the following dashboards:

Adjust the names and tags as needed.

• For VERIFY add the filter: transaction_type is VERIFY

• For UPDATE add the filter: transaction_type is UPDATE

• For IDENTIFY add the filter: transaction_type is IDENTIFY and latent is false

• For LATENT add the filter: transaction_type is IDENTIFY and latent is true

With the five dashboards created, enter each one and set the time range to be displayed by clicking on the calendar icon located in the upper right corner of the screen.

Next, click on Share and then on Copy link. Save the link, as it will be used later.

Repeat the operation for the five dashboards.

At the end of each link, add the following information:

&hide-filter-bar=true&show-time-filter=true&embed=true

For example, the link:

http://172.16.0.185:5601/app/lens#/edit/a0a936d5-4e92-4015-b3e7-37810c2a114a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))

Will become:

http://172.16.0.185:5601/app/lens#/edit/a0a936d5-4e92-4015-b3e7-37810c2a114a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

Repeat the operation for the five obtained links.

Save the links, as they will be used in the next step.

Configuring the Dashboards on SmartSense

Open the the SmartSense configuration file, config.properties, located in the /var/lib/tomcats/smart-sense/conf folder:

vim /var/lib/tomcats/smart-sense/conf/config.properties

Find the # SMARTSENSE - ELK CONFIGURATION section.

For each property (linkEnroll, linkIdentify, linkIdentifyLatent, linkUpdate, linkVerify), insert the corresponding dashboard link obtained earlier. For example:

linkEnroll=http://172.16.0.185:5601/app/lens#/edit/a0a936d5-4e92-4015-b3e7-37810c2a114a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkUpdate=http://172.16.0.185:5601/app/lens#/edit/25d53ee8-7adc-4b06-b05d-f38bfda39c66?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkVerify=http://172.16.0.185:5601/app/lens#/edit/8bfa1546-7990-4ed3-baae-86e421a60aef?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkIdentify=http://172.16.0.185:5601/app/lens#/edit/0d5edf08-ca78-40fc-ac5f-59ca91d07412?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

linkIdentifyLatent=http://172.16.0.185:5601/app/lens#/edit/e3f84cc5-68dd-4c76-a84e-d209da2e777a?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d/d,to:now))&hide-filter-bar=true&show-time-filter=true&embed=true

Save and close the file.

After all steps in the Elastic Stack installation procedure are completed, refer back to the SmartSense Server Configuration manual to complete the configuration.