1. Basic Application Handling

The purpose of this document is to give Griaule application/services and required components basic knowledge to all users. After reading the document, the user should be able to manage the applications and perform basic troubleshooting.

1.1. Architecture

It is possible to simplify the GBDS architecture to four main components and the Integrator application. Each of those components communicates with each other as shown in the figure below. Those components are:

  • GBDS: API, Matcher, and Notifier. Griaule main Applications.
  • Services: ETR, Cardscan, and BEST. Client and Web services that will interact directly with Griaule apps.
  • RDB: Relational Database. Used to store reference metadata.
  • Hadoop: Collection of Open-source software designed for distributed storage and big data processing. Also, handles the communication between Griaule apps.
  • Integrator: Client application used to perform specific activities or customizations using Griaule apps.
../../../_images/archFlow.png

1.2. Directories and Commands

This section will show the directories where the files are located, the initialization commands, and how to perform health checks.

1.2.1. GBDS Management

The GBDS is composed of the GBDS Api, the Marcher, and the Notifier.

1.2.1.1. API Handling

Operation Terminal Command
Start service gbsapid start
Stop service gbsapid stop
Status service gbsapid status
Restart service gbsapid restart
Directories Path
Home Directory /var/lib/griaule/gbsapi
Configuration Directory /etc/griaule/conf/gbsapi
Main Configuration File /etc/griaule/conf/gbsapi/gbdsapi.properties
Log Directory /var/log/griaule/gbsapi/
Main Log File /var/log/griaule/gbsapi/gbsapi.log

1.2.1.2. Matcher Handling

Operation Terminal Command
Start 1 Node /var/lib/griaule/gbds/scripts/start-gbds.sh
Stop 1 Node /var/lib/griaule/gbds/scripts/kill-gbds.sh
Start Cluster /var/lib/griaule/gbds/scripts/start-cluster.sh
Stop Cluster /var/lib/griaule/gbds/scripts/kill-cluster.sh
Directories Path
Home Directory /var/lib/griaule/gbds
Configuration Directory /etc/griaule/conf/gbds
Main Configuration File /etc/griaule/conf/gbds/application.conf
Log Directory /var/log/griaule/gbds/
Main Log File /var/log/griaule/gbds/gbds.log

1.2.1.3. Notifier Handling

Operation Terminal Command
Start service gbsnotifierd start
Stop service gbsnotifierd stop
Status service gbsnotifierd status
Restart service gbsnotifierd restart
Directories Path
Home Directory /var/lib/griaule/gbsnotifier
Configuration Directory /etc/griaule/conf/gbsnotifier
Main Configuration File /etc/griaule/conf/gbsnotifier/notifier.properties
Log Directory /var/log/griaule/gbsnotifier/
Main Log File /var/log/griaule/gbsnotifier/notifier.log

1.2.1.4. Health Checks

This section describes some methods to monitor or troubleshoot the applications.

  • Confirming the application started successfully.

    At the terminal, you should see lines similar to:

API:

{date} {time},257 [INFO ] Http11NioProtocol - Starting ProtocolHandler ["http-n io-8085"]
{date} {time},262 [INFO ] NioSelectorPool - Using a shared selector for servlet write/reading
{date} {time},823 [INFO ] API - Started API in {time} seconds (JVM running for {time})

Matcher:

{date} {time},180 [INFO ] TaskConsumer - Starting consumer using consumer group: spt-02-gbds-tasks
{date} {time},182 [INFO ] PriorityConsumer - subscribed to gbds-tasks-GOD_PRIORITY
{date} {time},184 [INFO ] PriorityConsumer - subscribed to gbds-tasks-HIGHEST_PRIORITY
{date} {time},186 [INFO ] PriorityConsumer - subscribed to gbds-tasks-HIGHER_PRIORITY
{date} {time},187 [INFO ] PriorityConsumer - subscribed to gbds-tasks-HIGH_PRIORITY
{date} {time},189 [INFO ] PriorityConsumer - subscribed to gbds-tasks-DEFAULT_PRIORITY
{date} {time},191 [INFO ] PriorityConsumer - subscribed to gbds-tasks-LOW_PRIORITY
{date} {time},192 [INFO ] PriorityConsumer - subscribed to gbds-tasks-LOWER_PRIORITY
{date} {time},197 [INFO ] PriorityConsumer - subscribed to gbds-tasks-LOWEST_PRIORITY

Notifier:

{date} {time},418 Notifier - group-ETR - notifier-$a - creating notifier
{date} {time},424 NotificationConsumer - group-ETR - notification-supervisor-group-ETR is ready. Start consuming
  • Check if application procces is running.
ps -ef | grep gbsapi | grep -v grep
ps -ef | grep matcher | grep -v grep
ps -ef | grep gbsnotifier | grep -v grep

Note

gbsapid and gbsnotifierd can be checked via service <appd> status.

  • Check if logs are being updated properly with the latest/current timestamps
tailf /var/log/griaule/gbsapi/gbsapi.log
tailf /var/log/griaule/gbds/gbds.log
tailf /var/log/griaule/gbsnotifier/notifier.log
  • Check if API is processing properly (CURL/HTTP)

API:

curl -v http://localhost:8085/gbds/v2/operations/ping

DB Coneection:

curl -v http://localhost:8085/gbds/v2/exceptions/EndDate=1400000000000

1.2.2. Services Management

Composed by Best Server, CardScan Web, and ETR Web.

1.2.2.1. Best Handling

Operation Terminal Command
Start systemctl start tomcat@best.service
Stop systemctl stop tomcat@best.service
Status systemctl status tomcat@best.service
Directories Path
Home Directory /var/lib/tomcats/best
Configuration Directory /var/lib/tomcats/best/conf
Main Configuration File /var/lib/tomcats/best/conf/config.properties
Log Directory /var/lib/tomcats/best/logs
Main Log File /var/lib/tomcats/best/logs/catalina-<date>.log

1.2.2.2. CardScan Handling

Operation Terminal Command
Start systemctl start tomcat@cardscan.service
Stop systemctl stop tomcat@cardscan.service
Status systemctl status tomcat@cardscan.service
Directories Path
Home Directory /var/lib/tomcats/cardscan
Configuration Directory /var/lib/tomcats/cardscan/conf
Main Configuration File /var/lib/tomcats/cardscan/conf/config.properties
Log Directory /var/lib/tomcats/cardscan/logs
Main Log File /var/lib/tomcats/cardscan/logs/catalina-<date>.log

1.2.2.3. ETR Handling

Operation Terminal Command
Start systemctl start tomcat@etr.service
Stop systemctl stop tomcat@etr.service
Status systemctl status tomcat@etr.service
Directories Path
Home Directory /var/lib/tomcats/etr
Configuration Directory /var/lib/tomcats/etr/conf
Main Configuration File /var/lib/tomcats/etr/conf/config.properties
Log Directory /var/lib/tomcats/etr/logs
Main Log File /var/lib/tomcats/etr/logs/catalina-<date>.log

1.2.2.4. Health Checks

This section describes some methods to monitor or troubleshoot the services.

  • Confirming the services started successfully.

    At the terminal, you should see lines similar to:

BEST:

{date} {time} spt02 server[5860]: {date} {time} org.apache.coyote.AbstractProtocol start
{date} {time} spt02 server[5860]: INFO: Starting ProtocolHandler ["http-nio-8123"]
{date} {time} spt02 server[5860]: {date} {time} org.apache.catalina.startup.Catalina start
{date} {time} spt02 server[5860]: INFO: Server startup in {time} ms
{date} {time} spt02 java[5860]: CaptureManager: waiting for some USB event...
{date} {time} spt02 server[5860]: {date} {time} [Pooling thread] INFO com.griaule.best.service.SearchService - Start pooling verification of pending/processing searchs
{date} {time} spt02 server[5860]: {date} {time} [Pooling thread] INFO com.griaule.best.service.SearchService - End of polling: # searches checked.

Note

After started, BEST should start pooling verifications searches from time to time.

CardScan:

{date} {time} spt02 server[3657]: {date} {time} org.apache.coyote.AbstractProtocol start
{date} {time} spt02 server[3657]: INFO: Starting ProtocolHandler ["http-nio-8087"]
{date} {time} spt02 server[3657]: {date} {time} org.apache.catalina.startup.Catalina start
{date} {time} spt02 server[3657]: INFO: Server startup in (time) ms
{date} {time} spt02 java[3657]: CaptureManager: waiting for some USB event...
{date} {time} spt02 server[3657]: {date} {time} [Profile queue thread] INFO com.griaule.cardscan.service.ProfileService - Get Next [GBDS_READY, GBDS_OK] profiles to process

Note

After started, CardScan should check if there’s any profile to process from time to time.

ETR:

{date} {time} spt02 server[790]: {date} {time} org.apache.coyote.AbstractProtocol start
{date} {time} spt02 server[790]: INFO: Starting ProtocolHandler ["http-nio-8089"]
{date} {time} spt02 server[790]: {date} {time} org.apache.catalina.startup.Catalina start
{date} {time} spt02 server[790]: INFO: Server startup in {time} ms

Note

After started, ETR should try to synchronize all exceptions and treatments.

  • For ETR and CardScan web, if you just installed the service or if you removed the service server directory for a redeploy, after a successful start, run the setup script for some additional configuration.
/var/lib/tomcats/setup.sh <app>

Where <app> can be etr or cardscan.

1.2.3. Relational Database Management

Composed by MySQL or MariaDB.

1.2.3.1. Database Handling

Operation Terminal Command
Start service mysqld start
Stop service mysqld stop
Status service mysqld status
Restart service mysqld restart
Directories Path
Home Directory /var/lib/mysql
Configuration Directory /etc/my.cnf.d
Main Configuration File /etc/my.cnf
Log Directory /var/log
Main Log File /var/log/messages

1.2.3.2. Health Check

This section describes some methods to monitor or troubleshoot the database operation.

  • Confirm service is up and running.

Execute the status operation in the terminal. It is expected something similar to:

[user] service mysqld status
Redirecting to /bin/systemctl status mysql.service
 mariadb.service - MariaDB {version} database server
    Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/mariadb.service.d
            migrated-from-my.cnf-setting.conf
    Active: active (running) since {date} {time}; {past time}
    Docs: mand:mysql(8)
        https://mariadb.com/kb/en/library/systemd/
 Maind PID: 15061 (mysqld)
    Status: "Taking your SQL request now..."
    Cgroup: /system.slice/mariadb.service
            15061 /usr/sbin/mysqld --basedir=/usr
  • Check if service process is runniong.
ps -ef | grep mysqld | grep -v grep
  • Check if there are any errors in the logs.
less /var/log/messages | grep mysqld

Important

It is necessary to stop GBDS applications before stopping or restarting this service.

GBDS will NOT automatically reconnect.

Danger

Stopping the database while GBDS is processing can cause issues and data corruption.

If any configuration change was made, restart it to take place.

1.3. Hadoop Management

Hortonworks Data Platform (HDP) services used in GBDS is composed of 6 services, each with a specific function:

  • HDFS - Distributed Filesystem.
  • HBASE - Non-Relational database for Big Data.
  • KAFKA - Queue System. Handles data transfer between apps.
  • ZOOKEEPER - Handle all HDP services management and synchronization.
  • RANGER + SOLR - Security layer. Handle user authentication and audit.

All these services are managed by a graphical user interface (GUI) named AMBARI.

AMBARI Server runs in the master node, AMBARI Agents (Client) run on all servers.

1.3.1. HDP Services Handling

This section will describe how to handle the services with the AMBARI GUI. It is important to notice that all services are sensible, and some critical services like HBASE and HDFS may take some time to stabilizes, especially in big clusters. This is important when initializing or restarting the services.

  • Access AMBARI GUI by http://<hostname>:8080/#/login

    The default username and password are both admin.

  • Service Information is displayed in the left sidebar.

    ../../../_images/ambariStatus.png
    • Green means the service is up and running.
    • Red means the service is stopped.
    • A red dot with numbers on the left side means alerts were found for the specified service.
    • If a service is in maintenance mode (suitcase with + symbol in the middle), the alerts will be suppressed.

  • Start, Stop and Restart services.

    To start/stop/restart the service, the user can click on the ... symbol at the left sidebar, options to start, stop, or restart all services will be displayed.

    ../../../_images/servicesMenu.png

    To start, stop, or restart one service at a time, click in the service and the upper right, click on the ACTIONS button, then select the desired action.

    ../../../_images/actionsMenu.png

  • Following Background Process.

    To follow the background processes, click on the engine icon in the top menu bar.

    ../../../_images/backgroundProcess.png

1.3.2. AMBARI Handling

To start, stop or verify server or agent status, follow the command described in the table below.

Operation Terminal Command
Agent Start ambari-agent start
Agent Stop ambari-agent stop
Agent Status ambari-agent status
Server Start ambari-server start
Server Stop ambari-server stop
Server Status ambari-server status

1.3.3. Logs

Since HDP Services are open-source, the user can find the documentation online. If there is an issue, to help troubleshooting, follow some logs locations:

Directories Path
HDFS /var/log/haddop/hdfs
HBASE /var/log/hbase
ZOOKEEPER /var/log/zookeeper
KAFKA /var/log/kafka
RANGER /var/log/ranger
AMBARI Agent /var/log/ambari-agent
AMBARI Server /var/log/ambari-server