Technical Manual

Introduction

The purpose of this document is to describe the operation of the solution GBS PSBio, the main components, modes of operation, configurations and main procedures. Note that the GBS PSBio solution comprises the SPID Client modules, AC Module and PSBio Module, which will be detailed later.

This manual is updated for PSBio version 5.1.1

Architecture

Introduction

The complete GBS PSBio solution is composed of 3 main modules: SPID Client, AC Module and PSBio Module (Biometric Service Provider).

The modules will be detailed in the following sections, however, basically the following functionalities are assigned to each module:

SPID Client

It is responsible for biometric collection, operator authentication and generation of biometric collection reports.

AC Module

It is responsible for receiving biometric collections coming from the workstations, accessing the IDN Generation Service and forwarding to PSBio. In addition, the AC Module also stores the Certification Authority Image Database and performs operator registration/deletion.

PSBio Module

It is responsible for executing biometric searches and implementing PSBio business rules.

It is important to emphasize that the AC and PSBio Modules work as an integration layer on top of the GBS Cluster, Griaule's ABIS (Automated Biometric Identification System).

SPID Client

SPID Client is the software responsible for:

Perform biometric collections (face and fingerprint)
Register operators
Authenticate operators
Generate biometric collection reports.

Usage

Refer to the SPID Client User Manual for more details on using the SPID Client.

Biometric Collection

During biometric collection, the SPID Client verifies the quality of biometrics, captures biometrics if they meet the configured quality standards and builds the biometric package that will be sent to the AC Module.

Data Encryption

The biometric data from the queries are encrypted and stored on the local machine in an H2 database, both for collections performed in online mode and in offline mode.

For this, the biometric collection client software (Griaule SPID Client) encrypts the data package sent to the server using a random AES key, which is encrypted with an RSA key.

On the server, this data package is decrypted using the RSA private key. The call to the server will be made using a secure HTTPS channel without the need for a certificate embedded in the collection software.

Pointing to AC Module

It is possible to configure which AC Module the SPID Client should point to by editing the file C:\Griaule\SPID\conf\GBSSpid2.properties. To make such configuration the field server.url must be configured with the AC Module URL:

# GBS Server connection
server.url=https://<hostname_modulo_AC>:8082/gbs-spid-server/service/cluster
server.username=admin
server.password=admin

Logs

The application logs can be found in C:\Griaule\SPID\log.

AC Module

The AC module is responsible for receiving transactions coming from the workstations, replacing CPF with IDN (through access to the IDN Generation Service) and forwarding transactions to the PSBio Module.

Details on how to configure pointing to PSBio and mutual HTTPS authentication AC-PSBio are presented in the procedures described in section Toolkits.

In the AC Module, the biometric package is opened and the biometric and biographical data of applicants and operators are saved. This module stores biometric information, CPFs and IDNs.

In addition to being authenticated during biometric collection in the SPID Client, operators' biometrics are also authenticated in the AC Module when the transaction arrives at the server.

Control Panel

The AC module has a control panel, accessed via username and password, in which the following actions can be performed:

Query performed transactions and the result of each
Add / remove operators authorized to log in to the SPID Client.

Transactions:

Transactions are filtered by:

Start date and End date (all transactions within the period will be displayed)
Transaction type:
- Registration
- Search
- Updates
Status
Client CPF
Operator CPF

Operator Registration:

Operator CPFs can be inserted in a list or individually through the addition field. For list insertion, CPFs must be separated by comma, space or line breaks.

The numbers can be entered in the format with dots and hyphen or without separators.

AC Module Configuration: spid.yaml

The configuration file for the AC Module operation is /etc/griaule/spid/properties/spid.yaml. Below are the main properties described:

spid:

authenticationEnabled: Defines the use of authentication in SPID.
Example Value: false
caName: Common name of the certificate used.
Example Value: ac1.griaule.com
documentID: Document used as the key that is sent to the system.
Example Value: documentID
decryptionKeyPath: Path to the file with the encryption key.
Example Value: /etc/griaule/spid/conf/data_private.key
operator:
- deduplicate: Boolean property (true or false) that defines whether operator registration will be performed with or without deduplication. If the AC Module is not operating together with the PSBio Module, operator deduplication (when enabled) is performed against the entire operator and client database. If the AC Module is operating together with the PSBio Module, operator deduplication (when enabled) is performed against the operator database present in the AC Module.
  Example Value: false

hadoop:

zookeeperPath: Path to connect to the server zookeeper, required for communication with the database.
Example Value: localhost:2181

idn:

serviceUrl: IDN service path.
Example Value: http://url:8081/gbs-spid-server/service/idn

gbds:

host: Path to the GBS Cluster service that is consumed
Example Value: localhost
port: Port for the GBS Cluster service that is consumed
Example Value: 8085
useSSL: Defines whether the security protocol should be used in the connection to GBDS
Example Value: false
authenticationEnabled: Use of authentication in GBDS
Example Value: false
authenticationExpiration: Token expiration time
Example Value: 600000
username: User registered in GBDS
Example Value: admin
password: Password of the user registered in GBDS
Example Value: admin

psbio:

active: Defines whether PSBIO is active or not.
Example Value: true
name: Name of the PSBio that is connected. The name must be exactly the same name present in the certificate sent by PSBio.
Example Value: psbio.griaule.com
apiUrl: Path to the psbio server API.
Example Value: https://url:8444/gbs-psbio-server/service/ac-api
hubUrl: Path to the psbio server HUB.
Example Value: https://url:8444/gbs-psbio-server/service/hub
dirUrl: Path to the psbio server directory.
Example Value: https://url:8444/gbs-psbio-server/service/directory

spidx:

organizationName: Certification Authority Name
Example Value: ac1
organizationCallback: SPIDX end-of-collection callback
Example Value: callback
organizationHostname: AC hostname that will be used for SPIDX to notify the SPID Server.
Example Value: ac1
host: Path to the SPIDX service
Example Value: spidx
port: Port for the SPIDX service
Example Value: 8090
qualityThreshold: Fingerprint capture quality threshold
Example Value: 50

server:

port: Port where the PSBio control panel will run without using SSL. If TLS is used, the default port value is 8444.
Example Value: 8082
ssl:
- protocol: Security protocol to be used.
  Example Value: TLS
- client-auth: Type of authentication that the server must have with the client. For example, the value want defines that the server will request an authentication certificate from the client, but it is not mandatory that the client possesses one.
  Example Value: want
- key-store: Path of the keystore with the PSBio certificates used for secure connection with AC module and other PSBios.
  Example Value: /etc/griaule/psbio/keystore/ac1.griaulebiometrics.com.pfx
- key-store-password: PSBio keystore password.
  Example Value: password
- trust-store: Path of the repository with the authorized certificates for secure connection.
  Example Value: /etc/griaule/psbio/keystore/cacerts
- trust-store-password: Password of the certificate repository for secure connection.
  Example Value: changeit

security:

require-ssl: Defines whether the security protocol should be used in the application
Example Value: false

legacy:

http-port: Legacy access port
Example Value: 8081

AC Module Configuration: controlpanel.properties

The configuration file for the operation of the AC Module Control Panel is /etc/griaule/spid/properties/controlpanel.properties. Below are the main properties described:

app.mode

Description: Used for controlling the level of information that is printed in the log.

Default Value: RELEASE

server.http.host

Description: Path to the AC module service.

Default Value: localhost

server.http.port

Description: Port where the AC module service is running.

Default Value: 8082

server.port

Description: Port where the AC Module control panel service will run.

Default Value: 58086

Authentication between AC Module-HSM (IDN Generation)

The keystore files, which contain the AC module's public and private x509 certificates in JKS format, are located by default in the folder /etc/griaule/spid/keystore.

SPID Client Automatic Update Configuration

The AC Module allows configuration of automatic updates for SPID Client applications used for biometric collection. In short, this feature allows updating all workstations that connect to the AC Module without the need to update them one by one. The configurations allow updating with or without user confirmation. That is, in the first case the user is asked whether they want to update the SPID Client. In the second case the application is updated directly, without user confirmation. For more details, refer to section AC Module configuration for automated SPID update.

PSBio Module

The PSBio Module is responsible for implementing PSBio business rules according to ICP-Brasil regulations.

Fault Tolerance

To make use of fault tolerance at the PSBio layer, this module must be installed and configured on all nodes of the GBS Cluster. Fault tolerance is performed through leader election.

When the PSBios of the same cluster start, only one of them is elected leader and only the leader can perform “proactive” activities, such as handling system queues.

All PSBios (including the leader) can receive requests. If the leader fails, the other PSBios detect it and one of them takes leadership.

In integration with the PSBio Module, in case one of the cluster nodes fails, subsequent requests must be forwarded to the subsequent node that assumes leadership.

Location of PSBio Module x509 Certificate Files

The PSBio Module x509 certificate files are located in /etc/griaule/psbio/keystore.

PSBio Module Configuration: config.properties

The configuration file for the PSBio Module operation is /etc/griaule/psbio/properties/config.properties. Below are the main properties described:

gbds.host

Description: Path to the GBS Cluster service that is consumed.

Default Value: localhost

gbds.port

Description: Port for the GBS Cluster service that is consumed.

Default Value: 8085

zookeeper.path

Description: Path to connect to the server zookeeper, required for communication with the database.

Default Value: localhost:2181

kafka.path

Description: Path to connect to the server kafka, required for using metrics with ELK.

Default Value: localhost:9092

psbio.name

Description: Name of the PSBio itself. The value of this field is assigned by ITI uniquely for each PSBio.

Default Value: Griaule-1

psbio-info.path

Description: Path to the JSON in DOC ICP 5.03 format containing information about the other PSBios on the network.

Default Value: /etc/griaule/psbio/conf/psbio-info.json

ac-info.path

Description: Path to the JSON containing information about the accepted AC Modules.

Default Value: /etc/griaule/psbio/conf/ac-info.json

[
	{
		"ACId": "ac1.griaulebiometrics.com",
		"ACName": "ac1.griaulebiometrics.com",
		"ACEndpoint": "https://ac1.griaulebiometrics.com:8444/gbs-spid-server/service/notify"
	}
]

connection.timeout

Description: Timeout in seconds that the module waits for connections it makes with the AC module, with other PSBios and with the GBS Cluster.

Default Value: 10

read.timeout

Description: Timeout in seconds that the module waits for connections it makes with the AC module, with other PSBios and with the GBS Cluster.

Default Value: 10

queue.interval

Description: Time between scans to check transactions in the pending queue.

Default Value: 60

queue.size

Description: Maximum number of transactions executed in parallel in the pending queue.

Default Value: 1000

cache.start

Description: Start time of the cache rebuild process, in UTC.

Default Value: 1

cache.final

Description: End time of the cache rebuild process, in UTC.

Default Value: 9

pending.operations.delay

Description: Interval between checking pending operations among PSBios, in minutes.

Default Value: 60

server.http.port

Description: Port where the PSBio control panel will run without using SSL.

Default Value: 8084

server.port

Description: Port where the PSBio APIs will run using SSL.

Default Value: 8444

server.ssl.key-store

Description: Path of the keystore with the PSBio certificates used for secure connection with AC module and other PSBios.

Default Value: /etc/griaule/psbio/keystore/psbio1.griaulebiometrics.com.pfx

server.ssl.key-store-password

Description: PSBio keystore password.

Default Value: password

server.ssl.trust-store

Description: Path of the repository with the authorized certificates for secure connection.

Default Value: /etc/griaule/psbio/keystore/cacerts

server.ssl.trust-store-password

Description: Password of the certificate repository for secure connection.

Default Value: changeit

resend-transaction.timeout

Description: (UNIX) Timestamp of the time of updating PSBio v2 to PSBio v3, if applicable. All pending transactions up to this timestamp will be resent to GBDS for reprocessing.

Default Value: 0

external-bases.path

Description: Path to the json that contains information about external databases.

kafka.topic.elk.active

Description: Flag for sending messages to ELK.

Default Value: false

Hub and Directory Service End-point

PSBio has two modules responsible for communication with other PSBios: the Biometric Hub and the Directory Service. The first is responsible for transmitting and receiving binary files or XMLs with transactions and biometric images.

The Directory Service implements the synchronization API between PSBios, described in item 3.3.6 of DOC-ICP-05.03 and is responsible for keeping synchronization with other PSBios. The modules' end-points are described below:

Biometric Hub: https://<hostname>/gbs-psbio-server/service/hub

Where hostname is the IP of the server where the PSBio Module is installed.

Directory Service: https://<hostname>/gbs-psbio-server/service/directory

Where hostname is the IP of the server where the PSBio Module is installed.

PSBio Control Panel

Connection to the PSBio control panel is made via HTTPS protocol. Thus, the x509 certificate ( .cer file) must be added to the browser so that the connection is possible. The certificate can be located in the directory indicated in section Location of PSBio Module x509 Certificate Files.

IDN Generator

For environments that use the IDN generator, procedures for installation, configuration, stop and start of the service are described below.

Installation and Configuration

The files provided for installation are:

idnservice.tar.gz
idnservice_*.tar.gz
idnservice
gbs-spid-idnservice_*.jar
config.properties

Files containing ‘*’ have variations depending on the HSM manufacturer and software version.

In addition to the provided files, if necessary, the keystore must be imported according to the HSM manufacturer's manual and saved with the name keystore.jks.

Installation:

All steps must be executed as root.

Move the files to the machine where the IDN generator will be configured.
Extract the compressed files into the listed directories:
- idnservice.tar.gz in /var/lib/griaule
- idnservice_*.tar.gz in /etc/griaule
Example:
```
tar -zxvf idnservice.tar.gz -C /var/lib/griaule/
tar -zxvf idnservice_*.tar.gz -C /etc/griaule/
```
Move the file gbs-spid-idnservice_*.jar to /var/lib/griaule/idnservice:
```
mv gbs-spid-idnservice_*.jar /var/liv/griaule/idnservice
```

Install the file idnservice:

mv idnservice /etc/init.d/
chmod 0755 /etc/init.d/idnservice

Install the file keystore.jks:

mv keystore.jks /etc/griaule/idnservice/keystore/

Install the file config.properties:

mv config.properties /etc/griaule/idnservice/properties/

Configuration:

Edit the file /etc/griaule/idnservice/properties/config.properties and adjust the alias of the key to be used (defined in the HSM) and the defined operating mode. Example:

entry.alias=IDN
operation.mode=GRIAULE_HOM
server.port=8084

Start the IDN Generator

The IDN generator is installed as a service, and can be started with the command:

service idnservice start

Initialization and operation can be followed by the log file:

tail -qF /var/log/griaule/idnservice/idnservice.log

Stop the IDN Generator

To stop the IDN generation service, use the command:

service idnservice stop

Monitoring and Recovery

For monitoring and automatic recovery purposes in case of loss of communication with the HSM, a script is provided that must be inserted into crontab as user root; it can be configured to send email in case of failure.

Run crontab -e and insert:

* * * * * /var/lib/griaule/idnservice/scripts/monitor-idnservice.sh | mail -E -s “Subject” email1,email2

The messages sent are:

Failure detected, restarting IDN Service
Restart was successful
Unable to communicate with the Service
Start was successful

Main Ports

The main communication ports used by GBS PSBio are described in the diagram below. These ports must be unrestricted by firewall in both directions so that communication between applications is not impaired.

The following ports must be open for the machines of the teams that manage the GBS Cluster:

80    - Apache
8080  - Ambari API
8081  - Cluster API
8082  - SPID PSBio API
8444  - PSBio Notification API
8088  - Monitoring
8042  - Monitoring
443   - Monitoring
6515  - On Demand Broker
18080 - Spark
19888 - Monitoring
50070 - HDFS
58086 - Control Panel
16010 - HBase
16020 - HBase
16030 - HBase

Main URLs

The URLs used by the applications are described below:

SPID Client: http://<hostname AC module>:8082/gbs-spid-server/service/cluster

If SSL authentication is used, change http to https in the URL above.

Operator Control Panel: https://<hostname AC module>:58086/gbs-spid-controlpanel/
PSBio Control Panel: https://<hostname PSBio module>/gbs-psbio-server/

To access the PSBio control panel it is necessary that the client workstation has the PSBio certificate to be able to authenticate (HTTPS Communication). Otherwise, the browser page will not load.

Agincourt: http://<host name GBS Cluster AC>:8081/gbscluster-api/rest/services/
GBS Apps: http://<host name GBS Cluster PSBio>:8081/gbscluster-api/rest/services/

The configuration of GBS Apps - Exception Handling to be used together with PSBio must be done carefully. The configuration and usage procedure is described in the document GBS ETR.

It is important to note that the AC Module and PSBio Module are composed of server clusters (GBS Cluster). Thus, the AC Module and PSBio Module APIs are not necessarily on the same server (hostname) as the GBS Cluster API for each of the Modules. This information is important so that application pointing is done correctly. The procedure on how to identify on which machine the AC/PSBio Module is installed will be described in later sections of this manual.

Operation of AC Module and PSBio Module

The AC module can operate in two ways: alone or together with the PSBio Module.

Operation of the AC Module without PSBio Module

When not connected to the PSBio module, the AC Module performs the registration of operators and clients in the AC Module's own database. As with registration, deduplication is also performed in the AC Module database, where all registrations are compared with the operator and client/applicant records. The indexing key used in the AC module is the CPF document.

Operation of the AC Module with PSBio Module

When the AC Module is connected to PSBio, the biometric transaction follows the complete flow:

Biometric collection performed in the SPID Client.
Sending of the transaction to the AC Module.
Storage of images in the AC Module.
IDN generation by querying the IDN generation service.
Sending the transaction to the PSBio Module, which will perform registration deduplication and send the transaction to the other PSBios.
When the transaction is finished, the result is returned to the AC Module, which makes the result available for SPID Client query.

At the architecture level, the main difference is that in this configuration (AC Module and PSBio) operator registration is performed in the AC Module while registration and deduplication of clients/applicants is performed in the PSBio Module

Exception Generation Dynamics

Below are described duplication cases in PSBio, together with statuses reported after exception handling.

Duplicate in registration transaction in the local database
- Registration IDN 1 and biometrics 1, in which registration is finalized in the local database.
- Registration IDN 2 and biometrics 1
  - Situation: In this scenario, a registration exception will be generated.
  - Status notified to AC: status of ENROLL_ANOMALY, accompanied by message “Enroll resulted in exception | tcn: <tcn> | idn: <idn> | Same biometrics found in local PSBio | elapsed time: <elapsed_time> s”.
    Registration presented a biometric exception and is awaiting handling in the GBS Apps application.
- Possible treatments:
  - Same biometrics: when the biometrics of the two registrations are the same, indicating suspicion of fraud. ATTENTION: the biometrics of both profiles will be sent to the blacklist (blacklist).
    Status notified to AC:
    For TCN 1 the status PERSON_FRAUD is notified, accompanied by the message “Enroll transaction marked as fraud | tcn: <tcn>| idn: <idn> | elapsed time: <elapsed_time> s”
    For TCN 2 the status FRAUD is notified.
  - Different biometrics: when the biometrics of the two registrations are not the same; that is, the system reported similarity, but the biometrics are not of the same person.
  - Recollection: when you want to reject the biometrics of the second registration. The second profile will be ignored and will not continue with the registration.
  - Incorrect enroll: when you want to ignore the biometrics of the first registration. The biometrics of the first profile will be deleted and the second profile will be registered.
Failure in update transaction in the local database
- Registration IDN 1 and biometrics 1, in which registration is finalized in the local database.
- Update IDN 1 and biometrics 2
  - Situation: In this scenario, an exception will be generated in an update transaction.
  - Status notified to AC: status of UPDATE_ANOMALY.
    Registration presented a biometric exception and is awaiting handling in the GBS Apps application.
  - Possible treatments:
    Same biometrics: when the biometrics of the registration and the update belong to the same person. This will allow the update to be processed.
    Status notified to the AC: For TCN 2 the status UPDATE_OK is notified.
    Different biometrics: when the biometrics of the registration and the update are not the same. If the original was the valid profile, select the option consider original valid; If the transaction that generated the exception was valid, select consider exception valid. Important: Rejected biometrics will be sent to the blacklist.
    Consider Original Valid:
    Status notified to AC:
    For TCN 2 the status FRAUD is notified.
    Consider Exception Valid: After handled, TCN 1 is notified as FRAUD and an IDE transaction is assembled, which is forwarded to be searched by the other PSBios. Upon completing the search, the AC is notified that the transaction of TCN 2 was completed successfully.
    Status notified to AC:
    For TCN 1 the status FRAUD is notified.
    For TCN 2 the status UPDATE_OK is notified.
    Recollection: when you want to refuse the update biometrics. The second profile will be ignored and will not continue with the registration.
    Incorrect enroll: when you want to ignore the registration biometrics. The biometrics of the first profile will be deleted and those of the update will be registered.
Registration transaction with duplicate found in cache
- Another PSBio registered IDN 1 and biometrics 1 in its local database.
- Origin PSBio registers IDN 2 with biometrics 1.
- Situation: when performing the biometric search for IDN 2, the same biometrics will be found in the origin PSBio's cache. Then the origin PSBio forwards the IDE packet to the other PSBios on the network and generates an exception to be handled in GBS Apps.
- Status notified to AC: ENROLL_ANOMALY together with message “ENROLL resulted in exception | tcn: <tcn> | idn: <idn> | Same biometrics found in PSBio <PSBio> | elapsed time: <elapsed_time> s”
  - Then the origin PSBio forwards the IDE packet to the other PSBios on the network.
- Possible treatments:
  - Same biometrics:
    When performing such handling, GBS Apps will indicate that such handling is not available for this situation, since such transaction must be handled by another PSBio, presenting the message: “Unable to treat exception. idn is registered in biometric cache”.
  - Different biometrics:
    When choosing the option of “different biometrics”, the exception will be handled locally and will wait for the response from the other PSBios to notify the AC module.
    If the origin PSBio receives all VRE packets with field SRF: “X” (Negative result for the biometrics sent by the PSBio), the origin PSBio will notify the ENROLL_OK status to the AC Module.
    If the origin PSBio receives at least 1 VRE packet with field SRF: “M” (Positive result for the biometrics sent by the PSBio), the origin PSBio will notify the ENROLL_ANOMALY status together with the message “Same biometric(s) found in PSBio <NOME_PSBIO>.”
  - Recollection:
    When choosing the option of “recollect”, the exception will be discarded so that a new collection can be made. The registration transaction will not be affected.
  - Incorrect enroll:
    When performing such handling, GBS Apps will indicate that such handling is not available for this situation, since such transaction belongs to another PSBio, presenting the message: “Unable to treat exception. idn is registered in biometric cache”
Update transaction with duplicate found in another PSBio
- Another PSBio registered IDN 1 and biometrics 1 in its local database.
- Origin PSBio registers IDN 1 with biometrics 2.
- Situation:
  - Upon receiving the update transaction, the origin PSBio will forward the UPR file to the PSBio that has IDN 1 in its local database and will wait for the result of that transaction.
  - Status notified to AC:
    If it is not possible to perform the biometric match in another PSBio, the status (for the AC Module) of UPDATE_ANOMALY will be notified accompanied by the message “Update resulted in exception | tcn: <tcn> | idn: <idn> | Different biometrics from registration | elapsed time: <elapsed_time> s”

Toolkits

Export of Biometrics from the AC Module

It is possible to export the biometrics collected and stored in the AC Module through the script get-biometrics. The input parameters of the script are the destination directory, the start date and end date, determining that the biometrics registered/updated between these two dates will be exported to the desired folder. The dates must be provided in the format DD/MM/YYYY.

The biometrics will be arranged in directories with the following organization:

output-dir
    |
    |
    +---yyyy-mm-dd
    |     |
    |     +---id-value
    |     |      |
    |     |      +---value_index_yyyymmdd.wsq
    |     |      |
    |     |      +---value_yyyymmdd.jpg
    |     |
    |     +---id-value
    |            |
    |            +---value_index_yyyymmdd.wsq
    |            |
    |            +---value_yyyymmdd.jpg
    |
    +---tguids.txt

In this directory structure the base is the directory provided when invoking the script. At the second level there are directories represented by the transactions' dates. Inside those there is one or more directories represented by the id, document name provided in that person's registration (pguid if there is no document), and the value for a person. That directory in turn contains the biometrics of the person in question. The fingerprint files will be in WSQ format with the file name being the person's identifier value, the finger index and the date. The face files will be in JPEG format with the file name composed by the person's identifier and the date. Inside the base directory there is also the file tguids.txt with the identification (tguid) of all transactions performed in the period.

The call to use this script must be made through:

python get-biometrics.py --out-dir OUTPUTDIR --ini-date INIDATE --end-date ENDDATE

SPID H2 Dump

Requirements:

The Java that will run the tool must have the Java Cryptography Extension (JCE) Unlimited Strength package installed.
The script must be executed from the level where the conf folder is located, which contains the database encryption and decryption keys.

Example: If the folder is extracted to C:\Users\usr39a\Desktop\gbs-spid-h2dump , the execution must be performed from that folder.

The script has three execution modes, indicated by the -mode parameter:

dump: Exports the biometrics present in the databases to a specific folder.
report: Queries the result of pending collections and generates three files with the collection statuses in the databases: one for final statuses, another for results with error and the third for results still pending.
resend: Resends the collections that do not have final statuses to the server.

Additional script parameters:

-dbDir

Folder containing the database files to be analyzed (only files with the extension .mv.db are analyzed).

-dbFile

Indicates a specific database file to be analyzed (must contain the extension .mv.db). It is mandatory in all modes that either this or the previous parameter (-dbDir) be present. If both exist, only -dbFile will take effect.

-url

Server service path. The URL is the same used by the SPID Client application. Mandatory for the report and resend modes.

-output (optional)

Directory where unique biometrics (only the first collection of each client identifier) exported will be saved. Only has effect in dump mode.

-outputUpdates (optional)

Directory where duplicate biometrics (if a collection with a given client identifier already exists exported in the previous directory) will be saved. Only takes effect in dump mode.

-operatorID (optional)

Filter that considers only collections with a specific operator identifier (the identifier must be without punctuation). Example: 12345678909.

-clientID (optional)

Filter that considers only collections with a specific client identifier (the identifier must be without punctuation).

-interval (optional)

Interval in seconds between each collection submission in resend mode. Note: Use values of 10 seconds or more.

Execution examples:

java -jar gbs-spid-h2dump.jar -mode report -dbDir J:/data/ -url http://192.168.0.140:8082/gbs-spid-server/service/cluster/

java -jar gbs-spid-h2dump.jar -mode resend -dbDir J:/data/ -url http://192.168.0.140:8082/gbs-spid-server/service/cluster/ -interval 10

java -jar gbs-spid-h2dump.jar -mode dump -dbFile J:/data/spid.mv.db -clientID 12345678909 -output collections

Procedures

Insertion of Profiles into the PSBio Blacklist

To insert images into the blacklist you must perform a trusted-Enroll operation directly in GBDS for each biometric fraud profile, via the URL:

http://<ip>:8085/gbds/v2/people/trusted

If the profile contains fingers and face, all biometrics must be sent in the same JSON package, analogous to the package used for enroll operations.

This call is a POST containing a JSON:

{
	"meta": {
		"timeout": -1
	},
	"data": {
		"keys": [
			{
				"id": "blacklist",
				"value": "BBBBBBBB-BBBB-4B07-CCCC-67B6E09F64C8"
			}
		],
		"biometric": [
			{
				"index": "(0/10)",
				"content": "<base64>",
				"source": "ORIGINAL",
				"type": "(FINGERPRINT/FACE)",
				"format": "(WSQ/JPEG)",
				"properties": {
					"width": <width>,
					"height": <height>,
					"resolution": <resolution>
				}
			}
		],
		"labels": [
			"BLACKLIST"
		]
	}
}

A random UUID must be generated for the “value” field. The indices used in this blacklist registration call are those in the table below:

Index

Biometric

Left little finger

Left ring finger

Left middle finger

Left index finger

Left thumb

Right thumb

Right index finger

Right middle finger

Right ring finger

Right little finger

Face

Obtain TCNs of Transactions Forwarded between PSBios

Sent Transactions

To obtain the transactions sent to a given PSBio, use the following command:

hbase shell

scan 'psbio_transactions',{FILTER => "SingleColumnValueFilter('f','dest', =, 'substring:GRIAULE') AND SingleColumnValueFilter('i','metadata', =, 'substring:originaryTCN')", MAXLENGTH => 300}

In this example, the result will be the list of all transactions sent to the PSBio GRIAULE.

Received Transactions

To obtain the transactions received from a given PSBio, use the following command:

hbase shell

scan 'psbio_transactions',{FILTER => "SingleColumnValueFilter('f','dest', =, 'substring:GRIAULE') AND SingleColumnValueFilter('i','metadata', !=, 'substring:originaryTCN')", MAXLENGTH => 300}

In this example, the result will be the list of all transactions received from PSBio GRIAULE.

List of Pending Transactions in Other PSBios

The query below allows obtaining the list of pending IDE transactions that other PSBios must process for the origin PSBio:

hbase shell << EOF > pending_1n_ORGANIZATION.out
scan 'psbio_orderPending',{MAXLENGTH => 300, FILTER => "SingleColumnValueFilter('i','dest',=,'binary:ORGANIZATION',true,true) AND SingleColumnValueFilter('i','status',=,'binary:PENDING',true,true)"}
EOF

Important: Replace ORGANIZATION in ...'dest',=,'binary:ORGANIZATION',true... with the name of the PSBio from which you want to obtain the pending items.

The generated list reports the TCNs that another PSBio owes processing for the origin PSBio.

In the example above, the output file will be pending_1n_ORGANIZATION.out.

Obtain TCNs and Status of Transactions Generated by the AC Module in the PSBio

hbase shell << EOF > outputFile.out
scan 'psbio_transactions',{MAXLENGTH => 300, TIMERANGE => [1517529600000,1518739199000], FILTER => "SingleColumnValueFilter('i','type',=,'substring:ENROLL',true,true)", COLUMNS => ['i:ac_sr','f:status','i:type']}
EOF

The numeric fields in the TIMERANGE above indicate the start and end timestamps in milliseconds, respectively, of the transactions for which you want to obtain the TCNs and results.

Configuration of the Pointing between AC Module and PSBio Module

To check which environment the PSBio Module and which the AC Module are installed in, just access the folder /var/lib/griaule:

In the case of the PSBio Module, there will be a folder called psbio.
In the case of the AC Module, there will be a folder called spid.

To point the AC Module to the PSBio Module, the following step-by-step must be followed:

Server with AC Module

Edit the file /etc/griaule/spid/properties/config.properties and replace <hostname> with the hostname of the instance where the PSBio module is running:

# name of the PSBio with which the Proxy will communicate,
# must be identical to the name configured in the
# config.properties file of the PSBio
psbio.name=server-psbio
psbio.api.url=https://<hostname>/gbs-psbio-server/service/ac-api
psbio.hub.url=https://<hostname>/gbs-psbio-server/service/hub
psbio.directory.url=https://<hostname>/gbs-psbio-server/service/directory

Note the information in ac.name.

Server with PSBio Module

Edit the file /etc/griaule/psbio/conf/ac-info.json, so that it matches the value configured in the AC module's config.properties:

[
	{
		"ACId": "ac1.griaulebiometrics.com",
		"ACName": "ac1.griaulebiometrics.com",
		"ACEndpoint": "https://ac1.griaulebiometrics.com:8444/gbs-spid-server/service/notify"
	},
]

The information in the config.properties of the PSBio Module and AC Module need to be consistent. The names must match, as well as the certificates. In addition, both environments need to be able to communicate through the configured ports (in both directions).

Driver, broker and GBS Cluster API confirmation

Once the above configurations have been made, it is necessary to confirm that the driver, broker and GBS Cluster API are up for both environments. This can be done through the SC API.

Access http://<hostname>:8081/gbscluster-api/rest/services/sc.

When accessing this URL, the driver and brokers for the GBS Cluster will be listed.

Restarting the AC Module and PSBio

This step must be done on both environments as root.

service psbio start
service spid start
service spid-cp start

SPID Client Configuration

Once the GBS Cluster services and the AC/PSBio Module are up, you need to configure the SPID Client to point to the AC Module.

To do this, edit the file C:\Griaule\SPID\conf\GBSSpid2.properties and change server.url as in the example below:

server.url=http://<hostname>:8082/gbs-spid-server/service/cluster/

Replace <hostname> with the hostname of the AC Module. The PC needs to be able to access the AC Module through port 8082.

Log Monitoring

Registrations can be monitored in the directories /var/log/griaule/spid and /var/log/griaule/psbio.

Configuration of the IDN Generation Service URL

It is possible to configure which IDN generation service URL the AC Module should point to. This can be done by changing the property idn: serviceUrl in the file /etc/griaule/psbio/properties/spid.yaml.

Configuration of Certificates for HTTPS Authentication AC-PSBio Module

Keystore Installation

Obtain the .keystore file, which is the private certificate that must be in JKS format. Basically it is a Java keystore with the client's private and public keys and also the public certificates of the signing chain.

Place the .keystore file in the folder /etc/griaule/psbio/keystore.

PSBio Configuration

Edit the file /etc/griaule/psbio/conf/config.properties and update the path and password fields. The ac.name field must also be changed to the Common Name of the AC module certificate. Example:

server.ssl.key-store=/etc/griaule/psbio/keystore/certificate.keystore
server.ssl.key-store-password=password
psbio.name=
ac.name=

Edit the file /etc/griaule/psbio/conf/psbio-info.json and change the property PSBioId to the Common Name of the AC module certificate being run at the associated address.

Importing the AC Public Certificate

For the PSBio module to recognize the AC and other PSBios' certificate, it is necessary that the AC or PSBio public certificate and its respective chain be imported. The files in .cer or .pem format must be imported into the list of accepted certificates, which is stored in the file /etc/griaule/psbio/keystore/cacerts. The import can be performed with the command below:

keytool  -import -trustcacerts -alias server-psbio10-w -file server-psbio10.cer -keystore /etc/griaule/psbio/keystore/cacerts -storepass changeit

The parameter -alias indicates the name of the certificate to be imported, and -file indicates the certificate file.

Importing the PSBio Public Certificate

For the AC module to recognize the PSBio certificate (mutual HTTPS authentication), it is necessary that the PSBio public certificate and its respective chain be inserted. For this, the files in .cer or .pem format must be imported into the list of accepted certificates, which is stored in the file /etc/griaule/spid/keystore/cacerts. The import can be performed with the command below:

keytool  -import -trustcacerts -alias server-psbio10-w -file server-psbio10.cer -keystore /etc/griaule/spid/keystore/cacerts -storepass changeit

The parameter -alias indicates the name of the certificate to be imported, and -file indicates the certificate file.

Configuration of Certificates for HTTPS Authentication between AC Module and IDN Generation Service

For mutual authentication to occur between the AC Module servers and the IDN Generation service server, it is necessary that the AC Module's public certificate be added to the IDN Generator and vice versa.

The import of the AC Module public certificate into the IDN Generator is not described in this manual, because the implementation of the IDN generation service may vary for each environment.

The import of the IDN Generator's public certificate should be performed similarly to that described in the section Importing the AC Public Certificate.

Changing the Java Version Running the Application

To specify a particular Java version to run the application, edit the file initialize-psbio.sh, which is located in /var/lib/griaule/psbio/scripts. Change the first line that contains /etc/griaule/psbio/jdk1.8.0_101/bin/java.

This modification is not recommended.

Database Cleanup

To perform database cleanup, you must follow the steps below and run the script truncate_hbase.sh.

This script will delete all data present in the database. Execution should only be performed in cases of full acceptance and understanding of the implications and impacts of its execution.

The script must be run as user griaule.

1. Stop the AC and PSBio services

service spid stop
service spid-cp stop

2. Run the script

truncate_hbase.sh

3. Restart the services

Wait a few minutes and restart the AC and PSBio Module services:

service spid start
service spid-cp start

Configuration of AC Module and PSBio Module operation

The AC Module can operate coupled or not coupled with the PSBio Module. In case of coupling, all client registrations will be sent and deduplicated solely by PSBio. If not coupled, all client registrations will be deduplicated in the AC Module itself.

To perform the configuration, the following procedure can be carried out:

Edit the file config.properties of the AC Module.
To enable coupling with PSBio, set the variable operation.mode=psbio
To disable coupling with PSBio, set the variable operation.mode=gbds

AC Module configuration for automated SPID update

From SPID version 1.4.0, two new calls were implemented for the AC module (check-version and update-version). In them, during SPID Client or SPID Services startup, the program checks the latest available version of the application, and may ask the user if they want to update the version on their computer.

To upload a new update to the server, the following procedures must be followed:

On the AC Module, in the folder /etc/griaule/spid/client/exe , upload the newest available version with the following name: **%COMPANY-NAME%-spid-client-vX-X-X.exe*, where X-X-X represents the version X.X.X. For example, for SPID version 1.4.12, the file must be:/etc/griaule/spid/client/exe/griaule-spid-client-v4-1-12.exe**
In the folder above (/etc/griaule/spid/client) there must be a file called version. In it, only one line of text should exist, with the number of the latest available version. This number must match that of some executable present in the folder /etc/griaule/spid/client/exe.
Two options are possible for the version present in the version file. The first is to save the number of the newest version without an asterisk at the end (format X.X.X). In this way, the SPID user will be asked whether or not they would like to update their version.
The other option is to save the number of the newest available version with an asterisk at the end (format X.X.X*). In this way, the SPID update will be mandatory for all users connected to that server, and the new version will be installed without the user's choice.

Configure AC Module to use local IDN generation

To point the AC Module to the local IDN generator, the following procedure must be performed:

Edit in the file /etc/griaule/spid/properties/config.properties the property idn.service.url to http://localhost:8082/gbs-spid-server/service/idn

Replication of PSBio Services in another Site

In cases of disaster recovery it is possible to continue PSBio services by importing the information from the main site into another backup site that has the GBS Cluster installed.

For this, you must maintain a routine of exporting the database data through the methodology detailed in the GBS Cluster documentation.

Logs

AC Module

On the servers that make up the AC module, monitoring and log tracking can be performed through the files below:

/var/log/griaule/spid/ac.log
/var/log/griaule/spid/stderr.out
/var/log/griaule/spid/stdout.out

PSBio Module

On the servers that make up a PSBio module, monitoring and log tracking can be performed through the files below:

/var/log/griaule/psbio/psbio.log
/var/log/griaule/psbio/stderr.out
/var/log/griaule/psbio/stdout.out

Monitoring

For more information about monitoring GBS PSBio, consult the GBS Cluster monitoring manuals.

Installation

PSBio Installation

Minimum requirements:

Available memory: 2 GB
Available disk space: 2 GB
GBS Cluster installed

Required packages:

gbs-psbio-server.tar.gz

Installation

Extract the contents of gbs-psbio-server.tar.gz and move the file gbs-psbio-server.jar to the directory /var/lib/griaule:

tar -xf gbs-psbio-server.tar.gz .
mv gbs-psbio-server.jar /var/lib/griaule/

Move the config.properties file to the directory /etc/griaule/psbio/properties:

mv config.properties /etc/griaule/properties/

Move the files psbio-info.json and ac-info.json to the directory /etc/griaule/psbio/conf:

mv *.json /etc/griaule/conf/

Move the initialize-psbio.sh file to the directory /var/lib/griaule/psbio/scripts and adjust its permissions:

mv initialize-psbio.sh /var/lib/griaule/psbio/scripts/
chmod 755 /var/lib/griaule/psbio/scripts/initialize-psbio.sh

Move the file psbio to the folder /etc/init.d and adjust its permissions:

mv psbio /etc/init.d/
chmod 755 /etc/init.d/psbio

Installation automation

There is an RPM for PSBio installation/update. The following steps are performed at installation time: Creates a folder /root/configs/ and copies the configuration files into it (config.sh ac-info.json config.properties initialize-psbio.sh psbio psbio-info.json) Extracts gbs-psbio-server.jar to /var/lib/griaule/psbio/ Therefore, it is necessary to manually copy and modify the psbio-info.json and ac-info.json files.

When running the config.sh script in /root/config.sh, for each case the user can choose between doing nothing or: Overwrite the psbio init service in /etc/init.d/ Overwrite initialize-psbio.sh in /var/lib/griaule/psbio/scripts/ Overwrite or merge config.properties If merging, the script will append configurations from the new file that are not present in the old file. It is necessary to change the values according to the environment.

rpm -ivh gbspsbioserver-RELEASE.x86_64.rpm
/root/configs/config.sh

Configuration

config.properties

Edit the file /etc/griaule/psbio/properties/config.properties as needed:

#
# GBS PSBio Server properties
# Copyright 2020 Griaule Biometrics
#

# Path to Zookeeper
zookeeper.path=localhost:2181

# GBDS Connection
gbds.host=gbds
gbds.port=8085

# GBDS Authentication
gbds.authentication.enabled=false
gbds.authentication.expiration.interval=600000
gbds.username=admin
gbds.password=admin

# Timeout of any connection in seconds
connection.timeout=30

# Path to data decryption private key
decryption.key.path=/etc/griaule/spid/conf/data_private.key

# Mode of operation
operation.mode=gbds

# Type of document used as key
document.id=documentID

# PSBio Connection
psbio.name=psbio.griaule.com
psbio.api.url=https://psbio.griaule.com:8444/gbs-psbio-server/service/ac-api
psbio.hub.url=https://psbio.griaule.com:8444/gbs-psbio-server/service/hub
psbio.dir.url=https://psbio.griaule.com:8444/gbs-psbio-server/service/directory

# ID of this AC
ac.name=ac1.griaulebiometrics.com

# IDN Service Connection
idn.service.url=http://localhost:8082/gbs-spid-server/service/idn

# Require Operator Validation
operator.validate=false

# Deduplicate Operator
operator.deduplicate=true

# Evidences
evidence.generate=true
evidence.path=/etc/griaule/spid/coletas
evidence.format=griaule

# Port configuration
server.port=8444
server.http.port=8082

# SSL
server.ssl.key-store=/etc/griaule/spid/keystore/ac1.griaulebiometrics.com.pfx
server.ssl.key-store-password=password
server.ssl.trust-store=/etc/griaule/spid/keystore/cacerts
server.ssl.trust-store-password=changeit

# Timestamp of version upgrade
resend-transaction.timeout=0

# External Bases Verification
external-bases.path=/etc/griaule/psbio/conf/external-bases.json

psbio-info.json

Edit the file psbio-info.json located at /etc/griaule/psbio/conf:

vim /etc/griaule/psbio/conf/psbio-info.json

The document will present entries similar to the entry below, for each PSBio. They should be configured in the pattern shown below:

[
	{
		"PSBioId": "server-psbioX",
		"PSBioName": "<hostname>",
		"nist_endpoint": "https://localhost/gbs-psbio-server/service/hub",
		"directory_endpoint": "https://localhost/gbs-psbio-server/service/directory"
	},
]

The PSBioId field is the name given to the ITI for each PSBio. The PSBioName field must be completed with the CommonName of the respective PSBio certificate. The two URLs reference the PSBio's own cluster.

ac-info.json

Edit the file ac-info.json located at /etc/griaule/psbio/conf:

vim /etc/griaule/psbio/conf/ac-info.json

The document will present entries similar to the entry below, for each AC. They must be configured in the pattern shown below:

[
	{
		"ACId": "acX.griaulebiometrics.com",
		"ACName": "acX.griaulebiometrics.com",
		"ACEndpoint": "https://ac1.griaulebiometrics.com:8444/gbs-spid-server/service/notify",
	},
]

The ACId and ACName fields must be completed with the CommonName of the respective AC certificate. The URL references the AC's notification endpoint.

Use the following commands to start, stop, restart or check the status of PSBio:

service psbio start
service psbio stop
service psbio restart
service psbio status

external-bases.json

Edit the file external-bases.json located at /etc/griaule/psbio/conf/ as needed:

[
	{
		"id": "DATAVALID",
		"host": "host",
		"port": 443
	}
]

AC Module Installation

Minimum requirements:

Available memory: 2 GB
Available disk space: 2 GB
GBS Cluster installed

Required packages:

spid.tar.gz
spid_etc.tar.gz
spid
spid-cp
gbs-spid-server_<version>.zip

Installation

Extract the contents of spid.tar.gz in the directory /var/lib/griaule:

tar -zxvf spid.tar.gz -C /var/lib/griaule/

Extract the contents of spid_etc.tar.gz in the directory /etc/griaule:

tar -zxvf spid_etc.tar.gz -C /etc/griaule/

Move the files spid and spid-cp to the folder /etc/init.d and adjust its permissions:

mv spid /etc/init.d
mv spid-cp /etc/init.d
chmod 755 /etc/init.d/spid*

Extract the package files gbs-spid-server and move them to the indicated locations:

unzip gbs-spid-server_<version>.zip
mv gbs-spid-server.jar /var/lib/griaule/spid
mv gbs-spid-controlpanel.jar /var/lib/griaule/spid
mv config.properties /etc/griaule/spid/properties
mv controlpanel.properties /etc/griaule/spid/properties

Configuration

spid.yaml

Edit the file /etc/griaule/spid/properties/spid.yaml as needed. An example of how the file can be configured:

# SPID Server Configuration
spid:
  authenticationEnabled: false
  caName: acdev1.griaule.com
  documentId: documentID
  decryptionKeyPath: /etc/griaule/spid/conf/data_private.key
  operator:
    deduplicate: false

hadoop:
  zookeeperPath: localhost:2181

idn:
  serviceUrl: http://localhost:8081/gbs-spid-server/service/idn

evidence:
  generate: true
  path: /etc/griaule/spid/coletas
  format: griaule

gbds:
  host: localhost
  port: 8085
  useSSL: false
  authenticationEnabled: false
  authenticationExpiration: 600000
  username: admin
  password: admin

psbio:
  active: true
  name: psbiodev.griaule.com
  apiUrl: https://localhost:8444/gbs-psbio-server/service/ac-api
  hubUrl: https://localhost:8444/gbs-psbio-server/service/hub
  dirUrl: https://localhost:8444/gbs-psbio-server/service/directory

spidx:
  organizationName: acdev1
  organizationCallback: callback
  organizationHostname: acdev1
  host: spidx
  port: 8090
  qualityThreshold: 50

# Additional Spring Configurations
server:
  port: 8082
  ssl:
    protocol: TLS
    client-auth: want
    key-store: /etc/griaule/spid/keystore/ac1.griaulebiometrics.com.pfx
    key-store-password: password
    trust-store: /etc/griaule/spid/keystore/cacerts
    trust-store-password: changeit

security:
  require-ssl: false

# Legacy Configuration
legacy:
  http-port: 8081

controlpanel.properties

Edit the file /etc/griaule/spid/properties/controlpanel.properties as needed:

#
# GBS SPID Control Panel properties
# Copyright 2020 Griaule Biometrics
#

# SPID Control Panel port
server.port=58086

# SPID Server Connection
server.http.host=localhost
server.http.port=8082

Use the following commands to start, stop, restart or check the status of the AC module and the Control Panel:

service spid start
service spid stop
service spid restart
service spid status

service spid-cp start
service spid-cp stop
service spid-cp restart
service spid-cp status

Backup Configuration

Copy the files backup-spid-server.sh and restore-spid-server.sh to the directory /var/lib/griaule/gbscluster/scripts/.

Add the cron backup job as user griaule. Run the command crontab -e and add the following line to the configuration:

0 23 * * * griaule /var/lib/griaule/gbscluster/scripts/backup-spid-server.sh

The script will back up the AC Module tables to the directory /home/griaule/backup/modulo-ac and will keep the last 5 days of backups. It is the client's responsibility to copy these files to external media.

To perform the restoration, extract the contents of the file tgz created by the backup to the directory /home/griaule/backup/modulo-ac and run the script restore-spid-server.sh as user griaule.

Contents of the backup-spid-server.sh script:

#!/bin/sh
### ====================================================================== ###
##                                                                          ##
##  Griaule Biometric PSBio data backup script                              ##
##                                                                          ##
### ====================================================================== ###

HDFS_BACKUP_DIR=griaule/backup/modulo-ac

hdfs dfs -test -d $HDFS_BACKUP_DIR
if [ $? == 0 ]; then
  echo "Clearing previous backup:"
  hdfs dfs -rm -R $HDFS_BACKUP_DIR
fi;

hbase org.apache.hadoop.hbase.mapreduce.Export 'proxy_operators' $HDFS_BACKUP_DIR/proxy_operators
hbase org.apache.hadoop.hbase.mapreduce.Export 'proxy_order' $HDFS_BACKUP_DIR/proxy_order
hbase org.apache.hadoop.hbase.mapreduce.Export 'proxy_people' $HDFS_BACKUP_DIR/proxy_people
hbase org.apache.hadoop.hbase.mapreduce.Export 'proxy_transactions' $HDFS_BACKUP_DIR/proxy_transactions

hdfs dfs -get griaule/backup/modulo-ac /home/griaule/backup/modulo-ac
tar -czvf /home/griaule/backup/modulo-ac.tgz /home/griaule/backup/modulo-ac
mv /home/griaule/backup/modulo-ac.tgz /home/griaule/backup/modulo-ac_`date +%d-%m-%Y`.tgz
rm -rf /home/griaule/backup/modulo-ac
find /home/griaule/backup/* -mtime +5 -exec rm {} \;

Contents of the restore-spid-server.sh script:

#!/bin/sh
### ====================================================================== ###
##                                                                          ##
##  Griaule Biometric PSBio data restore script       ##
##                                                                          ##
### ====================================================================== ###

HDFS_BACKUP_DIR=griaule/backup/modulo-ac

hdfs dfs -test -d $HDFS_BACKUP_DIR
if [ $? != 0 ]; then
  echo "HDFS Backup path not found:"
  echo $HDFS_BACKUP_DIR
  exit 1;
fi;

hdfs dfs -put /home/griaule/backup/modulo-ac griaule/backup/modulo-ac

hbase org.apache.hadoop.hbase.mapreduce.Import 'proxy_operators' $HDFS_BACKUP_DIR/proxy_operators
hbase org.apache.hadoop.hbase.mapreduce.Import 'proxy_order' $HDFS_BACKUP_DIR/proxy_order
hbase org.apache.hadoop.hbase.mapreduce.Import 'proxy_people' $HDFS_BACKUP_DIR/proxy_people
hbase org.apache.hadoop.hbase.mapreduce.Import 'proxy_transactions' $HDFS_BACKUP_DIR/proxy_transactions

Facelib Setup

Run the script facelib_setup.sh.

Contents of the script facelib_setup.sh:

#!/bin/bash

EXE_TIME=$(date +'%Y-%m-%d-%H%M%S')
LOG_PATH=$PWD/logs
LOG_FILE=$LOG_PATH/facelib_setup-${EXE_TIME}.log
function logger(){
    # FUNCTION PARAMETERS
    # logger functionName status [INFO, WARN, ERROR] logMessage
    # exemple: logger "hello_world" "INFO" "Hello World" first_log

    if [ ! -d $LOG_PATH ];then
        mkdir $LOG_PATH
    fi

    CURRENT_TIME=$(date +'%Y-%m-%d %H:%M:%S')
    FUNCTION=$1
    STATUS=$2
    LOG_MESSAGE=$3

    LOG_FORMAT="$CURRENT_TIME  -  $FUNCTION  -  $STATUS  -  $LOG_MESSAGE"

    # commit log message
    echo $LOG_FORMAT | tee -a $LOG_FILE

}

function install_pre_reqs(){
    # parameter
    PACKAGE=$1

    # installing package

    logger "environment_checker" "INFO" "Checking $PACKAGE installation"
    yum -y install $PACKAGE &>/dev/null
    OUTPUT_CODE=$?
    CHK_INSTALLATION=$(rpm -qa | grep $PACKAGE)

    # Checking if was installed

    if [ $OUTPUT_CODE -eq 0 ];then
        logger "environment_checker" "INFO" "$PACKAGE installation OK"
    else
        logger "environment_checker" "WARN" "$PACKAGE installation finalized with errors or warnings"
        if [ -z $CHK_INSTALLATION ];then
            logger "environment_checker" "ERROR" "$PACKAGE not installed due some error!!"
            logger "environment_checker" "INFO" "Please run \"yum -y install $PACKAGE\" and try again!!"
            exit 1
        fi
    fi
}

function environment_checker(){
    # This function is used only to check some requirements
    # of facelib
    logger "environment_checker" "INFO" "Staring environment check"

    # variables
    OS=$(head -1 /etc/os-release | awk -F '"' '{print $2}')
    OS_VERSION=$(head -2 /etc/os-release | tail -1 | awk -F '"' '{print $2}')
    OS_ID=$(head -3 /etc/os-release | tail -1 | awk -F '"' '{print $2}') # rhel or centos
    SPID_LIB_FOLDER="/var/lib/griaule/spid"
    USR_NAME=$(whoami)

    logger "environment_checker" "INFO" "OS: $OS $OS_VERSION"

    # Need run with root user

    if [[ ! $USR_NAME == "root" ]];then
        logger "environment_checker" "ERROR" "You're running script as $whoami!! Please run script as root."
        exit 1
    fi

    # Installing all pre recs with the function install_pre_reqs

    #TODO fill this list with all pre-reqs
    if [[ $OS_ID == "centos" ]];then
        PACKAGES_TO_INSTALL="wget curl OpenEXR-libs libraw1394-devel libusbx"
    elif [[ $OS_ID == "rhel" ]];then
        PACKAGES_TO_INSTALL="wget curl OpenEXR-libs libraw1394 libusbx"
    else
        logger "environment_checker" "ERROR" "your operating system has not been approved"
    fi
    for PACKAGE in $PACKAGES_TO_INSTALL;do
        install_pre_reqs $PACKAGE
    done

    # checking /var/lib/griaule/spid folder

    if [ ! -d $SPID_LIB_FOLDER ];then
        logger "environment_checker" "ERROR" "SPID not installed!!"
        exit 1
    fi

    # create griaule user

    logger "environment_checker" "INFO" "Checking griaule user"
    useradd griaule &>/dev/null
    OUTPUT_CODE=$?
    CHK_USR=$(grep griaule /etc/passwd)
    if [ $OUTPUT_CODE -eq 0 ] || [ ! -z $CHK_USR ];then
        logger "environment_checker" "INFO" "griaule user: OK"
    else
      logger "environment_checker" "ERROR" "Error creating griaule user"
      exit 1
    fi

    # Setting griaule user as owner

    logger "environment_checker" "INFO" "Changing SPID folders ownership to griaule"
    chown griaule: -R {/var/lib/griaule,/etc/griaule}

    logger "environment_checker" "INFO" "Environment check run successfully"

}

function facelib_setup(){
    logger "facelib_setup" "INFO" "Starting facelib setup"
    logger "facelib_setup" "INFO" "Downloading facelib from griaule repo"

    # download facelib from griauel repo
    if [ ! -e $PWD/bin.facelib-java.tar.gz ];then
        wget -q http://repo.griaule.com/griaule/services/spid/bin.facelib-java.tar.gz
    fi

    OUTPUT_CODE=$?
    GET_PUB_IP=$(curl ifconfig.me 2>/dev/null)

    # Verify connetion with repo
    if [ ! $OUTPUT_CODE -eq 0 ];then
        logger "facelib_setup" "ERROR" "Downloading facelib from griaule repo"
        if [ -z $GET_PUB_IP ];then
            logger "facelib_setup" "ERROR" "Maybe you're not connected with the internet. Please check your connection and try again!!"
            exit 1
        else
            logger "facelib_setup" "INFO" "Maybe you're not authorized in the Griaule repo. Please send your public ip $GET_PUB_IP to Griaule support requesting authorization"
            exit 1
        fi
    fi

    # Extracting tar.gz

    logger "facelib_setup" "INFO" "Starting facelib extraction"

    tar -xvzf bin.facelib-java.tar.gz -C /var/lib/griaule/spid/ &>/dev/null
    OUTPUT_CODE=$?
    if [ ! $OUTPUT_CODE -eq 0 ];then
  logger "facelib_setup" "ERROR" "extracting facelib to /var/lib/griaule/spid/"
        exit 1
    fi

    logger "facelib_setup" "INFO" "Facelib extracted successfully"

    # update spid script

    logger "facelib_setup" "INFO" "Update initialize spid scirpt"
    mv /var/lib/griaule/spid/scripts/initialize-spid.sh /var/lib/griaule/spid/scripts/initialize-spid.sh.bkp
    mv /var/lib/griaule/spid/bin.facelib-java/initialize-spid.sh /var/lib/griaule/spid/scripts/initialize-spid.sh
    chmod +x /var/lib/griaule/spid/scripts/initialize-spid.sh
    logger "facelib_setup" "INFO" "Initialize spid scirpt updated"

    logger "facelib_setup" "INFO" "Facelib setup run successfully"
}


# MAIN

environment_checker
facelib_setup

Upgrade

PSBio Upgrade

Version 4 Upgrade

Unpack the updated files into a temporary directory: gbs-psbio-server<version>.tar.gz

Stop the PSBio Module:

service psbio stop

Rename the previous version of the PSBio module, keeping it until the upgrade completes successfully:

mv /var/lib/griaule/psbio/gbs-psbio-server.jar /var/lib/griaule/psbio/gbs-psbio-server.jar.old

If you are migrating from a version earlier than 5.0 to version 5.0 or later, it is necessary to migrate the local database using the provided tool, the PSBio Database Migration Tool. Its execution is performed with the command below:

java -Dconfig.path={path}/pdbm.properties -jar {path}/psbio-database-migration-1.0.2-exec.jar >>{path}/log.out 2>> {path}/log.err

If necessary, consult the section Database Migration to Version 5.0 below for more details about this tool.

Install the new PSBio Module API:

tar -xf gbs-psbio-server_<version>.tar.gz
mv gbs-psbio-server.jar /var/lib/griaule/psbio

Check the file config.properties with the template sent for the last update.

Start the PSBio Module:

service psbio start

Database Migration to Version 5.0

When upgrading PSBio from a version earlier than 5.0 to version 5.0 or later it will be necessary to migrate the local database. Griaule provides a tool to perform this operation, the PSBio Database Migration Tool.

Unpack the package into a temporary directory on the server. The package has two files, the executable .jar and the configuration file pdbm.properties.

The fields of the configuration file are the following:

debug

(boolean) If enabled, logs all actions performed. Default: false.

zookeeper.path

(string) Zookeeper path.

migration.batch-size

(integer) Number of transactions fetched in each iteration. Default: 10000.

migration.transactions

(boolean) Enables migration of the psbio_transactions. Default: true.

migration.transactions.include-finalized

(boolean) Enables migration of finalized transactions. Default: false.

Configure the file pdbm.properties with the appropriate values and run the tool with the command:

java -Dconfig.path={path}/pdbm.properties -jar {path}/psbio-database-migration-1.0.2-exec.jar >>{path}/log.out 2>> {path}/log.err

PSBio Module Rollback Procedure

If the previous version still exists, /var/lib/griaule/psbio/gbs-psbio-server.jar.old:

Stop the PSBio Module:

service psbio stop

Remove the new version and rename the previous version of the PSBio module:

rm -f /var/lib/griaule/psbio/gbs-psbio-server.jar
mv /var/lib/griaule/psbio/gbs-psbio-server.jar.old /var/lib/griaule/psbio/gbs-psbio-server.jar

Check the config.properties file against the previous template.

Start the PSBio Module:

service psbio start

If the previous version is no longer available, follow the upgrade procedure using the previous version's package.

AC Module Upgrade

Version 2 Upgrade

Unpack the update contents gbs-spid-server<version>.zip into a temporary directory.

Stop the AC Module:

service spid stop
service spid-cp stop

Rename the previous version of the AC module and Control Panel, keeping them until the update completes successfully:

mv /var/lib/griaule/spid/gbs-spid-server.jar /var/lib/griaule/psbio/gbs-spid-server.jar.old
mv /var/lib/griaule/spid/gbs-spid-controlpanel.jar /var/lib/griaule/spid/gbs-spid-controlpanel.jar.old

Install the new version of the AC Module and Control Panel:

unzip gbs-spid-server_<version>.zip
mv gbs-spid-server.jar /var/lib/griaule/spid
mv gbs-spid-controlpanel.jar /var/lib/griaule/spid

Verify the files config.properties and controlpanel.properties against the templates sent for the last update.

Start the AC Module:

service spid start
service spid-cp start

AC Module Rollback Procedure

If the previous version still exists, /var/lib/griaule/psbio/gbs-spid-server.jar.old and /var/lib/griaule/spid/gbs-spid-controlpanel.jar.old:

Stop the AC Module:

service spid stop
service spid-cp stop

Remove the new version and rename the previous version of the AC module:

rm -f /var/lib/griaule/spid/gbs-spid-server.jar
rm -f /var/lib/griaule/spid/gbs-spid-controlpanel.jar
mv /var/lib/griaule/spid/gbs-spid-server.jar.old /var/lib/griaule/spid/gbs-spid-server.jar
mv /var/lib/griaule/spid/gbs-spid-controlpanel.jar.old /var/lib/griaule/spid/gbs-spid-controlpanel.jar

Check the files config.properties and controlpanel.properties against the previous template.

Start the AC Module:

service spid start
service spid-cp start

If the previous version is no longer available, follow the same upgrade procedure using the previous version's packages.

Last updated 2 months ago

Was this helpful?