GBDS Releases

GBDS 4.6.14.1535

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.13 or newer

Partial release with:

  • API

Bug Fixes

[BF-001][ALPHA-1603][API] Update on Biographic Base error treatment

When calling Biographic Base Update endpoint with POST /people/{pguid}/bio-base and Biographic Base service is not enabled, return processing error on API with proper code. Example:

{
    "errors": [
        {
            "type": "PROCESSING_ERROR",
            "code": "BIOGRAPHIC_BASE_SERVICE_ERROR",
            "message": "Biographic Base Service error: UNAVAILABLE"
        }
    ]
}

[BF-002][ALPHA-1606][API] Fixed duplicated keys on get transaction/get person

Due to errors in legacy migrations, people stored in XML within HBase transactions may contain repeated keys.

API now filters duplicate keys in both get transaction, get person and list transactions.

GBDS 4.6.13.1534

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: c404c72f4bbe41aa60030d527626fc72
GBDS redistribution MD5 Checksum: f1b0603bdd73df098cefc931adcba6eb
GBS Smart Sense Agent MD5 Checksum: af28096481686b5445de2f2b0a4f8e06
GBDS RDB Migration and Reextractor MD5 Checksum: 6b67cc709e41126134a4c5b771f456d1

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • RDB upgrade 4.6.6.1527 provided from 4.6.0.1520 on upgrade-rdb.md.
  • RDB upgrade 4.6.8.1530 provided from 4.6.6.1527 on upgrade-rdb.md.
  • Java client and API model 4.6.13 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.13.1534.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.13-exec.jar (with Facelib internal matcher upgraded).

Improvements

[IP-001][ALPHA-1599][API][GBDS] Verify post match on REFUSED transactions

When a transaction was REFUSED: on get/list transaction, only identify candidates were returned on field candidates. Verify post match results (exception candidates) were not returned.

Now, exception candidates are saved for refused transactions as well and returned on get/list transactions on an additional list (exceptions) with modifications:

  • Save refused transaction exceptions on HBase table anomalies as an exception transaction.

  • Return on get/list transaction a new list on:

    • get transaction: data->exceptions
    • list transactions: data->[]->exceptions

    With the same structure as candidates, but with only exception candidates.

[IP-002][ALPHA-1603][API] Update biographics on Biographic Base

To allow API to update biographics on Biographic Base Server, a new endpoint was added to API:

POST /people/{pguid}/bio-base

  • Request body:

    • keys: array of

      • id string
      • value string

    • biographics: array of

      • id string
      • type enum; values: TEXT, FACE
      • value string

  • Response:

    • 201: identity created on Biobase Server
    • 202: identity updated on Biobase Server
This new endpoint creates or updates biographics on the Biographic Base Server using BioBase configuration on API.
It uses any key provided to find a person/identity on server. If it does, it updates it with all biographics provided, keeping the existing ones.
If it does not find any person/identity with the keys provided, it creates a person/identity on the server with the keys and biographics provided.
It calls all servers until one responds with a 2xx http code. If none respond successfully, it returns the last server response call code.

Obs: On request body keys, API creates one more with pguid to pass to BioBase.

[IP-003][ALPHA-1604] Ginger updated to 1.8.6 and verify with ginger adjust resolution

Ginger was updated to version 1.8.6 on internal library and all ginger micro services.

New setting on gbdsapi.properties and application.conf was added (it can be placed on RDB gbds.settings as well, but only on GBDS/API boot changes will be applied):

  • gbds.search.verify.adjust-resolution default true

When this setting is turned on, verify and update on API and verify post match on GBDS will adjust resolution performing verify match, decreasing score on fingers and palms.

GBDS 4.6.12.1533

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.2 or newer

Partial release with:

  • GBDS API 4.6.12
  • GBDS Java Client 4.6.12 deployed on artifactory

Improvements

[IP-001][ALPHA-1602][API] Do not extract sequence control on trusted enroll/update

GBDS 4.6.11.1532

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: 14f128a011d37febbfd543892394d9d7
GBDS redistribution MD5 Checksum: 68ea5d2422cee7b69ffc1ab3c87ed16c
GBS Smart Sense Agent MD5 Checksum: d0618161c225d64609eefe65770c0461
GBDS RDB Migration and Reextractor MD5 Checksum: 805fc19dda9f26089645854950aeeeb2

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • RDB upgrade 4.6.6.1527 provided from 4.6.0.1520 on upgrade-rdb.md.
  • RDB upgrade 4.6.8.1530 provided from 4.6.6.1527 on upgrade-rdb.md.
  • Java client and API model 4.6.11 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.11.1532.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.11-exec.jar (with Facelib internal matcher upgraded).
  • Facelib extraction service upgrade provided directly through folder /var/lib/griaule/extractor/facelib (extractor microservice).
  • Facelib match service upgrade provided directly through folder /var/lib/griaule/extractor/facematch (matcher microservice).

Bug Fixes

[BF-001][ALPHA-1595][API] Fixed priority on trusted enroll/update

Also, delete person endpoint has now two new query parameters:

  • priority, values: all priorities, default GOD_PRIORITY

  • timeout, values:

    • -1 (sync)
    • 0 (async) (default)
    • >0 (sync with timeout)

[BF-002][ALPHA-1596][Ginger match service] Fixed palm match on ginger matcher micro service with DESERIALIZED_FULL mode

Obs: this mode (DESERIALIZED_FULL) is not meant to be used anymore. This is NOT a bug from current ginger matcher, optimized_gingermatch.

Improvements

[IP-001][ALPHA-1597][API][Facelib extraction service][Facelib match service] Face template v3 score normalization

Facelib match service, Facelib extraction service, and API internal facelib matcher upgraded, with new face template v3 match score normalized with face template v1.1 match score.

GBDS 4.6.10.1531

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.2 or newer

Partial release with:

  • GBDS Driver 4.6.10
  • GBDS Java Client 4.6.10 deployed on artifactory

Bug Fixes

[BF-001][ALPHA-1598][GBDS] Fixed palm identify preset when search is not latent and modality is PALMPRINT.

GBDS 4.6.9.1530

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: 8eb921d9ce4de532467d175bc1085d6e
GBDS redistribution MD5 Checksum: 4f678da34f1f5d8f5a3e7e2e96b5c369
GBS Smart Sense Agent MD5 Checksum: d54b3df6d9dc1dd1537d3f54e467808c
GBDS RDB Migration and Reextractor MD5 Checksum: 8789d841ed4e2a08e8c4885c0d9409d5

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • RDB upgrade 4.6.6.1527 provided from 4.6.0.1520 on upgrade-rdb.md.
  • RDB upgrade 4.6.8.1530 provided from 4.6.6.1527 on upgrade-rdb.md.
  • Java client and API model 4.6.9 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.9.1530.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.9-exec.jar.

Bug Fixes

[BF-001][ALPHA-1593][Micro services] Fixed palms too big failing on 1:N searches and enrolls in optimized matcher micro services.

Improvements

[IP-001][ALPHA-1592][GBDS] Verify post match

On enroll transactions with exceptions, GBDS can perform a verify enroll post match after the current identify enroll post match.

This is done to improve the score of exception candidates and to get more matches per candidate than just the matches coming from the identify.

Example: the identify of an enroll on a transaction brings 4 fingers, indexes 4, 5, 6, 7 with scores 42, 45, 47, 43. Verify post match upon the same transaction results on 6 fingers, indexes 3, 4, 5, 6 ,7, 8, with scores 44, 46, 49, 51, 47, 41.

The flag to turn it on is on rdb gbds.settings:

  • gbds.verifyPostMatch.enabled, type GBDS, default true

Obs: The verify post match is available on get/list exception for enrolls. Get transaction keeps returning the same identify candidate list as before.

New Features

[NF-001][ALPHA-1580] New GBDS Reextractor (replaces Batch Operations)

It uses GBDS Migration structure with extraction microservices.

It can reextract and simplify templates on HBase transactions and people tables, respectively.

It runs on a single node with a TGUID range on HBase transactions table according to settings.

Scripts and Logs

  • Starts with /var/lib/griaule/gbds-migration/scripts/start-reextractor.sh (starts GBDS Migration with --mode reextract).
  • Kills with /var/lib/griaule/gbds-migration/scripts/kill-migration.sh (kills any GBDS Migration running).
  • Generates log on /var/log/griaule/gbds-migration/gbds-reextract.log and error on /var/log/griaule/gbds-migration/reextract_console.out.

Settings

  • Uses /etc/griaule/conf/gbds-migration/gbds-migration.properties as configuration:

    • General configuration

      • gbds.reextract.nodeNumber: node number running Reextractor. It determines scan range on HBase based on total nodes.
      • gbds.reextract.totalNodes: total nodes running Reextractor.
      • gbds.reextract.totalScanRegions: Total number of regions to break scans; default is 256 (00-FF regions).
      • gbds.reextract.scanners.number: number of scanners (a scanner scans from HBase a range based on node number, total nodes and total scan regions).
      • gbds.reextract.workers.number: number of workers (a worker holds a transaction template extraction).
      • gbds.reextract.writers.number: number of writers (a writer gets the extraction result and writes back into transaction and people, if necessary).

    • Pipeline queue configuration

      • gbds.reextract.workers.inqueueMaxSize: queue size from scanner to worker, default 1000 (the bigger this size, more scans are made and held on workers, but more memory is allocated).
      • gbds.reextract.writers.inqueueMaxSize: queue size from worker to writer, default 1000 (the bigger this size, more scans are made and held on writers, but more memory is allocated).

    • Extraction microservices configuration

      • gbds.reextract.msextraction.ginger.number: ginger extraction microservices count (multiples of 10 recommended).
      • gbds.reextract.msextraction.face.number: face extraction microservices count (10 times less than ginger recommended).
      • gbds.reextract.msextraction.initialPort: local initial port. Do not conflict with API template extraction microservices ports (30000+), quality extraction microservices ports (31000) and GBDS matcher microservices ports (32000).
      • gbds.reextract.msextraction.maxTries: max tries on a single biometry.
      • gbds.reextract.msextraction.linkLibSegfault: flag to turn on seg fault lib debugger.
      • gbds.reextract.msextraction.checkTimeoutSecs: timeout to check if a microservice is up.
      • gbds.reextract.msextraction.logLevel: log level on GBDS Reextractor log for microservices: INFO, TIME or DEBUG.
      • gbds.reextract.msextraction.timeout: timeout of a single microservice call.

    • Modalities to reextract flags

      • gbds.reextract.modality.finger: true/false.
      • gbds.reextract.modality.face: true/false.
      • gbds.reextract.modality.palm: true/false.
      • gbds.reextract.modality.iris: true/false.
      • gbds.reextract.modality.newborn-palm: true/false.

    • Reextraction HBase Column Family configuration

      • gbds.reextract.cf.finger: finger cf name to receive extracted template on transactions and people.
      • gbds.reextract.cf.palm: palm cf name to receive extracted template on transactions and people.
      • gbds.reextract.cf.face: face cf name to receive extracted template on transactions and people.
      • gbds.reextract.cf.iris: iris cf name to receive extracted template on transactions and people.
      • gbds.reextract.cf.newborn-palm: newborn palm cf name to receive extracted template on transactions and people.

    • SQLite configuration

      • gbds.reextract.sqlite.filePath: file path for SQLite local database.
      • gbds.reextract.sqlite.resetOnStart: flag to reset SQLite on start true of false.

Obs: SQLite database local file is used to jump regions according to total scan regions configuration, along with each transaction modality cf flag saved on HBase.

  • It holds a table informing Reextractor which regions is being scanned and if a region is finished.

  • When Reextractor tries a new region, if this regions is already scanned and processed according to SQLite, this regions is skipped, speeding up the process.

  • If SQLite local database file is removed or reset flag is true when Reextractor started, it will perform a full scan on entire transactions table, looking only for modality cf flag on each row.

  • It can be monitored using the bash command:

    • sudo sqlite3 -header <file path to sqlite local database> "select * from range_process where total_scanned > 0"

GBDS 4.6.8.1529

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.2 or newer

Partial release with:

  • GBDS API 4.6.8
  • GBDS Java Client 4.6.8 deployed on artifactory

Bug Fixes

[BF-001][ALPHA-1591][API] Do not fail a transaction if any extraction issues occurs, allowing identify search 1:N with extraction issues, unless template extraction results in a no searchable template.

GBDS 4.6.7.1528

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: 9c5dcb37a7ed32c12fd8fb4ed068a995
GBDS redistribution MD5 Checksum: c91ba929e6ccf8de6037659d87d6862f
GBS Smart Sense Agent MD5 Checksum: 43d894e407ca95d08c52386fda1cef2e
GBDS RDB Migration MD5 Checksum: 222cf369cdf6c7727c8e4445b7924a04

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • RDB upgrade 4.6.6.1527 provided from 4.6.0.1520 on upgrade-rdb.md.
  • Java client and API model 4.6.7 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.7.1528.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.7-exec.jar.

Bug Fixes

[BF-001][ALPHA-1587][GBDS] Fixed processing any task above default priority before retain tasks (boot tasks).

GBDS boot could start processing kafka tasks before all templates are saved on memory if any tasks at the beginning of processing has priority above default.

Fixed to wait all boot tasks before processing any kafka task.

Obs: Log will show now how many people are registered on the node on every 10000 people (so boot log does not looks frozen).

[BF-002][ALPHA-1589][GBDS] Fixed updates treated on GBDS side, not API.

API can erroneously send to GBDS an enroll that is actually an update with keys. This happens when the original enroll for this update was not processed by GBDS when the update arrives on API. In this scenario where GBDS process the update, GBDS was generating exceptions with no matches. Or could even result in a FAILED transaction.

Fixed to change the enroll into an update and check exception correctly with previous enroll transaction.

To fix this issue, some changes were made on GBDS:

  • On enroll post match, inactive people are included on post match.
  • GBDS setting gbds.biometric.remove-inactive-people-from-enroll-result works now only on searches, update UL candidates and enrolls, not on updates treated on GBDS.
  • GBDS setting gbds.biometric.remove-inactive-people-from-enroll-result is now a hot setting (can be changed with GBDS on).
  • If the update on GBDS does not generates an exception, send back to all matchers an “update person biometrics” task to update correct PGUID on memory, leaving an inactive PGUID behind until GBDS reboots.
  • Fixed RDB exception creation on GBDS.
  • Even with GBDS setting gbds.biometric.remove-inactive-people-from-enroll-result as false, always remove inactive people from exception candidates on enroll, getting the candidates list correctly.
  • Get person status on HBase optimized: this is used to verify inactive people on GBDS setting gbds.biometric.remove-inactive-people-from-enroll-result or on remove inactive people from exception candidates on enroll.

Improvements

[IP-001][ALPHA-1588][Matcher microservices] Cleaning logs on matcher microservices.

  • Matcher microservice log on boot logs only at every 5000 people.
  • Matcher microservice log match with two lines only: one with parameters and one with pre drop and time.
  • The matcher microservice log files can now be truncated directly on system.
  • Some debug logs can be activated on matcher microservice log using parameter --debug-log-level true on script /var/lib/griaule/extractor/scripts/run-matcher.sh (default is false on this script).

GBDS 4.6.6.1527

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: fdd278c905a353cf7f3c24ce0bbeddf8
GBDS redistribution MD5 Checksum: d405662727cd33ea66424138f15fd1b7
GBS Smart Sense Agent MD5 Checksum: 7259e2ca3e84547fc0e67db089f17c99
GBDS RDB Migration MD5 Checksum: 5358589e57189816715795e35f6fba77

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • RDB upgrade 4.6.6.1527 provided from 4.6.0.1520 on upgrade-rdb.md.
  • Java client and API model 4.6.6 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.6.1527.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.6-exec.jar.

Bug Fixes

[BF-001][ALPHA-1579][API][GBDS] Fix update person check.

On update, check if the person exists, is active and not deleted.

On finishing enrolls with rdb error while saving exceptions, fixed transaction marked as failed and not active (do exclude from next boot and to be removed from enroll/search 1:N candidates).

[BF-002][ALPHA-1584][GBDS] Fixed reverse latent match using optimized ginger microservice.

[BF-003][ALPHA-1585][API] Fixed refused transactions on update with inconsistent keys exceptions.

[BF-004][ALPHA-1586][API] Fixed UL candidate list removing inactive candidates. All UL candidates are returned now.

Improvements

[IP-001][ALPHA-1582][GBDS] Better match microservice fail treatment.

When a match microservice has a connection error, try 3 times before failing its transaction, waiting 2s between tries. Configuration:

  • gbds.match.service.maxTries, type GBDS, default 3.

When a match microservice has 5 failed transactions by connection error in a row, stop GBDS to prevent failing every transaction from that point. Configuration:

  • gbds.match.service.maxConnectionErrors, type GBDS, default 5.

This last setting was provided with upgrade-rdb.md on release, in the 4.6.6.1527 section.

GBDS 4.6.5.1526

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.2 or newer

Partial release with:

  • API
  • GBDS Java Client 4.6.5 deployed on artifactory

Bug Fixes

[BF-001][ALPHA-1569][API] Fixed add key validation: allow key of other person if this person is deleted.

[BF-002][ALPHA-1573][API] Fixed enroll/update/trusted/add keys when a person has a biographic without value stored as key| or just key.

[BF-003][ALPHA-1574][API] Fixed update on exception that was updating HBase people table with exception templates before exception treatment.

[BF-004][ALPHA-1581][API] Fixed list transaction with labels filter where labels contains :. Now, only key and biographic filters use : (or |) as key/value separators.

GBDS 4.6.4.1525

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: e9ece8991e41aa1fa9677e79bca34024
GBDS redistribution MD5 Checksum: f4b85a396447d7d11798ce0ee74e9fe5
GBS Smart Sense Agent MD5 Checksum: 1d0b976c4688748a66bce7fa46c7217d
GBDS RDB Migration MD5 Checksum: 702324c0f335ded3fdaa1d0c284d4861
Batch Operations MD5 Checksum: ecc5ddb05c51ea810410d3872dfcb00b

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • Java client and API model 4.6.4 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.4.1525.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.4-exec.jar.

Bug Fixes

[BF-001][ALPHA-1577][GBDS] Fixed remove task from backpressure queue on corner situations, such as:

  • Already processed tasks twice
  • Failed enroll/search/trusted enroll twice
  • Stopped matcher supervisor
  • On error when loading templates from HBase

Log task elapsed time when removing from backpressure queue.

Improvements

[IP-001][ALPHA-1572][Batch Operations] Added two features:

  • New configuration: gbds.operations.extraction.retries. Defines the number of retries for worker extraction operation.

    • Default: 2
    • Min: 0
    • Max: 10

  • Added sleep when creating workers.

[IP-002][ALPHA-1575][GBDS] Ginger updated to 1.8.5.

Cab match optimized up to 3x.

Obs: Match microservices (optimized ginger and face) must be upgraded for this feature to work.

GBDS 4.6.3.1524

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.2 or newer

Partial release with:

  • API
  • GBDS Java Client 4.6.3 deployed on artifactory

Bug Fixes

[BF-001][ALPHA-1570][API] Fixed queries performance for list transaction and list people.

  • On list transactions, removed subselects with exists for key/biographic/label filter and use inner join for these filters.
  • On list people, removed distinct when select has no key/biographic/label filter.

GBDS 4.6.2.1523

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.2 or newer

Partial release with:

  • API
  • GBDS Java Client 4.6.2 deployed on artifactory

Improvements

[IP-001][ALPHA-1571][API] Biographic Base

API now sends TGUID and external IDs on every get/list transaction/person to Biobase server.

Fixed GBDS Java Client.

  • Added biographicBase query parameter on:

    • Get transaction
    • List transaction
    • Get person

GBDS 4.6.1.1522

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: 031cfb499e89f37bf701162dfd5d9022
GBDS redistribution MD5 Checksum: 6318e91a0535524f88e1f2842acf1fa1
GBS Smart Sense Agent MD5 Checksum: dd70f7680200ed704e99f315642efdcf
GBDS RDB Migration MD5 Checksum: 8fec4fd7417f431afa850aaa2f962aa9

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • Java client and API model 4.6.1 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.1.1522.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.

Bug Fixes

[BF-001][ALPHA-1561][Smartsense Agent] Fixed Smartsense Agent swagger and GBS Services endpoint to verify rest provided service among existing ones.

[BF-002][ALPHA-1562][Batch Operations] Fixed batch operations start script: added OpenVINO path on linux library path.

Improvements

[IP-001][ALPHA-1563][Batch Operations] Added configuration for batch operations.

  • gbds.operations.simplify.activate-person, default true:

    • If true activate person on simplify.
    • If false, leave it as found.

GBDS 4.6.0.1520

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.6.0 or newer
GBDS release MD5 Checksum: c608e5290efdf0b48dfe303e21d66877
GBDS redistribution MD5 Checksum: abf258065771d52d6fab73f3e8d00cd4
GBS Smart Sense Agent MD5 Checksum: 33f4a1c9a79f4d02786175b040ad8fe1
GBDS RDB Migration MD5 Checksum: 8af6f7e313eed39afc084249cf741cb3

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.6.0.1520 provided from 4.5.17.1516 on upgrade-rdb.md.
  • Java client and API model 4.6.0 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.6.0.1520.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds.
  • Fast API upgrade with gbscluster-api-4.6.0-exec.jar.

Bug Fixes

[BF-001][ALPHA-1558][Client] Fixed external ID objects on create people trusted for GBDS Client.

New Features

[NF-001][ALPHA-1551][API] External Biographic Base

GBDS API can now look up for biographics on an external server called Biographic Base.

When configured and running, access to this external biographic base server (let’s call it Biobase Server) allows API to retrieve biographics (and possibly a face image) to show on get transaction, get person, list transaction and list people calls.

Internally, the API will look up on the Biobase Server for any person that is returned on any response using all the keys of the person, expecting biographics and possible an external face from the Biobase Server.

Any API response with a person in it (like get transaction) will have its biographics from the GBDS database unified with the biographics retrieved from Biobase Server using its keys.

When the Biobase Server returns a face for a person, the API will try to interpret the face as an image and add it to the biometrics, without a template, just the image (ORIGINAL source, FACE type, format according to image).

The API can look on several Biobase Servers, using round robin balance to call them.

API calls to a Biobase Server are authenticated using client ID and secret. If the Biobase Server does not implement authentication, the API will assume that the authentication is ok.

Important: Access to the Biobase Server DOES NOT impact any get/list transaction/person. All these calls will always return, as they used to do, regardless of the Biobase Server.

The API Biobase Server configuration includes timeouts for gets and for lists to ensure that biographic retrieval will not take too long.

[CF-001] Configuration

To configure the Biobase Server access, all configurations are made on the RDB gbds.settings table and replicated onto the gbdsapi.properties file of the API. The behavior is determined by:

  • gbds.biographicBase.enabled: Type API. Default false: Flag to turn Biobase Server access on/off. When it is off, biographicBaseStatus is not returned on get/list transaction/person calls.

  • gbds.biographicBase.endpoints: Type API. Default empty: Comma separated list with all Biobase Server URLs, with host and port. Example: http://biobase-server-ip:8130,http://biobase-server-ip:8131

  • gbds.biographicBase.get.timeout.ms: Type API. Default 500: Biobase Server get request timeout. It is used on API get transaction or get person calls.

  • gbds.biographicBase.list.timeout.ms: Type API. Default 500: Biobase Server list request timeout. It is used on API list transaction or list people calls. On list calls, Biobase Server is called only once for all transactions/people that will be returned.

  • gbds.biographicBase.logLevel: Type API. Default INFO: Log level on Biobase Server calls. Values are:

    • INFO: logs server request URL.
    • NONE: no server request logs.
    • TIME: logs server request URL and elapsed time.
    • DEBUG: logs server request URL, elapsed time and request/response bodies.

  • gbds.biographicBase.clientID: Type API. Default empty: Authentication client ID.

  • gbds.biographicBase.clientSecret: Type API. Default empty: Authentication client Secret.

  • gbds.biographicBase.lookAllServers: Type API. Default false: Flag to look on all server in cases of unauthorized or not found people.

    • For authorization:

      • When it is ON and a Biobase Server returns unauthorized, the API will search all others servers trying to authenticate.
      • When it is OFF, the API will look only at the current Biobase Server for authentication, returning unauthorized or authorized.

    • For not found people:

      • When it is ON and a Biobase Server call does not return all people, the API will try all others servers until all people are found. If all servers’ consolidated people list still misses any person even after going through all servers, it leaves it as it is.
      • When it is OFF, the API will look only at the current Biobase Server for all people in the request. If a person on the people list is still missing, it leaves it as it is.

All configurations can be hot changed using API configuration endpoints. Biobase Server endpoint settings too, if Biobase Server is disabled and then enabled again.

Get or list transaction/people

On these endpoints:

  • Get transaction: GET /people/transactions/{tguid}
  • List transactions: GET /people/transactions
  • Get person: GET /people/{pguid}

A new query parameters was added:

  • biographicBase, boolean, not required. Behavior:

    • If absent: API will perform as configured.
    • If present, true: API will perform as configured.
    • If present, false: API will NOT try to get biographics from Biobase Server, even if configured to do so.

On the list people endpoint, the same parameter was added, but on the payload.

  • List people: POST /people/list

Example:

{
    "data" : {
        "operator" : "OR",
        "pageIndex" : 0,
        "pageSize" : 10,
        "includeAnomalies" : true,
        "paginationCount" : true,
        "personFields" : ["ALL_FIELDS"],
        "restrictions" : [
            {
                "type" : "KEY",
                "id" : "cpf",
                "value" : "001",
                "matchMode" : "EXACT"
            }
        ],
        "biographicBase" : false
    }
}

Person object

The API Person object has a new attribute to show the Biobase Server status when it is on:

  • biographicBaseStatus, type string, with values:

    • UNAVAILABLE: When all Biobase Servers are off.
    • TIMEOUT: When at least one Biobase Server is ON but the call timed out.
    • UNAUTHORIZED: If the lookAllServers conf is ON, it indicates that all running servers returned that the API authentication is unauthorized. If the lookAllServers conf is OFF, it indicates that the server that received the biographic request returned that the API authentication is unauthorized.
    • INVALID_DATA: The Biobase Server returned that the data request from the API is invalid. It is not supposed to happen if Biobase Server is implemented according to the Biographic Base API.
    • NOT_FOUND: If the lookAllServers conf is ON, it indicates that the person keys were not found on all Biobase Servers configured. If the lookAllServers conf is OFF, it indicates that person keys were not found on the server that received the biographic request.
    • OK: The Biobase Server returned biographics/face for the given person keys.

Example:

  • Biographics returned from the Biobase Server: OK (only biographics). Biobase Server biographics have their IDs prepended with bs-. For example, bs-name and bs-surname, as shown below:
{
    "data": [
        {
            "tguid": "878001FB-3A01-47EB-900C-DC5E153E922F",
            "pguid": "8E3560AF-37FD-44B4-BDA8-9C5950A65237",
            "status": "ENROLLED",
            "timestamp": 1697640635319,
            "progress": 100.0,
            "person": {
                "pguid": "8E3560AF-37FD-44B4-BDA8-9C5950A65237",
                "lastEnrollTguid": "878001FB-3A01-47EB-900C-DC5E153E922F",
                "timestamp": 1697640635319,
                "keys": [
                    {
                        "id": "cpf",
                        "value": "002"
                    }
                ],
                "biographics": [
                    {
                        "id": "name",
                        "value": "xxx"
                    },
                    {
                        "id": "surname",
                        "value": "yyy"
                    },
                    {
                        "id": "bs-name", <-- Biobase Server biographic
                        "value": "Jane"
                    },
                    {
                        "id": "bs-surname", <-- Biobase Server biographic
                        "value": "Doe"
                    }
                ],
                "biometric": [
                    ...
                ],
                "labels": [
                    "xz"
                ],
                "biographicBaseStatus": "OK" <-- Biobase Server status
            },
...
  • Biographics and face returned from the Biobase Server: OK. Biobase Server biographics have their IDs prepended with bs-. For example, bs-name and bs-surname, as shown below. There’s no special attribute on added face.
{
    "data": [
        {
            "tguid": "7F1BCCE8-980E-4220-86C3-E2A16D40E6BF",
            "pguid": "50DD9759-2C5B-47B8-BE0E-68F3E423AF24",
            "status": "ENROLLED",
            "timestamp": 1698671850280,
            "progress": 100.0,
            "candidates": [
                ...
            ],
            "person": {
                "pguid": "50DD9759-2C5B-47B8-BE0E-68F3E423AF24",
                "lastEnrollTguid": "7F1BCCE8-980E-4220-86C3-E2A16D40E6BF",
                "timestamp": 1698671850280,
                "keys": [
                    {
                        "id": "cpf",
                        "value": "001"
                    }
                ],
                "biographics": [
                    {
                        "id": "name",
                        "value": "xxx"
                    },
                    {
                        "id": "surname",
                        "value": "yyy"
                    },
                    {
                        "id": "bs-name", <-- Biobase Server biographic
                        "value": "John"
                    },
                    {
                        "id": "bs-surname", <-- Biobase Server biographic
                        "value": "Doe"
                    }
                ],
                "biometric": [
                    ... // other biometric images
                    {
                        "source": "ORIGINAL", <-- this is GBDS face image
                        "type": "FACE",
                        "format": "JPEG",
                        "properties": {
                            "width": 640,
                            "height": 480,
                            "resolution": 300,
                            "ratio": 0.0,
                            "matcherId": 0,
                            "extractorId": 0
                        },
                        "index": 10,
                        "content": "..."
                    },
                    ... // other biometric templates
                    {
                        "source": "TEMPLATE", <-- this is GBDS face template
                        "type": "FACE",
                        "format": "UNKNOWN",
                        "index": 10,
                        "content": "...",
                        "quality": 0
                    },
                    {
                        "source": "ORIGINAL", <-- this is Biobase Server face image, without template
                        "type": "FACE",
                        "format": "JPEG",
                        "properties": {
                            "width": 640, <-- width and height are computed by API with face image buffer from Biobase Server
                            "height": 480,
                            "resolution": 0,
                            "ratio": 0.0,
                            "matcherId": 0,
                            "extractorId": 0
                        },
                        "index": 10,
                        "content": "..."
                    }
                ],
                "labels": [
                    "xz"
                ],
                "biographicBaseStatus": "OK"
            },
...
  • Biographics not found on the Biobase Server:
{
    "data": [
        {
            "tguid": "CB9DDDE2-8390-4BF1-B5BA-1D30000354BB",
            "pguid": "BAAF26CB-1AAA-4ABC-8084-F05EDF9BE1B7",
            "status": "ENROLLED",
            "timestamp": 1698254428790,
            "progress": 100.0,
            "candidates": [
                ...
            ],
            "person": {
                "pguid": "BAAF26CB-1AAA-4ABC-8084-F05EDF9BE1B7",
                "lastEnrollTguid": "CB9DDDE2-8390-4BF1-B5BA-1D30000354BB",
                "timestamp": 1698254428790,
                "keys": [
                    {
                        "id": "cpf",
                        "value": "SC-03"
                    },
                    {
                        "id": "cpf-2",
                        "value": "SC-003"
                    },
                    {
                        "id": "rg",
                        "value": "rg-001"
                    },
                    {
                        "id": "titulo",
                        "value": "titulo-001"
                    },
                    {
                        "id": "titulo-2",
                        "value": "titulo-002"
                    }
                ],
                "biographics": [
                    {
                        "id": "other",
                        "value": "Joan"
                    }
                ],
                "biometric": [
                    ...
                ],
                "labels": [
                    "sc"
                ],
                "biographicBaseStatus": "NOT_FOUND"
            },
...
  • Biobase Server timeout:
{
    "data": [
        {
            "tguid": "CB9DDDE2-8390-4BF1-B5BA-1D30000354BB",
            "pguid": "BAAF26CB-1AAA-4ABC-8084-F05EDF9BE1B7",
            "status": "ENROLLED",
            "timestamp": 1698254428790,
            "progress": 100.0,
            "candidates": [
                ...
            ],
            "person": {
                "pguid": "BAAF26CB-1AAA-4ABC-8084-F05EDF9BE1B7",
                "lastEnrollTguid": "CB9DDDE2-8390-4BF1-B5BA-1D30000354BB",
                "timestamp": 1698254428790,
                "keys": [
                    {
                        "id": "cpf",
                        "value": "SC-03"
                    },
                    {
                        "id": "cpf-2",
                        "value": "SC-003"
                    },
                    {
                        "id": "rg",
                        "value": "rg-001"
                    },
                    {
                        "id": "titulo",
                        "value": "titulo-001"
                    },
                    {
                        "id": "titulo-2",
                        "value": "titulo-002"
                    }
                ],
                "biographics": [
                    {
                        "id": "other",
                        "value": "Joan"
                    }
                ],
                "biometric": [
                    ...
                ],
                "labels": [
                    "sc"
                ],
                "biographicBaseStatus": "TIMEOUT"
            },
...
  • Biobase server down or unreachable:
{
    "data": [
        {
            "tguid": "CB9DDDE2-8390-4BF1-B5BA-1D30000354BB",
            "pguid": "BAAF26CB-1AAA-4ABC-8084-F05EDF9BE1B7",
            "status": "ENROLLED",
            "timestamp": 1698254428790,
            "progress": 100.0,
            "candidates": [
                ...
            ],
            "person": {
                "pguid": "BAAF26CB-1AAA-4ABC-8084-F05EDF9BE1B7",
                "lastEnrollTguid": "CB9DDDE2-8390-4BF1-B5BA-1D30000354BB",
                "timestamp": 1698254428790,
                "keys": [
                    {
                        "id": "cpf",
                        "value": "SC-03"
                    },
                    {
                        "id": "cpf-2",
                        "value": "SC-003"
                    },
                    {
                        "id": "rg",
                        "value": "rg-001"
                    },
                    {
                        "id": "titulo",
                        "value": "titulo-001"
                    },
                    {
                        "id": "titulo-2",
                        "value": "titulo-002"
                    }
                ],
                "biographics": [
                    {
                        "id": "other",
                        "value": "Joan"
                    }
                ],
                "biometric": [
                    ...
                ],
                "labels": [
                    "sc"
                ],
                "biographicBaseStatus": "UNAVAILABLE"
            },
...

[NF-002][ALPHA-1557][API] Add keys to person/last transaction

A new endpoint has been added to the API to allow new keys to be added to a person.

[EP-001] POST /people/{pguid}/trusted/keys

  • Path parameters:

    • pguid: person GUID to have keys added.

  • Request body:

    • data: list of keys to be added, each key is a pair:

      • id: new key id.
      • value: new key value.

Example:

{
    "data" : [
        {
            "id" : "titulo-003", "value" : "titulo-003"
        }
    ]
}

Validations:

  • This features is available only when BoB is off.
  • List of keys must be provided and not empty.
  • pguid must be a valid person GUID from GBDS database.
  • Person that will have keys added must not have any key id already.
  • Any key id/value provided must not be found on any other person on GBDS database.

When person has provided keys added, this operations are made:

  • On HBase people table, row keys are added with keys id:value, pointing to person GUID and current person transaction.
  • On HBase transactions table, column search-keys:orgKeys is updated
  • On HBase transactions table, person XML is updated
  • On RDB, a new person version is created on gbds.people_version, gbds.fields (with new added keys) and gbds.biometric tables
  • On RDB table gbds.transaction_fields, the new keys are inserted.

Response:

  • HTTP status 400 on any validation error:

    Example:

    {
    "errors": [
        {
            "type": "VALIDATION_ERROR",
            "code": "INVALID_JSON_ATTRIBUTE_VALUE",
            "message": "Key 'titulo-003' already exists on person with pguid A65CC74A-D716-46BE-B538-D0BC82322147"
        }
    ]
}

  • HTTP status 200 with:

    • pguid: person changed
    • lastEnrollTguid: current person transaction changed
    • keys: list of all keys on person after keys added.

    Example:

    {
    "data": {
        "pguid": "A65CC74A-D716-46BE-B538-D0BC82322147",
        "lastEnrollTguid": "7337600A-3B61-47FE-9712-EDF9A52F418B",
        "keys": [
            {
                "id": "cpf",
                "value": "SC-05"
            },
            {
                "id": "titulo",
                "value": "titulo-002"
            },
            {
                "id": "titulo-002",
                "value": "titulo-002"
            },
            {
                "id": "titulo-003",
                "value": "titulo-003"
            }
        ]
    }
}

GBDS 4.5.20.1519

Partial release with API only.

Bug Fixes

[BF-001][ALPHA-1559][API] Fixed external ID name and key being swapped on trusted enroll/update endpoint.

GBDS 4.5.19.1518

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 3e63d352acc0601b2901722863cbcafa
GBDS redistribution MD5 Checksum: e020cbd8f5cd7d006aede0d15f759d4d
GBS Smart Sense Agent MD5 Checksum: eba0a4a51853e635f9cac63a0eacfe48
GBDS RDB Migration MD5 Checksum: bb72f753d969a71198802071d6294c76

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md
  • RDB upgrade 4.5.13.1512 provided from 4.5.8.1504 on upgrade-rdb.md
  • RDB upgrade 4.5.17.1516 provided from 4.5.13.1512 on upgrade-rdb.md
  • Java client and API model 4.5.19 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.19.1518.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds
  • Fast API upgrade with gbscluster-api-4.5.19-exec.jar

Bug Fixes

[BF-001][ALPHA-1555][API] Fixed verify on people without biometrics in the last transaction (BoB off).

Obs: That happens when a person’s last transaction is a no biometric trusted update with BoB on.

Improvements

[IP-001][ALPHA-1554][API] Verify performance optimized.

  • Monitoring:

    • Better verify instrumentation on log, with register and load times (less overhead time);
    • Better extraction metrics on log;
    • Better debug logs;

  • Performance:

    • Do not load person XML for verification;
    • Do not update transaction on HBase during verification, only before return result;
    • Fix not save on RDB identify search transactions;

GBDS 4.5.18.1517

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 8aa88dd09e451eda23113378c0d73846
GBDS redistribution MD5 Checksum: 96a1b51129c50075006bddd888346133
GBS Smart Sense Agent MD5 Checksum: bd1c8a59d0015f46865d068d4d627dfe
GBDS RDB Migration MD5 Checksum: fc30eca7c6a4a654a1e3727c91539811
Batch operations MD5 Checksum: fd7e36d9fede143fbc9bee85a6d9dcc4

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md
  • RDB upgrade 4.5.13.1512 provided from 4.5.8.1504 on upgrade-rdb.md
  • RDB upgrade 4.5.17.1516 provided from 4.5.13.1512 on upgrade-rdb.md
  • Java client and API model 4.5.18 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.18.1517.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds
  • Fast API upgrade with gbscluster-api-4.5.18-exec.jar

Bug Fixes

[BF-001][ALPHA-1552][API] Fixed quality control with biometric edition.

On quality analysis, rejected transaction has now its date updated.

[BF-002][ALPHA-1553][Batch Operations] Fixed batch operations for GBDS versions 4.1 and above. Startup was missing RDB transaction quality classes.

GBDS 4.5.17.1516

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 76285c83fc5f4ca6d668d13ea7c4c2b6
GBDS redistribution MD5 Checksum: a6683a8fb1df34cd40500653c6642088
GBS Smart Sense Agent MD5 Checksum: 67924abf7941860438511ec076ae2a4f
GBDS RDB Migration MD5 Checksum: 2f1a92dca85d3f2f6da44ab7efa802c5

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md
  • RDB upgrade 4.5.13.1512 provided from 4.5.8.1504 on upgrade-rdb.md
  • RDB upgrade 4.5.17.1516 provided from 4.5.13.1512 on upgrade-rdb.md
  • Java client and API model 4.5.17 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.17.1516.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1546][GBDS] Fixed update settings load/reload on GBDS to deal with updates that came from MIR.

These updates are treated on GBDS, not API, so the following settings had their type changed to BOTH on RDB table gbds.settings:

  • gbscluster.update.consider.fingerprints
  • gbscluster.update.consider.faces
  • gbscluster.update.faces.verify.matchthreshold
  • gbscluster.update.minimum.fingers

[BF-002][ALPHA-1547][API] Fixed API allowing null labels directly on HTTP or using client.

Any null label on label list is now denied on request.

On GBDS client class PersonBuilder, the labels method withLabels(null) now leaves label set with null.

[BF-003][ALPHA-1549][API] Fixed get transaction/person with no biometrics.

Fixed get transaction with no biometrics, regardless BoB on or off and get person that last transaction has no biometrics when BoB is off.

When performing trusted update with no biometrics (only available with BoB on), API will save previous transaction with biometrics on this update transaction.

Obs: Previous transaction can be current transaction on trusted update or any previous one (in case of multiple trusted updates with no biometrics).

So any get transaction on transactions with no biometrics or get person that last transaction has no biometrics (again, with BoB off) will use this biometric-tguid saved on transaction to retrieve biometrics.

[BF-004][ALPHA-1550][API] Fixed get exception with TGUID/PGUID when PGUID is not the first exception for TGUID.

GBDS 4.5.16.1515

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 5316d3427ba08f9260f0c8c7e08c192f
GBDS redistribution MD5 Checksum: 02a7b742d632cd42fc7b6a64ca779f22
GBS Smart Sense Agent MD5 Checksum: 8b43c670f7fd437d4d2ed28dd38cfd90
GBDS RDB Migration MD5 Checksum: 365a7fa1c37ddf253b371ebb17184fb1

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md
  • Java client and API model 4.5.16 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.16.1515.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1543][API] Fixed double verification on update using POST /v2/people with some key on a person with keys

Obs: Update using PUT on /v2/people/{pguid} with or without keys on person or on request is already ok

[BF-002][ALPHA-1544][API][GBDS] Fixed ginger 1.8.3 serialize/deserialize

Ginger upgraded to 1.8.4 on libgbds and all ginger micro services

[BF-003][ALPHA-1545][API] Fixed get transaction when HBase table people does not have column family auxiliary

GBDS 4.5.15.1514

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: fb100602c93a36af9471f906c1525a85
GBDS redistribution MD5 Checksum: 31e5fdf4a858f22f26ba6d4d6f5f6e3a
GBS Smart Sense Agent MD5 Checksum: 9c3f9a88f84268131d94888a7b235731
GBDS RDB Migration MD5 Checksum: f5f174482c4c5543260fa816a87a9e6d

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md
  • Java client and API model 4.5.15 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.15.1514.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1536][GBDS] Fixed bad ULs on boot

GBDS boot now skips bad ULs (corrupted ULs with templates that fails on deserialization)

API still fails on create/update UL with corrupted templates

Obs: ginger was updated to version 1.8.3 on libgbds.so and all micro services

[BF-002][ALPHA-1541][GBDS] Fixed UL candidates add/update with micro matchers

Improvements

[IP-001][ALPHA-1532][API] Get/list transaction new query parameter

On get/list transaction a new query parameters was added:

  • biometricFields=INDEX|ALL_FIELDS

To return all biometric fields or only their indexes

On get/list transaction and get person, performance was optimized with less HBase access

GBDS 4.5.14.1513

Partial release with quality micro extractors only

Finger Quality Service MD5 Checksum: 0ced73b47932d035544d8dfdd606fa85
Face Quality Service MD5 Checksum: fc13bba4347b35f233585439ccce1363

  • Java client and API model 4.5.14 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.14.1513.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Improvements

[IP-001][ALPHA-1526][API] Quality extractors no longer validate request with API to boost quality extraction.

GBDS 4.5.13.1512

Partial release with API only.

Improvements

[IP-001][ALPHA-1525][API] Improvements on Quality extraction on background

  • Added RDB settings:

    • gbds.extraction.quality.service.rows-on-select, type API, default 20
    • gbds.extraction.quality.service.submitted-queue-factor, type API, default 3

  • Quality extraction queue kept full.

Instead of submitting a transaction block for extraction (using setting gbds.extraction.quality.service.rows-on-select of size 20 by default) to thread pool (pool size is the maximum number of quality extractions active, by modality) and waiting for its ending before submitting another block, now background extraction processor keeps the processing queue size at an average of 3 times (the setting gbds.extraction.quality.service.submitted-queue-factor) the thread pool size.

It keeps the background quality extraction processing full and doesn’t use a lot of memory by submitting all pending transactions at once.

GBDS 4.5.12.1511

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 90e1a690104e64e62336457ff4a8c16c
GBDS redistribution MD5 Checksum: 1e58b819cf3ff83c14bcfa3baafd6617
GBS Smart Sense Agent MD5 Checksum: b6ac07b7f07ebdacef771d64ebb9bd1c
GBDS RDB Migration MD5 Checksum: 8547a85a4c11b0174476703e8373517f

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united in one file: upgrade-rdb.md, each upgrade for a version has its own section in this file.
  • RDB upgrade to 4.5.0.1484 (from 4.4.2) provided on upgrade-rdb.md.
  • RDB upgrade to 4.5.0.1488 (from 4.5.0.1484) provided on upgrade-rdb.md.
  • RDB upgrade to 4.5.0.1491 (from 4.5.0.1488) provided on upgrade-rdb.md.
  • RDB upgrade to 4.5.8.1504 (from 4.5.0.1491) provided on upgrade-rdb.md.
  • Java client and API model 4.5.12 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.12.1511.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1521][API] Fixed get person after trusted update with no biometrics.

Use last transaction always as person keys/biographics/biometrics/auxiliary images/labels, only searches for all that on history transactions if last transaction has no biometrics.

[BF-002][ALPHA-1522][API] Background quality extraction fixes:

  • Fixed capability to use all configured quality micro extractors on background (use thread poll on java to use it all).
  • Fixed NPE consolidating extraction quality elapsed time on background.
  • Fixed quality extraction only for enroll and update transactions, even if other transactions has flags set to 0 on RDB.

[BF-003][ALPHA-1523][ALPHA-1524][GBDS] Fixed micro service matcher search with label filter for face and iris.

Obs: This fix requires that [IP-002] is applied, it means, RPM must be installed to upgrade OpenVINO.

Improvements

[IP-001][ALPHA-1509][Face Detect Service]

Added 3rd parameter (in secs) on start-detect-service.sh to hold before start processing.

Obs: This improvement requires that [IP-002] is applied, it means, RPM must be installed to upgrade OpenVINO.

[IP-002][ALPHA-1510][API][GBDS][Face Detect Service]

Face lib openVINO upgraded to version 2023.1 on face extraction service, face matcher service, face detection service and GBDS 3 native lib.

Obs: This improvement only is applied if RPM is installed.

GBDS 4.5.11.1510

Partial release with API and RDB Migration only.

Bug Fixes

[BF-001][ALPHA-1512][RDB Migration] Fixed transaction not found on RDB protection on fix pending labels mode

Stack trace of each error is now logged on both check pending and fix pending label modes

[BF-002][ALPHA-1521][API] Fixed get person when last person transaction has no biometrics.

From now on, last considered transaction for a person is the last person transaction with at least one biometric.

To find this last person transaction with at least one biometric, API now look up on person history, from last to first, for a transaction with biometrics.

Obs: This behavior used to occur when a trusted update with no biometrics is made (BoB on for this) to manipulate person keys/biographics/labels. Since to do this BoB has to be on, when BoB is back to off, the get person used to fail (no biometric found on last transaction).

[BF-003][ALPHA-1522][API] Fixed background quality extraction parallel distribution and processing

Fixed background quality extraction distribution to multiple APIs where quality extraction API id were not being set properly on RDB table gbds.transactions

GBDS 4.5.10.1509

Partial release with RDB Migration only.

Bug Fixes

[BF-001][ALPHA-1519][RDBMigration] Exceptions import on mode anomalies fixed when entrant transactions does not have search result or search result format is invalid

GBDS 4.5.10.1508

Partial release with API, gbds.tar.gz and RDB Migration only.

Bug Fixes

[BF-001][ALPHA-1512][RDB Migration]

Fixed processing old transactions on checkPending and fixPendingLabels; fixed command line option checkGroupFactor

[BF-002][ALPHA-1515][API]

Fixed retro compatibility with older GBDS Driver versions when processing new enrolls on API 4.5.10.1507+

GBDS 4.5.10.1507

Partial release with GBDS API and RDB Migration only.

Bug Fixes

[BF-001][ALPHA-1508][API]

Fixed Get transaction when quality extraction result is missing or absent.

This happens on transactions before quality extraction implementation. On later transactions, even with quality extraction disabled, its quality extraction result is present but empty.

[BF-002][ALPHA-1511][API]

Update on broken people

Allow update on broken people in these conditions:

  • People that have active transactions as failed (with field active true or not)
  • People that have row key key:value pointing to a failed transaction, even when transaction guid on person points correctly

To allow this, update verification allows FAILED transactions to be the current transaction of an update operation.

[BF-003][ALPHA-1515][API]

Fixed template extraction with low/zero minutiae

On enroll/update/search, removed 0 minutiae templates from extraction and allowed 1:1 verify with extraction issues.

When fingerprint quality extraction is made and some template has 0 minutiae extracted, send the enroll transaction to MIR.

Extraction has better issue messages.

[BF-004][ALPHA-1516][API][RDB Migration]

Fixed insertion of transaction fields on RDB table gbds.transaction_fields with same key/biographic name and more than one label. This fix allows transaction listing with multiple labels to MIR.

To fix PENDING quality transactions on RDB, run RDB Migration with new mode:

--mode fixPendingLabels --checkTimeRange yyyy-MM-dd-HH-mm-ss,yyyy-MM-dd-HH-mm-ss

  • Argument checkTimeRange can be omitted or only start or end can be informed, leaving a comma to indicate which one.
  • Be careful: this mode runs a scan over transactions HBase table, then filtering PENDING ones.

To fix APPROVED quality transactions on RDB, run script:

insert into gbds.transaction_fields (fkey, `type`, fvalue, transaction_id)
select distinct f.fkey fkey, f.`type` `type`, f.fvalue fvalue, t.id transaction_id
  from gbds.fields f, gbds.biometrics b, gbds.transactions_ref tr, gbds.transactions t
 where b.person_version = f.person_version and f.person_id = b.person_id and b.transaction_id = tr.id
   and tr.tguid = t.tguid
   and f.`type` = 'LABEL' -- comment this line to fix biographics too
   and not exists (select 1 from gbds.transaction_fields f2
                    where f2.fvalue = f.fvalue and f2.transaction_id = t.id
                      and f2.`type` = f.`type`
                      and f2.fkey <=> f.fkey)

Obs: On RDB table gbds.fields is already ok

GBDS 4.5.9.1506

Partial release with RDB Migration only

Bug Fixes

[BF-001][ALPHA-1513][RDB Migration]

Exceptions migration fix:

  • Run always on all anomalies on HBase
  • Can create or update exception on RDB
  • Get correct reference transaction guid looking at candidates on entrant transaction pointed at anomaly: transaction-guid

To perform anomaly migration again, we recommend:

  • Create backup table for gbds.exceptions, such as CREATE TABLE GBDS.EXCEPTION_BCK AS SELECT * FROM GBDS.EXCEPTIONS
  • Truncate gbds.exceptions: DELETE FROM GBDS.EXCEPTIONS
  • Then run RDB migration with --mode anomalies

Improvements

[IP-001][ALPHA-1512][RDB Migration]

Check pending mode

RDB Migration tool with new mode for checking pending transactions on HBase:

--mode checkPending: check pending transactions

Other parameters:

--checkTimeRange yyyy-MM-dd-HH-mm-ss,yyyy-MM-dd-HH-mm-ss: time range to check; start and/or end can be omitted. Default is no time range

  • Ex: only start: --checkTimeRange 2023-07-07-00-00-00,
  • Ex: only end: --checkTimeRange ,2023-07-08-00-00-00

--checkGroupFactor 10: group factor on pending transactions (default is 10)

Output will be on log like:

"578 pending transactions"
"10 -> 20: 123"
"20 -> 30: 456"

GBDS 4.5.9.1505

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: b0a9a899d6125fd563af091e37550cb8
GBDS redistribution MD5 Checksum: c74eb2616d84c4700eaf953bd2fc00a5
GBS Smart Sense Agent MD5 Checksum: ea133582c14b7061004697a62e67a605

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.

  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md

  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md

  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md

  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md

  • Java client and API model 4.5.9 deployed on artifactory.

  • Fast GBDS Driver upgrade with gbds-4.5.9.1505.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

  • The matcher services can be upgraded individually using matcher-service on folders:

    • Face matcher service: /var/lib/griaule/extractor/facematch
    • Optimized ginger matcher service: /var/lib/griaule/extractor/optimized_gingermatch
    • Micro services scripts: /var/lib/griaule/extractor/scripts
    • GBDS start script: /var/lib/griaule/gbds/scripts

Improvements

[IP-001][ALPHA-1496][GBDS]

1:N loop performance improvement

Parallel calls to matcher microservices inside a transaction, grouping templates by modality (including UL modalities), for enroll/search were implemented.

With this feature, on 1:N enroll/search, calls to optimized ginger and face matcher micro services occur at the same time inside a transaction, removing the overhead time on Java to process the ginger response before calling face matcher micro service. For this to work inside matcher micro services, these ones now have interprocess mutex to serialize ginger and face calls internally. The mutex is shared between the same Java matcher optimized ginger and face micro services.

[CF-001] Configurable on application.conf or on RDB table gbds.settings:

  • gbds.match.service.templateSend.parallelByModality, default true

Obs: old settings gbds.match.service.templateSend.qtde and gbds.match.service.templateSend.method not used anymore

GBDS 4.5.8.1504

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 13f1dfcc4eda0ae98a2057c6f4dd83d2
GBDS redistribution MD5 Checksum: 4c3cde554c08f0ac1b52585615b6abc8
GBS Smart Sense Agent MD5 Checksum: 9ec620854743f748b8c4f060926da215

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • RDB upgrade 4.5.8.1504 provided from 4.5.0.1491 on upgrade-rdb.md
  • Java client and API model 4.5.8 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.8.1504.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1473][Smartsense agent]

Fixed GBS Service endpoint response:

  • Fixed default BCC version endpoint
  • Fixed default Intelligence version endpoint and installation path

[BF-002][ALPHA-1491][Face detect service]

Added 3 re tries to open existing file to check MD5

[BF-003][ALPHA-1492][Smartsense Agent][API]

Fixed connection error message on API and Smartsense agent:

  • API endpoints /version and /status
  • Smartsense agent endpoint /matcher-status

[BF-004][ALPHA-1494][Smartsense agent]

Fixed 400 error response with body describing error

[BF-005][ALPHA-1495][GBDS]

Fixed sink and enroll post matchers increase

  • HBase connection clean up

    • GBDS and API used to disconnect/connect HBase clients and check availability multiple times during boot, now it just opens the connection
    • HBase clients fixed with resources leak protection
    • Better debug logs on HBase connection creation

  • Sink increase fix

    • Do not keep reconnecting HBase when boot finishes
    • Sink oriented logs for sink, enroll/search result processor and enroll business

  • Enroll post matcher increase

    • Do not use akka proxy on enroll post matchers when micro services is enabled
    • Reconnection on error: max tries up to 10 with 5s sleep and clean contexts (people stored on enroll post matcher)

[BF-006][ALPHA-1498][Extraction microservices]

Fixed micro extractors header size limit up to 8kb, so larger tokens used on API can be passed to micro extractores as well

[BF-007][ALPHA-1503][API]

Fixed fragment image being sent on kafka task to GBDS on create/update UL, causing kafka fail to enqueue it (kafka claim that package gets to big). Image is not sent anymore, just fragment template.

Improvements

[IP-001][ALPHA-1472][API]

Comments column length on RDB for exception treatment upgraded to 1024, now validated on treat endpoint (cropped if bigger than 1024)

[IP-002][ALPHA-1502][GBDS]

GBDS Monitor endpoint /status/monitor speed up

Endpoint /status/monitor speed up, calling now directly matcher metrics and matcher micro service clients to get matcher sizes by modality. Do not uses akka task to matchers anymore.

[IP-003][ALPHA-1504][GBDS]

Enroll post matcher selection using round robin instead of tguid hash to result a better enroll pm distribution

[IP-004][ALPHA-1506][Matcher microservices]

Optimized matcher micro service updated

  • Optimized matcher micro service updated with lib ginger 1.8.2
  • On lib ginger: Faster cab vector match

GBDS 4.5.7.1503

Partial release

Bug Fixes

[BF-001][ALPHA-1493] Fixed backpressure queue lock

  • When some error occurs on enroll post matcher, that forwards task as enroll failed
  • When a task is trying to be reprocessed coming from kafka

GBDS 4.5.6.1502

Partial release

Improvements

[BF-001][ALPHA-1497] Added get/set on additional headers on client

GBDS 4.5.6.1501

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: a8fd10d17b70058a951e486dd68a445d
GBDS redistribution MD5 Checksum: 956300464cc9583858df764af57684f1
GBS Smart Sense Agent MD5 Checksum: 1df6250796e4551aa6adfa330b5c1abd

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has its own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • Java client and API model 4.5.6 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.6.1501.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1490] Fixed processing already processed tasks on back-pressure queue. It now removes correctly from queue on local consolidator

GBDS 4.5.5.1500

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 883e4fd277eaa78e4df4db23c90b241e
GBDS redistribution MD5 Checksum: c058fd7164bbe795de5426113b83c58e
GBS Smart Sense Agent MD5 Checksum: 190e454eea5b38b19cba5fc2fcb48e7f

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • Java client and API model 4.5.5 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.5.1500.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1479] On all extractors, quality and matcher micro services: Fixed optimization flag to a more secure one

[BF-002][ALPHA-1486] Fixed peaks, ridges and segments extraction on micro service ginger extractor.

These ones are used on enroll extraction, not on verify extraction.

[BF-003][ALPHA-1487] Fixed total time on RDB when enroll post match is enabled

Improvements

[IP-001][ALPHA-1480] Internal and micro service ginger extractor with maximum minutiae set to 256

[IP-0012][ALPHA-1483] Better logs on remove person, create, update and remove ULs

GBDS 4.5.4.1499

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 60060a1c34ab90bd5dc1d1c04c1b5835
GBDS redistribution MD5 Checksum: bb11b01878996d4991fbb23e162c5b8c
GBS Smart Sense Agent MD5 Checksum: 80dd531579ca06eefb0b5bb65f4c0a70

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has it own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • Java client and API model 4.5.4 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.4.1499.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1481] Fixed Solve UL when palmprint modality is not enabled

GBDS 4.5.3.1498

Partial release

Bug Fixes

[BF-001][ALPHA-1477] Fixed concurrent list modification on task consumer queue (back-pressure per node)

[BF-002][ALPHA-1461] Fixed latent search with post match

Search result was 0 if some candidate transaction is not found on HBase when performing post match. Fixed this issue skipping tguids that search post match fails to load from HBase.

GBDS 4.5.2.1496

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: e9c4fb6a39dba95a6f8d6c0974d2de49
GBDS redistribution MD5 Checksum: a7daf4727c16112657e8cbf2cd5349c1
GBS Smart Sense Agent MD5 Checksum: 8a57529087029db29b635cb048fe6d42

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has its own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • Java client and API model 4.5.2 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.2.1496.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1475] Fixed reconnection with HBase on enroll post matcher if HBase connection is lost

GBDS 4.5.1.1494

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: aa150d393c7f1fcaeefa2e42f997d402
GBDS redistribution MD5 Checksum: f89f641bfa6632197e58003e10d28c77
GBS Smart Sense Agent MD5 Checksum: 879fde93824cec3db0242202e6826655

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has its own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • Java client and API model 4.5.1 deployed on artifactory.
  • Fast GBDS Driver upgrade with gbds-4.5.1.1494.tar.gz with GBDS Driver jar files only. Tar files relative to /var/lib/griaule/gbds

Bug Fixes

[BF-001][ALPHA-1455] Fixed corrupted RPM for micro service binaries

[BF-002][ALPHA-1465] Enroll post matcher fixed. It now uses always deserialized full template format for ginger

Improvements

[IP-001][ALPHA-1454][ALPHA-1458] Ginger micro service improvements

  • Better logs on micro service with elapsed time and pre drop information
  • Fixed cab on micro service
  • Ginger lib update with deserialization fix for pre drop
  • Better candidate list manipulation
  • Better overall performance (improved compilation flags)

[IP-002][ALPHA-1457] Do not remove by default all inactive people from enroll results.

[IP-003][ALPHA-1466] Matcher creation speed up on boot

Some improvements and fixes were made on GBDS while assembling matchers:

  • [CF-001] Configuration renamed:

    • from: gbds.boot.matcher.creation.sleepTime
    • to: gbds.boot.matcher.creation.sleepTime.ms
    • Default 500, minimum 0, maximum 30000

  • Fixed native face instance creation: only one native instance now is created, made only on Driver, not on all matchers anymore

  • Fixed global consolidator creation lookup wait between tries. This can reduce matcher assembly lag from 80~90s to 5s when the setting gbds.node.actors-start.timeout is set to 400s, meaning you can reduce on boot about 20% of the time pointed on this setting.

GBDS 4.5.0.1491

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 4becaabc4ea896df3fd6a7380a19318b
GBDS redistribution MD5 Checksum: 2bc6063630e1948558a130e60dfc2a4c
GBS Smart Sense Agent MD5 Checksum: 889e3e0e2c542bd19a9d777ad8809fcb

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has its own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1491 provided from 4.5.0.1488 on upgrade-rdb.md
  • Java client and API model 4.5.0 re deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.

Bug Fixes

[BF-001][ALPHA-1445] GBDS boot fixes

[CF-001] Fixed sleep time between matcher creation on application.conf setting: gbds.boot.matcher.creation.sleepTime

[CF-002] Setting gbds.node.matchers.start.timeout maximum value adjusted to 240s

Memory monitor

Added a memory monitor that logs java JVM memory usage every 10 seconds.

[CF-003] On application.conf or RDB gbds.settings, use setting: gbds.memory-monitor, default false

[BF-002][ALPHA-1447] Fixed Smartsense Agent kill script

[BF-003][ALPHA-1448] Fixed face search result when identify is sent as latent; do crop after post match only for fingerprints and palmprints

[BF-004][ALPHA-1449] Fixed memory increase while performing latent search 1:N with matcher MS.

Matcher MS now keeps only max candidate sized list on memory, always sorted by score.

[BF-005][ALPHA-1451] Fixed akka configuration for nodes with more than 32 matchers.

Akka parallelism configuration exposed on application.conf so it can be changed on nodes with more than 32 matchers. Number of matchers x 2 is recommended.

Configuration exposed:

akka {
    ...
    actor {
        ...
        default-dispatcher {
            type = "Dispatcher"
            executor = "default-executor"
            default-executor {
                fallback = "fork-join-executor"
            }
            fork-join-executor {
                parallelism-min = 8
                parallelism-factor = 1.0
                parallelism-max = 64
            }
        }
    }
}

prio-dispatcher {
    mailbox-type = "com.griaulebiometrics.gbds.driver.mailbox.PriorityMailbox"
    type = "Dispatcher"
    executor = "default-executor"
    default-executor {
        fallback = "fork-join-executor"
    }
    fork-join-executor {
        parallelism-min = 8
        parallelism-factor = 1.0
        parallelism-max = 64
    }
}

[BF-006][ALPHA-1452] BEST Face Detect Service: better error logging on python and better error flow when detecting faces

[BF-007][ALPHA-1425] Fixed priority order that tasks are distributed, removed or have consuming verified on task consumer queue (from GOD to LOWEST)

Improvements

[IP-001][ALPHA-1450] Matcher creation and start API log clean up

[IP-002][ALPHA-1453] Watchdog new settings

Watchdog log mode and level were added using application.conf and RDB gbds.settings. These settings can be hot changed using RDB

  • gbds.watchdog.interval, integer, default 1, min 1, max 60 (minutes)

  • gbds.watchdog.log.mode

    • TGUID_MATCHER_MAP (default)
    • MATCHER_TGUID_MAP

  • gbds.watchdog.log.level

    • INFO
    • DEBUG (default)

With log mode TGUID_MATCHER_MAP watchdog logs a list of TGUIDs that have enroll/search running and for each TGUID the list of matchers running it with elapsed time so far.

With log mode MATCHER_TGUID_MAP watchdog logs a list of matchers that have enroll/search running and for each matcher the list of TGUIDs that this matcher is running search for, with elapsed time so far for each TGUID.

This information is logged only with watchdog log level DEBUG. On watchdog log level INFO, only the count of matchers running searches are logged.

GBDS 4.5.0.1488

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: db9b4525d31252a60152a448f8b28375
GBDS redistribution MD5 Checksum: 3431796520f0d331b071800d08f76e9f
GBS Smart Sense Agent MD5 Checksum: f20bc7da66207035bbcc70c3aec33552

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has its own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • RDB upgrade 4.5.0.1488 provided from 4.5.0.1484 on upgrade-rdb.md
  • Java client and API model 4.5.0 re deployed on artifactory. It might be necessary to erase the local ~/.m2/repository folder for this version to use it.

Bug Fixes

[BF-001][ALPHA-1428] Watch dog fixes:

  • Watch dog less log on INFO and full log on DEBUG.
  • [CF-001] Watch dog log interval configurable: gbds.watchdog.interval (in minutes; default 1, minimum 1, maximum 60)

[BF-002][ALPHA-1442] BEST Face Detect Service

Added additional information on endpoint /status.

Improvements

[IP-001][ALPHA-1440] MS boot improvements

[CF-001] Check timeout on MS client start configurable (API file gbdsapi.properties, GBDS file application.conf or RDB table gbds.settings)

  • gbds.match.service.checkTimeoutSecs on GBDS for matchers
  • gbds.extraction.quality.service.checkTimeoutSecs on API for quality extraction,
  • gbds.extraction.service.checkTimeoutSecs on API for template extraction
  • All settings default is 10s.

[CF-002] Matcher creation sleep default now is 3s, configurable on application.conf

  • gbds.boot.matcher.creation.sleepTime

On single port matcher clients (ginger and optimized ginger), check only 1 client, no need to check them all (all points to the same port).

[IP-002][ALPHA-1443] Sorted list transactions

When list transactions is made on RDB (when filter has qualityStatus and/or enrollStatus) there’s a new query parameter to sort list and pagination:

  • orderBy=[CREATED_ASC|CREATED_DESC]

    CREATED_ASC is default.

GBDS 4.5.0.1484

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.5.0 or newer
GBDS release MD5 Checksum: 0275f63c402149a5a724d48d44cebd39
GBDS redistribution MD5 Checksum: fa091cf3d9a06817c2e9948bbd2b2281
GBS Smart Sense Agent MD5 Checksum: 8946bf6044db510da3400112f1ee5b94

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • All RDB upgrades are now united on one file: upgrade-rdb.md, each upgrade for versions has its own session on this file.
  • RDB upgrade 4.5.0.1484 provided from 4.4.2 on upgrade-rdb.md
  • Java client and API model 4.5.0 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1427] Do not create these HBase column families if not configured to do so:

  • fingerprints
  • faces
  • iris
  • voice
  • palmprints

[BF-002][ALPHA-1428] Fixed watchdog logging more than once per node.

Fixed add/remove tasks on watchdog with multiple matchers.

[BF-003][ALPHA-1428] Fixed GBDS monitor endpoint gets status for Smartsense Agent. It now uses kafka tasks as God tasks (faster).

[BF-004][ALPHA-1433] Fixed memory leak on fingerprint/palmprint extraction (it used to get a new extractor per web request) and face (face image not destroyed on native code).

[BF-005][ALPHA-1434] Fixed unnecessary logs on matching.

[BF-006][ALPHA-1436] Fixed template and quality extraction micro services deployed on API.

When RDB System Configuration Agent is off (setting gbds.rdbSystemConfiguration.enabled), look only at gbdsapi.properties file to deploy template extraction micro services (setting gbds.extraction.service, default value in file is true) and deploy quality extraction micro services (setting gbds.extraction.quality.service, default value in file is false).

Improvements

[IP-001][ALPHA-1419] UL new statuses: enqueued and fail, due to GBDS matcher micro service optimized memory control

UL creation and update used to have UNSOLVED status right from API, before GBDS matcher processing. This could be problematic if ginger matcher micro service storage is full or almost full for new ULs, because an UL creation could not be deployed in this case. To fix this, UL create/update now sets UL status as ENQUEUED until it is processed, changing to UNSOLVED or FAILED.

Create/update UL Workflow:

  • UL status is now ENQUEUED on HBase and RDB right after API receives it and sends it to GBDS.
  • UL status is changed from ENQUEUED to UNSOLVED or FAILED when GBDS finishes UL create/update processing.
  • UL status becomes UNSOLVED if processing is ok.
  • UL status becomes FAILED when matcher micro service memory is full on create/update UL.
  • UL with status FAILED can be updated to become UNSOLVED if GBDS accepts it on memory.
  • Get UL returns new UL status and possible fail reason as failReason on GET /ul.
  • Solve UL is unchanged (API solves it on HBase and RDB, GBDS removes it from memory).
  • Other UL endpoints are unchanged as well.

[IP-002][ALPHA-1409] API ID and GBDS Version is now stored on HBase and RDB and returned on get transaction/search.

[BD-001] On RDB, table gbds.transactions has one new column:

  • gbds_version: GBDS version that processed the transaction.

On get transaction/search, two new attributes are returned:

{
    "data": {
        ...
        "apiID": "api1",
        "gbdsVersion": "4.5.0.1484",
        ...
    }
}
  • apiID: API ID that processed the enroll/search transaction.
  • gbdsVersion: GBDS version that processed the transaction.

[IP-003][ALPHA-1425] Back pressure per node.

Back pressure mechanism changed in two ways:

  • Queue processing from cluster to node. The back pressure queue on nodes does not wait for full cluster processing for one transaction anymore. It waits only for the search 1:N local node processing to request new tasks.

  • Queue by priority. The back pressure queue consuming can be disabled or enabled by priority when this queue is full or available.

    Example: If back pressure maximum tasks is 10 (by priority now) and Kafka queues have 10+ tasks on DEFAULT_PRIORITY. Kafka task consuming on DEFAULT_PRIORITY queue is stopped after default queue reaches 10 tasks and will enabled only when this queue size is less than 10 tasks, but all other task consuming queues are still enabled, because its back pressure queues are empty.

[CF-001] Back pressure per node configuration:

It can be done on application.conf or on RDB gbds.settings:

  • gbds.node.max-loaded-tasks: Maximum tasks loaded by priority on node. Default is 100, minimum is 10, maximum is 20000.
  • gbds.cluster.kafka.max-tasks-per-poll: Maximum task enqueued on kafka poll. Default is 1, minimum is 1, maximum is 1000.

The max-tasks-per-poll setting can be higher than max-loaded-tasks now, because all tasks polled from Kafka are enqueued on back pressure queues, but this queues loads only tasks until the size configured on max-loaded-tasks.

When a task is completed, back pressure loaded next task on queue if necessary and request new ones to Kafka (by enabling task consuming) if back pressure queue size is less than max-loaded-tasks.

This way, back pressure queues has always max-loaded-tasks tasks enqueued and loaded and max-tasks-per-poll enqueued (if Kafka queue has enough tasks, of course).

[IP-004][ALPHA-1429] Return times on get transaction and get search: extraction elapsed, extraction quality elapsed and search elapsed

Get transaction/search can now return extraction template/quality and search times:

{
    "data": {
        ...
        "extractionElapsed": 7404,
        "extractionQualityElapsed": 7890,
        "searchElapsed": 6118
    }
}

[IP-005][ALPHA-1431] Labels can be replaced or accumulated on update and trusted update.

Default behavior is still to update labels, to accumulate them along update transactions. This can be changed using:

{
    "meta" : {
        "labels" : "ACCUMULATE"
    },
    ...
}
  • meta->labels: ACCUMULATE, REPLACE (default is ACCUMULATE)

This works for update and trusted update.

[IP-006][ALPHA-1438] Unique upgrade RDB file.

All RDB upgrades are now on upgrade-rdb.md markdown file.

Each GBDS version has a session ## <version>[.<release>] and the upgrade script is under code block marked with ~~~~sql and ~~~~ (markdown syntax for SQL).

For example:

## 4.1.1
~~~~sql
ALTER TABLE gbds.transactions ADD manually_reviewed TINYINT(1) DEFAULT 0 NOT NULL AFTER global_quality;

update gbds.settings set svalue = '<sample-url>' where skey = 'gbds.transparency.email-notifier.url' and svalue = '';

commit;
~~~~

Current active GBDS version could have session separated by release. Older versions will have all upgrades joined into single version session.

New features

[NF-001][ALPHA-1426] Smartsense Agent new Endpoint: Kafka queues

A new endpoint was created on Smartsense Agent:

GET /kafka: It returns all Kafka queues with committed and offset values

  • Query parameter: consumerGroup, not required, default value is gbds-tasks-committer (Kafka queue consumer group for entire cluster).
    It can be any other group, for instance, gbds-tasks, that is Kafka queue consumer group for a specific node.

  • Return: 200 OK

{
    "consumerName": "gbds-tasks-committer",
    "topics": [
        {
            "endOffset": 1,
            "currentOffset": 1,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-LOWER_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-HIGH_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-DEFAULT_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-LOW_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-LOWEST_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-HIGHER_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-HIGHEST_PRIORITY"
        },
        {
            "endOffset": 0,
            "currentOffset": 0,
            "lag": 0,
            "partion": 0,
            "topic": "gbds-tasks-GOD_PRIORITY"
        }
    ]
}

[NF-002][ALPHA-1435] BEST Face Detect Service

BEST Face Detect Service is a service made for face detection, tracking, extraction, matching and ID consolidation on videos. It is used on BEST with video cases.

This service is file based, with no database. All output generated is available on the folder configured on service start.

Installation

Update environment with gbds-distribution provided.

Run

Start script: /var/lib/griaule/extractor/scripts/start-detect-service.sh <output folder>

Stop script: /var/lib/griaule/extractor/scripts/kill-all-facedetect-service.sh

Endpoints

[EP-001] POST /v1/detection/startToProcess

Starts a face detection process. It receives a video’s list of paths to be processed. It returns a GUID to retrieve process status later.

Request:

{
    "number_of_best_faces": 5,
    "frames_appearing_filter": 30,
    "frames_step": 3,
    "match_threshold": 65,
    "md5_check_timeout": 60,
    "videoPaths" : [
        {
            "filePath": "/tmp/tse/videoFolder1/",
            "md5": "7b9eb0d5ce8b62730ffc53a229334bca"
        },
        {
            "filePath": "/tmp/tse/videoFolder2/",
            "md5": "b002b7afdfee0e136edac8c58b46fc0d"
        },
        {
            "filePath": "/tmp/tse/videoFolder3/",
            "md5": "7bc232087692b9e9ab908edb7a04b9fb"
        },
        ...
    ]
}

Attributes:

  • number_of_best_faces: number of faces to keep from each ID consolidated

  • frames_appearing_filter: minimum number of frames that a face detected appears to be considered on result

  • frames_step: number of frames to drop after a frame processed while detecting. It accelerates the video process.

  • match_threshold: face match threshold to ID consolidation

  • md5_check_timeout: timeout on MD5 checking

  • videoPaths: list of video paths to be processed:

    • filePath: full video file path
    • md5: MD5 provided to be checked internally

Response:

201 Created:

{
    "processID": "85e20280-4602-4007-b5e4-c8bd61ffb53d"
}

400 Bad request:

{
    "message": "video path(s) X not found"
}

503 Server busy:

{
     "error": "A process has already taken place (85e20280-4602-4007-b5e4-c8bd61ffb53d).. Wait until it ends.",
     "statusCode": 503
}

[EP-002] GET /v1/detection/status/{processID}?page=0&size=30

It gets a process status and result

  • Path parameters:

    • processID: process id provided on startProcess

  • Query parameters:

    • page: page number to paginate. Default is 0
    • size: page size to paginate: Default is 20

Response:

200 OK while processing:

{
     "status": "CHECKING_MD5_HASHES","
     "status_percent": 20,
     "number_of_best_faces": 2
}

Fields:

  • status: process status. Processing values: CHECKING_MD5_HASHES, EXTRACTING_FACE_TEMPLATES, TRACKING_FACES_AND_QUALITIES, COLLECTING_IDENTITIES, ERROR
  • status_percent: process progress 0 to 100
  • number_of_best_faces: number of best faces configured for this processing

200 OK with process completed:

{
    "status": "DONE",
    "status_percent": 100,
    "total_identities": 234,
    "number_of_best_faces": 5,
    "result": {
        "identities": [
            {
                "identity_id": "id_001",
                "bestFaces": [
                    {
                        "imageBase64": "base64_image1",
                        "video_file": "video_file_name.mp4",
                        "quality": 100,
                        "video_fps": 30,
                        "original_frame": "base64_frame_image",
                        "frame_number": 12,
                        "xmin": 9,
                        "ymin": 9,
                        "xmax": 10,
                        "ymax": 10,
                        "video_width": 1280,
                        "video_height": 720
                    },
                    ...
                ],
                "appearances": [
                    {
                        "video_file": "video_file_name.mp4",
                        "video_fps": 30,
                        "timestamp_ranges": [
                            {
                                "start": 233,
                                "end": 555
                            },
                            {
                                "start": 666,
                                "end": 988
                            },
                            ...
                        ]
                    },
                    ...
                ]
            }
        ]
    }
}

Fields:

  • status: process done status: DONE

  • status_percent: always 100

  • total_identities: number of consolidated IDs

  • number_of_best_faces: number of best faces configured for this processing

  • result: process result:

    • identities: list of IDs. An ID has:

      • identity_id: identify guid, it just enumerated all ids
      • bestFaces: list of all best faces, sorted from best to worst
      • imageBase64: cropped best face image on base 64
      • video_file: video file where this face was found
      • quality: face quality
      • video_fps: frames per second of video
      • original_frame: original frame image on base 64
      • frame_number: frame number where this face was found
      • xmin, ymin, xmax, ymax: box that surrounds cropped face image on original frame
      • video_width, video_height: video dimensions

    • appearances: list of appearances of this ID along all videos processed.

      • video_file: video file where this ID was found

      • video_fps: frames per second of video

      • timestamp_ranges: list of frame ranges on this video appearance:

        • start: start frame on video
        • end: end frame on video

400 Bad request:

{
    "error": "PID 28e2567c-d675-44b1-be1c-ad1936c68775 not found!",
    "statusCode": 400
}

[EP-003] GET /v1/detection/errors/{processID}

It returns all errors on processing. It can be called at any time while processing or after the process is completed.

Response:

200 OK:

{
    "errors": [
        "error1",
        "error2",
        ...
    ]
}

400 Bad request:

{
    "error": "PID 28e2567c-d675-44b1-be1c-ad1936c68775 not found!",
    "statusCode": 400
}

GBDS 4.4.7.1493

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.4.0 or newer

GBDS release MD5 Checksum: ec2b4abc27a09cf76e8190def0b58261

GBDS redistribution MD5 Checksum: 95d3088bc0bed694642ad44a04415fd3

GBS Smart Sense Agent MD5 Checksum: 89cc777802734b60c3297b6326758ead

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0 / 4.4.0 up to 4.4.6
  • Java client and API model 4.4.6 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1455] Fixed corrupted RPM for micro service binaries

[BF-002][ALPHA-1465] Enroll post matcher fixed. It now uses always deserialized full template format for ginger

Improvements

[IP-001][ALPHA-1454][ALPHA-1458] Ginger micro service improvements

  • Better logs on micro service with elapsed time and pre drop information
  • Fixed cab on micro service
  • Ginger lib update with deserialization fix for pre drop
  • Better candidate list manipulation

Obs: ginger-matcher-service and optimized-ginger-matcher-service recompiled on this build due to ginger lib not updated properly

[IP-002][ALPHA-1457] Do not remove by default all inactive people from enroll results.

GBDS 4.4.6.1490

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.4.0 or newer

GBDS release MD5 Checksum: 3ce43c0a38bebff0bc468cbbd96950a5

GBDS redistribution MD5 Checksum: 3c0a8e9d44fa1b4d507dc847482fda0e

GBS Smart Sense Agent MD5 Checksum: b353a11e5f819eab9dac6e8d62a93b12

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0 / 4.4.0 up to 4.4.5
  • Java client and API model 4.4.6 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1449] Fixed memory increase while performing latent search 1:N with matcher MS. Matcher MS now keeps only max candidate sized list on memory, always sorted by score.

GBDS 4.4.3.1486

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.4.0 or newer

GBDS release MD5 Checksum: 08680dc92dac397be4c076ee7f7810b1

GBDS redistribution MD5 Checksum: 51ac01ed5af9c8f547ce2de6d4a97c0f

GBS Smart Sense Agent MD5 Checksum: 70082364b6460639a4d45532c43c0656

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0 / 4.4.0 up to 4.4.1
  • RDB upgrade from 4.4.2.1482 provided
  • Java client and API model 4.4.3 deployed on artifactory.

Improvements

[IP-001][ALPHA-1440] Micro service boot improvements

  • Check timeout on MS client start configurable:

    • gbds.match.service.checkTimeoutSecs for matchers
    • gbds.extraction.quality.service.checkTimeoutSecs for quality extraction
    • gbds.extraction.service.checkTimeoutSecs for template extraction
    • default 10s

  • Matcher creation sleep default now is 3s, configurable on application.conf:

    • gbds.boot.matcher.creation.sleepTime

  • On single port matcher clients (ginger and optimized ginger), check only 1 client, no need to check them all (all points to the same port).

GBDS 4.4.2.1482

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.4.0 or newer

GBDS release MD5 Checksum: e56f3f32fbbf271a9ae26333286cc5e3

GBDS redistribution MD5 Checksum: 59a9c83e9f2cdc283ab3ea52e55f6e3e

GBS Smart Sense Agent MD5 Checksum: f2cff3b075eca88f5d163747271e5ea1

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0 / 4.4.0 up to 4.4.1
  • RDB upgrade from 4.4.1.1476 provided
  • Java client and API model 4.4.2 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1412] Fixed new scripts inclusion

New GBDS scripts were missing:

  • /var/lib/griaule/gbds/scripts/status-cluster.sh
  • /var/lib/griaule/gbds/scripts/monitor-gbds.sh

[BF-002][ALPHA-1401] 4.4.1 Reverted People scan region track

People scan region track is back on HBase transactions table, because this table has better region distribution.

[BF-003][ALPHA-1417] Fixed get transactions when deleting biometric with no Matcher Micro service enabled. If Matcher Micro service is on, return an error message.

Improvements

[IP-001][ALPHA-1411] UL boot bad region features:

Settings on application.conf:

  • [CF-001] Skip UL boot entirely: gbds.ul.boot.scan.enabled default true
  • [CF-002] Skip UL bad regions: gbds.boot.scan.ignoreErrorsOnRegion default true

[IP-002][ALPHA-1413] Added Kafka queue logging on every task consuming poll.

  • [CF-001] Log enabled by gbds.log.diagnose. It could be on RDB gbds.settings (setting type GBDS) or just on application.conf.

[IP-003][ALPHA-1416] Matcher MS template sending optimized.

It was added three ways to send templates (by modality) to matcher MS:

  • One by one parallel
  • One by one serial
  • All at once serial (default)

Settings can be made on RDB gbds.settings (setting type GBDS) or just on application.conf:

  • [CF-001] gbds.match.service.templateSend.qtde

    • ONE_BY_ONE
    • ALL_AT_ONCE (default)

  • [CF-002] gbds.match.service.templateSend.method

    • PARALLEL
    • SERIAL (default)

These settings are not present on gbds.settings and/or application.conf by default. Can be inserted on RDB if needed to be changed.

4.4.1 Fix Segmentation fault fix when using ginger MS or optimized ginger MS with ONE_BY_ONE, PARALLEL (only mode present on GBDS 4.4.1)

It was also added a setting on RDB gbds.settings and/or application.conf to debug future segmentation faults on micro services:

  • [CF-001] gbds.extraction.service.linkLibSegfault (RDB setting type API): to turn on seg fault tracking on extraction MS
  • [CF-002] gbds.extraction.quality.service.linkLibSegfault (RDB setting type API): to turn on seg fault tracking on quality extraction MS
  • [CF-003] gbds.match.service.linkLibSegfault (RDB setting type GBDS): to turn on seg fault tracking on matcher MS

[IP-004][ALPHA-1421] Inconsistent keys improvements

Some consistent key checking were improved:

  • API does not allow duplicated key IDs on enroll/update anymore

  • Exceptions are now generated on inconsistent keys:

    • Update with inconsistent keys (1:1):

      • Occurs when the entrant has a key id that already belongs to the reference person but with a different value (this key id/value is not found on base)

      • Occurs also on a replace person (PUT /v2/people/{pguid}), when it has a key with different value from the reference

      • Treatment:

        • APPROVE activate entrant and goes to memory
        • REJECT leaves as it is

    • Update with multiple people loading (1:N):

      • Occurs when the entrant keys points to multiple people. Example: An entrant with keys cpf:x and rg:y that point a reference person with cpf:x and other person with rg:y

      • Treatment:

        • Only one treatment can be APPROVE
        • All REJECT on involved exceptions leaves as it is
        • The APPROVE exception activates the entrant, updates the reference with all info, goes to memory and all people other than this one is deactivated and removed from memory (but not delete it)

GBDS 4.4.1.1476

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.4.0 or newer

GBDS release MD5 Checksum: d96b968c1d9c4561f1cc878bca09e062

GBDS redistribution MD5 Checksum: 3bf8eb24a2ab35fd66d9f59fc8d4d355

GBS Smart Sense Agent MD5 Checksum: 8a5b8eb93bb6f9d546961af528463949

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0 / 4.4.0
  • Java client and API model 4.4.1 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1401] Fixed region list track on people HBase table, not using transaction HBase table on people boot scan

[BF-002][ALPHA-1404] Fixed GBDS validation message on boot for ginger optimized template format with matcher microservices off

[BF-003][ALPHA-1406] Fixed match result on search 1:N with invalid pguids when using optimized ginger template format. It used to cause no exception generation

Improvements

[IP-001][ALPHA-1402] GBDS people boot features and fail saves

Some settings on application.conf were added:

[CF-001] Added people boot scan ignore error on region

  • gbds.boot.scan.ignoreErrorsOnRegion

    Default value: true

  • gbds.boot.scan.ignoreErrorsOnRegion.maxTries

    Default value: 5

Using this feature, GBDS will skip bad regions on people boot scan, with maximum tries configured.

[CF-002] Added people boot scan maximum rows to be scanned per node

  • gbds.boot.scan.maxRowsPerNode

    Default value: 0

Using this feature, GBDS will stop people scan on a node when all scanners for that node reach the configured number of rows.

[CF-003] Added people boot scan delayer

  • gbds.boot.scan-delayer.rows

    Default value 0

  • gbds.boot.scan-delayer.secs

    Default value 0

Using this feature, GBDS will wait the configured amount of seconds whenever reaching the configured number of scanned rows. This feature works per node.

[IP-002][ALPHA-1405] GBDS scripts for start/stop/monitor/status GBDS and Cluster updated

  • start-cluster: Added cluster already running and minimum memory validations.

    Memory verification is formatted in MB and minimum memory on script provided is 0 MB, to be adjusted for each client.

  • kill-gbds: better running process verification for GBDS and children (GBDSProcess and matcher microservices).

    Matcher microservices verification added as well.

  • monitor-gbds: Added for memory monitoring.

    Memory verification is formatted in MB and minimum memory on script provided is 0 MB, to be adjusted for each client.

  • start-gbds: Added memory monitor execution.

  • status-gbds: Added for cluster monitoring.

[IP-003][ALPHA-1407] Identify search optimizations

Search 1:N and enrolls were optimized on GBDS Driver and matchers with microservices:

  • Better akka serialization
  • Search 1:N performance fixed with no microservice http/json overhead on the same modality. Overhead still remains on a matcher search between modalities and between transactions.
  • Exception validation on enrollment result process with no HBase transaction get. It used to take several seconds, up to minutes, to finish an enroll. Now it is immediate.
  • Coincident fingers to ignore face feature used to request a HBase transaction get too. It was removed and this information now comes from people on memory from matchers.
  • Several logs removed on search 1:N main match loop and log changed to debug on enroll finish process.

[IP-004][ALPHA-1408] RDB update scripts with no DROPs

RDB upgrade scripts since 4.2.0 (to be used from 4.1.x and above) with no DROP TABLE, ALTER TABLE, or DROP COLUMN. Auxiliar script with these DROPs provided.

Existing script affected:

  • upgrade-rdb-4.3.0.sql: all DROPS were commented

New drop script provided:

upgrade-rdb-4.3.0-drops.sql: only DROPs SQL here

GBDS 4.4.0.1472

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.3.0 or newer

GBDS release MD5 Checksum: 11b5bd43d4afcc02da20de3e94dbf2db

GBDS redistribution MD5 Checksum: 0210abb2557c655a7a2aa774774276d0

GBS Smart Sense Agent MD5 Checksum: 34f82e4c3795977eef080374831e19cf

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0 / 4.4.0.1471
  • Java client and API model 4.4.0 re deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.

Bug Fixes

[BF-001][ALPHA-1398] Fixed update when reference uses face template version 1, 1.1 or 2 and update transaction has only images (multiple face template extraction) or new face template version 3. This bug appears first on GBDS 4.3.0.

GBDS 4.4.0.1471

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.3.0 or newer

GBDS release MD5 Checksum: 1e677ca7ee1d69e0e463de135dfb3009

GBDS redistribution MD5 Checksum: 342f67aa6e28c406f31068f07fd94821

GBS Smart Sense Agent MD5 Checksum: cc19db2cc64361b3545d2b3da45748f4

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • No RDB upgrade from 4.3.0
  • Java client and API model 4.4.0 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1285][ALPHA-1365] Removed unnecessary info logs on MS and native codes

[BF-002][ALPHA-1383] Fixed key duplication on enroll/update, trusted enroll/update

[BF-003][ALPHA-1380] Better message when finishing enroll with error

[BF-004][ALPHA-1384] Fixed latent search match time on table gbds.transactions

[BF-005][ALPHA-1388] Fixed clear-rdb script with default value for face multiplicity

[BF-006][ALPHA-1389] Fixed GBDS Search Engine Driver Spring Boot failure on some machines (fixed servlet-api-2.5 packaging)

Improvements

[IP-001][ALPHA-1382] Allow list transaction with specific labels, no labels or all

On GET /v2/people/transactions API accepts now three ways to inform label filter:

  • One or more labels: use regular query parameter: labels=<labelx>[&labels=<labely>...]
    Ex: /v2/people/transactions?qualityStatus=APPROVED&labels=label1&labels=label2
  • No labels at all (list all transactions): Do not pass any labels query parameter
    Ex: /v2/people/transactions?qualityStatus=APPROVED
  • NEW Transactions with no labels: use empty labels query parameter: labels=
    Ex: /v2/people/transactions?qualityStatus=APPROVED&labels=
    or even: /v2/people/transactions?qualityStatus=APPROVED&labels=&manuallyReviewed=true

Some operators can now me used on labels on the same way it used to be applied for keys and biographics:

On a label query parameters:

Operators:

  • labels=[or]label1: applies OR operator with previous label
  • labels=[and]labels1: applies AND operator with previous label (default behavior)

Filter modifiers:

  • Exact: labels=label1 or labels=label1[exact] - search transactions that have exactly the label on parameter
  • At start: labels=label1[atstart] - search transactions with labels that starts with the parameter
  • At end: labels=label1[atend] - search transactions with labels that ends with the parameter
  • Anywhere: labels=label1[anywhere] - search transactions with labels that contains the parameter

[IP-002][ALPHA-1385] Watchdog

Now GBDS Search Engine Driver logs search/enroll being processed every minute when background search are being made on native code.

[IP-003][ALPHA-1386] SmartSense Agent

SmartSense Agent does not duplicate internal calls to each host/port on port sweep endpoint

[IP-004][ALPHA-1395] Ginger Memory Optimization

Ginger matcher microservice was refactored to optimize memory use

[CF-001] New ginger template format for gbds.template.memory.format

A new ginger template format was introduced: OPTIMIZED

When this format is used, ginger matcher microservice will use a new optimized ginger template format internally and will take over control of memory allocation to reduce search performance degradation over time.

On OPTIMIZED format, matcher MS pre allocates all memory for template storage by ginger modality. For that, some configurations must be made on application.conf (and on the file only, not RDB):

[CF-002] Pre allocation configuration

  • gbds.fingerprint.memory-storage.pre-aloc=500mb
  • gbds.palmprint.memory-storage.pre-aloc=500mb
  • gbds.newborn-palmprint.memory-storage.pre-aloc=500mb
  • gbds.ul-fingerprint.memory-storage.pre-aloc=500mb
  • gbds.ul-palmprint.memory-storage.pre-aloc=500mb
Default value for each configuration is 500mb.
The system understands Kilobytes (k, kb), Megabytes (m, mb) and Gigabytes (g, gb), not case sensitive.
The value is for the entire node and divided equally between all ginger matcher microservices on that node.
So, if some given node has, for instance, 3 matchers, and the fingerprint memory tells the system to pre allocate 2400mb, each ginger matcher MS on that node will pre allocate 800mb.

Obs: face and iris storage are still the same

[IP-005] API endpoint /v2/status now each matcher returns a map with memory use and allocation

On OPTIMIZED ginger template format, size returns memory used for templates and total returns all memory allocated by the ginger matcher MS.

On all other ginger template formats and for face and iris, size returns memory used for templates and total return memory allocated by SO for that MS process.

[EP-001] API endpoint /v2/status

{
    "data": {
        "configuredMatchers": 1,
        "peopleCount": 4,
        "ulCount": 0,
        "nodes": [
            {
                "hostname": "gbds-39",
                "monitorPort": 9100,
                "status": "RUNNING",
                "configuredMatchers": 1,
                "peopleCount": 4,
                "ulCount": 0,
                "memory": {
                    "FINGERPRINT": {
                        "size": 37216,
                        "total": 59817984
                    },
                    "FACE": {
                        "size": 8256,
                        "total": 22159360
                    }
                },
                "activeMatchers": [
                    {
                        "name": "matcher-0",
                        "peopleCount": 4,
                        "ulCount": 0,
                        "memory": {
                            "FINGERPRINT": {
                                "size": 37216,
                                "total": 59817984
                            },
                            "FACE": {
                                "size": 8256,
                                "total": 22159360
                            }
                        }
                    }
                ]
            }
        ]
    }
}

[IP-006][ALPHA-1396] SmartSense Agent

Better log format and log rotation were implemented.

[IP-007][ALPHA-1397] Allow API to skip get result transaction saving

Now get result transaction can be configured to not be saved on HBase.

GBDS 4.3.2.1483

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.3.0 or newer

Files released:

  • /var/lib/griaule/gbsapi/gbscluster-api-4.3.2-exec.jar
  • /var/lib/griaule/gbds/gbds-4.3.2.jar
  • /var/lib/griaule/gbds/lib/gbscluster-native-4.3.2.jar
  • /var/lib/griaule/gbds/lib/gbscluster-server-components-4.3.2.jar
  • /var/lib/griaule/gbsapi/scripts/start-api.sh

Remove old versions of these files in order to update.

Obs:

  • Major RDB upgrade from 4.2.x to 4.3.0 provided
  • No RDB Upgrade from 4.3.1.1473
  • Java client and API model 4.3.2 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1433] Fixed memory extraction leak on ginger (it used to get a new extractor per web request) and face (face lib image not destroyed on native)

[BF-002][ALPHA-1434] Fixed on [ALPHA-1285][ALPHA-1365]: Removed unnecessary extraction and matching logs

GBDS 4.3.1.1473

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.3.0 or newer

GBDS release MD5 Checksum: 85b91fc0f8cd725f066364269b44c6c9

GBDS redistribution MD5 Checksum: 310543ee9c831f550a0a3181f89409a2

GBS Smart Sense Agent MD5 Checksum: 2825dd7e13fd235b002ce442f8dc7e9f

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.2.x to 4.3.0 provided
  • No RDB Upgrade from 4.3.0.1470
  • Java client and API model 4.3.1 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1398] Fixed update when reference person uses face template version 1, 1.1 or 2 and update transaction has only images (with multiple face template extraction or only newest face template extraction) or new face template version 3. In these cases, the update failed with an Internal error and the transaction was not processed.

GBDS 4.3.0.1470

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.3.0 or newer

GBDS release MD5 Checksum: 109fbbd47835bc235f853d26a53ab252

GBDS redistribution MD5 Checksum: 36d8bac98bd393652231ab95bf3127a8

GBS Smart Sense Agent MD5 Checksum: 5fdb56792a64305ecacd522f5b9993d0

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.2.x to 4.3.0 provided
  • RDB Upgrade 4.3.0.1470 replaces 4.3.0.1469
  • Java client and API model 4.3.0 re-deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.

Bug Fixes

[BF-001][ALPHA-1371] Fixed upgrade-rdb-4.3.0 script

[BF-002][ALPHA-1373] Fixed search and total time on table gbds.transactions

[BF-003][ALPHA-1376] API endpoints for GBDS Node Monitor with no need of authentication

[BF-004][ALPHA-1377] Reduced INFO logs on GBDS

[BF-005][ALPHA-1378][ALPHA-1379] GBS Node Monitor http timeout request reduced on GET /monitor/status for SmartSenseAgent

[BF-006][ALPHA-1285] Fixed ethernet overload on matchers when using matcher microservice

Improvements

[IP-001][ALPHA-1375] Minimum coincident fingers on enroll

On application.conf, setting changed from gbds.biometric.face.exception.min-finger-matches-to-ignore-face to gbds.biometric.face.exception.minimum.coincident-fingers.ignore.face

Now, on enrollment, exception analysis doesn’t look at minimum fingers matches to ignore face, but does look minimum coincident fingers to ignore face.

[IP-002][ALPHA-1365] Facelib upgrade and face multi template support

New facelib face template format available with more accuracy and more precision.

By default, GBDS will use this new format on extractions and matches, but GBDS can now also work with face templates up to GBDS 4.1. (Face template format labeled TPT_FORMAT_2 from GBDS 4.2 not supported).

To work with both old and new face templates, GBDS will extract a face big template with both formats in it and use it to compare with new or old reference ones, depending on each enroll.

This feature can be turned off and GBDS will then work only with the new face template format.

To configure it, go to gbdsapi.properties or RDB table gbds.settings, depending on whether RDB System Configuration Agent is off or on, respectively.

[CNF-001] On gbdsapi.properties:

gbds.template.face.multiplicity=ONLY_NEWEST

[CNF-002][DB] On gbds.settings:

gbds.template.face.multiplicity, type API, default ONLY_NEWEST

The values are:

  • ONLY_NEWEST only the new face template format is used. Only new enrolls (GBDS 4.3+) can be searched with this option.
  • MULTIPLE API will extract both old and new face template formats from a face biometry in order to enable comparison with old reference enrolls.

GBDS 4.3.0.1469

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.3.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.3.0 or newer

GBDS release MD5 Checksum: 10598c04c56055a901580cba79f6619d

GBDS redistribution MD5 Checksum: f0681ccb674c752cd8f229e19cb10347

GBS Smart Sense Agent MD5 Checksum: f7b124efa7c15c9275ac0397b47f8ae0

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.2.x to 4.3.0 provided
  • Java client and API model 4.3.0 deployed on artifactory.

Bug Fixes

[BF-001][ALPHA-1318] Fixed extensive log when pending enrolls are caught on GBDS boot.

[BF-002][ALPHA-1319] Fixed verify search 1:1 with more than one key on any order.

[BF-003][ALPHA-1321] Fixed extract quality on microservices for ALL images provided.

It used to extract quality only on images without template. Now extracts quality for all images.

  • Fixed on MIR: when performing quality analysis approvement with biometric edition, provided biometric image quality now is copied into new transaction’s biometrics.

[BF-004][ALPHA-1322] Fixed unnecessary validation on biometrics on get person endpoint GET /v2/people

[BF-005][ALPHA-1337] Fixed GBDS boot and enroll/update/trusted people distribution.

GBDS used to send a person biometrics to one matcher on enroll/update and to another matcher on GBDS boot. It used to have one hash distribution on boot and other on enroll/update/trusted and it could result in people duplication on updates after boot. GBDS now send any person biometrics to the same matcher even in boot or enroll/update/trusted.

Performance and recovery not changed.

[BF-006][ALPHA-1339] GBDS log4j upgraded to 2.17.1

According to Log4j - Apache Log4j Security Vulnerabilities log4j needed to be upgraded from 2.16.0 to 2.17.1 due to vulnerabilities.

  • 2.17.0: CVE - CVE-2021-45105
  • 2.17.1: CVE - CVE-2021-44832

[BF-007][ALPHA-1342][ALPHA-1344] On script start-gbds.sh, changed std out redirection from create to append file on console.out, so it can be truncated.

[BF-008][ALPHA-1356] Fixed match and total time on verify

Match time now is calculated only inside verify match process.

Total time was disregarding extraction time, now is calculated over entire verify process, since endpoint request until response.

[BF-0009][ALPHA-1360] Face quality extraction fixes:

  • Fixed face quality extraction when image fails
  • Fixed face quality extraction when enroll has no fingerprints

[BF-0010][ALPHA-1362] Fixed reload configurations on GBDS from API endpoint for enroll configurations.

[BF-0011][ALPHA-1369] Fixed GBDS crash on reverse latent match with no biometrics on update.

Improvements

[IP-001][ALPHA-1288][CNF] Minimum coincident fingers on update

On application.conf, setting changed from gbds.biometric.face.exception.min-finger-matches-to-ignore-face to gbds.biometric.face.exception.minimum.coincident-fingers.ignore.face

Now, on update, exception analysis don’t look to minimum fingers matches to ignore face, but do look minimum fingers coincident fingers to ignore face.

[IP-002][ALPHA-1303][DB] RDB table gbds.transactions upgraded

Added columns on gbds.transactions:

  • latent
  • ul

Now these columns are used to flag latent and UL searches. Not used on enroll/updated transactions.

[IP-003][ALPHA-1315] Multi API control on RDB

[DB] Created RDB table gbds.apis to multi API control.

Table structure:

  • api-id: string
  • hostname: string
  • port: integer
  • type: enum (LEADER, RUNNER or null)

The RDB setting gbds.extraction.quality.api.uniqueId.list, type API, is now obsolete.

Use RDB gbds.apis table to control how many API’s are configured on environment and how many of these API’s can perform quality extraction.

On API boot, each API will look itself on table. If not found with hostname/ip and port, it inserts itself there and assign a GUID as API id, but with no type, meaning not ready for quality extraction. Changes directly on RDB table will be parsed every 15 minutes.

Some endpoints were added to control this table from any API:

List APIs

[EP-001] GET /v2/apis

List APIs:

Returns all APIs configured:

Response:

200 OK:

{
    "data": [
        {
            "apiId": "api1",
            "hostname": "gbds-giolo",
            "port": 8085,
            "type": "LEADER"
        }
    ]
}

[EP-002] POST /v2/apis

Create/Update API:

Created or updates an API on table.

Request:

{
    "data": {
        "apiId": "api2",
        "hostname": "gbds-77",
        "port": 8085,
        "type": "LEADER"
    }
}

Response:

201 CREATED:

{
    "status": "CREATED",
    "data": {
        "apiId": "api2",
        "hostname": "gbds-77",
        "port": 8085,
        "type": "LEADER"
    }
}

202 ACCEPTED:

{
    "status": "UPDATED",
    "data": {
        "apiId": "api2",
        "hostname": "gbds-77",
        "port": 8085,
        "type": "LEADER"
    }
}

Error codes:

  • 400 MISSING_REQUIRED_ENTITY_ATTRIBUTE: API is required
  • 400 MISSING_REQUIRED_ENTITY_ATTRIBUTE: API ID is required
  • 400 MISSING_REQUIRED_ENTITY_ATTRIBUTE: API hostname is required
  • 400 MISSING_REQUIRED_ENTITY_ATTRIBUTE: API port is required

If type is LEADER and other API is already LEADER, other API type changes to RUNNER.

[EP-003] DELETE /v2/apis/{apiId}

Remove API:

Removes an API from table

Response:

  • 204 REMOVED
  • 404 NOT_FOUND

[IP-004][ALPHA-1317] Kill GBDS orphan processes on kill-gbds.sh

If any GBDS matcher process became orphan on GBDS shutdown, kill-gbds.sh will kill it too.

[IP-005][ALPHA-1320] Better log for received endpoints with subject name if exists and query parameters

API used to log received endpoints as:

Endpoint: /gbds/v2/people/pguid

Now logs as:

Endpoint: /gbds/v2/people/pguid?key_id=titulo&key_value=123456

With all query parameters received.

[IP-006][ALPHA-1336] RDB reduce

Many changes were made on API and GBDS Search Engine Drive to reduce RDB access when BoB is off:

  • Enroll/update/trusted

    • Verify if person exists for update is now using HBase
    • Save quality extraction JSON on HBase

  • Get person

    • load from HBase from last transaction if BoB is off
    • Get pguid for person from keys is now using HBase
    • Get quality properties from HBase transactions even if BoB is on

  • Get transaction

    • Load from HBase from last transaction if BoB is off
    • Get quality properties from HBase transactions even if BoB is on

  • Search 1:1

    • Load from HBase from last transaction if BoB is off
    • Save verify transaction on RDB only at the end of the process

  • Search 1:N

    • Do not re consolidate search result (because of master record) if BoB is off
    • Only RDB inserts/updates on GBDS Search Engine, no select access

  • Post match

    • Load from HBase from last transaction if BoB is off

  • People transparency

    • Cached people transparency list on memory, update every 15 minutes or on /v2/people-transparency/list

  • SQL improvements

    • Transaction update directly on RDB
    • Reduced SQL selects before any SQL entity insert/update

  • RDB clear-rdb

    • Removed gbds.transaction_quality column properties_json

  • RDB upgrade-rdb

    • Removed gbds.transaction_quality column properties_json
    • Set finger and face quality extracted flags from 1 to 0 for re-extraction

  • RDB quality background quality extraction

    • Do quality extraction for fingerprint and face at same time
    • Save quality json on HBase transactions

[IP-007][ALPHA-1338][DB] Quality table optimization

RDB tables gbds.transaction_quality and gbds.transaction_quality_properties are no longer being used. So, quality properties that is used to be stored as name-value rows on gbds.transaction_quality_properties are now stored as columns on gbds.transaction_fingerprint_quality for fingerprints and on gbds.transaction_face_quality for faces.

By default, clear-rdb.sql and upgrade-rdb-4.3.0.sql will create these tables with current default fingerprint and face property columns. If any new property appears in the future, quality extractor will automatically create new columns at API runtime (on enroll/update and on background quality extraction too).

[IP-008][ALPHA-1343][DB] RDB table gbds.transactions upgraded

Added columns on gbds.transactions:

  • api_id: store which API receive that transaction to process
  • num_fingers: number of fingerprints on transaction
  • num_faces: number of faces on transaction. Actually, this column could be read as face present or not (0 or 1).

[IP-009][ALPHA-1358] Kafka settings updated

  • Kafka buffer size on poll: gbds.cluster.kafka.max-tasks-per-poll maximum limit updated to 1000
  • Back pressure: gbds.node.max-loaded-tasks maximum limit updated to 20000

[IP-010][ALPHA-1359] Added number of faces to face cost on face quality extraction

[IP-011][ALPHA-1361] When an image fails on any face quality extraction, send enroll to MIR

New Features

[NF-001][ALPHA-1285] Matchers microservices

GBDS now can have matcher microservices for biometric storage and search, resulting in better memory control and better search and boot performance.

It is configured on application.conf or RDB table gbds.settings, depending on whether RDB System Configuration Agent is off or on, respectively.

[CNF-001] On application.conf:

  • gbds.match.service.enabled=false
  • gbds.match.service.initialPort=32000
  • gbds.match.service.logLevel=NONE
  • gbds.match.service.maxTries=3
  • gbds.match.service.timeout=10000

[CNF-002][DB] On gbds.settings:

  • gbds.match.service.enabled, type GBDS, default false
  • gbds.match.service.initialPort, type GBDS, default 32000
  • gbds.match.service.logLevel, type GBDS, default NONE
  • gbds.match.service.timeout, type GBDS, default 10000
  • gbds.match.service.maxTries, type GBDS, default 3

Each configuration means:

  • gbds.match.service.enabled: Flag to turn matcher microservices on/off. If off, GBDS will perform as in previous versions.

  • gbds.match.service.initialPort: Initial port on node to start match services.

  • gbds.match.service.logLevel: Match services log level on GBDS.

    • NONE logs nothing.
    • INFO logs scripts and request urls.
    • TIME logs scripts, requests urls and time elapsed.
    • DEBUG logs it all and turn on service log tail.

  • gbds.match.service.timeout: Match services request timeout (seconds).

  • gbds.match.service.maxTries: Match services maximum tries to fail a store/match call.

Match microservices do not have a hostname configuration, it is always localhost and GBDS node always controls all start/stop match microservice.

GBDS has RDB setting gbds.driver.logLevel as well that, even if matcher logLevel is NONE, it can log entire person/UL boot. If both logLevel and gbds.driver.logLevel are false, no person/UL boot is logged.

At GBDS node boot start, it will kill all match services process on node and starts as many as necessary matcher microservices (using initial port configured) according to the number of now logical matchers configured on application.conf:

  • gbds.node.matchers.number=xx

Each logical matcher starts three matcher microservices:

  • ginger, storing:

    • fingerprints
    • palmprints
    • new born palmprints
    • UL fingerprints
    • UL palmprints

  • facelib, storing

    • faces

  • girl, storing

    • irises

Kill script kill-gbds.sh now kills every match microservice process that remains active after GBDS node shutdown.

[NF-002][ALPHA-1341] GBDS Node Monitor

Each GBDS Search Engine Node is now also a small web service that can monitor itself.

[CNF-001] Default port is 9100 but it can be configured at (and only at, not on RDB) application.conf:

gbds.monitor.port=9100

Now each node has these endpoints:

  • GET /monitor/status/driver
  • GET /monitor/status
  • GET /monitor/version

[EP-001] GET /monitor/status/driver

Status driver

Response:

200 OK:

{
    "data": {
        "configuredMatchers": 4,
        "nodes": [
            {
                "hostname": "gbds-39",
                "monitorPort": 9100,
                "configuredMatchers": 4
            }
        ]
    }
}

[EP-002] GET /monitor/status

Status

Response:

200 OK:

{
    "data": {
        "hostname": "gbds-39",
        "monitorPort": 9100,
        "status": "RUNNING",
        "configuredMatchers": 4,
        "peopleCount": 28,
        "ulCount": 0,
        "activeMatchers": [
            {
                "name": "matcher-0",
                "peopleCount": 7,
                "ulCount": 0
            },
            {
                "name": "matcher-1",
                "peopleCount": 6,
                "ulCount": 0
            },
            {
                "name": "matcher-2",
                "peopleCount": 8,
                "ulCount": 0
            },
            {
                "name": "matcher-3",
                "peopleCount": 7,
                "ulCount": 0
            }
        ]
    }
}

[EP-003] GET /monitor/version

Version

Response:

200 OK:

{
    "data": {
        "version": "5.0.0-SNAPSHOT",
        "build": "5000"
    }
}

GBDS node now logs each step on boot and log status:

  • NONE
  • SPRING_START
  • CLUSTER_ASSEMBLY
  • MATCHERS_ASSEMBLY
  • PEOPLE_BOOT
  • UL_BOOT
  • KAFKA_START
  • RUNNING
  • SHUTTING_DOWN

On RUNNING status, logs elapsed time to perform entire boot.

API endpoint changes:

  • GET /v2/status/driver
  • GET /v2/status
  • GET /v2/version

Those endpoints were changed to use monitor, not kafka, to request node information

[CNF-002] For API, GBDS Node Monitor can be configured on gbds.settings:

gbds.monitor.url, type API, default 'http://<some-gbds-cluster-node-hostname>:9100'

[EP-004] GET /v2/version

Version endpoint

Use one monitor to retrieve all node hosts and check version on all nodes

Response:

200 OK:

{
    "data": {
        "api": {
            "version": "5.0.0-SNAPSHOT",
            "build": "5000",
            "apiId": "api1"
        },
        "searchEngine": {
            "version": "5.0.0-SNAPSHOT",
            "build": "5000"
        }
    }
}

[EP-005] GET /v2/status

Status endpoint

Use one monitor to retrieve all node hosts and retrieve status from each one

Response:

200 OK:

{
    "data": {
        "configuredMatchers": 4,
        "peopleCount": 28,
        "ulCount": 0,
        "nodes": [
            {
                "hostname": "gbds-39",
                "monitorPort": 9100,
                "status": "RUNNING",
                "configuredMatchers": 4,
                "peopleCount": 28,
                "ulCount": 0,
                "activeMatchers": [
                    {
                        "name": "matcher-0",
                        "peopleCount": 7,
                        "ulCount": 0
                    },
                    {
                        "name": "matcher-1",
                        "peopleCount": 6,
                        "ulCount": 0
                    },
                    {
                        "name": "matcher-2",
                        "peopleCount": 8,
                        "ulCount": 0
                    },
                    {
                        "name": "matcher-3",
                        "peopleCount": 7,
                        "ulCount": 0
                    }
                ]
            }
        ]
    }
}

[EP-006] GET /v2/status/driver

Driver status endpoint

Retrieve all node hosts

Response:

200 OK:

{
    "data": {
        "configuredMatchers": 4,
        "nodes": [
            {
                "hostname": "gbds-39",
                "monitorPort": 9100,
                "configuredMatchers": 4
            }
        ]
    }
}

[NF-003][ALPHA-1345][EP] API endpoint for transaction report

Added new API endpoint:

GET /v2/people/transactions/report

Query parameters:

  • start: initial date on period, optional.
    Format: yyyy-MM-dd HH:mm:ss
    If absent, API uses one month ago from request timestamp.
    If present, date is set to start of the day.

  • end: final date on period, optional.
    Format: yyyy-MM-dd HH:mm:ss
    If absent, API uses current request timestamp.
    If present, date is set to end of the day.

  • byApiId: flag to group results by API or not.
    This endpoint groups transaction information using RDB table gbds.transactions, giving transaction count, average time for extractions, search and total operation time by transaction type and API that received that transaction (check API load balance)

Response

200 OK:

{
    "data": [
        {
            "count": 28,
            "extractionTimeAvg": 602.0714,
            "extractionTimeMin": 199,
            "extractionTimeMax": 4744,
            "extractionQualityAvg": 247.6786,
            "extractionQualityMin": 203,
            "extractionQualityMax": 317,
            "matchAvg": 180.0714,
            "matchMin": 0,
            "matchMax": 1732,
            "totalAvg": 1424.3571,
            "totalMin": 605,
            "totalMax": 7215,
            "type": "ENROLL",
            "apiId": "api1"
        },
        {
            "count": 50,
            "extractionTimeAvg": 141.42,
            "extractionTimeMin": 108,
            "extractionTimeMax": 264,
            "extractionQualityAvg": 0.0,
            "extractionQualityMin": 0,
            "extractionQualityMax": 0,
            "matchAvg": 319.88,
            "matchMin": 0,
            "matchMax": 2187,
            "totalAvg": 736.62,
            "totalMin": 162,
            "totalMax": 3492,
            "type": "IDENTIFY",
            "latent": false,
            "ul": false,
            "apiId": "api1"
        },
        {
            "count": 6,
            "extractionTimeAvg": 125.6667,
            "extractionTimeMin": 115,
            "extractionTimeMax": 160,
            "extractionQualityAvg": 0.0,
            "extractionQualityMin": 0,
            "extractionQualityMax": 0,
            "matchAvg": 54.0,
            "matchMin": 0,
            "matchMax": 138,
            "totalAvg": 490.8333,
            "totalMin": 190,
            "totalMax": 1409,
            "type": "IDENTIFY",
            "latent": true,
            "ul": false,
            "apiId": "api1"
        },
        {
            "count": 8,
            "extractionTimeAvg": 1806.0,
            "extractionTimeMin": 190,
            "extractionTimeMax": 4840,
            "extractionQualityAvg": 240.5,
            "extractionQualityMin": 207,
            "extractionQualityMax": 278,
            "matchAvg": 76.875,
            "matchMin": 0,
            "matchMax": 353,
            "totalAvg": 3337.75,
            "totalMin": 714,
            "totalMax": 7011,
            "type": "UPDATE",
            "apiId": "api1"
        },
        {
            "count": 25,
            "extractionTimeAvg": 495.12,
            "extractionTimeMin": 164,
            "extractionTimeMax": 4682,
            "extractionQualityAvg": 0.0,
            "extractionQualityMin": 0,
            "extractionQualityMax": 0,
            "matchAvg": 176.68,
            "matchMin": 58,
            "matchMax": 943,
            "totalAvg": 226.96,
            "totalMin": 75,
            "totalMax": 1412,
            "type": "VERIFY",
            "latent": false,
            "ul": false,
            "apiId": "api1"
        }
    ]
}

Fields on each object on array:

  • Grouping properties

    • type: type of transactions groups
    • latent: true if search was a latent. false if not; absent if it was an enroll/update
    • ul: true if search was a UL search. false if not; absent if it was an enroll/update
    • apiId: API id that received those transactions

  • Aggregation properties

    • count: transaction count on period
    • extractionTimeAvg: average extraction template time on transactions on period, in milliseconds
    • extractionTimeMin: minimum extraction template time on transactions on period, in milliseconds
    • extractionTimeMax: maximum extraction template time on transactions on period, in milliseconds
    • extractionQualityAvg: average extraction quality time on period, in milliseconds
    • extractionQualityMin: minimum extraction quality time on period, in milliseconds
    • extractionQualityMax: maximum extraction quality time on period, in milliseconds
    • matchAvg: average search time on period, in milliseconds
    • matchMin: minimum search time on period, in milliseconds
    • matchMax: maximum search time on period, in milliseconds
    • totalAvg: average total time on period, in milliseconds
    • totalMin: minimum total time on period, in milliseconds
    • totalMax: maximum total time on period, in milliseconds

If parameter byApiId is true, endpoint will group results by API id; if it is false or absent, apiId property is not returned and results reflects all transactions on that period.

[NF-004][ALPHA-1355] Kafka check.

Added new script /var/lib/griaule/gbds/scripts/kafka-check.sh to check Kafka on node.

It will only start kafka on node and exit, informing if kafka connection is ok on that node. This GBDS boot mode do not check/waits for cluster assembly, do not start matchers and do not boot people or UL.

[NF-005][ALPHA-1347] GBS Smart Sense Agent

GBS suite now have an agent running on every node to monitor GBDS servers called GBS Smart Sense Agent. This agent provides information on any GBS node to GBS Smart Sense.

Binaries found on

/var/lib/griaule/gbs-smartsense-agent/gbs-smartsense-agent-<version>-exec.jar

Scripts:

  • /var/lib/griaule/gbs-smartsense-agent/scripts/smartsense-status.sh
  • /var/lib/griaule/gbs-smartsense-agent/scripts/start-smartsense.sh
  • /var/lib/griaule/gbs-smartsense-agent/scripts/stop-smartsense.sh
  • /var/lib/griaule/gbs-smartsense-agent/scripts/tail-smartsense.sh

Log is found on

  • /var/log/griaule/gbs-smartsense-agent/console.out
  • /var/log/griaule/gbs-smartsense-agent/smartsense.log

Some alias to help:

  • alias smartsenseconf='vim /etc/griaule/conf/gbs-smartsense-agent/application.properties'
  • alias smartsenselogt='/var/lib/griaule/gbs-smartsense-agent/scripts/tail-smartsense.sh'
  • alias smartsensestart='/var/lib/griaule/gbs-smartsense-agent/scripts/start-smartsense.sh'
  • alias smartsensestatus='/var/lib/griaule/gbs-smartsense-agent/scripts/smartsense-status.sh'
  • alias smartsensestop='/var/lib/griaule/gbs-smartsense-agent/scripts/stop-smartsense.sh'

All alias specified on /etc/griaule/conf/gbs-smartsense-agent/suggestedaliases.txt

Multi API configuration

Smart sense Agent uses a json to verify multiple APIs on the same host:

/etc/griaule/conf/gbs-smartsense-agent/multiapi.json

Format:

{
    "number-of-configured-apis": 2,
    "ports": [
         8085, 8086
    ]
}

Implementations can use this file to get how many APIs must go up and on which ports.

Endpoints

[EP-001] GBS Services

Retrieves information from GBS products

  • Bravo products

    • MIR
    • BCC
    • BEST
    • CardScan
    • etc

  • GBDS API (one or more than one per node)

  • GBDS Search Engine

POST /gbs-services

Request body:

List of GBS services to be retrieved with

  • name
  • port
  • version endpoint
  • installation path
  • configuration path

Example:

{
    "gbs-services": [
        {
            "name": "MIR",
            "port": 7777,
            "version-endpoint": "/v3/version",
            "installation-path": "/var/lib/tomcats/mir",
            "conf-path": "/var/lib/tomcats/mir/conf/config.properties"
        }
    ]
}

Response

200 OK:

{
    "gbs-services": [
        {
            "name": "MIR",
            "port": 7777,
            "version": "4.3.0",
            "installed": "true",
            "status": "online",
            "memory-usage" : "152045025"
        }
    ]
}

For GBDS APIs, multi APIs on host can be configured on

/etc/griaule/conf/gbs-smartsense-agent/multiapi.json

JSON Example:

{
    "number-of-configured-apis": 2,
    "ports": [
         8085, 8086
    ]
}

[EP-002] Node Resources

Retrieves resource information from current node.

GET /node-resources

Response:

200 OK:

{
    "total-ram": "123456",
    "used-ran": "123456",
    "avalable-ram": "123456",
    "number-of-cores": 8,
    "temperatures": [
        {
            "id": "Package 0",
            "current": "+69.0",
            "high": "+80.0",
            "crit": "+100.0"
        },
        {
            "id": "Core 0",
            "current": "+56.0",
            "high": "+80.0",
            "crit": "+100.0"
        }
    ],
    "total-disk": "123456",
    "used-disk": "123456",
    "available-disk": "123456",
    "disk-partitions": [
        {
            "filesystem": "/dev/sda5",
            "size": "123456",
            "used", "123456",
            "available": "123456",
            "use%": "90%",
            "mount": "/"
        }
    ],
    "hfds": {
        "filesystem": "hfds://gbds-35:8020",
        "size": "123456",
        "used": "123456",
        "available": "123456",
        "use%": "14%"
    },
    "network-card-uptime": "Wed Oct 19 21:50:00 BRT 2022",
    "throughtput-rx": "5678.0",
    "throughtput-tx": "2145.0",
    "bandwidth": null
}

[EP-003] Configuration Files

Generates a tar.gz with configuration files

POST /configuration-files

Request:

{
    "base-conf-path": "/etc/griaule/conf",
    "individual-files": [
        "/etc/griaule/conf/gbds/application.conf",
        "/etc/griaule/conf/gbsapi/gbdsapi.properties"
    ]
}

Obs: use base conf path or individual files.

Response:

200 OK: binary tar.gz file

[EP-004] Port Sweep

Sweeps ports and returns their status

POST /port-sweep

Request:

  • hostnames: list of hostnames to sweep ports
  • ports: list of ports on each hostname to sweep. Each field here can be a single port or a range of ports (use -)
{
    "hostnames": [
        "localhost"
    ],
    "ports": [
        "8080-8083",
        "9000"
    ]
}

Response:

200 OK:

{
    "results": [
        {

            "hostname": "localhost",
            "ports": [
                {
                    "port-no": 8080,
                    "status": "PORT_LISTENING"
                },
                {
                    "port-no": 8081,
                    "status": "PORT_CLOSED"
                },
                {
                    "port-no": 8082,
                    "status": "PORT_CLOSED"
                },
                {
                    "port-no": 8083,
                    "status": "PORT_CLOSED"
                },
                {
                    "port-no": 9000,
                    "status": "PORT_CLOSED"
                }
            ]
        }
    ]
}

[EP-005] Matcher Status

Gets the expected number of matchers and lists online matchers

GET /matcher-status

Response

200 OK:

{
    "configured-matchers": 1,
    "matches": [
        {
            "name": "matcher-0"
        }
    ]
}

[EP-006] Load Balancing

Gets information on API load balancing

  • By api id
  • By transaction type
  • By Latent / UL

GET /load-balancing

Query parameters

  • start-time: start time in UNIX epoch long. If missing, sets to one month ago
  • end-time: end time in UNIX epoch long. If missing, sets to current time
  • by-api-id: boolean, breaks result by API

Response:

200 OK:

{
    "data": [
        {
            "count": 25,
            "extractionTimeAvg": 125.6667,
            "extractionQualityAvg": 0.0,
            "matchAvg": 54.0,
            "totalAvg": 490.8333,
            "type": "IDENTIFY",
            "latent": true,
            "ul": false,
            "apiId": "api1"
        }
    ]
}

GBDS 4.2.0.1466

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.2.0 or newer

GBDS release MD5 Checksum: 4fa1bf0f9a250aded23566a99e3e6344

GBDS redistribution MD5 Checksum: 1f325e04512147f53ef9e57c19b041b0

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.2.0 re deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.2.0.1464

Bug Fixes

[BF-001][ALPHA-1301] Fixed set manually reviewed to true on quality control approval with edition

[BF-002][ALPHA-1304] Fixed bad image returning 500 on face template extractors.

This bug used to make template extractors to be restarted at each bad image passed.

Beside that, SO could hold the port that an extractor is using on restart, so a delay was added on the extractor restart process.

[BF-003][ALPHA-1305] Fixed wrong finger global quality on transactions with fingerprints and faces.

[BF-004][ALPHA-1306] Fixed update with no keys on enroll with no keys.

[BF-005][ALPHA-1307] Fixed wrong enumeration values message when validating enumerations on endpoints.

[BF-006][ALPHA-1310] Fixed quality extraction on background distribution for ALL API’s, not only LEADER.

[BF-007][ALPHA-1311] Fixed read properties from RDB setting table if RDB configuration agency is on and read properties only from file if RDB configuration agency is off

Fixed on template and quality extractions too (sync and background)

[BF-008][ALPHA-1314] Fixed quality control with edition losing external id’s from pending transaction to new transaction.

GBDS 4.2.0.1464

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.2.0 or newer

GBDS release MD5 Checksum: 8968313c5ba345760f96a7f1732d2cb7

GBDS redistribution MD5 Checksum: 8f6c79180aa6c0176140a8ef91bceb75

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.2.0 re deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.2.0.1463

Bug Fixes

[BF-001][ALPHA-1286] Fixed trusted enroll with no biometric on endpoint PUT /v2/people/{pguid}/trusted with BoB enabled

[BF-002][ALPHA-1287] Fixed empty candidate list for latent identify search

[BF-003][ALPHA-1296] Fixed identify/verify with save search information on RDB turned off

[BF-004][ALPHA-1297] Better error message when GBDS Driver is offline on endpoint GET /v2/version

[BF-005][ALPHA-1299] MAJOR FIX: Post matching was looking up templates on the last transaction only. Fixed to look up for templates on master record on RDB, not only last transaction

GBDS 4.2.0.1463

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.2.0 or newer

GBDS release MD5 Checksum: ca7b4bf452b3d49ff96b3887ec41a550

GBDS redistribution MD5 Checksum: 5c01f443f71fe4baf05056ab21f5f75e

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.2.0 re deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.2.0.1461

Bug Fixes

[BF-001][ALPHA-1281] Fixed re enroll of deleted person or refused transaction (after exception treatment)

[BF-002][ALPHA-1284] Fixed post matching re crop on candidates with score below threshold on latent searches, not only on UL searches

GBDS 4.2.0.1461

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.2.0 or newer

GBDS release MD5 Checksum: 0c360f4896329ab9df240e5faa499bcf

GBDS redistribution MD5 Checksum: 6e898c35069f4f882ca3b1523133c34a

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.2.0 re deployed on artifactory. It might be necessary to erase local ~/.m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.2.0.1460

Bug Fixes

[BF-001][ALPHA-1280] Fixed trusted enroll meta on GBDS Java Client to include on json only not null values.

On TrustedEnrollMeta and TrustedEnrollMetaBuilder, labels renamed to labelsOperation. Json itself is not changed.

GBDS 4.2.0.1460

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: b349f7a6073dd9cebae2bbd003845c54

GBDS redistribution MD5 Checksum: b5c31544483519be3e7c5d6c65c3a2a2

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.2.0 deployed on artifactory.
  • No RDB upgrade needed from 4.2.0.1458.

Bug Fixes

[BF-001][ALPHA-1277] Fixed CRLF on extractor micro service scripts

[BF-002][ALPHA-1278] Fix get person with key and get person pguid with key directly on HBase, not using RDB

On endpoints:

/v2/people/{expression}: expression can be a PGUID or key:value; If it is key:value, it looks directly on HBase to retrieve pguid and then retrieve person from RDB.

/v2/people/pguid?key_id=<id>&key_value=<value>: key is now retrieved from HBase and not from RDB.

[BF-003][ALPHA-1279] Fixed extraction services startup.

Fixed tensorflow lib import location to /var/lib/griaule/tensorflow (looking up for version 2, not version 1 anymore) that was preventing extraction services to initialize

GBDS 4.2.0.1458

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: b349f7a6073dd9cebae2bbd003845c54

GBDS redistribution MD5 Checksum: b5c31544483519be3e7c5d6c65c3a2a2

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.2.0 deployed on artifactory.
  • There’s a RDB upgrade from 4.1.x

Bug Fixes

[BF-001][ALPHA-1275] Fixed get transaction and get people endpoint with enrollFields that excludes person from response returning internal error

[BF-002][ALPHA-1276] Fixed enroll with high load on cluster with more than one node

Improvements

[IP-001][ALPHA-1200] Choose template serialization type that GBDS will retain on memory

Now you can choose how fingerprint templates are stored on memory on each matcher when GBDS starts.

There are three options:

  • DESERIALIZED_FULL - Deserialized: This is the default option, that used to be the only option until now. Templates are stored on matchers deserialized on memory with minutiae and segments.
  • SERIALIZED_MINUTIAE_SEGMENTS - Serialized with minutiae and segments: Templates are stored on matchers serialized on memory with minutiae and segments.
  • SERIALIZED_MINUTIAE - Serialized with minutiae: Templates are stored on matchers serialized on memory with minutiae only. Segments are re-extracted from template on each search made. This affects search (identify) time.

Configurations

This configuration can be done on properties file or RDB table. Mainly this settings are applied using RDB table, with RDB Configuration agent on.

  • On application.conf:

gbds.template.memory.format=DESERIALIZED_FULL or SERIALIZED_MINUTIAE_SEGMENTS or SERIALIZED_MINUTIAE

Note

DESERIALIZED_FULL is DEFAULT if absent

  • On RDB table gbds.settings: (With RDB Configuration Agent turned on)

gbds.template.memory.format, type GBDS, value default DESERIALIZED_FULL

Obs: The use of this setting on RDB table will only take effect after GBDS boot.

[IP-002][ALPHA-1264] Multiple API background quality extraction

Quality extraction on background now can be made not only with one API, but with as many as configured.

On all configured API’s, one of then is designed to distribute transactions to have quality extraction made and all of then can perform quality extraction on transactions assigned to each API.

The distributor API has type LEADER and all API’s that runs quality extraction on background has type RUNNER.

The API’s to be used are defined using API id. API ids are defined on gbdsapi.properties, on setting: api.uniqueId. If an API doesn’t have this setting, an id (random UUID) is automatically created.

RDB changes

RDB table gbds.transactions has now a new column quality_extraction_api_id that is the API id that will perform or performed quality extraction.

Configuration

RDB setting on gbds.settings was changed from gbds.extraction.quality.api.uniqueId to:

gbds.extraction.quality.api.uniqueId.list, type API, value default empty

This setting is now described as: API ids that will process quality extraction on background. Could be comma separated (first is leader) or json: [{“apiId”:”<api_id>”,”type”:”LEADER|RUNNER”}]

The setting value could be API ids comma separated list or a json.

  • Comma separated: First API id is the LEADER and all others are RUNNER.
  • JSON:
[
    {
        "apiId":"<api_id>",
        "type":"LEADER|RUNNER"
    }
]
  • apiId is always required
  • type is optional and defined as LEADER if no other leader in json is defined yet or then is RUNNER
  • Only one LEADER can be defined.
  • apiId cannot be empty

The quality background extraction is made by modality if these settings are turned on:

gbds.fingerprints.extraction.quality.background

gbds.faces.extraction.quality.background

The extraction process operates on all gbds.transactions rows that have columns finger_quality_extracted and face_quality_extracted with value 0, using setting above.

After extraction, these columns are updated to 1.

Note

If you want to re-extract quality on certain transaction, update back these two columns to 0 and set to null column quality_extraction_api_id so a new API grabs the extraction process.

If you don’t want ever that a transaction has quality extraction, set these two columns to null.

[IP-003][ALPHA-1265] Trusted update with no biometric

Now a trusted updated can be made with no biometrics if BoB (Best of Biometrics) is turned on.

If BoB is off, the trusted enroll is denied as it used to be.

[IP-004][ALPHA-1268] Face template format configuration

The GBDS can now work with new face template format, known as format 2.

By now, GBDS can work with legacy face template format (known format 1) OR new format 2. It’s not interchangeable, ie, faces enrolled with face template format 1 cannot be searched with faces extracted with face template format 2 and vice versa.

Configuration

The face template format can be configured only on application.conf and gbdsapi.properties:

gbds.biometric.face.template.format=TPT_FORMAT_1 or TPT_FORMAT_2

Note

Both API and GBDS configurations for face template format must be the same.

[IP-005][ALPHA-1269] Possibility to turn on/off saving search transactions on RDB

Since GBDS 4.1 all searches can be stored on RDB on table gbds.transactions having type VERIFY or IDENTIFY.

Now you can turn off this feature.

Configuration

RDB settings on gbds.settings are added:

gbds.searches.verify.saveOnRdb.enabled, type BOTH, value default true

gbds.searches.identify.saveOnRdb.enabled, type BOTH, value default true

These settings can be changed on real time using endpoint POST /v2/configuration:

{
    "application" : "GBDS_API_AND_DRIVER",
    "configurations" : {
        "gbds.searches.verify.saveOnRdb.enabled" : "false",
        "gbds.searches.identify.saveOnRdb.enabled" : "false"
    }
}

Note

With RDB Configuration Agent off, these settings can be set directly on application.conf and gbdsapi.properties.

[IP-006][ALPHA-1272] Added new endpoint /v2/version

The endpoint GET /v2/version was added:

  • Response:

    • 200 OK:
{
    "data": {
        "api": {
            "version": "4.2.0",
            "build": "1457",
            "apiId": "api1"
        },
        "searchEngine": {
            "version": "4.2.0",
            "build": "1457"
        }
    }
}
  • If Driver is down, only API version is informed and an error is returned along with api info:
"error": {
    "type": "INTERNAL_ERROR",
    "code": "DRIVER_OFFLINE",
    "message": "Enroll task is taking too long"
}

[IP-007][ALPHA-1273] Face quality threshold for Quality Control

Quality Control can now verify face image quality threshold on enroll/update.

When an enroll or update has a face and its image quality is below face quality threshold, the transaction is sent to MIR.

Configuration

The threshold is configured on RDB setting:

gbds.enroll.face.min.quality, type API, value default 50

gbds.update.face.min.quality, type API, value default 50

This feature is only enabled when quality extraction is turned on and made by API, with RDB settings on gbds.settings:

gbds.extraction.quality.service

gbds.faces.extraction.quality.api

These settings must be true for the face quality threshold to be considered.

Again, all settings can be changed on real time with configuration endpoint POST /v2/configuration:

{
    "application" : "GBDS_API",
    "configurations" : {
        "gbds.extraction.quality.service" : "true",
        "gbds.faces.extraction.quality.api" : "true",
        "gbds.enroll.face.min.quality" : "65",
        "gbds.update.face.min.quality" : "65"
    }
}

Note

With RDB Configuration Agent off, these settings can be set directly on gbdsapi.properties.

GBDS 4.1.12.1481

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: 3571418a39fb9d1bc716ef64190962c1

GBDS redistribution MD5 Checksum: 4102734cbac28cada4b07cf9d6f3d9bf

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • Java client and API model 4.1.12 deployed on artifactory.
  • No RDB upgrade needed since 4.1.7

Improvements

[IP-001][ALPHA-1421] Inconsistent keys improvements

Some key validation were implemented:

  • GBDS API does not allow duplicated key ids on enroll/update anymore

  • Inconsistent keys on update now generates two kinds of exception:

    • Update with inconsistent keys (1:1)

      Occurs when the entrant has a key id that already belongs to reference person but with a different value (this key id/value is not found on base). On treatment:

      • Only APPROVE or REJECT
      • APPROVE activate entrant and goes to memory
      • REJECT leaves as it is

    • Update with multiple people loading (1:N):

      Occurs when the entrant keys points to multiple people. Ex: entrant with keys cpf:x and rg:y and there is already a person with cpf:x and OTHER person with rg:y. On treatment:

      • Only APPROVE or REJECT
      • Only one treatment can be APPROVE
      • If all involved exceptions are treat as REJECT, existing people are leaved as it is
      • If there’s one and only one APPROVE exception treatment then API activates the entrant, updates the reference with all info (keys, biographics, biometrics, labels), entrant goes to memory and all people other than the one approved is deactivated and removed from memory (but not deleted)

GBDS 4.1.7.1468

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: 1230f900e20fa4308e3814f24fc8e91b

GBDS redistribution MD5 Checksum: 5929a34fd474ec8d33570652a7dac8d4

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 deployed on artifactory.
  • No RDB upgrade needed from 4.1.6.1467

Bug Fixes

[BF-001][ALPHA-1322] Fixed unnecessary validation on biometrics on get person

GBDS 4.1.6.1467

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: 1230f900e20fa4308e3814f24fc8e91b

GBDS redistribution MD5 Checksum: 5929a34fd474ec8d33570652a7dac8d4

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 deployed on artifactory.
  • No RDB upgrade needed from 4.1.5.1465

Bug Fixes

[BF-001][ALPHA-1319] Fixed search 1:1 with keys: any key in any order can be used to find the person being verified

GBDS 4.1.5.1465

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: 760ad63364799c77b3d559ccc6d704c9

GBDS redistribution MD5 Checksum: 9b11a294fe2477689c997b97224b5dbf

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 deployed on artifactory.
  • No RDB upgrade needed from 4.1.4.1462

Bug Fixes

[BF-001][ALPHA-1284] REVERTED fix: post matching re crop on candidates with score below threshold on latent searches, not only on UL searches

GBDS 4.1.4.1462

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: 15117e620d46ff56d97d2afb7055877b

GBDS redistribution MD5 Checksum: 020a88a40535667521a4ea7f21b8cdca

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 deployed on artifactory.
  • No RDB upgrade needed from 4.1.3.1459

Bug Fixes

[BF-001][ALPHA-1281] Fixed re enroll of deleted person or refused transaction (after exception treatment)

[BF-002][ALPHA-1284] Fixed post matching re crop on candidates with score below threshold on latent searches, not only on UL searches

GBDS 4.1.3.1459

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: 03ef76f5c9596bca1fb3f38596f47870

GBDS redistribution MD5 Checksum: 86e69d120e89383f9745e78df8dc20d7

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

Major RDB upgrade from 4.0.x to 4.1.x provided RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided. No RDB upgrade needed from 4.1.1.1456

Bug Fixes

[BF-001][ALPHA-1277] Fixed CRLF on extractor micro service scripts

[BF-002][ALPHA-1278] Fix get person with key and get person pguid with key directly on HBase, not using RDB

On endpoints:

/v2/people/{expression}: expression can be a PGUID or key:value; If it is key:value, it looks directly on HBase to retrieve pguid and then retrieve person from RDB.

/v2/people/pguid?key_id=<id>&key_value=<value>: key is now retrieved from HBase and not from RDB.

GBDS 4.1.2.1457

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: e9c8c7db059c78880104968f377b9337

GBDS redistribution MD5 Checksum: bc5564454c501da079ae618bd3fc8090

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 redeployed on artifactory. It might be necessary to erase local .m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.1.1.1456

Bug Fixes

[BF-001][ALPHA-1276] Fixed enroll with high load on cluster with more than one node

GBDS 4.1.1.1456

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: fdcdfbe868a33f69f7c5cc9e2d288bbe

GBDS redistribution MD5 Checksum: ed390d57da70ed89b1513c146a8cd768

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 re deployed on artifactory. It might be necessary to erase local .m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.1.1.1455

Bug fixes

[BF-001][ALPHA-1271] Fixed saving manually reviewed information on gbds.transactions table when connected to MariaDB

GBDS 4.1.1.1455

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.1 or newer

GBDS release MD5 Checksum: a8215dec547c93aabc5c12a7f22b1193

GBDS redistribution MD5 Checksum: 0e37d0823fb382a78e8dbfedddc4aa1e

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 re deployed on artifactory. It might be necessary to erase local .m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.1.1.1454

Bug Fixes

[BF-001][ALPHA-1258] Quality analysis approval with template edition (biometrics provided on approvement) was not setting manually reviewed flag to true. Fixed.

[BF-002][ALPHA-1266] API does not log stack trace anymore on get people endpoint /v2/people when person is not found

GBDS 4.1.1.1454

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: f6ef9f73958819ad4fd38e5eaa271d38

GBDS redistribution MD5 Checksum: 16a96ad2e0ba140407726391ba7cfdc3

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

Obs:

  • Major RDB upgrade from 4.0.x to 4.1.x provided
  • RDB upgrade from 4.1.0.1452 up to 4.1.1.1453 provided.
  • Java client and API model 4.1.1 re deployed on artifactory. It might be necessary to erase local .m2/repository folder for this version to use it.
  • No RDB upgrade needed from 4.1.1.1453

Bug fixes

[BF-001][ALPHA-1258] GBDS Client list transaction call with manually reviewed parameter on ListEnrollTransactionsRequest fixed

GBDS 4.1.1.1453

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: 68b898d5558fb07719b4f0205a2f48ca

GBDS redistribution MD5 Checksum: cb394c7a99d81812859221f74673c3db

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.1 deployed on artifactory.

There is a RDB upgrade from 4.1.0.1452 to 4.1.1.1453

Improvements

[IP-001][ALPHA-1258] List transactions for MIR with parameter manuallyReviewed.

Endpoint /v2/people/transactions can now have query parameter manuallyReviewed, true, false or null/absent.

Transactions are now manually reviewed if they once have the status PENDING (caught by quality control) and they were approved or rejected.

Bug Fixes

[BF-001][ALPHA-1259] Fixed bug on UL latent search post match where it used to show candidates below threshold

Obs: UL latent searches with post match first does regular search and then applies post matching. So, if a candidate is cropped by score on regular search, even if post match could elevate this score above threshold, this candidate will not be shown. Operator must lower the threshold to get this candidate.

[BF-002][ALPHA-1260] Fixed HBase connection being lost on close enroll or search calls

[BF-003][ALPHA-1263] Fixed update notifications not being sent after a successful update (enrolled or exception)

GBDS 4.1.0.1452

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: 51baed2581fab4b500dc0d5ade1d0e35

GBDS redistribution MD5 Checksum: d4ffd825419d77502a1bf3486d869e3b

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.0 deployed on artifactory.

No RDB upgrade need from 4.1.0.1447 up to 4.1.0.1452

Bug Fixes

[BF-001][ALPHA-1238] Fixed update exception treatment for specific event sequence: same finger, delete profile, different fingers

GBDS 4.1.0.1451

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: ab543d39892dc8897c6330a211d3aeb8

GBDS redistribution MD5 Checksum: 2fcb94ebb843680be9c8f76084007bb1

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.0 deployed on artifactory.

No RDB upgrade need from 4.1.0.1447 up to 4.1.0.1451

Bug Fixes

[BF-001][ALPHA-1230] Fixed minimum extractors on remove extractor endpoint: For templates, at least one extractor must remain. For quality, all extractors can be removed.

[BF-002][ALPHA-1231] Fixed extractor issues messages on update endpoint when no extractor is available for a given biometric type

[BF-003][ALPHA-1232] Better error message on API start when checking extractors (template or quality) running on same port

[BF-004][ALPHA-1234] Fixed extractor client service order to be the same every API boot

[BF-005][ALPHA-1235] Fixed zero resolution attribute on biometric properties when extracting templates with quality extraction turned on

[BF-006][ALPHA-1236] Fixed pguid change unnecessarily on enroll/update made by GBDS Driver (when API assumes an enroll, not an update).

Inactivation of all RDB transactions when a person is deleted.

Sync response fixed on enroll/update with pguid is changed from Driver.

[BF-007][ALPHA-1237] Fixed get transaction endpoint for transactions with REFUSED status

Improvements

[IP-001][ALPHA-1233] GBDS has now GHDP files archived on /var/lib/griaule/gbds/ghdp-libs.

To run GBDS on GHDP environment, follow the instructions on readme file, that is:

To upgrade from HDP3 to GHDP, remove these files from /var/lib/griaule/gbds/lib:

hbase-annotations-2.1.3-tests.jar
hbase-client-2.1.3.jar
hbase-common-2.1.3.jar
hbase-common-2.1.3-tests.jar
hbase-hadoop2-compat-2.1.3.jar
hbase-hadoop2-compat-2.1.3-tests.jar
hbase-hadoop-compat-2.1.3.jar
hbase-hadoop-compat-2.1.3-tests.jar
hbase-http-2.1.3.jar
hbase-mapreduce-2.1.3.jar
hbase-metrics-2.1.3.jar
hbase-metrics-api-2.1.3.jar
hbase-procedure-2.1.3.jar
hbase-protocol-2.1.3.jar
hbase-protocol-shaded-2.1.3.jar
hbase-replication-2.1.3.jar
hbase-server-2.1.3.jar
hbase-server-2.1.3-tests.jar
hbase-shaded-miscellaneous-2.1.0.jar
hbase-shaded-netty-2.1.0.jar
hbase-shaded-protobuf-2.1.0.jar
hbase-testing-util-2.1.3.jar
hbase-zookeeper-2.1.3.jar
hbase-zookeeper-2.1.3-tests.jar

and replace with all jar files from this folder.

GBDS 4.1.0.1450

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: b37ed1a00529cf2ed840e67d223db8be

GBDS redistribution MD5 Checksum: 8caab36c8a0d9ad1199f5b16a98fb222

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.0 deployed on artifactory.

No RDB upgrade need from 4.1.0.1447, 4.1.0.1448 and 4.1.0.1449

Bug Fixes

[BF-001][ALPHA-1228] Fixed get transaction with status RESENT_ENROLL

GBDS 4.1.0.1449

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: eb5739816e0786d6eb187cac04e303a8

GBDS redistribution MD5 Checksum: e891e23ec63afcbc7daefc14f52dbe1a

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.0 deployed on artifactory.

No RDB upgrade need from 4.1.0.1447 and 4.1.0.1448

Bug Fixes

[BF-001][ALPHA-1227] Fixed saving old transactions (pre quality extraction) on any HBase operation HOT FIX

GBDS 4.1.0.1448

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: f2ede06305b8421e07c217ee2dd14f28

GBDS redistribution MD5 Checksum: 0c951773b9ce48858e0be71655d1ede8

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.0 deployed on artifactory.

No RDB upgrade need from 4.1.0.1447

Bug Fixes

[BF-001][ALPHA-1199] SSL settings commented by default on gbdsapi.properties (SSL is off by default)

[BF-002][ALPHA-1222] Extracted image quality returned on get and list people and get and list transactions

[BF-003][ALPHA-1223] Genuine update fixed on API (when 1:1 is made). API used to perform update ok, but generating exceptions against its own enroll.

[BF-004][ALPHA-1224] On endpoint GET /extraction-control/list and GET /extraction-quality-control/list, fixed to list all extraction template and quality services, not only enabled ones

[BF-005][ALPHA-1225] Fixed quality re extraction on background for old transactions (pre GBDS 4.1.0) when operation time on rdb table gbds.transactions is null

[BF-006][ALPHA-1226] Fixed saving search start time for enroll/search on GBDS Driver when transaction does not exist or HBase connection is off and protect save search elapsed time for enroll/search for the same reason

GBDS 4.1.0.1447

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 4.1.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.1.0 or newer

GBDS release MD5 Checksum: b265241591f6c46e4f5f965725de00f2

GBDS redistribution MD5 Checksum: d02daafb53e3d6a3172e48dc89711079

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. Major RDB upgrade from 4.0.x to 4.1.x provided.

Java client and API model 4.1.0 deployed on artifactory.

New Features

[NF-001][ALPHA-1166] New endpoint for link resent enroll:

Endpoint: PUT /v2/people/transactions/linkResentEnroll/{tguid}/newTguid/{newTguid}

  • Request: no body

  • Response:

    • 200 OK
    • 400 BAD REQUEST
    • 422 MISSING_TGUID
    • 422 INVALID_TRANSACTION_STATUS

On java client:

LinkResentEnrollRequest req = LinkResentEnrollRequestBuilder.aBuilder()
        .withTguid(tguid)
        .newTguid(newTguid)
        .build();
LinkResentEnrollResponse response = client.linkResentEnroll(req);
return response.isSuccessful();
  • Save newTguid on hBase transaction: transaction:new-resent-tguid (returned on get/list transaction)

[NF-002][ALPHA-1167] List transactions with qualityStatus and/or enrollStatus on RDB using table gbds.quality

Now the endpoint GET /v2/people/transactions can be used to retrieve not only pending/approved quality transactions on RDB but also any enroll/update transactions through RDB, using query parameter enrollStatus.

Enum values are:

  • ENQUEUED
  • PROCESSING
  • ENROLLED
  • EXCEPTION
  • FAILED
  • REFUSED
  • RESENT_ENROLL
  • PENDING

[NF-003][ALPHA-1107] Safe shutdown for API and GBDS Driver

Endpoint: POST /v2/shutdown

  • Request
{
    "data": {
        "shouldShutdownAPI" : true,
        "shouldShutdownDriver" : true
    }
}

  • Response:

    • 200: OK
    • 500: Internal error

Added a new script on: /var/lib/griaule/gbsapi/scripts/shutdown-driver-through-kafka.sh to gracefully shutdown Driver via shell

[NF-004][ALPHA-1107][ALPHA-1106] Finger/Face quality extraction on micro service

All fingerprints and faces can now have their quality attributes and properties extracted on micro service called Extraction Quality Service

An API can use several quality extraction services to perform quality extraction. Each quality extraction service can be of 2 types:

  • finger: for fingerprints
  • face: for faces

When using quality extraction services, API will balance extraction HTTP REST calls. If a quality extraction service returns 500 or refuses connection or gives a timeout, API extraction service call balancer disables the client calling this extraction service. It can be turned on again on quality extraction control endpoint PUT described below.

There’s no minimum number of quality extraction services to be up at API boot. It can be increased with quality extraction control endpoints.

Quality extraction will set properties for a transaction always on json on table gbds.transaction_quality, that can be retrieved on get/list people or transaction, inside each biometric.

Setting flag gbds.extraction.quality.fillTransactionQualityPropertiesTable does quality properties to be stored also on table gbds.transaction_quality_properties for each transaction, on key/value format, for better report. But does not affect get/list people or transaction.

Quality Extraction Service API configuration

Some settings configures it on RDB gbds.settings or on gbdsapi.properties, if RDB setting agent is off (setting gbds.rdbSystemConfiguration.enabled):

  • General flags for quality extraction:

    • gbds.extraction.quality.service, type API, default true: Flag to enable quality extraction. This flag can be turned on and off freely with API up.
    • gbds.extraction.quality.api.uniqueId, type API, default empty: API unique ID that will perform background quality extraction, described below.
    • gbds.extraction.quality.fillTransactionQualityPropertiesTable, type API, default false: Store quality properties on table gbds.transaction_quality_properties, on key/value format for better report

  • Flag by modality:

    • gbds.faces.extraction.quality.api, type API, default true: Flag to turn on quality extraction for faces on enroll/update, trusted too, aligned with template extraction
    • gbds.faces.extraction.quality.background, type API, default false: Flag to turn quality on quality extraction for faces on background, described below.
    • gbds.fingerprints.extraction.quality.api, type API, default true: Flag to turn on quality extraction for fingerprints on enroll/update, trusted too, aligned with template extraction
    • gbds.fingerprints.extraction.quality.background, type API, default false: Flag to turn quality on quality extraction for fingerprints on background, described below.
    • If API and background flags were turned off, assumes no quality extraction needed for new enrolls/updates

Quality Extraction Microservice configuration:

  • gbds.extraction.quality.service.finger.count: type API, default 1: Initial finger quality extraction service count

  • gbds.extraction.quality.service.face.count: type API, default 1: Initial face quality extraction service count

  • gbds.extraction.quality.service.initialPort: type API, default 31000: Initial port to start quality extraction services

  • gbds.extraction.quality.service.logLevel: type API, default INFO: Quality extraction services log level on API.

    • INFO logs scripts
    • DEBUG turns on service log tail for each micro service on API log

  • gbds.extraction.quality.service.timeout: type API, default 60: Quality extraction services request timeout (seconds)

  • gbds.extraction.quality.service.hostname: type API, default localhost: Quality extraction services host name.

    • If localhost, API will take control of services start/kill
    • If anything else, the operator must take care of getting up and down micro services on each port configured.

  • gbds.extraction.quality.service.maxTries: type API, default 3: Quality extraction services maximum tries to fail a quality extraction

API takes care of getting quality extraction micro services up and down if hostname is localhost, same as template micro service extractors.

On gbdsapi.properties:

gbds.extraction.quality.service=true
gbds.extraction.quality.api.uniqueId=
gbds.extraction.quality.fillTransactionQualityPropertiesTable=false
gbds.faces.extraction.quality.api=true
gbds.faces.extraction.quality.background=false
gbds.fingerprints.extraction.quality.api=true
gbds.fingerprints.extraction.quality.background=false
gbds.extraction.quality.service.finger.count=1
gbds.extraction.quality.service.face.count=1
gbds.extraction.quality.service.initialPort=31000
gbds.extraction.quality.service.logLevel=INFO
gbds.extraction.quality.service.timeout=60
gbds.extraction.quality.service.hostname=localhost
gbds.extraction.quality.service.maxTries=3

Basically, setting gbds.extraction.quality.service turns on/off micro service quality extraction. If it is on, all quality extractions can be done if fingerprint and/or face flag are on too, either on API or background.

gbds.extraction.quality.service.finger|face.count settings controls how many quality extractors of each type API will expect to be up and running. The sum of all quality extractor services can be zero.

gbds.extraction.quality.service.initialPort setting informs initial port to API lookup for quality extractors (finger count first, then face count).

Ex: initial port is 31000, finger count is 3, face count is 1. API will look for extractors:

http://localhost:31000, type finger
http://localhost:31001, type finger
http://localhost:31002, type finger
http://localhost:31003, type face

gbds.extraction.quality.service.hostname setting tells where quality extractors are for API. If is ‘localhost’, API will take control of start/kill quality extractor services on the same host.

API uses api.uniqueId setting on gbdsapi.properties (not present on RDB) to start/kill its own extractors, not interfering with quality extractors of another API on the same machine. Obs: if api.uniqueId is not present or is empty, API created one using a unique GUID.

If hostname is anything but ‘localhost’, API will only lookup extractors using initial port and counts described above.

gbds.extraction.quality.service.logLevel setting controls extractor log levels on API log (if extractors is controlled by API):

  • INFO: logs scripts output on API log
  • DEBUG: logs scripts output and extractor log directly on API log

gbds.extraction.quality.service.timeout setting controls HTTP timeout for each extractor client on API.

Quality Extraction Service file structure

All quality extraction service files are located at /var/lib/griaule/extractor

Extractor names for quality here are: fingerquality and facequality

Some scripts are available:

  • /var/lib/griaule/extractor/scripts/start-extractor.sh: Starts a quality extraction service on machine with these parameters, in this order:

    • gbds_api_id: API unique ID
    • gbds_api_host: API host
    • gbds_api_port: API port
    • native_extraction_service_lib: extraction service type (fingerquality, facequality)
    • native_extraction_service_port: service port

  • /var/lib/griaule/extractor/scripts/kill-extractor.sh: Kills one quality extraction service on machine using these parameters:

    • gbds_api_id: API unique ID
    • gbds_api_port: API port
    • native_extraction_service_lib: extraction service type (fingerquality, facequality)
    • native_extraction_service_port: service port

  • /var/lib/griaule/extractor/scripts/kill-all.sh: Kills all extraction service (template AND quality extractor) of one API or all APIs on the machine. If no parameter is provided, kills all template and quality extraction services on the machine. Parameters:

    • gbds_api_id: API unique ID (optional)
    • gbds_api_port: API port (optional)

Log files are located at /var/log/griaule/extractor and has this file name template: log_<native_extraction_service_lib:fingerquality|facequality>_<native_extraction_service_port>_<gbds_api_id>.log

Note

Remember, if setting gbds.extraction.quality.service.hostname is ‘localhost’, API will use these scripts to start/kill extractors. If not, the operator will have to start/kill extractors for himself and configure API settings to look up quality extractors correctly.

Quality Extraction Service on background

Quality extraction on background is turned on with:

  • General flag: gbds.extraction.quality.service
  • Background modality flags: gbds.faces.extraction.quality.background and gbds.fingerprints.extraction.quality.background
  • API unique id according to settings: gbds.extraction.quality.api.uniqueId
  • Any row on table gbds.transactions (changes described below) with columns finger_quality_extracted and/or face_quality_extracted that has value false (0)

If the API unique id matches the unique id for any API looking up RDB, this API will perform quality extraction on background.

When background quality extraction is on, if on the RDB table gbds.transactions any row has its columns finger/face quality extracted reset to 0, new quality extraction is made again on that transaction.

On the same logic, right after API boot, if there’s rows on the RDB table gbds.transactions with these same flags as 0, all that ones have quality extraction made.

  • POST /v2/extraction-quality-control/add
  • DELETE /v2/extraction-quality-control/remove
  • PUT /v2/extraction-quality-control/0/{operation}

Quality Extraction Control Endpoints

Some endpoints are provided to list/add/remove extraction services (when API controls it, also start/kill extraction services):

  • GET /v2/extraction-quality-control/list: Lists all quality extraction services configured on API.

  • Response:

    • 200 OK
{
    "data": [
        {
            "id": 0,
            "url": "http://localhost:31000/v1",
            "enabled": true,
            "library": "FACE",
            "message": "Service client initialized"
        },
        {
            "id": 1,
            "url": "http://localhost:31001/v1",
            "enabled": true,
            "library": "FINGER",
            "message": "Service client initialized"
        }
    ]
}

Every time a quality extraction is made, the message on the quality extraction service client that made the quality extraction indicates the action and time elapsed.

  • POST /v2/extraction-quality-control/add: Adds a quality extraction service to API look up (starts it if configured):

Request:

{
    "library" : "FINGER",
    "count" : 1
}

Parameters:

  • library: FINGER, FACE

  • count: how many quality extraction services to add

  • Response:

    • 200 OK
{
    "data": [
        {
            "id": 2,
            "url": "http://localhost:31002/v1",
            "enabled": true,
            "library": "FINGER",
            "message": "Service client initialized"
        }
    ]
}
  • PUT /v2/extraction-quality-control/{quality-extraction-service-id}/{operation}

Parameters:

  • quality-extraction-service-id: id for a quality extraction service, as list endpoint returned.

  • operation: ENABLE, DISABLE or RESTART

  • Response:

    • 200 OK, if operation is ok
    • 422 Unprocessable entity, if quality extraction service is already enabled/disables
{
    "errors": [
        {
            "type": "PROCESSING_ERROR",
            "code": "INVALID_PARAMETER_COMBINATION",
            "message": "Client #0 is already disabled"
        }
    ]
}
  • DELETE /v2/extraction-quality-control/remove

Request:

{
    "library" : "FINGER",
    "count" : 1
}

Parameters:

  • library: FINGER, FACE

  • count: how many quality extractions to remove

  • Response:

    • 200 OK
{
    "data": [
        {
            "id": 2,
            "url": "http://localhost:31002/v1",
            "enabled": false,
            "library": "FINGER",
            "message": "Extractor service removed by user"
        }
    ]
}

If there’s no quality extractor service for that library, it fails:

  • 400 Bad request
{
    "errors": [
        {
            "type": "VALIDATION_ERROR",
            "code": "INVALID_PARAMETER_COMBINATION",
            "message": "ERROR: Not enough service clients of library finger to remove"
        }
    ]
}

Table refactoring

For better RDB structure data reading, some tables were renamed.

  • Table gbds.transactions were renamed to gbds.transactions_ref
  • Table gbds.quality renamed to gbds.transactions.
  • Table gbds.quality_fields renamed to gbds.transaction_fields.
  • Table gbds.quality_properties renamed to gbds.transaction_quality.

Some tables were upgraded too, with more information:

  • Table gbds.transactions: Now transactions can hold enroll/update transaction and verify/identify searches too. So these modifications were made:

    • Column finger_properties_extracted renamed to finger_quality_extracted, boolean, 0 or 1: If false, fingerprints for this transaction could be extracted on background, if true, quality extraction were made
    • Column face_properties_extracte renamed to face_quality_extracted, boolean, 0 or 1: If false, faces for this transaction could be extracted on background, if true, quality extraction were made
    • Column properties_extraction_priority renamed to priority: priority for this transaction
    • Column _timestamp renamed to created: transaction creation timestamp
    • Column pguid now can be null (searches does not have pguid)
    • Column enroll_status now can be null (searches does not have enroll status)
    • Column quality_status now can be null (searches does not have quality status)
    • Column search_status added for verify/identify transactions
    • Column updated added with last modification timestamp for this transaction
    • Column active added with active flag for this transaction (enroll or last update)
    • Column extraction_time added with elapsed time for template extractions for this transaction
    • Column extraction_quality_time added elapsed time for quality extractions for this transaction
    • Column match_time added with elapsed match time for this transaction (1:1 for update on API or 1:N for enroll on Driver)
    • Column total_time added with total operation time (minus queue waiting time. It is the sum of endpoint and process time on API and GBDS Driver queue retrieving, match and consolidation time)
    • Column type added with transaction type, ENROLL, UPDATE, VERIFY or IDENTIFY
    • Column finger_global_quality added with quality for all fingerprints for this transaction
    • Column global_quality added with quality for all biometrics (fingerprints and faces) for this transaction (not implemented yet)

  • Table gbds.transaction_fields: one row by field for transaction on table gbds.transactions

    • Column quality_id renamed to transaction_id

  • Table gbds.transaction_quality: one row by biometric with quality extracted for transaction on table gbds.transactions

    • Columns nfiq and contrast dropped
    • Column face_json renamed to properties_json to hold all properties for fingerprint and face on json format, used on get/list endpoints
    • Column quality_id renamed to transaction_id
    • Column quality_num renamed to template_quality with template quality for biometric (fingerprints)
    • Column image_quality added with image quality for biometric (faces)
    • Column minutiae_count added with fingerprint minutiae count

  • Table gbds.transaction_quality_properties added: one row by fingerprint or face property for each biometric on transaction on table gbds.transaction_quality. This table is filled on API only if setting gbds.extraction.quality.fillTransactionQualityPropertiesTable is true.

    • Primary key column id
    • Key for table gbds.transaction_quality on column transaction_quality_id
    • Key for table gbds.transactions on column transaction_id
    • Column idx with biometric index
    • Column name with property name, string
    • Column value with property value, string

Caution

with this last table gbds.transaction_quality_properties filled with all fingerprints/faces properties, RDB space could be increased immensely.

Improvements

[IP-001][ALPHA-1170] New behavior for update/trusted update on biographics and labels.

  • Do not accumulate biographics with same key but different values
  • Accumulate labels as always, but this behavior can be changed using trusted enroll/update:
{
    "data": {
        ...
    },
    "meta": {
        ...
        "labels": "REPLACE"
    }
}

Values for labels are: REPLACE|ACCUMULATE

[IP-002][ALPHA-1193] Do not log scanned regions on scan boot metrics during scan, only at end; log which scan are being made (person or UL)

[HIP-003][ALPHA-1192] Transparency and email notifier unitary tests

[IP-004][ALPHA-1199] Properties from API, GBDS Driver and Notifier revision and cleanup

Files changed:

  • /etc/griaule/conf/gbsapi/gbdsapi.properties
  • /etc/griaule/conf/gbds/application.conf
  • /etc/griaule/conf/gbsnotifier/notifier.properties

Bug Fixes

[BF-001][ALPHA-1197] RDB Dump and upgrades fixes

Dump RDB script already fixed when provided with this release:

  • host column on table gbds.settings size fixed to 255
  • create griaule user fixed on MariaDB/MySQL
  • setting gbds.rdbSystemConfiguration.enabled changed to true

Upgrades fixed:

[BF-002][ALPHA-1204] API could send and INTERNAL ERROR in some updates with no exception. Fixed.

[BF-003][ALPHA-1216] Transparency configuration reload on GBDS Driver

The transparency functionality configuration on GBDS Driver were not being reloaded on boot, or on any other configuration reload event. Therefore, transparency service were only looking up on file, which is updated on boot, but only after file configuration reading. Then, if RDB is changed, only on the next boot transparency service would look the right rdb updated configurations.

[BF-004][ALPHA-1218] Fix missing candidates bug on latent search with post matching on

Post matching was not sorting the candidate list, but since transparency functionality all match algorithms must do it. Therefore, API cropped that unsorted list and sent away some high scores candidates.

[BF-005][ALPHA-1219] Fixed pagination count field on list exceptions

Exception list pagination were having count filed according to internal grouping of enroll tguid/match person pguid, where before an enroll tguid showed all match person pguids as a tree.

So the count field used to show enroll tguids count, not considering enroll tguid, match person pguid pairs, as is today’s list. Fixed to show pair count. All other fields are correct.

GBDS 4.0.2.1446

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: c8cdc68fbde8266caf2126cbe548dd64

GBDS redistribution MD5 Checksum: 2a1112ceb91e956da6e654d5bab3eb32

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump for this release were provided. No RDB upgrade needed from 4.0.1.1445

Bug Fixes

[BF-001][ALPHA-1198] API used to save gbdsapi.properties with RDB settings always on /etc/griaule/cong/gbdsapi.properties even if start script changes this location. Fixed to save gbdsapi.properties on correct file location, if changed on script.

[BF-002][ALPHA-1202] When RDB settings configuration agency were off, GBDS does not read settings from file, but from default code values. Fixed properties reading from file (gbdsapi.properties, application.conf) when RDB settings configuration agency is off.

[BF-003][ALPHA-1203] Get/list people fallback templates reading fix. When a person used to have templates only on fallback column families on HBase, API used to ignore these column families. Fixed to read regular columns families (tpts by default) and then, if absent, from fallback column families (fingerprints, palmprints, face, iris, newborn-palmprints)

[BF-004][ALPHA-1204] Due to GBDS kafka queue being large, API could send the same enroll twice as new enroll. First one is processed as enroll and second one is processed as an update, but if 1:N search result registers other people candidates (erroneously), several update exceptions could be registered. Fixed to retain only the enroll itself on candidate list (this makes Driver process the search for an enroll turned into update as a verify)

GBDS 4.0.1.1445

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: f748644fb363e033991e2bb95338c781

GBDS redistribution MD5 Checksum: e85b2ecb1233ccc8e92cf76edd90fdc1

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump script for this release was provided. No RDB upgrade needed from 4.0.0.1444

Bug Fixes

[BF-001][ALPHA-1184] Pguid from an enroll could be retrieved from a previously deleted/recollected enroll when some operation on this last one was not completed. Fixed along with ALPHA-1186

[BF-002][ALPHA-1187] Entrant enroll on an exception could copy reference person keys. Fixed along with ALPHA-1186

[BF-003][ALPHA-1188] Fixed bug on quality analysis with edition when an update is made (endpoint PUT /v2/people/{pguid} using different keys or endpoint POST /v2/people with existing keys). GBDS used to generate an enroll exception, but now treats it as an update correctly.

[BF-004][ALPHA-1190] Fixed adding history event on enroll with exception, not added anymore; verify search (that used to return error here) now looks for tguid on hbase people, not history events.

[BF-005][ALPHA-1191] Fixed verify search tguid on each candidate, that was looking only on the last transaction (through history). Looking now on best of biometrics if on; if off, look on the last transaction (transaction biometric map stored on RDB gbds.biometrics)

GBDS 4.0.0.1444

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: db6ee2966b31c8be8b4c65de8f1a1ec1

GBDS redistribution MD5 Checksum: e4a865556a6e14df97f3b894b9380eac

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

All RDB dumps and upgrade scripts are the same as release 4.0.0.1441

Bug Fixes

[BF-001][ALPHA-1185] Fixed assign treatment to user error message on metadata. It shows “tguid” incorrectly. Fixed to show “user”

[BF-002][ALPHA-1186] CRITICAL Fixed sequence control template extraction using extraction micro service.

GBDS 4.0.0.1443

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: 1503bd422742069612f59255038670d9

GBDS redistribution MD5 Checksum: 6ff69a1a841e961c29b6fc6f5050042b

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

All RDB dumps and upgrade scripts are the same as release 4.0.0.1441

Bug Fixes

[BF-001][ALPHA-1179] Fixed bug on person verification (search with pguid) when this person is an entrant on an exception (used to have no history)

[BF-002][ALPHA-1180] Fixed bug on iris extraction micro service. Using right serialization type to compare irises.

[BF-003][ALPHA-1182] Fixed bug on micro service extraction while removing extractor. When removing an extractor via endpoint while an extraction is made, API used to run into a deadlock. Deadlock fixed.

[BF-004][ALPHA-1183] On face/iris identify, latent search parameters can now be provided with match threshold, that will be used as threshold on search. OBS: face/iris does not have latent search, only regular search, but this fix made threshold passing to be possible through latent search parameters.

GBDS 4.0.0.1442

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: e55cb4f2fa279ac03442fd0c5e77aa8d

GBDS redistribution MD5 Checksum: a65ee4446fbd8e8fd5a6213d493e5aac

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

All RDB dumps and upgrade scripts are the same as release 4.0.0.1441

Bug Fixes

[BF-001][ALPHA-1171] Extractor services are now verified if it is up and running on API when needed.

On startup, if any extractor service does not complete initialization or does not respond after initialization, API fails to boot.

On endpoint GET /v2/extraction-control/list all extraction services are checked. If any do not respond, API disables it.

On endpoint PUT /v2/extraction-control/{id}/{operation} if operation is ENABLE or RESTART, extraction service is checked as well. If it does not respond, API disables it.

On endpoint POST /v2/extraction-control/add API checks all extraction services being added. If any do not respond, abort all inclusion and none extraction services are added.

On extraction, if service returns some error that will get this service to restart, if this restart fails to check, API disables it.

[BF-002][ALPHA-1172] Allow enroll/updates (and trusted enroll/update) to proceed with transactions with some extraction issues (not all biometrics, that remains failing the transaction). This behavior is now the same as GBDS 3.

GBDS 4.0.0.1441

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: e3fbf6a4583e62a2d671bde41e00edbb

GBDS redistribution MD5 Checksum: c6e638a1fd8400f67a6df4f02e8ba933

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

RDB dump and upgrade script for this release were provided.

Bug Fixes

[BF-001][ALPHA-1164] Validate extractors as them sum of all extractor counts on startup and on remove (minimum of one on total)

Previously, at least one extractor type must be configured. Now there must be at least one extractor at all, of any type.

[BF-002][ALPHA-1165] Extractor last execution log now is named log-xxx.oldExecution.YYYY-mm-dd-hh-MM-ss

[BF-003][ALPHA-1165] On extraction service returning HTTP error code 500, the extractor client on API is restarted and biometric being extracted is sent to the next extractor.

[BF-004][ALPHA-1165] On API shutdown, all API extractors are also turned off.

[BF-005][ALPHA-1169] On extraction service network error (refuse, timeout, etc), extractor client on API is restarted (same logic as BF-003)

[BF-006][ALPHA-1169] Maximum tries on a biometric extraction. If it crashes or makes an extraction service return HTTP code 500, set biometric and transaction as FAILED.

Default maximum number of tries is 3 using the configuration gbds.extraction.service.maxTries, type API

[BF-007][ALPHA-1169] Save and notify failed transaction on trusted enroll/update

GBDS 4.0.0.1439

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: 0bd587d1ee4e8b0e26743d972fc17977

GBDS redistribution MD5 Checksum: 31ae5d74caf838e68829b5793c75b0b8

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

All RDB dumps and upgrade scripts are the same as release 4.0.0.1438

Bug Fixes

[BF-001][ALPHA-1160] Fix extraction treatment configuration on ignoring face match having enough configured fingerprints.

Currently, if the API property gbds.biometric.face.exception.min-fingers-to-ignore-face is present, GBDS only considers the number of fingerprints on the update request to check if it should ignore face match to generate update exception.

Now this property was changed to gbds.biometric.face.exception.min-finger-matches-to-ignore-face and it checks for total fingerprint matches on update to check if it should ignore face match to generate update exception.

Legacy property gbscluster.enroll.minimum.fingers.ignore.face was changed to gbscluster.enroll.minimum.finger-matches.ignore.face.

[BF-002][ALPHA-1163] Fix face extraction service

If a face image with no face or invalid data was sent to face extraction service, this one returns http code 500.

Fixed to return http code 400 on bad image and no face found.

GBDS 4.0.0.1438

Compatibility

  • HBase: Entries created with GBDS 3.22.0 or newer
  • MySQL: Transactions registered with 3.25.0 or newer
  • Kafka: Compatible with entries created with GBDS 3.24.0 or newer
  • GBDS Client: 4.0.0 or newer

GBDS release MD5 Checksum: 85f61feb2105e7ac6b18312567c104e1

GBDS redistribution MD5 Checksum: 97330f57e268b391c402c3502d418d1b

From now on, GBDS release RPMs will have two files:

  • gbscluster-version-release-x86_64.rpm
  • gbds-distribution-version-release-x86_64.rpm

The first is the regular release, with the last changes. The second is a distribution file, with files required to an environment configured for the first time.

New Features

[HNF-001][ALPHA-1017]: Future finger and face quality RDB structure available on release (dump on clear-rdb-4.0.0.1438.sql and upgrade on upgrade-rdb-4.0.0.1438.sql)

[NF-002][ALPHA-1106] Change log level on endpoint and/or directly on RDB.

To change log level on API/GBDS Driver there’s now three ways to perform it:

  1. Endpoint POST /v2/log:
{
    "data" : [
        {
            "component": "API",
            "logLevel": "INFO"
        },
        {
            "component": "DRIVER",
            "logLevel": "INFO"
        }
    ]
}

Parameters:

  • data: is a list of a change log object. It can contain only one object (for API or DRIVER) or two, for both components.
  • component: API, DRIVER
  • logLevel: TRACE, DEBUG, INFO, WARN, ERROR, OFF, ALL

Response:

200 OK:

{
    "data": {
        "apiMessage": "API log level was not changed because the previous one was the same",
        "driverMessage": "Driver log level was not changed because the previous one was the same"
    }
}

200 OK:

{
    "data": {
        "apiChanged": true,
        "driverChange": [
            {
                "matchersChanged": {
                    "matcher-0": true,
                    "matcher-1": true
                },
                "nodeName": "gbds-137"
            }
        ]
    }
}
If API and/or DRIVER log level is not changed, message as above is returned.
If API log level is changed, apiChanged is returned as true.
If DRIVER log level is changed, driverChanged is return with a list of all matchers changed. Node cluster that performed changed is returned too.

  1. Directly on RDB gbds.settings table:

GBDS will look up these settings:

  • gbds.api.logLevel, type API, default INFO, for API log level
  • gbds.driver.logLevel. type GBDS, default INFO, for DRIVER log level

Any changes directly on RDB will affect API/DRIVER within T-15 minutes. Alternatively, if RDB setting agent is off, files application.conf and gbdsapi.properties could contain the same settings and it will take affect on API/DRIVER restart, respectively.

  1. Using configuration endpoint: POST /v2/configuration:

API:

{
    "application" : "GBDS_API",
    "configurations" : {
        "gbds.api.logLevel" : "INFO"
    }
}

Response:

200 OK

Driver:

{
    "application" : "GBDS_DRIVER",
    "configurations" : {
        "gbds.driver.logLevel" : "INFO"
    }
}

Response:

200 OK

Works the same as item 2, but immediately.

[NF-003][ALPHA-1146] Log load/unload people and UL on Driver boot.

Use RDB gbds.settings setting:

  • gbds.log.loadUnload: type GBDS, default false

To log all load and unload memory operations on GBDS Driver for all people and ULs.

[NF-004][ALPHA-1149] Extraction on micro service

All API template extractions now can be done on a micro service called Extraction Service. An API can use several extraction services to perform extractions. Each extraction service can be of 3 types:

  • ginger: for fingerprints, palmprints and newborn palmprints
  • face: for faces
  • girl: for irises

When using extraction services, API will balance extraction HTTP REST calls. If an extraction service returns 500 or refuses connection or gives a timeout, API extraction service call balancer disables the client calling this extraction service. It can be turned on again on extraction control endpoint PUT described below.

Extraction Service API configuration

Some settings configures it on RDB gbds.settings or on gbdsapi.properties, if RDB setting agent is off (setting gbds.rdbSystemConfiguration.enabled):

  • gbds.extraction.service, type API, default true: Flag to enable biometric extraction through service
  • gbds.extraction.service.face.count, type API, default 1: Initial facelib extraction service count
  • gbds.extraction.service.ginger.count, type API, default 5: Initial ginger extraction service count
  • gbds.extraction.service.girl.count, type API, default 1: Initial girl extraction service count
  • gbds.extraction.service.hostname, type API, default ‘localhost’: Extraction services host name. If localhost, API will take control of services start/kill
  • gbds.extraction.service.initialPort, type API, default 30000: Initial port to start extraction services
  • gbds.extraction.service.logLevel, type API, default INFO: Extraction services log level on API. INFO logs scripts and DEBUG turn on service log tail
  • gbds.extraction.service.timeout, type API, default 60: Extraction services request timeout (seconds)

On gbdsapi.properties: gbds.extraction.service=true gbds.extraction.service.face.count=1, gbds.extraction.service.ginger.count=5, gbds.extraction.service.girl.count=1, gbds.extraction.service.hostname=localhost, gbds.extraction.service.initialPort=30000, gbds.extraction.service.logLevel=INFO, gbds.extraction.service.timeout=60.

Basically, setting gbds.extraction.service turns on/off micro service extraction. If it is on, all extractions are made on extraction services. If it is off, all extractions work as on GBDS 3.

gbds.extraction.service.ginger|facelib|girl.count settings controls how many extractors of each type API will expect to be up and running. gbds.extraction.service.initialPort setting informs initial port to API lookup for extractors (ginger count first, then face count then girl count).

Ex: initial port is 30000, ginger count is 3, face count is 1 and girl count is 1. API will look for extractors:

  • http://localhost:30000, type ginger
  • http://localhost:30001, type ginger
  • http://localhost:30002, type ginger
  • http://localhost:30004, type face
  • http://localhost:30005, type girl

gbds.extraction.service.hostname setting tells where extractors are for API. If is ‘localhost’, API will take control of start/kill extractor services on the same host.

API uses api.uniqueId setting on gbdsapi.properties (not present on RDB) to start/kill its own extractors, not interfering with extractors of another API on same machine. Obs: if api.uniqueId is not present or is empty, API created one using a unique GUID.

If hostname is anything but localhost, API will only lookup extractors using initial port and counts described above.

gbds.extraction.service.logLevel setting controls extractor log levels on API log (if extractors is controlled by API):

  • INFO: logs scripts output on API log
  • DEBUG: logs scripts output and extractor log directly on API log

gbds.extraction.service.timeout setting controls HTTP timeout for each extractor client on API.

Extraction Service file structure:

All extraction service files are located at /var/lib/griaule/extractor

Some scripts are available:

  • /var/lib/griaule/extractor/scripts/start-extractor.sh: Starts an extraction service on machine with these parameters, in this order:

    • gbds_api_id: API unique ID
    • gbds_api_host: API host
    • gbds_api_port: API port
    • native_extraction_service_lib: extraction service type (ginger, facelib, girl)
    • native_extraction_service_port: service port

  • /var/lib/griaule/extractor/scripts/kill-extractor.sh: Kills one extraction service on machine using these parameters:

    • gbds_api_id: API unique ID
    • gbds_api_port: API port
    • native_extraction_service_lib: extraction service type (ginger, facelib, girl)
    • native_extraction_service_port: service port

  • /var/lib/griaule/extractor/scripts/kill-all.sh: Kills all extraction services of one API or all APIs on machine. If no parameter is provided, kill all extraction services on machine. Parameters:

    • gbds_api_id: API unique ID (optional)
    • gbds_api_port: API port (optional)

Log files are located at /var/log/griaule/extractor and has this file name template: log_<native_extraction_service_lib:ginger|facelib|girl>_<native_extraction_service_port>_<gbds_api_id>.log

Obs: Remember, if setting gbds.extraction.service.hostname is ‘localhost’, API will use these scripts to start/kill extractors. If not, the operator will have to start/kill extractors for himself and configure API settings to look up extractors correctly.

Extraction Control Endpoints:

Some endpoints are provided to list/add/remove extraction services (when API controls it, also start/kill extraction services):

  • GET /v2/extraction-control/list: Lists all extraction services configured on API.

Response:

200 OK

{
    "data": [
        {
            "id": 2,
            "url": "http://localhost:30002/v1",
            "enabled": true,
            "library": "GIRL",
            "message": "Extractor service client created"
        },
        {
            "id": 0,
            "url": "http://localhost:30000/v1",
            "enabled": true,
            "library": "GINGER",
            "message": "Extractor service client created"
        },
        {
            "id": 1,
            "url": "http://localhost:30001/v1",
            "enabled": true,
            "library": "FACE",
            "message": "Extractor service client created"
        }
    ]
}

Every time an extraction is made, the message on the extraction service client that made the extraction indicates the action and time elapsed.

  • POST /v2/extraction-control/add: Adds an extraction service to API look up (starts it if configured):

Request:

{
    "library" : "GINGER",
    "count" : 1
}

Parameters:

  • library: GINGER, FACE, GIRL
  • count: how many extraction services to add

Response:

200 OK

{
    "data": [
        {
            "id": 3,
            "url": "http://localhost:30003/v1",
            "enabled": true,
            "library": "GINGER",
            "message": "Extractor service client created"
        }
    ]
}
  • PUT /v2/extraction-control/{extraction-service-id}/{enabled:true|false}

Parameters:

  • extraction-service-id: id for an extraction service, as list endpoint returned.
  • enabled: true or false to enable or disable an extraction service

Response:

200 OK if operation is ok

422 Unprocessable entity if extraction service is already enabled/disables

{
    "errors": [
        {
            "type": "PROCESSING_ERROR",
            "code": "INVALID_PARAMETER_COMBINATION",
            "message": "Client #0 is already disabled"
        }
    ]
}
  • DELETE /v2/extraction-control/remove

Request:

{
    "library" : "GINGER",
    "count" : 1
}

Parameters:

  • library: GINGER, FACE, GIRL
  • count: how many extractions to remove

Response:

200 OK

{
    "data": [
        {
            "id": 3,
            "url": "http://localhost:30003/v1",
            "enabled": true,
            "library": "GINGER",
            "message": "Extractor service removed by user"
        }
    ]
}

If count would leave API with less than one extraction of that type, it fails:

400 Bad request

{
    "errors": [
        {
            "type": "VALIDATION_ERROR",
            "code": "INVALID_PARAMETER_COMBINATION",
            "message": "At least one extraction service client of library ginger must remain"
        }
    ]
}

Improvements

[IP-001][ALPHA-1116] Transparency improvement: Multiple e-mail send for face identify search requests.

If configured, GBDS will send only one e-mail at a given interval with all face identify search requests that happened on that interval.

Additional configuration on RDB for Email Notifier Service (table enotifier.settings):

  • mail.identify.request.multiple: default true; Turns on/off multiple e-mail sending with face identify search request notifications
  • mail.identify.request.multiple.period: default 1h; period to consolidate e-mail notifications
  • mail.identify.request.multiple.subject: default on dump and upgrade RDB script: Subject for multiple notifications e-mail
  • mail.identify.request.multiple.template: default on dump and upgrade RDB script: Body template for multiple notifications e-mail The e-mails packs a zip file with all images listed.
  • Image list zip file attached can contain a maximum of 24MB. Above that, the listing is broken down into more than one e-mail.

[IP-002][ALPHA-1159] Get search on /v2/people/searches/{tguid} with invalid tguid (tguid from other type of transactions, such as close session, get person, get exception(s), etc.) now returns as:

{
    "errors": [
        {
            "type": "PROCESSING_ERROR",
            "code": "INVALID_TRANSACTION",
            "message": "Transaction is not a search transaction"
        }
    ]
}

Bug Fixes

[BF-001][ALPHA-1145] Fix configuration endpoint so API and GBDS Driver are updated. Today only API configurations were updated