GBDS

5.2.

5.2.0

New Features

[NF-001] affects [API][GBDS][GBS Migration] Template Storage MinIO for verify 1:1 authentication

Use of MinIO storage to persist people templates and use it on search verify 1:1 if HBase connection is off or if configuration is on for template storage use.

Storage

Use of MinIO storage. It requires a MinIO instance, from single-node, single-drive up to multi-node, multi-drive, according to client needs.

Configurations

GBDS

All RDB configurations:

Setting name
Type
Default
Description

gbds.templateStorage.enabled

BOTH

false

Use template storage on search verify

gbds.minio.active

BOTH

false

Connect on MinIO

gbds.minio.host

BOTH

MinIO host name

gbds.minio.port

BOTH

MinIO port

gbds.minio.username

BOTH

MiniIO user name

gbds.minio.password

BOTH

MinIO password

GBDS Migration

On gbds-migration.properties:

  • gbds.templateStorage.activeOnly=false Migrate only active people

  • gbds.templateStorage.peopleTemplates=true Migrate people template.

    • If true, get template from HBase people.

    • If false, get templates from HBase person transaction.

      • If person has biometric map on people’s table column enroll:biometricMap, get template from all transactions that composes biometric map

      • If person does not have biometric map, get templates from active transaction, pointed by guids:transaction-guid

Behavior

Settings

There’s two flag settings: Template storage enabled and MinIO active.

Template storage enabled indicates that GBDS must process verify searches through MinIO storage.

MinIO active flag indicates if MinIO connection must be made. If false, even if verify searches tries storage, it will not be made, resulting in Person not found.

You can be connected on MinIO, but use only HBase regularly until a HBase fault. If not connected to MinIO, even on HBase fault, verify search is not possible. If connected, people are stored on MiniIO on enroll/update.

Storage on MinIO

With MinIO active, on enroll/update/trusted, person templates (templates that goes to HBase people table) are saved into MinIO as well, on a bucket named “people“.

If the bucket does not exists, GBDS will created it.

HBase stored templates are not changed, being persisted as usual.

On enroll/update/trusted flow, an HBase error does not turn on template storage for verify searches.

On update, with template storage enabled true, person is checked if exists using MinIO storage, not HBase people table.

Verify

When template storage setting is enabled, GBDS will look up person template using MinIO storage, that will have the same templates stored on HBase people table after migration. If MinIO storage does not have person templates, an error Person Not Found is thrown.

If verify search is requested using keys, GBDS will search for PGUID on RDB using query for each key provided:

select t.pguid from gbds.transaction_fields tf 
 inner join gbds.transactions t on tf.transaction_id = 
 where tf.field_type = 'KEY' and fkey = :key_id and tf.fvalue = :key_value
  and ((t.enroll_status = 'ENROLLED' and t.active = 1) or t.enroll_status = 'EXCEPTION')
  and t.transaction_type in ('ENROLL','UPDATE')

If no PGUID is retrieved from any key, search will return person not found.

When performing verify searches with template storage enabled false, GBDS will look up people templates using last active person transaction or biometric map, as usual.

To turn template storage usage, you can turn it on on RDB settings or it will be automatically turned on on any HBase connection fault, after 5 error attempts, on any step of a verify search flow.

Obs: The action of turning template storage after 5 HBase errors is temporary while API is up only, it does not reflect on RDB settings.

Structure

People templates are stored on MiniIO bucket “people”, using format:

XX/YY/<PGUID>.zip

  • XX is the first two PGUID chars

  • YY is the next two PGUID chars

  • ZIP structure:

    • All templates as <index>.tpt file

    • Metadata is metadata.json, containing:

      • pguid, string

      • tguid, string

      • active, boolean

      • enrollStatus, enum

      • biometricMap, map of tguid, list of integer

Migration

GBDS Migration have a new mode to migrate people to MinIO: templateStoragePeople

Range can be applied on CLI:

  • --range <start>:<end>

    • Start is required, can be one char or more

    • End is optional

Migration will scan HBase people table and store templates on bucket “people”.

If bucket does not exists, it is created.

Endpoints

No endpoints are changed.

Improvements

[IP-001] affects [GBDS Migration] Reextraction with tguids and pguids list

Reextraction can be made now with tguids and pguids list.

Configurations

  • gbds.reextract.tguidsFile = <tguids file path>

  • gbds.reextract.pguidsFile = <pguids file path>

File format:

  • One or more guid per line

  • Comma separated if more than one guid on same line.

If any of these lists is provided, no check on hbase flag is made, only SQLite file to skip entire range.

So we recommend reset SQLite file with tguids/pguids lists.

[IP-003] affects [GBDS][API] Start/stop Scripts upgraded

API and GBDS scripts upgraded according to scripts used by support

On API, haproxy is disabled by defaylt

Matcher changelog

  • Fix memory leaks when receiving unsupported face template

  • Reduce duplication of template metadata (pguid, tguid and labels)

  • Print version in the service CLI and logs

  • Set unsupported memory allocations to 0 bytes as default

[IP-004] affects [API] Trust, Lights out

Set lights out status to error if Biobase is unavailable or Biobase get biograics results on time out

This way, admin operator can reprocess lights out when Biobase is available again via RDB update, changing column lights_out_status on gbds.exception_group table

[IP-005] affects [API] Trust

Better operation log on RDB table gbds.operation_log when creating exception and exception group

On each one, log target and tguid, for better inconformity creation track

[IP-006] affects [API][GBDS] Lib multibio and matcher multibio versions

Lib multibio and matcher multibio versions added to:

  • API GET /gbds/v2/version endpoint

    • data

      • api

        • version

        • build

        • apiId

        • apiType

        • libMultibio: new, got from lib multibio embedded into API exec jar

      • searchEngine

        • version

        • build

        • libMultibio: new, got from lib multibio embedded into GBDS’s native jar

        • matcherMultibio: new, got from optimized ginger matcher service process log, running on GBDS cluster node

    • Obs: If any version/libMultibio/matcherMultibio differs on any GBDS node monitor version that API list, API will throw an error indicating divergent versions on GBDS cluster nodes

  • GBDS GET /monitor/version endpoint

    • data

      • version

      • build

      • libMultibio: new, got from lib multibio embedded into native jar

      • matcherMultibio: new, got from optimized ginger matcher service process log on any matcher running

API get version example:

{
    "data": {
        "api": {
            "version": "5.2.0-SNAPSHOT",
            "build": "5000",
            "apiId": "api1",
            "apiType": "LEADER",
            "libMultibio": "0.3.6"
        },
        "searchEngine": {
            "version": "5.2.0-SNAPSHOT",
            "build": "5000",
            "libMultibio": "0.3.6",
            "matcherMultibio": "0.3.6"
        }
    }
}

GBDS monitor status example:

{
    "data": {
        "version": "5.2.0-SNAPSHOT",
        "build": "5000",
        "libMultibio": "0.3.6",
        "matcherMultibio": "0.3.6"
    }
}

[IP-007] affects [API] Quality Analysis

Added BoB response status and tguid on quality analysis treatment.

On any quality analysis treatment, BoB status and tguid were added:

Ex:

  • Without edition

{
    "data": {
        "bobTguid": "57B34B4F-58A2-414E-B385-A5503BB782B4",
        "bobStatus": "ENQUEUED"
    }
}

  • With edition:

{
    "data": {
        "newTransactionGUID" : "08CBDBBB-1C4C-490D-A71B-8FBC2A0FC0E3",
        "bobTguid": "57B34B4F-58A2-414E-B385-A5503BB782B4",
        "bobStatus": "ENQUEUED"
    }
}

[IP-008] affects [API] Trust: use on BoB flag

Added flag useOnBoB on enroll/update/trusted meta section to flag to use the transaction on forced BoB and trust keep treatment

When performing Trust enroll treatment, if all transactions being used are set with useOnBoB = false, use last transaction biometrics.

When performing forced BoB trusted update, if query transactions is set with useOnBoB = false, do not use any biometric from update query transaction, use all from reference transaction.

Enroll/update/trusted payload example:

{
    "meta" : {
        ...
        "useOnBoB" : true,
        ...
    },
    "data": {
        ...
    }
}

Bug Fixes

[BF-001] affects [GBDSClient]

Fixed get final notification by pguid URL on GBDS Client

[BF-002] affects [API] Trust

Allow enroll keep treatments with no keys. All keys informed are validated, but not that presence of at least one anymore.

[BF-003] affects [API] Delete person permanent

Fixed physical delete of a person with exception or pending quality.

For that to be fixed:

  • Delete tguids from HBase (history) and RDB (transactions table)

  • Delete from RDB operation log table and notifications table only if has tguids

  • Delete exceptions/candidates/decisions and groups;

[BF-004] affects [GBDS Migration]

Fixed transaction template simplify on person. Simplification now will be made:

  • if tguid is current person tguid

  • if tguid is current person tguid’s biometric-tguid (covering trusted updates with no biometric that points to previous transaction via biometric-tguid column)

[BF-005] affects [API] Enroll/update with four fingers sequence control

Fixed enroll/update with sequence control indexes 940, 941 (four fingers sequence control) with ginger template type GRIAULE_2025.

Internally, API will not extract sequence control images using old extractor and will compare templates on libmultibio with indexes 0 to 9, not 900 to 909.

[BF-006] affects [API][GBDS] Ginger template simplification

Fixed template simplify only for ginger template type GRIAULE_2024 or prior; do not try to simplify ginger templates on GRIAULE_2025 or GRIAULE_2026

[BF-007] affects [GBDS Migration] Performance and fixes for migration

Mode --anomalies from HBase:

  • Added missing index for exception group and timestamp on exceptions table

  • Skip if no aguid, no anomaly date or any other error

  • If reference person does not exists, inserts it, with protection if any error occurs inserting it

  • If entrant transaction does not exists, inserts it, with protection if any error occurs inserting it

  • Create/update exception group organizations

  • Parallel processing with confs:

    • gbds.anomalies.threadPoolSize, default 16

    • gbds.anomalies.threadPoolQueueSize, default 100

  • Fixed group target migration

  • Added configurations for no hit threshold for enroll/update on finger/face:

    • gbds.csi.enroll.finger.nohit.threshold, default 40

    • gbds.csi.enroll.face.nohit.threshold, default 50

    • gbds.csi.update.finger.nohit.threshold, default 40

    • gbds.csi.update.face.nohit.threshold, default 50

  • A finger/face biometric with score below enroll or update threshold (current threshold configuration) but upper than these new settings, has decision UNCERTAIN, below these new settings, is NO_HIT

  • Fixed default reference tguid from RDB and/or HBase for not found NO HIT exception candidate generation

Mode --anomalies from RDB:

  • List performance optimized from RDB

  • Same fixes and performance improvements above

Mode --processCSI

  • List performance optimized from RDB

  • Parallel processing with confs:

    • gbds.processCSI.threadPoolSize, default 16

    • gbds.processCSI.threadPoolQueueSize, default 100

Mode --refused

  • List performance optimized from RDB

  • Parallel processing with confs:

    • gbds.refused.threadPoolSize, default 16

    • gbds.refused.threadPoolQueueSize, default 100

[BF-008] affects [API] Trust

Fixed organization cache every minute on API for Trust

[BF-009] affects [GBDS Migration] GBDS Reextractor

Fixed: Check for template simplify in these cases:

  • Transaction is last (current) person transaction with biometric (that was configured to be reextracted by modality);

  • Transaction is the biometric-tguid transaction for person (pointed by last (current) person transaction with column biometric-tguid);

  • NEW Transaction is not the biometric-tguid transaction for person, but is the last person transaction with biometric (configured to be reextracted by modality)

[BF-010] affects [API] Trust

Fixed Trust Best of Biometrics trusted update; face quality is now used to get better face, not only the latest, if using face template GRIAULE_2025.

Obs: Face template GRIAULE_2024 or prior does not have quality information, so Trust Best of Biometrics trusted update will continue to use latest face.

[BF-011] affects [API] Liveness

Fixed liveness heuristic to accept JPEG2000 images

[BF-014] affects [API]

Fixed delete permanent (physically) a person with only one pending enroll transaction

[BF-015] affects [API] Biobase

Fixed sending related external ids to Biobase

To do so:

  • Biobase model:

    • Endpoint list biographics: POST /biographicBase/biographics

    • Added type to RequestKey on list biographics, to differ keys, external ids and related external ids

      • RequestKey:

        • key

        • value

        • type: enum

          • TGUID

          • PGUID

          • KEY

          • EXTERNAL_ID

          • RELATED_EXTERNAL_ID

    • bump to 1.2.0

Ex:

{
    "ids" : [
        {
            "id" : "01",
            "keys" : [
              { "key": "cpf", "value": "17081647669-182", "type" : "KEY" },
              { "key": "rg", "value": "rg-182", "type" : "KEY" },
              { "key": "tguid", "value": "0C2A05E2-3945-459A-9393-FED297CD1D78", "type" : "TGUID" },
              { "key": "pguid", "value": "A0C3B7D6-2BC4-4A63-83A8-0A349BCA7E48", "type" : "PGUID" },
              { "key": "id01", "value": "key01", "type" : "EXTERNAL_ID" },
              { "key": "id02", "value": "key02", "type" : "RELATED_EXTERNAL_ID" }
            ]
        }
    ]
}

Obs: new field type does not affect current Biobase servers, being retrocompatible.

  • API:

    • bump biobase-model-1.2.0

    • Send to Biobase service related external ids on get/list transaction, get/list people, lights out, qualtiy analysis list (as request key type RELATED_EXTERNAL_ID)

    • Send others fields, as tguid/pguid, keys and external ids with respective request key type, as described above

[BF-016] affects [API] Trust

Fixed MISMATCH exception fusion decison as null on RDB table gbds.exceptions

[BF-017] affects [API][GBDS] Trust / Refused / Previous history

Fixed previous history on some scenarios:

  • Keep previous history on updates

  • On refused resend, is refused is an update, keep previous history from current person transaction

  • On refused resend, if refused is an enroll, make previous history with tguid and pguid

  • Get all transactions using tguid and pguid for previous history (Trust treatments and refused resending)

[BF-022] affects [GBDS Migration] GBDS Reextrator

Fixed set transaction as error on any reextraction error

Set better max tries exception message for extraction, quality extraction and reextraction

Re-start extractors and keep it down on error while restarting

[BF-023] affects [API] RDB missing indexes

Fix: added missing indexes to clear-rdb that already was on upgrade-rdb

[BF-024] affects [API] Titulo key validation

Fixed key titulo validation; allow only titulo key with 12 numeric characters, no more, no less, besides current checksum validation

[BF-026] affects [API} Trust

Fixed race condition in Trust biometric treatment where two users could process different biometrics for the same exception and this would make Trust unable to correctly analyze whether the inconformity should change its target from BIOMETRIC to BIOGRAPHIC, MISMATCH or INCONCLUSIVE

To do that, API will get exception again from database after each treatment, to have all information up to date for analysis.

[BF-027] affects [GBDS Migration]

Migration scripts updated, to allow GBDS Migration of exceptions/quality during GBDS 4.7 execution.

Scripts down to:

  • upgrade-rdb-4.x-to-5.1-part1-before-4-stops.sql

  • upgrade-rdb-4.x-to-5.1-part2-after-4-stops.sql

  • upgrade-rdb-4.x-to-5.1-part3-after-5-starts.sql

Before this, migration used to rename some columns:

  • gbds.transactions, column type to transaction_type

  • gbds.transaction_fields, column type to field_type

  • gbds.fields, column type to field_type

  • gbds.exceptions, column status to exception_status

Causing error on GBDS 4.7 execution if migration is made along with it.

New script part 1 (upgrade-rdb-4.x-to-5.1-part1-before-4-stops.sql) only create these new columns, keeping old ones, and set its values the same as old ones. This allows GBDS 4.7 and migration to run simultaneously.

Columns created:

  • gbds.transactions, column transaction_type

  • gbds.transaction_fields, column field_type

  • gbds.fields, column field_type

  • gbds.exceptions, column exception_status

On delta migration, script part 2 (upgrade-rdb-4.x-to-5.1-part2-after-4-stops.sql) updates these new columns with GBDS 4.7 difference.

After GBDS 5.1 start, script part 3 (upgrade-rdb-4.x-to-5.1-part3-after-5-starts.sql) drops old columns.

Columns dropped:

  • gbds.transactions, column type

  • gbds.transaction_fields, column type

  • gbds.fields, column type

  • gbds.exceptions, column status

5.1.

5.1.12

Improvements

  • Ginger extractor updated to 2.3.0

Bug Fixes

  • Fixed a bug where update exceptions generated by two profiles with the same face and different fingers couldn't be treated

  • Fixed a bug where refused transactions were being sent as updates instead of enrolls after reject treatments

  • Fixed a collateral bug from updates with the same face and different fingers (5.1.12) where it wasn't possible to generate biometric targets nonconformities.

    • from fusion decision "UNCERTAIN" to target "BIOMETRIC", both enroll and update

    • from fusion decision "MISMATCH", treted as "NO_HIT", to target "BIOMETRIC", only update, when one modality is "HIT" and the other is "NO_HIT" with no uncertain biometric

5.1.11

New Features

  • A new validation mechanism was introduced for biometric image used in Liveness search to ensure no data was altered during transmission. This mechanism applies exclusively to requests made to the endpoint POST /v2/people/searches with search type set to LIVENESS.

Payloads (specifically the content of biometric images) in version 2 are now validated using a cryptographic HMAC signature.

Improvements

  • Support for new extractors

    • Face templates simplification are no longer made on transactions/people

    • Finger templates are no longer mande on people

    • Quality control and sequence control are made using multibio comparison

  • Query performance for list groups have been improved

Bug Fixes

  • Fixed a bug where update transactions received from an update caused by the enroll of a person with non-deduplicated keys were being sent as enroll transactions instead of update

5.1.10

New Features

  • Now group pending can be configured to go to analysis or be directly treated

    • TRUE: group treatment can go to pending status, according to user permisisons over all group organizations

    • FALSE : group treatment is always final, regardless user permissions over all group organizations

Improvements

  • Native library Facepad urgraded to 2.3.1, which is used on liveness analysis

  • Matcher update

    • New face quality analysis for UPDATE, which now can return face UNCERTAIN

Bug Fixes

  • Fixed bug where the search result was changing depending on the number of candidates

  • Fixed a bug where wasn't possible to delete a person when this person was reactivated using a trusted update

5.1.9

Bug Fixes

  • Fixed a bug where UPDATE exceptions treated with only its reference being kept were getting lost

  • Now listing people with "include anomalies = false" returns correctly only people not related to an anomaly

5.1.8

New Features

  • Fusion (1:1 and 1:N) decision by biometry and mismatch

    • If micro service and libmultibio returns MISMATCH, exception and group is BIOMETRIC_MISMATCH automatically

    • If no match is returned by fusion, API and Trust will assume NO HIT

Bug Fixes

  • Now enroll status is set correctly to FAILED on entrant transactions when a Trust treatment for update occurs

5.1.7

Bug Fixes

  • [TRUST] Fixed a bug where refused enrolls were being sent as updates

  • Fixed a bug where listing people with endDate filter would return error 500

  • Now if the setting for BoB (Best of Biometrics) use in an update is absent, it is set to true automatically

5.1.6

Improvements

  • Now "get group" returns a flag indicating if the group is treatable by the user (user has to have the appropriate permissions)

  • Prioritization now is only available for groups in analysis, pending, refused created or ready to send

Bug Fixes

  • Fixed a bug REFUSED status changes were not validating the user permissions

5.1.5

Bug Fixes

  • Fixed bug where update transactions in profiles that were created a long time ago were taking too long

  • Fixed a bug where Trust was using biometrics from rejected transactions on Enroll Unify treatments

  • Verify and update 1:1 searches now uses correctly ALL_INDEXES and SAME_INDEX during its execution

4.7.

4.7.28

Improvements

  • Reextraction can be made now with TGUIDs and PGUIDs lists

    • File format: one or more GUID per line, comma separated if more than one on same line

Bug Fixes

  • Fixed a bug where simplification wasn't being made when the latest transaction didn't had biometrics

Improvements

  • Priority change now is only available for groups in analysis, pending, refused or ready to send

4.7.27

Improvements

  • Ginger extractor upgraded to v2.3.0

    • On GBS Reextractor, Ginger extractor was upgraded to 2.3.0 for reextraction only

4.7.26

New Features

  • Added support for new extractors (Ginger v2.2.0 and Facelib 3.1.1). The current extractors remain fully supported and available

4.7.25

Bug Fixes

  • Fixed a bug where minutias searched with the previous template were shown in the wrong position

4.7.24

Bug Fixes

  • Fixed bug related to UL searches

    • Updated the post-match filtering logic to limit the results to the top X highest scores.

Last updated

Was this helpful?