# Database Encryption ## Prerequisites The procedure described in this manual depends on the following requirements: * [Apache Ranger™ installation procedure](https://docs.griaule.com/gbs/en/auxiliary-tools/apacheranger) * [Luna Cloud HSM installation procedure (Optional)](https://docs.griaule.com/gbs/en/auxiliary-tools/lunacloudhsm) ## Create key for encryption ### User permission To encrypt HBase, it is necessary to add permissions for the user who administers HDFS. In the browser, access Ranger at `http://:6080`. * User: `keyadmin` * Password: `` {% hint style="info" %} In this example, we will use the user `hadoop` {% endhint %} When accessing the Ranger KMS Dashboard, click on the repository `kmsdev` of the KMS, which will redirect to the *policies* of *kmsdev*. Click to edit the `policy 1`. When redirected, click the + icon inside Allow Conditions and fill it out as follows: * Select Role: *No change* * Select Group: *No change* * Select User: `hadoop` * Permissions: `Decrypt EEK`, `Generate EEK`, `Get`, `Get Keys`, `Get Metadata` * Delegate Admin: *Checked* To test access to the server, run the following commands to list the keys and the key metadata, if a key exists: ```sh hadoop key list hadoop key list -metadata ``` The aim of the test is to ensure that access will not be denied, whether a key exists or not. ### Create key Still on the Ranger KMS dashboard, click Encryption and then Key Manager. In Select Service, choose the repository `kmsdev`. To create a key, click Add New Key and fill it out as follows: * Key Name: `hbase` * Cipher: *Do not change* * Length: `256` * Description: optional field * Attributes: *Do not change* By clicking save, the key will be created and can be accessed by the server for encryption. To verify, connect to the server with the user who has permission to `get key` and run the following commands: ```sh hadoop key list hadoop key list -metadata ``` ## Create encryption zone To encrypt the Hadoop database, it is necessary to create an encryption zone. That way, all data inserted into this zone will be redirected by HDFS to the KMS for encryption or decryption, depending on the request sent and the user's permissions in the KMS. To create an encryption zone, the path must be empty. Therefore, it is necessary to rename the folder `data` of HBase and create a folder with the same name. ```sh hdfs dfs -mv /apps/hbase/data /apps/hbase/data-bkp hdfs dfs -mkdir /apps/hbase/data /apps/hbase/data hdfs dfs -ls /apps/hbase/ # Found 2 items # drwxr-xr-x - hadoop hadoop 0 2023-02-17 14:02 /apps/hbase/data # drwxr-xr-x - hadoop hadoop 0 2023-02-15 11:58 /apps/hbase/data-bkp ``` Then run the following command to set the path `/apps/hbase/data` as an encryption zone: ```sh hdfs crypto -createZone -keyName hbase -path /apps/hbase/data ``` The return should be: ``` Added encryption zone /apps/hbase/data ``` To verify, run the command to list the zones: ```sh hdfs crypto -listZones ``` This command should return the following message: ``` /apps/hbase/data hbase ``` Where `/apps/hbase/data` is the encryption zone and `hbase` is the key. After defining the encryption zone, copy all the contents of the folder `/apps/hbase/data-bkp` into the encryption zone `/apps/hbase/data`. ```sh hdfs dfs -cp /apps/hbase/data-bkp/* /apps/hbase/data/ ``` To verify the encryption operation, start HBase and connect to the HBase Shell, then run the following file read commands: Inside the encryption zone: ```sh hdfs dfs -cat /apps/hbase/data/hbase.id ``` The return should be: ``` PBUF $24b1ef30-14b6-46d0-b46b-f49c9c109cb1 ``` Outside the encryption zone: ```sh hdfs dfs -cat /.reserved/raw/apps/hbase/data/hbase.id ``` The return should be: ``` ▒w▒▒▒▒▒,U▒▒▒k▒y▒,*▒*|▒~▒/U▒U▒▒▒|# ``` To get information about the file encryption, run the following command: ```sh hdfs crypto -getFileEncryptionInfo -path /apps/hbase/data/hbase.id ``` The return should be: ```properties { cipherSuite: { name: AES/CTR/NoPadding, algorithmBlockSize: 16 }, cryptoProtocolVersion: CryptoProtocolVersion{ description='Encryption zones', version=2, unknownValue=null }, edek: 0b79b49c77335747824d78e97e5e0bf2a54f428ddc81faf6dc220c4ecea7c7de, iv: ee35b713df8994046758372cee3eeea0, keyName: hbase, ezKeyVersionName: hbase@0 } ``` ## Key rollover procedure {% hint style="warning" %} It is not necessary to stop services for the key rollover and re-encryption. Read the complete procedure below for more information. {% endhint %} ### Key Rollover Procedure To create a key, click Add New Key and fill it out as follows: To change the encryption key, access the Ranger KMS dashboard and click Encryption and then Key Manager. In Select Service, choose the repository `kmsdev`. On the key used for the encryption zone, on the right side of the screen in Action, click the icon described as Rollover. When clicking Rollover, you will need to confirm the operation in a pop-up. After confirming the *Rollover*, Hadoop will continue using the previous key, but the new key will also be available in the Ranger KMS. For this reason it is not necessary to stop services for this procedure. To check the availability of the keys and their respective versions, run the following command: ```sh curl http://localhost:9292/kms/v1/key/hbase/_versions?user.name=hadoop ``` The return should be: ```json [ { "material": "NIoXKS9lHMIPBUhIARV3V5TCzTj12IHEOjVwD00R8NM", "name": "hbase", "versionName": "hbase@0" }, { "material": "RjAvLWzEu7BuRvPpXs7K2Q8EqRTa_gJsDm-NF8D_HSc", "name": "hbase", "versionName": "hbase@1" }, ... ] ``` In this return, note that Hadoop is using the key `hbase@0`, but the key `hbase@1` is already available in the KMS. ### Re-encryption procedure The re-encryption procedure can be done while the system is fully operational in two aspects: #### The type of encryption Hadoop uses TDE (*transparent data encryption*). This type of encryption acts on the database only at the file level, allowing the data to be available to applications without the need for encryption or decryption on every operation. #### Encryption information in the file Each encrypted file has a header or package with the encryption information, including the version of the key used. This information is consulted directly in any encryption procedure. {% hint style="info" %} To check the file encryption information, run the following command: ```sh hdfs crypto -getFileEncryptionInfo -path /apps/hbase/data/hbase.id ``` The return should be: ```properties { cipherSuite: { name: AES/CTR/NoPadding, algorithmBlockSize: 16 }, cryptoProtocolVersion: CryptoProtocolVersion{ description='Encryption zones', version=2, unknownValue=null }, edek: 0b79b49c77335747824d78e97e5e0bf2a54f428ddc81faf6dc220c4ecea7c7de, iv: ee35b713df8994046758372cee3eeea0, keyName: hbase, ezKeyVersionName: hbase@0 } ``` {% endhint %} To re-encrypt the encryption zone after the *Rollover* of the key, run the following commands: * To list the encryption zones: ```sh hdfs crypto -listZones ``` * To re-encrypt the desired zone with the new key: ```sh hdfs crypto -reencryptZone -start -path /apps/hbase/data/ ``` The return should be a message confirming the re-encryption request: ``` re-encrypt command successfully submitted for zone: /apps/hbase/data/ action: START ``` After the request, Hadoop will start re-encrypting the data with the updated key provided by Ranger KMS. To check the status of the re-encryption, run the following command: ```sh hdfs crypto -listReencryptionStatus ``` The return should be a report with the status of the operation: ``` Zone Name Status EZKey Version Name Submission Time Is Canceled? Completion Time Number of files re-encrypted Number of failures Last File Checkpointed /apps/hbase/data Completed hbase@1 2023-02-17 15:43:15,429-0300 false 2023-02-17 15:43:15,623-0300 28 0 ``` To verify if the encryption information has been updated in the files, run the command `-getFileEncryptionInfo`: ```sh hdfs crypto -getFileEncryptionInfo -path /apps/hbase/data/hbase.id ``` If the key change has been successfully performed, the field `ezKeyVersionName` will reflect the name of the new version: ```properties { cipherSuite: { name: AES/CTR/NoPadding, algorithmBlockSize: 16 }, cryptoProtocolVersion: CryptoProtocolVersionc{ description='Encryption zones', version=2, unknownValue=null }, edek: 995afbc575fa88ff0ef72a908e0caa8397c18b2df852cb86e35fcc4577ed257b, iv: ee35b713df8994046758372cee3eeea0, keyName: hbase, ezKeyVersionName: hbase@1 } ```