Description:

Enabling Customer Managed Keys (CMK) ensures that both data at rest and data in transit within Azure Databricks are encrypted using keys that are managed by the customer rather than relying on Microsoft's managed keys. This configuration enhances security by allowing the customer to control the lifecycle and access to the encryption keys. CMK is essential for organizations that need to meet strict compliance requirements or require more granular control over their encryption.

Rationale:

Encryption at rest protects stored data, while encryption in transit secures data being transferred across networks. By using CMK, organizations ensure that the encryption keys used for both types of encryption are fully under their control. This provides a higher level of security and compliance with regulatory standards, as organizations are responsible for key management, rotation, and access policies.

Impact:

Enabling CMK encryption increases security by controlling how keys are generated, stored, and accessed. It may introduce additional overhead in terms of key management, such as ensuring keys are rotated regularly, managing access controls, and addressing any potential key access issues. It may also introduce slight performance overhead due to the encryption and decryption processes, but the security benefits far outweigh this.

Default Value:

By default, Azure Databricks uses Microsoft Managed Keys (MMK) for encryption at rest and in transit. Enabling Customer Managed Keys (CMK) for Databricks requires manual configuration and is available with Databricks premium plans.

Pre-requisites:

  • Azure account with Azure Databricks workspace.

  • Azure Key Vault configured to store the Customer Managed Keys.

  • The user must have Databricks Admin permissions or equivalent access to configure encryption settings.

  • Azure Active Directory integrated with Azure Databricks for identity management.

Audit:

  1. Sign in to Azure portal as a Databricks Admin or equivalent role.

  2. Navigate to the Azure Databricks workspace.

  3. Verify the encryption settings to ensure that both data at rest and data in transit are encrypted using Customer Managed Keys (CMK).

Implementation steps(Automated):

  1. Sign in to Azure portal:

    • Use an account with Databricks Admin or Global Admin permissions.

  2. Create and Configure Customer Managed Keys (CMK) in Azure Key Vault:

    • In Azure Key Vault, create a new key or select an existing key that will be used for encryption.

    • Ensure the Key Vault is in the same region as your Databricks workspace.

    • Configure Access Policies to allow the Azure Databricks workspace to access the keys.

  3. Enable CMK for Azure Databricks:

    • Navigate to your Azure Databricks workspace in the Azure portal.

    • Under the Databricks workspace, go to the Admin Console and select Encryption.

    • Choose the option to use Customer Managed Keys (CMK) for encryption.

    • Select Azure Key Vault and the Customer Managed Key you created earlier.

    • Apply the configuration to enable data at rest encryption using CMK.

  4. Enable CMK for Data in Transit:

    • Ensure that Transport Layer Security (TLS) is enabled for securing data in transit between Databricks services and external resources.

    • Configure your Key Vault to manage the TLS keys for encryption in transit.

    • You can configure your custom TLS certificates to manage encrypted communications between worker nodes, clusters, and external services.

  5. Automate Key Rotation:

    • Set up automatic key rotation policies in Azure Key Vault to rotate keys periodically to maintain a secure environment.

    • Configure Azure Key Vault to automatically regenerate the keys at defined intervals (e.g., 90 days).

  6. Verify and Monitor:

    • After setting up CMK encryption, test your environment by verifying that new data stored in Databricks is encrypted with CMK.

    • Use Azure Monitor or Azure Security Center to track encryption status and compliance.

    • Review audit logs to ensure encryption settings are enforced.

  7. Automate Monitoring and Compliance:

    • Set up automated alerts to notify administrators if there are any issues with encryption compliance, such as expired keys, failed key access, or issues with the key vault.

Backout Plan (Automated):

  1. Revert to Microsoft Managed Keys (MMK):

    • Sign in to Azure Databricks as a Databricks Admin.

    • Navigate to the Admin Console and go to Encryption settings.

    • Switch the configuration from Customer Managed Keys (CMK) to Microsoft Managed Keys (MMK).

    • Save and apply the changes.

  2. Revoke Access to Key Vault:

    • In Azure Key Vault, remove or modify the access policy that allows Azure Databricks to access the encryption keys.

    • This will ensure Databricks no longer uses the CMK for data encryption.

  3. Test Databricks Access:

    • After reverting to MMK, test the Databricks environment to confirm that no unauthorized access occurs and that the encryption settings have been reverted.

References: