Description:

Network Security Groups (NSGs) are used to control inbound and outbound traffic to Azure resources. When deploying Azure Databricks in a customer-managed Virtual Network (VNet), it is essential to configure NSGs for the Databricks subnets to manage the flow of network traffic. This configuration ensures that only authorized traffic is allowed to access Databricks resources, enhancing security by isolating Databricks environments from unauthorized network access.

Rationale:

Configuring NSGs for Databricks subnets helps enforce security controls by allowing you to define rules that specify which traffic is allowed or denied based on IP addresses, ports, and protocols. This is crucial for preventing unauthorized access to Databricks clusters and protecting sensitive data from exposure. Additionally, NSGs help with compliance by ensuring that only authorized resources within the network can communicate with Databricks.

Impact:

Properly configured NSGs help to secure the Databricks environment by restricting access to only trusted network resources. However, incorrect or overly restrictive rules can lead to connectivity issues, disrupting legitimate traffic or preventing required services from functioning. It is important to ensure that the rules allow necessary traffic while blocking unauthorized or unnecessary access.

Default Value:

By default, NSGs are not applied to Databricks subnets. You must manually configure NSGs to define the inbound and outbound traffic rules for these subnets.

Pre-requisites:

  • Azure account.

  • Azure Databricks workspace deployed in a customer-managed VNet.

  • Azure Virtual Network (VNet) and subnets associated with Databricks.

  • User must have appropriate permissions to manage Network Security Groups (e.g., Network Admin, Owner, Databricks Admin).

Audit:

  1. Sign in to the Azure portal as a Network Admin, Owner, or Databricks Admin.

  2. Navigate to the Azure Databricks workspace and find the subnet where Databricks is deployed.

  3. Verify that NSGs are properly configured for the Databricks subnet by reviewing the inbound and outbound traffic rules.

  4. Ensure that only authorized traffic is allowed based on the security policies defined for the Databricks environment.

Implementation Steps:

  1. Sign in to the Azure portal with Network Admin, Owner, or Databricks Admin privileges.

  2. Identify the Databricks subnet within your Azure Virtual Network (VNet):

    • Go to Azure Databricks workspace and identify the subnet assigned to it.

  3. Create or modify the Network Security Group (NSG):

    • In the Azure portal, navigate to Network Security Groups.

    • Either create a new NSG or modify an existing NSG.

  4. Configure inbound and outbound rules for the Databricks subnet:

    • Inbound rules: Define which sources (e.g., IP addresses, VNets) are allowed to connect to the Databricks subnet.

      • Allow access from trusted IP ranges or internal resources.

      • Block any public or unauthorized access.

    • Outbound rules: Specify which destinations Databricks can communicate with (e.g., Azure Storage, SQL Databases).

      • Allow traffic to Azure services that Databricks requires, while blocking unnecessary external access.

  5. Apply the NSG to the Databricks subnet:

    • In the NSG settings, associate the NSG with the appropriate subnet used by Databricks.

  6. Verify connectivity:

    • After applying the NSG, ensure that Databricks can access required services while ensuring that unauthorized traffic is blocked.

Backout Plan:

  1. Sign in to the Azure portal as a Network Admin, Owner, or Databricks Admin.

  2. Navigate to the Network Security Group (NSG) associated with the Databricks subnet.

  3. Remove or modify the NSG rules that were configured for the subnet.

    • Set the rules to a more permissive configuration or remove the NSG association from the subnet.

  4. Test the Databricks connectivity to ensure it is functioning correctly.

  5. Save and apply the changes.

References: