Description:
Configuring diagnostic log delivery for Azure Databricks ensures that key operational and security logs are captured and stored for monitoring, auditing, and troubleshooting purposes. These logs include information about cluster activities, job execution, user activities, and system health, which are essential for identifying issues, ensuring compliance, and optimizing performance.
Rationale:
Diagnostic logs are crucial for maintaining operational efficiency and security in a Databricks environment. By enabling log delivery to a Log Analytics workspace, Azure Storage, or Event Hub, organizations can centralize log storage, apply monitoring rules, and create automated alerts based on log events. This configuration also helps meet compliance requirements by maintaining an auditable log of activities and system events.
Impact:
Enabling diagnostic log delivery ensures that logs are collected in a centralized location for easy access and analysis. However, this may introduce additional storage costs depending on the volume of logs generated. It’s also essential to manage log retention policies to avoid excessive storage costs while ensuring that logs are kept for the required duration.
Default Value:
By default, diagnostic log delivery is not enabled for Azure Databricks. This must be manually configured to ensure that logs are delivered to a central monitoring solution.
Pre-requisites:
Azure account with access to Azure Databricks workspace.
Admin access to configure diagnostic log delivery in Databricks.
A configured Log Analytics workspace or Azure Storage Account for log storage.
A user must have the Databricks Admin role or appropriate permissions.
Audit:
Sign in to the Azure portal as a Databricks Admin or Global Admin.
Navigate to the Azure Databricks workspace and review the Diagnostic Settings.
Verify that diagnostic logs are being delivered to a Log Analytics workspace, Azure Storage, or Event Hub as configured.
Implementation Steps:
Sign in to the Azure portal with Databricks Admin or appropriate permissions.
Navigate to the Azure Databricks workspace:
Open the Azure Databricks workspace in the Azure portal.
Configure Diagnostic Log Delivery:
In the Databricks workspace, go to the Diagnostics Settings.
Select + Add diagnostic setting to create a new configuration.
Select Log Delivery Destination:
Choose the destination where you want the logs to be delivered. The available options are:
Log Analytics workspace: Ideal for advanced monitoring and querying with Azure Monitor.
Azure Storage Account: Good for long-term log storage and backup.
Event Hub: Useful for streaming logs to external systems or third-party services.
Choose Logs and Metrics to Collect:
Select the types of logs you want to capture:
Audit Logs: Logs related to user and administrative activities.
Cluster Logs: Logs related to cluster creation, termination, and execution.
Job Logs: Logs from Databricks jobs, including run status, errors, and execution times.
Driver and Worker Logs: Detailed logs of the Databricks clusters, including system health and performance metrics.
Configure Retention and Storage:
Specify how long the logs should be retained in your selected destination.
Set retention policies to automatically delete old logs based on your organization’s requirements.
Save and Apply the Configuration:
Review your settings and click Save to enable log delivery.
Validate the Configuration:
After configuration, validate that logs are being delivered correctly by querying the logs in Log Analytics or checking the logs in the Azure Storage Account or Event Hub.
Backout Plan:
Sign in to the Azure portal as a Databricks Admin or Global Admin.
Navigate to the Azure Databricks workspace and open the Diagnostic Settings.
Remove or disable the diagnostic setting that you previously configured.
Verify that logs are no longer being delivered to the specified destination (e.g., Log Analytics workspace, Azure Storage, or Event Hub).
If necessary, delete any stored logs from the chosen destination to prevent unnecessary storage costs.