Description

Ensure that log-based metric filters and corresponding alerting policies are configured to detect and notify on Compute Engine configuration changes, such as instance creation, deletion, or modification of instance metadata, network settings, or firewall rules affecting Compute Engine resources.

Rationale

Without proactive monitoring, unauthorized or unexpected changes to Compute Engine configuration, whether from a compromised credential, a misconfigured automation pipeline, or insider action, may go unnoticed until they cause an incident or are discovered during a routine audit. Configuring log metric filters against Cloud Audit Logs, paired with alerting policies, ensures that security and operations teams are notified promptly when significant Compute Engine configuration changes occur, enabling timely investigation and response.

Impact

  • Provides near real-time notification of significant Compute Engine configuration changes

  • Improves the ability to detect unauthorized or unexpected changes quickly

  • Requires ongoing tuning to avoid alert fatigue from expected, routine configuration changes

Default Value

No log metric filters or alerting policies for Compute Engine configuration changes are configured by default; these must be created explicitly in Cloud Logging and Cloud Monitoring.

Pre-Requisites

  • Cloud Audit Logs (Admin Activity) enabled and being received for Compute Engine (enabled by default)

  • A Cloud Monitoring notification channel (email, Slack, PagerDuty, etc.) configured to receive alerts

  • IAM permissions to create log-based metrics and alerting policies (e.g., Logging Admin, Monitoring Admin)

Test Plan

  1. Sign in to the Google Cloud Console at https://console.cloud.google.com.

  2. Select the relevant project.

  3. Navigate to Logging > Log-based Metrics.

  4. Verify that a metric exists filtering for Compute Engine configuration change events (for example, resource.type="gce_instance" AND protoPayload.methodName matching insert/delete/update operations).

  5. Navigate to Monitoring > Alerting and verify an alerting policy is configured against that metric with an appropriate notification channel.

  6. Verify the alert has been tested or has fired appropriately in response to a known change.

  7. If no such metric or alert exists, follow the implementation steps.

Implementation Steps

  1. Sign in to the Google Cloud Console at https://console.cloud.google.com.

  2. Select the relevant project.

  3. Navigate to Logging > Log-based Metrics and select Create Metric.

  4. Define a Counter metric with a filter targeting Compute Engine admin activity events, such as resource.type="gce_instance" combined with relevant protoPayload.methodName values (e.g., compute.instances.insert, compute.instances.delete, compute.instances.setMetadata).

  5. Save the metric.

  6. Navigate to Monitoring > Alerting and select Create Policy.

  7. Select the log-based metric created above as the condition, define an appropriate threshold (for example, any count greater than 0 within a defined window), and attach a notification channel.

  8. Save the alerting policy and verify it triggers correctly, for example by making a test configuration change.

Backout Plan

  1. Sign in to the Google Cloud Console at https://console.cloud.google.com.

  2. Select the relevant project.

  3. Navigate to Monitoring > Alerting, select the alerting policy created for Compute Engine configuration changes, and delete or disable it.

  4. Navigate to Logging > Log-based Metrics, select the corresponding metric, and delete it if it is no longer needed.

Reference