Description

This rule verifies that alerting is configured for the core health signals of your DigitalOcean infrastructure: CPU utilization, memory utilization, disk utilization, and endpoint downtime. DigitalOcean splits this across two products, so a complete implementation uses both. Resource Alerts (INSIGHTS -> Monitoring) cover per-Droplet metrics such as CPU, memory, and disk utilization and require the DigitalOcean metrics agent on the target Droplets. Uptime (a separate monitoring service) covers downtime by probing a URL or IP over HTTP, HTTPS, or ping and alerting when the endpoint stops responding.

The rule inspects your account for alert policies and uptime checks/alerts and matches them against a checklist of expected signal types per resource class. It flags resources that have no CPU, memory, or disk threshold alert, and public endpoints that have no downtime alert.

  • Droplets (or tags) with no CPU utilization alert policy.

  • Droplets (or tags) with no memory utilization alert policy.

  • Droplets (or tags) with no disk utilization alert policy.

  • Public endpoints or Droplets with no Uptime check or no downtime alert attached to an existing check.

  • Alert policies that exist but are disabled (enabled=false).

Rationale

Without baseline alerting, ordinary operational failures degrade or take down a service silently. A runaway process pins CPU, a memory leak triggers the OOM killer, logs or a database fill a disk to 100%, or a host stops answering entirely, and the first signal is an angry customer rather than a page to the on-call engineer. Threshold alerts on CPU, memory, and disk give early warning while there is still time to intervene; a downtime alert catches the total-failure case that resource metrics cannot report because the agent itself may be down.

These are the minimum signals that turn an outage from a discovery into a notification. They also create an auditable record that the operator has active monitoring coverage over production infrastructure.

  • CPU/memory/disk alerts catch gradual degradation before it becomes an outage.

  • Downtime (Uptime) alerts catch total failures that metric agents cannot report.

  • Alerting shrinks mean-time-to-detect and provides evidence of monitoring coverage.

Impact

Pros:

  • Early warning on resource exhaustion before it causes an outage.

  • Immediate notification when an endpoint or host goes fully down.

  • Faster mean-time-to-detect and mean-time-to-respond.

  • Auditable evidence that production infrastructure is actively monitored.

Cons:

  • Poorly tuned thresholds cause alert fatigue from false positives.

  • Resource alerts require the metrics agent installed and running on each Droplet.

  • Uptime checks and, on some plans, additional alert volume can carry cost.

  • Ongoing effort to keep policies attached as new Droplets and tags are created.

Default Value

DigitalOcean creates no resource alert policies or uptime checks by default; DigitalOcean-provided Droplet images ship with the metrics agent preinstalled, but no thresholds or downtime alerts are configured until you create them.

Pre-Requisite

  • DigitalOcean Control Panel access with permission to manage monitoring (Owner, or a member/agent role with the equivalent rights).

  • A DigitalOcean API token, or doctl authenticated (doctl auth init), to check or create policies programmatically.

  • The DigitalOcean metrics agent installed on target Droplets (required for CPU/memory/disk resource alerts).

  • Knowledge of which Droplets/tags are production and which public URLs or IPs represent user-facing endpoints.

  • Notification destinations decided in advance (email addresses and/or a Slack integration).

Remediation

Test Plan

Using the DigitalOcean Control Panel:

  1. Sign in to cloud.digitalocean.com.

  2. In the left menu click INSIGHTS, then Monitoring.

  3. Review the list of Resource Alert policies on the Monitoring page.

  4. Confirm that policies exist covering CPU Utilization Percent, Memory Utilization Percent, and Disk Utilization Percent for your production Droplets or their tags, and that each is enabled.

  5. In the left menu open Uptime, and review your uptime checks.

  6. Open each check that fronts a public endpoint and confirm it has a downtime alert configured (type 'down').

  7. Pass: Enabled alert policies exist for CPU, memory, and disk on every production Droplet/tag, and each public endpoint has an Uptime check with a downtime alert.

  8. Fail: Any production Droplet/tag is missing a CPU, memory, or disk alert, or any public endpoint lacks an Uptime check with a downtime alert (or a policy exists but is disabled).

Using doctl or the API:

List existing alert policies and uptime checks/alerts and confirm the required signal types are present.

doctl monitoring alert list
curl -s -X GET "https://api.digitalocean.com/v2/monitoring/alerts" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" | jq '.policies[] | {uuid, type, enabled, entities, tags}'
doctl monitoring uptime list
curl -s -X GET "https://api.digitalocean.com/v2/uptime/checks" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN"
doctl monitoring uptime alert list <check-uuid>
curl -s -X GET "https://api.digitalocean.com/v2/uptime/checks/<check-uuid>/alerts" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN"

Pass if the policies list contains enabled alerts of type v1/insights/droplet/cpu, v1/insights/droplet/memory_utilization_percent, and v1/insights/droplet/disk_utilization_percent covering your production entities/tags, and each public endpoint's uptime check has an alert of type 'down'. Fail if any of these signal types is missing, disabled, or does not cover the intended resources.

Implementation Steps

Using the DigitalOcean Control Panel:

  1. Sign in to cloud.digitalocean.com and open INSIGHTS -> Monitoring.

  2. Click Create Resource Alert.

  3. Under 'Select metric & set threshold', choose CPU Utilization Percent, set the rule to 'is above', enter a threshold (for example 80), and pick a duration such as 5 min.

  4. Under 'Select Droplets or Tags', choose the production Droplets or a tag that covers them (tags auto-include future Droplets).

  5. Under 'Select alert notification method', enable Email and/or Slack.

  6. Name the alert and click Create Resource Alert.

  7. Repeat for Memory Utilization Percent (for example above 85) and Disk Utilization Percent (for example above 85-90).

  8. In the left menu open Uptime and click Create Uptime Check; enter the target URL or IP, choose the check type (HTTP/HTTPS/ping) and regions, and save.

  9. On the new check, add an alert of type 'Down' with your notification destinations to be paged when the endpoint stops responding.

Using doctl or the API:

Create the missing alert policies and uptime checks/alerts with doctl or the API.

doctl monitoring alert create --type "v1/insights/droplet/cpu" --compare GreaterThan --value 80 --window 5m --tags production --emails [email protected] --description "CPU > 80%"
doctl monitoring alert create --type "v1/insights/droplet/memory_utilization_percent" --compare GreaterThan --value 85 --window 5m --tags production --emails [email protected] --description "Memory > 85%"
doctl monitoring alert create --type "v1/insights/droplet/disk_utilization_percent" --compare GreaterThan --value 85 --window 5m --tags production --emails [email protected] --description "Disk > 85%"
curl -s -X POST "https://api.digitalocean.com/v2/monitoring/alerts" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" -H "Content-Type: application/json" -d '{"type":"v1/insights/droplet/cpu","compare":"GreaterThan","value":80,"window":"5m","enabled":true,"tags":["production"],"alerts":{"email":["[email protected]"]},"description":"CPU > 80%"}'
doctl monitoring uptime create "app endpoint" --target https://app.example.com --type https --regions us_east,us_west --enabled true
doctl monitoring uptime alert create <check-uuid> --name "app down" --type down --period 2m --emails [email protected]

Backout Plan:

  1. If an alert proves too noisy, edit it (INSIGHTS -> Monitoring -> Resource Alerts -> ... -> Edit, or doctl monitoring alert update) to raise the threshold or widen the window rather than deleting it.

  2. To remove a resource alert, open Resource Alerts, click ..., choose Delete, type the alert name and confirm (or doctl monitoring alert delete <uuid>).

  3. To remove a downtime alert, delete it from the Uptime check (or doctl monitoring uptime alert delete <check-uuid> <alert-uuid>); delete the check itself with doctl monitoring uptime delete <check-uuid>.

  4. Record which policies were removed so monitoring coverage can be restored intentionally.

References:

  1. Monitoring Features

  2. How to Create, Edit, and Delete Resource Alerts

  3. doctl monitoring alert

  4. DigitalOcean Uptime

  5. doctl monitoring uptime alert create