Skip to main content

Alerting

If you are using Conductor, you can configure automatic alerts when certain failure conditions are met. You can configure alerts either in Conductor directly or on Conductor-exported metrics using your existing observability stack.

info

Alerts require at least a DBOS Teams plan.

Creating Alerts​

You can create new alerts (or view or update your existing alerts) from your application's "Alerting" page on the DBOS Console.

Alerts

Currently, you can create alerts for the following failure conditions:

  • If a certain number of workflows (parameterizable by workflow type) fail in a set period of time.
  • If a workflow remains enqueued for more than a certain period of time (parameterizable by queue name), indicating the queue is overwhelmed or stuck.
  • If an application is unreponsive (no connected executors, or connected but unresponsive executors).

If multiple applications share a system database, each application's alerts consider only the workflows it owns.

You may also specify an application to receive the alert—this does not need to be the same as the application that generated the alert. For some failure conditions (e.g., unresponsive application), the application receiving the alert is required to be different from the one generating it.

Receiving Alerts​

You can register an alert handler in your application to receive alerts from Conductor. Your handler can log the alerts or forward them to another system, such as Slack or PagerDuty.

Only one alert handler may be registered per application, and it must be registered before launching DBOS. If no handler is registered, alerts are logged automatically.

The handler receives three arguments:

  • rule_type: The type of alert rule. One of WorkflowFailure, SlowQueue, or UnresponsiveApplication.

  • message: The alert message.

  • metadata: Additional key-value string metadata about the alert. The metadata keys depend on the rule type:

    WorkflowFailure:

    • workflow_name: The workflow name filter, or * for all workflows.
    • failed_workflow_count: The number of failed workflows detected in the time window.
    • threshold: The configured failure count threshold.
    • period_secs: The time window in seconds.

    SlowQueue:

    • queue_name: The queue name filter, or * for all queues.
    • stuck_workflow_count: The number of workflows that have been in the queue for longer than the time threshold.
    • threshold_secs: The enqueue time threshold in seconds.

    UnresponsiveApplication:

    • application_name: The application name.
    • connected_executor_count: The number of executors currently connected to this application.

Example logging alerts:

from dbos import DBOS

@DBOS.alert_handler
def handle_alert(rule_type: str, message: str, metadata: dict[str, str]) -> None:
    DBOS.logger.warning(f"Alert received: {rule_type} - {message}")
    for key, value in metadata.items():
        DBOS.logger.warning(f"  {key}: {value}")

Example forwarding alerts to Slack using incoming webhooks

@DBOS.alert_handler
def handle_alert(rule_type: str, message: str, metadata: dict[str, str]) -> None:
    webhook_url = os.environ.get("SLACK_WEBHOOK_URL")

    slack_text = f"*Alert: {rule_type}*\n{message}\n" + "\n".join(f"• {k}: {v}" for k, v in metadata.items())

    try:
        resp = requests.post(webhook_url, json={"text": slack_text}, timeout=10)
        resp.raise_for_status()
    except requests.RequestException as e:
        DBOS.logger.error(f"Failed to send Slack alert: {e}")

Example forwarding alerts to PagerDuty using the Events API:

@DBOS.alert_handler
def handle_alert(rule_type: str, message: str, metadata: dict[str, str]) -> None:
    routing_key = os.environ.get("PAGERDUTY_ROUTING_KEY")

    payload = {
        "routing_key": routing_key,
        "event_action": "trigger",
        "payload": {
            "summary": f"{rule_type}: {message}",
            "severity": "error",
            "source": "my-app",
            "custom_details": metadata,
        },
    }

    try:
        resp = requests.post(
            "https://events.pagerduty.com/v2/enqueue",
            json=payload,
            timeout=10,
        )
        resp.raise_for_status()
    except requests.RequestException as e:
        DBOS.logger.error(f"Failed to send PagerDuty alert: {e}")

See the Python reference for more details.

Metrics-Based Alerts​

If you scrape Conductor metrics into a monitoring system such as Prometheus (with Alertmanager), Datadog, or Grafana, you can define alerts directly on DBOS metrics. This is an alternative to the Conductor-managed alerts above, useful if you already run a monitoring and alerting stack.

Here are some example alerting conditions written in in PromQL:

Elevated workflow failures: more than 10 workflows failing per minute for an application:

sum by (application) (dbos_conductor_v1_workflow_failed_rate) * 60 > 10

Stuck queue: a workflow has been waiting in a queue for more than 5 minutes:

time() - dbos_conductor_v1_workflow_oldest_enqueued_timestamp_seconds > 300

Application offline: no healthy executors are connected for an application:

absent(dbos_conductor_v1_executor_count{application="my-app", status="HEALTHY"})

Detecting an offline application is a "missing data" condition: a fully disconnected application emits no executor series at all. In PromQL, use absent() with an explicit application label for each app you want to monitor.