Monitor details

When viewing all monitors, you can select an individual monitor to view its details.

After selecting an individual monitor, you can view all details for that monitor and filter query results to individual time series and related change events. You can also view a monitor’s history of changes to understand when alerts triggered and which notifications sent.

If a monitor has any active muting rules, Chronosphere indicating the number of active muting rules and when they expire. Chronosphere displays a banner indicating Alerts for this monitor are being muted if a monitor is actively muted or was muted within the last hour.

Related alerts

The top of each monitor includes a Related alerts section to provide insight into data patterns across multiple alerts. This summary includes the number of new alerts that triggered, which labels were impacted, and identifies related alert patterns across other monitors and service level objectives (SLOs). For example, this section might include a summary like this:

In the same time window, 124 new alerts with similar labels were triggered, and
came from 50 monitors and 20 SLOs.

This information can help determine whether alerts for a single monitor indicate an issue with a narrow or broad impact. For example, does this issue impact one or two labels or alerts, or many alerts firing simultaneously across many monitors with a single label in common?

Click any of the labels, monitors, or SLOs in the summary to display a list of related entities, which you can navigate to for further exploration.

Viewing related alerts can help identify alerts that are false alarms or flapping. For example, a single monitor that’s triggered several new alerts in the last hour for the same label set likely indicates that an alert is triggering or resolving in rapid succession. This alert is likely flapping, and can be muted with a muting rule, or might need reconfigured. If the alert summary includes many unique labels or signal values, the issue might be escalating and require more attention.

If the related alerts summary seems noisy, such as indicating too many alerts, Chronosphere suggests the following actions:

1. Configure signals to create groups of notifications for similar alerts when a monitor alert triggers or resolves.
2. Create derived labels to standardize on one name for the same service or component, and reduce the number of alerts that alerts generate per label.
3. Shape your metric data to reduce the metrics you persist, identify problematic metrics and labels, downsample data, and reduce cardinality.

Duplicate a monitor

Sometimes, you want to reuse a monitor with minor changes. Rather than recreating a monitor and manually configuring each setting, duplicate the monitor, make the necessary changes, and save the new monitor.

To duplicate a selected monitor:

1. Click the three vertical dots icon.
2. Click Duplicate monitor.
3. In the Duplicate monitor pane, edit the monitor.

View a monitor’s history of changes

To view the history of changes for a selected monitor, click the three vertical dots icon.

Click Version history to display a panel with two tabs:

• Code config: Displays a code representation of the selected entity as of the time of the selected revision.
• Code diff: Displays a Git-style diff of the most-recent change made to the entity, in Chronosphere API format. To compare the selected revision to another revision in the history, click the Compare With dropdown and select the timestamp of the revision that you want to compare.
  • Click Unified to see the diff stacked horizontally.
  • Click Split to see changes side by side.

You can see both who made and the method used for the last change at the top of the list of changes.

To view a revision in the history, click any entry in the list of timestamped revisions. The timestamps default to your local timezone.

You can view unchanged lines within the diff by clicking the Expand X lines links.

Filter query results

When viewing an individual monitor, you can filter the results plotted on the graph. To display the query for the monitor, click the Show Query toggle.

1. Change the displayed error state by clicking either All, All Alerting, or Critical Only.

2. From the time range selector, change the time range of the query results to modify the displayed time period.

3. In the Series Legend, select one or more series in the list to filter the resulting time series that the graph displays. See Series legend for more information.

4. Use the Change Events dropdown menu or advanced filtering syntax to display change events related to the selected monitor.

   You can create a change event such as a broadcast to inform other users of an upcoming outage, or a third-party change event in addition to the predefined change event categories.

5. Click the Alert History tab to display a history of alerts generated by the monitor.

6. Display additional monitor details by clicking the three vertical docs icon for the selected monitor:

   • Monitor Specifications includes details such as the data source, slug, and time interval for the monitor.
   • Conditions are tests that the results of a monitor query are evaluated against, with a related severity.
   • Notification Policies contain rules for notifications, which send to specified endpoints (such as PagerDuty, email, and Slack) when an alert triggers, and optionally when the alert resolves.
   • Edit Monitor opens a panel with tools for editing a monitor’s configuration and notification policy, labels, query, signals, conditions, and annotations.
   • Duplicate Monitor copies the current monitor in a new panel, where you can duplicate the monitor, make changes, and save it as a new monitor.
   • Version History opens the history of changes for the monitor. See View a monitor’s history of changes for more information.
   • Manage Connections opens a panel where you can manage connections between the selected monitor and any service or collection.

7. To examine and modify the query and the metric the monitor uses, click Open in Explorer to open the query in the Metrics Explorer.

8. To mute the selected monitor, a signal from a monitor, or any stored time series, click Mute to add a muting rule. When you create a muting rule from a monitor, the muting rule’s Name field prepopulates with the monitor’s name.

9. To investigate suspicious patterns, click DDx to open metrics differential diagnosis.

Display change events

Use the Change Events dropdown menu or advanced filtering options to display change events on the Query Results panel for a monitor. Use basic filtering to select a predefined query that displays change events, or use advanced filtering to define a query if you know details about the change event you’re searching for.

Change events use category icons for ease of visibility into different event types.

Basic filtering

From the Change Events dropdown menu, select one of the following predefined queries to display change events:

• Broadcasts
• Broadcasts, deploy events in production, and feature flag updates in production
• Broadcasts, deploy events in production, feature flag updates in production, and alerts for the selected monitor
• All events

Click any change event icon in the graph on the Query Results panel to open a more detailed view of change events for the selected time period. This list includes the event time, category, and title for each change event. Click any change event to view detailed information for a specific change event. When viewing this list, you can also click View in Changes Explorer to open the selected filter in Changes Explorer.

To learn more about change event details, refer to change event details.

Advanced filtering

If you know some of the criteria to search for, you can use advanced filtering to define a filter that displays specific change events. Use the lightweight filtering syntax to define which change events you want to display for the selected monitor.

The filtering syntax supports the following keys:

• category: Event category, such as alerts, broadcasts, or deploys.
• labels: Label metadata you’ve annotated your change events with, such as build, environment, or description.
• source: Event source, such as alert_history, buildkite, or deployer.
• title: Event title, either partial or complete.
• type: Event type, such as build_started, deploy_start, or monitor_muted.

To use advanced filtering:

1. Click Filter Events to display the filter box.
2. Define a filter based on the change event attributes you want to filter on.

For example, the following filter returns all infrastructure change events where the build completed on the master branch:

category = "infrastructure" AND type = "build_finished" AND labels.build.branch = "master"

1. Click Filter, or press Shift+Enter (Shift+Return on macOS) to run the query.

2. In the query result histogram, click any change event icon to view more information about each change event.

3. When viewing change event details, click a change event from the list to view more detailed information.

   To learn more about change event details, refer to change event details.

Create a change event

You can create a change event directly from the details page for a specific monitor:

1. On the monitor details page, click + Create Event.
2. Follow the instructions to create a change event.

Connect monitors to services or collections

Services and collections can own monitors, but you can also connect a monitor to any service or collection.

1. In the navigation menu select Alerts > Monitors.
2. Click the name of the monitor you want to edit.
3. To the right of the monitor’s name, click the three vertical dots icon and select Manage connections.
4. Add or remove a connection.

Series legend

The series legend displays labels for all metrics displayed in the graph as either a list or table view.

Both views display the label keys and values for each series and include the current alert status (clear, warning, critical).

You can filter the list of the resulting time series with the Search Series search box.

Click an item in the list to isolate the related line on the graph. Clicking the item again clears the selection. You can Control+click (Command+click on macOS) to choose multiple items.

Annotations

The annotations defined for the monitor, such as runbooks, links to related dashboards, data links to related traces, and documentation links. See Annotations for more information.

Alert history

The Alert History tab next to Monitor Info displays a history of alerts generated by the monitor.

You can change the order of the history by clicking the Timestamp (UTC) column header, and then clicking the chevron to toggle display of the JSON payload for an alert.

The page also lets you filter the history by event type, or toggle the scope of the Alert history between the currently selected signal or all signals.

Alert event payload

The Alert History tab displays the values captured when the alert fired. The payload fields are primarily defined in the monitor data model.

• monitorSlug: The monitor’s unique slug.
• eventType: The type of event triggered. Valid alert types are:
  • Alert triggered
  • Alert resolved
  • Notification sent (alert triggered)
  • Notification failed (alert triggered)
  • Notification sent (alert resolved)
  • Notification failed (alert resolved)
  • Alert muted
  • Alert unmuted
• createdAt: The time the alert fired.
• signal: See signals. Contains one or more name and value.
• details: Contains severity as defined by the notification policy and notifier. notifier values contain the notifier name and slug.