Drop rules

Based on your amount of available storage and rate limits, you might want to reduce the amount of metrics that you continue to store so that you’re keeping only those metrics you need. For example, using Telemetry Usage Analyzer, you might find a metric has a low utilization score. Metrics with high volume and low utilization are candidates for dropping.

Use drop rules to omit incoming metrics based on labels, and have Chronosphere Observability Platform persist only those metrics you need. Define filters to specify the metric or label names you want to drop. You can also drop metrics based on a data point value.

The Collector scrapes data from an app, and sends the data to Chronosphere where you can apply drop rules. Any metrics that weren’t impacted by drop rules are then aggregated in Observability Platform.

Drop rules affect data from the point of rule creation onward. When you query for the metric:

• Data can display in autocomplete searches for a period of approximately two hours.
• Queries covering the time period before the drop rule took effect display the metric until the retention period expires.

View and filter created drop rules

To view the drop rules defined on your instance, you must have administrative privileges.

Select from the following methods to view your available drop rules.

Create a drop rule

Use drop rules to drop the ingestion of certain metrics that you don’t need to persist. For example, dropping metrics from a specific cluster, or test data.

Drop rules can have multiple filter conditions on different labels, making it possible to drop a subset of the series matching a particular metric name. Use multiple filters for cases where only a subset of a metric’s data is used. For example, if a metric is used in an alert but examines only series related to error statuses, you can drop non-error status series.

You can use glob syntax in your drop rule to match specific values. Use the value_based_drop object to define a value-based drop rule that drops metrics based on a target data point value.

Drop rules have these definition limitations:

• The rule can’t be longer than 4,096 characters.
• A drop rule slug must contain only alphanumeric characters, dashes, or underscores. The allowed regular expression for matching is [a-zA-Z0-9-_]. Spaces aren’t allowed.
• Each rule name must be unique.

A metric that matches any drop rule is dropped, regardless of overlap with drop rules created using other methods. For example, if a metric matches a drop rule created with Terraform that also matches a subset of a drop rule created with the CreateDropRule endpoint, the metric is dropped.

For example, you might create a drop rule containing kubernetes_cluster:dev to match and drop lower value metrics containing this key:value pair. To exclude specific values that also match this rule, use a single filter with an AND logical operator to specify a “does not equal” match. The following drop rule matches any metric containing kubernetes_cluster:dev, but not metrics that also contain the specified container_name values:

"kubernetes_cluster:dev AND container_name:!{log_service|data_service}"

Select from the following methods to create a drop rule.

Matching with glob syntax

Verify your glob syntax with Live Telemetry Analyzer to ensure your query matches the correct metrics. For example, the following drop rule for the CreateDropRule API drops any metric containing the service label with a value of api that also has a cluster label that begins with dev.

{
  "drop_rule": {
    "slug": "api-drop-rule",
    "name": "api drop rule",
    "mode": 1,
    "filters": [
      {
        "name": "service",
        "value_glob": "api"
      }
      {
        "name": "cluster",
        "value_glob": "dev*"
      }
    ]
  }
}

To keep only metrics with a specific label value, use glob syntax to specify a “does not equals” match with an exclamation mark ! to match specified values. For example, the following drop rule drops any metric containing the service label with a value of api, that doesn’t also have an env label with a value of production. This drop rule also applies if the env label isn’t set.

{
  "drop_rule": {
    "slug": "api-drop-rule",
    "name": "api drop rule",
    "mode": 1,
    "filters": [
      {
        "name": "service",
        "value_glob": "api"
      }
      {
        "name": "env!",
        "value_glob": "production"
      }
    ]
  }
}

Use an asterisk * to match metrics with a particular label, regardless of the value. Conversely, to drop only metrics that don’t have a particular label, use ! to specify a “does not equal” match, in conjunction with an asterisk * to match all values. For example, the following drop rule drops any metric containing the service label with a value of api that doesn’t also have an env label.

{
  "drop_rule": {
    "slug": "api-drop-rule",
    "name": "api drop rule",
    "mode": 1,
    "filters": [
      {
        "name": "service",
        "value_glob": "api"
      }
      {
        "name": "env!",
        "value_glob": "*"
      }
    ]
  }
}

Define a value-based drop rule

Use the value_based_drop object to define a value-based drop rule. The target_drop_value parameter specifies the target data point value at which to drop metrics.

You can include the drop_nan_value parameter to drop data point values that are Not a Number (NaN). If set to true, Chronosphere drops NaN data points, along with any published staleness markers.

If you set the drop_nan_value parameter to true in conjunction with the value_based_drop parameter, Observability Platform drops data points matching the value or any NaN values.

If you want to drop only NaN values, set drop_nan_value to true without a corresponding value_based_drop parameter. When no value is set for drop_nan_value, all values are dropped, including NaN values.

Disable a drop rule

Select from the following methods to disable a drop rule.

Delete a drop rule

Select from the following methods to delete a drop rule.