Redpanda Cloud DataSet destination plugin
The Redpanda Cloud destination plugin
(name: kafka, alias: RedPandaCloud) lets you send your telemetry pipeline
data to the Redpanda Cloud service.
Supported telemetry types
This plugin for Chronosphere Telemetry Pipeline supports these telemetry types:
| Logs | Metrics | Traces |
|---|---|---|
Configuration parameters
Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.
General
| Name | Key | Description | Default |
|---|---|---|---|
| Brokers | brokers | Required. List of Kafka Brokers. For example, 192.168.1.3:9092, 192.168.1.4:9092 | none |
| Topics | topics | Required. Single entry or comma-separated list of topics that Telemetry Pipeline uses to send messages to Kafka. If only one topic is set, that one will be used for all records. If multiple topics exist, the topic set in the record by Topic_Key is used. | calyptia-core |
| Format | format | Required. Specifies a data format. Accepted values: json, msgpack. | json |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| Security Protocol | rdkafka.security.protocol | The Security Protocol used to communicate with Redpanda. | sasl_ssl |
| SASL Mechanism | rdkafka.sasl.mechanisms | The SASL authentication mechanism for the API. | SCRAM-SHA-256 |
| Redpanda Username | rdkafka.sasl.username | Your Redpanda username. | none |
| Redpanda Password | rdkafka.sasl.password | Required. Password for your Redpanda username. | none |
| Message Key | message_key | Optional key to store the message. | none |
| Message Key Field | message_key_field | If set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used if set. | none |
| Timestamp Key | timestamp_key | Sets the key to store the record timestamp. | none |
| Timestamp Format | timestamp_format | Sets the format to iso8601 or double. | double |
| Body Key | body_key | Specifies the key that contains the body. | none |
| Queue Full Retries | queue_full_retries | Telemetry Pipeline queues data into rdkafka library. If the underlying library can’t flush the records the queue might fill and block new addition of records. This option sets the number of local retries to enqueue the data. The interval between each retry is 1 second. Setting queue_full_retries to 0 to set an unlimited number of retries. | 10 |
Extended librdkafka parameters
This plugin uses the librdkafka (opens in a new tab)
library. Certain configuration parameters available through the Telemetry
Pipeline UI are based on librdkafka settings. These parameters generally use the
rdkafka. prefix.
In addition to the parameters available through the Telemetry Pipeline UI, you can
customize any of the
librdkafka configuration properties (opens in a new tab)
by adding them to a pipeline configuration file. To do so, append the rdkafka.
prefix to the name of that property.
For example, to customize the socket.keepalive.enable property, add the
rdkafka.socket.keepalive.enable key to your configuration file.