TELEMETRY PIPELINE
Confluent Cloud

Confluent Cloud source plugin

The Confluent Cloud source plugin (name: kafka, alias: confluent_cloud) lets you retrieve data from Confluent Cloud and ingest it into a telemetry pipeline.

This is a pull-based source plugin.

Supported telemetry types

This plugin for Chronosphere Telemetry Pipeline supports these telemetry types:

LogsMetricsTraces

Configuration parameters

Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.

General

NameKeyDescriptionDefault
Confluent Cloud Bootstrap ServersbrokersRequired. The Confluent Cloud bootstrap found within the configuration settings.[YOUR_BOOTSTRAP_SERVER].confluent.cloud:9092
Confluent Cloud TopictopicsRequired. The Confluent Cloud topic to read information from.none
Confluent Cloud API Keyrdkafka.sasl.usernameYour Confluent Cloud API key.none
Confluent Cloud API Secretrdkafka.sasl.passwordRequired. Your Confluent Cloud API secret.none

Advanced

NameKeyDescriptionDefault
Minimum Queued Messagesrdkafka.queued.min.messagesThe minimum number of messages per topic and partition that Telemetry Pipeline tries to maintain in the local consumer queue.10
Session Timeout (ms)rdkafka.session.timeout.msHow long Telemetry Pipeline waits before terminating a session connection.45000
Security Protocolrdkafka.security.protocolThe security protocol for Azure Event Hub. If you require OAuth or OpenID, contact Chronosphere Support.SASL_SSL
SASL Mechanismrdkafka.sasl.mechanismThe transport mechanism for the SASL connection.PLAIN

Security and TLS

NameKeyDescriptionDefault
TLStlsIf true, enables TLS/SSL. If false, disables TLS/SSL. Accepted values: true, false.false
TLS Certificate Validationtls.verifyIf on, and if tls is true, enables TLS/SSL certificate validation. If off, disables TLS/SSL certificate validation. Accepted values: on, off.on
TLS Debug Leveltls.debugSets TLS debug verbosity level. Accepted values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose).1
CA Certificate File Pathtls.ca_fileAbsolute path to CA certificate file.none
Certificate File Pathtls.crt_fileAbsolute path to certificate file.none
Private Key File Pathtls.key_fileAbsolute path to private key file.none
Private Key Path Passwordtls.key_passwdPassword for private key file.none
TLS SNI Hostname Extensiontls.vhostHostname to be used for TLS SNI extension.none

Other

This parameter doesn’t have an equivalent setting in the Telemetry Pipeline web interface, but you can use it in pipeline configuration files.

NameKeyDescriptionDefault
nonebuffer_max_sizeSets the size of the memory buffer available to that pipeline. If your records exceed the size of the default memory buffer (4 MB), you must increase the buffer to accommodate them.4M

Extended librdkafka parameters

This plugin uses the librdkafka (opens in a new tab) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the rdkafka. prefix.

In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the librdkafka configuration properties (opens in a new tab) by adding them to a pipeline configuration file. To do so, append the rdkafka. prefix to the name of that property.

For example, to customize the socket.keepalive.enable property, add the rdkafka.socket.keepalive.enable key to your configuration file.

Don’t use librdkafka properties to configure a pipeline’s memory buffer. Instead, use the buffer_max_size parameter.