Kafka source plugin

You can use the Kafka source plugin (name: kafka, alias: kafka_input) to ingest data from your Apache Kafka instances into a telemetry pipeline.

This is a pull-based source plugin.

Supported telemetry types

This plugin for Chronosphere Telemetry Pipeline supports these telemetry types:

Logs	Metrics	Traces

Configuration parameters

Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.

General

Name	Key	Description	Default
Brokers	brokers	Required. Single of multiple lists of Kafka brokers. For example, 192.168.1.3:9092, 192.168.1.4:9092.	none
Topics	topics	Required. Single entry or list of topics separated by comma (,) that Telemetry Pipeline uses to receive messages from Kafka.	none

Advanced

Name	Key	Description	Default
Minimum Queued Messages	rdkafka.queued.min.messages	Minimum number of messages per topic and partition Telemetry Pipeline tries to maintain in the local consumer queue.	10
Group ID	rdkafka.group.id	The client group ID string. All clients with the same group ID belong to the same group.	none

Other

This parameter doesn’t have an equivalent setting in the Telemetry Pipeline web interface, but you can use it in pipeline configuration files.

Name	Key	Description	Default
none	buffer_max_size	Sets the size of the memory buffer available to that pipeline. If your records exceed the size of the default memory buffer (4 MB), you must increase the buffer to accommodate them.	4M

Extended librdkafka parameters

This plugin uses the librdkafka (opens in a new tab) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the rdkafka. prefix.

In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the librdkafka configuration properties (opens in a new tab) by adding them to a pipeline configuration file. To do so, append the rdkafka. prefix to the name of that property.

For example, to customize the socket.keepalive.enable property, add the rdkafka.socket.keepalive.enable key to your configuration file.