Kafka Service Connection

   Journey Manager (JM) The transaction engine for the platform. /  Event-Based ArchitectureEvent-Based Architecture |   System Manager / DevOps |  22.10 This feature was introduced in 22.10.

Manager allows you to create and configure a Kafka service connection to connect to an Apache KafkaApache Kafka is a distributed event store and stream-processing platform, which provide a unified, high-throughput, low-latency platform for handling real-time data feeds. For more information, see https://kafka.apache.org/intro producer. Various services can use this service connection to publish events. Typically, you need this service connection for building services based on the Event-Based Architecture.

To configure a service connection:

1. Select Services > Service Connections.
2. Locate a Kafka service connection or click New to create a new service connection.

   

3. Provide a unique service connection name in the Name field.
4. Enter an organization name in the Organization field or leave it blank.
5. Select Kafka as a service connection type from the Type dropdown list.
6. Enter a server URL, where a Kafka producer is installed, in the Kafka Server URL field, for example, my-kafka-host.com
7. Provide a user name to connect to a Kafka producer in the Kafka Username field.
8. Provide a user password to connect to a Kafka producer in the Kafka Password.
9. Confirm the user password in the Confirm Kafka Password field.
10. Enter Kafka producer parameters to establish a connection in the Config Parameters field. For more information, see producer configuration.
11. Click Save to update the changes.

Kafka Producer Configuration

The Kafka service connection supports the limited set of parameters that you can use to configure a connection to the Apache Kafka producer. Some of the parameters are set to default values in Manager and some are in a producer. We recommend checking Kafka documentation to learn about these parameters and their default values.

Supported producer configuration parameters:

• acks - default is all
• enable.idempotence - default is true
• client.id - default is JM-Producer
• compression.type
• client.dns.lookup
• connections.max.idle.ms
• delivery.timeout.ms
• max.block.ms
• max.request.size
• metadata.max.age.ms
• metadata.max.idle.ms
• partitioner.class
• reconnect.backoff.max.ms
• reconnect.backoff.ms
• request.timeout.ms
• retries
• retry.backoff.ms

Next, learn about other service connection types.