Documentation
API
Q&A
Read
AnnouncementsArticlesWebsite tips
Learn
Getting StartedManagerMaestroAnalyticsExchangeWorkspaces
Resources
VideosDownloadsExchange PackagesRelease Notes
Ideas
Contact
Platform
Manager
Maestro
Exchange
Workspaces
Analytics
Springboard

Kafka Service Connection

   Journey Manager (JM) The transaction engine for the platform. /  Event-Based ArchitectureEvent-Based Architecture |   System Manager / DevOps |  22.10 This feature was introduced in 22.10.

Manager allows you to create and configure a Kafka service connection to connect to an Apache KafkaApache Kafka is a distributed event store and stream-processing platform, which provide a unified, high-throughput, low-latency platform for handling real-time data feeds. For more information, see https://kafka.apache.org/intro producer. Various services can use this service connection to publish events. Typically, you need this service connection for building services based on the Event-Based Architecture.

To configure a service connection:

  1. Select Services > Service Connections.
  2. Locate a Kafka service connection or click New to create a new service connection.
    Manager configure AWS S3 service connection
  3. Provide a unique service connection name in the Name field.
  4. Enter an organization name in the Organization field or leave it blank.
  5. Select Kafka as a service connection type from the Type dropdown list.
  6. Enter a server URL, where a Kafka producer is installed, in the Kafka Server URL field, for example, my-kafka-host.com
  7. Provide a user name to connect to a Kafka producer in the Kafka Username field.
  8. Provide a user password to connect to a Kafka producer in the Kafka Password.
  9. Confirm the user password in the Confirm Kafka Password field.
  10. Enter Kafka producer parameters to establish a connection in the Config Parameters field. For more information, see producer configuration.
  11. Click Save to update the changes.

Kafka Producer Configuration

The Kafka service connection supports the limited set of parameters that you can use to configure a connection to the Apache Kafka producer. Some of the parameters are set to default values in Manager and some are in a producer. We recommend checking Kafka documentation to learn about these parameters and their default values.

Note

Parameters are version specific, so check the Kafka version and Kafka client library version to ensure you use the correct parameters.

Supported producer configuration parameters:

acks
default is all
enable.idempotence
default is true
client.id
default is JM-Producer
compression.type
client.dns.lookup
connections.max.idle.ms
delivery.timeout.ms
group.id
To join a group, so listeners don't read an event multiple times.  |  24.04 This feature was introduced in 24.04
max.block.ms
max.request.size
metadata.max.age.ms
metadata.max.idle.ms
partitioner.class
reconnect.backoff.max.ms
reconnect.backoff.ms
request.timeout.ms
retries
retry.backoff.ms

Next, learn about other service connection types.