--- alias: installation-guide-planning-and-installation-guides-kafka description: "Describes what you must prepare to connect to Kafka, including the deployment, the authentication methods, and the system topics" --- # Kafka Connection Setup This section describes what you must prepare to configure the connection to Kafka. ## Deployment The MES uses Kafka for event streaming. Ensure that the Kafka deployment complies with the [[system-requirements-kafka#minimum-system-requirements|Minimum System Requirements]] defined for the platform. The MES supports two authentication methods to communicate with Kafka: - Mutual TLS, with client certificates. - SASL Plain, with a username and password. !!! info For an adequate operation connection setup for Kafka under Critical Manufacturing MES, the user must have the permissions described in [[installation-guide-accountsandsecurity]]. When running MES on OpenShift, we recommend using a version of Red Hat Streams for Apache Kafka based on Apache Kafka 4.0.0 or higher. Please contact Red Hat at [Red Hat Support](https://www.redhat.com/en/services/support) for detailed instructions on setting up AMQ Streams. It is also possible to use Confluent Cloud. Please contact Confluent at [Confluent Support](https://support.confluent.io/hc/en-us) for detailed instructions on setting up Kafka. ## Authentication Methods The required settings depend on the authentication method. The following tables list the settings of each method. ### Using Mutual TLS Authentication | Item | Description | Usage | Planning and installation guide | Checked | |-------------------------------|------------------------------------------------|-----------------------------------|--------------------------------------------|---------| | Cluster Certificate Authority | The certificate authority of the Kafka cluster | Connects to Kafka with mutual TLS | [[installation-guide-accountsandsecurity]] | | | User public key | The public key of the Kafka user | Connects to Kafka with mutual TLS | [[installation-guide-accountsandsecurity]] | | | User private key | The private key of the Kafka user | Connects to Kafka with mutual TLS | [[installation-guide-accountsandsecurity]] | | Table: Kafka connection setup using mutual TLS authentication ### Using SASL SSL Plain | Item | Description | Usage | Planning and installation guide | Checked | |----------------|--------------------------------|---------------------------------------|--------------------------------------------|---------| | Kafka Username | The username of the Kafka user | Connects to Kafka with SASL SSL Plain | [[installation-guide-accountsandsecurity]] | | | Kafka Password | The password of the Kafka user | Connects to Kafka with SASL SSL Plain | [[installation-guide-accountsandsecurity]] | | Table: Kafka connection setup using SASL SSL Plain ## System Topics The installation creates the Kafka topics that the system uses, and the Host creates the topics of each IoT Event Definition. The following table lists the system topics and when the installation creates them: | Topic | Installed when | Partitions | Retention | |--------------------------------------------|-----------------------------------------------|---------------|-----------| | `__workqueue_operationevents` | Always | 1 | 90 days | | `_dp_shids_ordered` | The installation includes Data Platform | 1 | 90 days | | `_dp_replication_shids` | The installation includes Data Platform | Setting or 10 | 90 days | | `_dp_replication_events` | The installation includes Data Platform | Setting or 10 | 90 days | | `_dp_iotevents_dlq` | The installation includes Data Platform | 1 | 90 days | | `__workqueue_datamanager` | The installation includes Data Platform | 10 | 90 days | | `__workqueue_clickhouse` | The installation includes Data Platform | 1 | 90 days | | `_edp_shids_ordered` | The license includes Enterprise Data Platform | 1 | 360 days | | `_edp_replication_events` | The license includes Enterprise Data Platform | Setting or 10 | 360 days | | `_edp_replication_shids` | The license includes Enterprise Data Platform | Setting or 10 | 360 days | Table: System Kafka topics created by the installation The system name in the topic names is in lowercase. ### Partitions The **Kafka Topic Max Partitions** setting of the Customer Environment sets the maximum number of partitions of the system topics. The setting is optional, and it is available in the Kafka settings with the advanced configuration level. To set the number of partitions of each topic, the installation compares the value in the table with the setting: - When the table shows a number and the setting has a value, the installation uses the lower of the two. A topic with one partition keeps one partition. - When only one of them has a value, the installation uses that value. - When neither has a value, the installation uses 10. The setting also caps the number of partitions of the topics that the Host creates for IoT Event Definitions. Kafka cannot remove partitions. When a topic already has more partitions than the resolved value, the installation logs a warning and keeps them. When it has fewer, the installation adds partitions. ### Retention The installation sets the retention of each system topic to the value in the table. When the new value is shorter than the current one, the installation logs a warning, because Kafka deletes the messages that are older than the new retention. If you change the retention of a system topic in Kafka, the next installation sets it back to the value in the table. ### Installations That Restore the Databases When an installation restores the databases, the system topics must not exist. A topic that already exists holds messages of the previous environment, so the installation fails with the following error: ```log Kafka topic '' already exists but a clean installation (database restore) occurred. Delete the stale topics before installing. ``` Delete the system topics of the environment before you run the installation again. !!! note The installation creates topics, adds partitions, and changes the retention of topics. The Kafka user of the system needs the permissions listed in [[system-requirements-kafka#access-control-lists-acls|Access Control Lists (ACLs)]].