Kafka Connection Setup#
This section describes what you must prepare to configure the connection to Kafka.
Deployment#
The MES uses Kafka for event streaming. Ensure that the Kafka deployment complies with the Minimum System Requirements defined for the platform.
The MES supports two authentication methods to communicate with Kafka:
- Mutual TLS, with client certificates.
- SASL Plain, with a username and password.
Info
For an adequate operation connection setup for Kafka under Critical Manufacturing MES, the user must have the permissions described in Accounts and Security.
When running MES on OpenShift, we recommend using a version of Red Hat Streams for Apache Kafka based on Apache Kafka 4.0.0 or higher. Please contact Red Hat at Red Hat Support ↗ for detailed instructions on setting up AMQ Streams.
It is also possible to use Confluent Cloud. Please contact Confluent at Confluent Support ↗ for detailed instructions on setting up Kafka.
Authentication Methods#
The required settings depend on the authentication method. The following tables list the settings of each method.
Using Mutual TLS Authentication#
| Item | Description | Usage | Planning and installation guide | Checked |
|---|---|---|---|---|
| Cluster Certificate Authority | The certificate authority of the Kafka cluster | Connects to Kafka with mutual TLS | Accounts and Security | |
| User public key | The public key of the Kafka user | Connects to Kafka with mutual TLS | Accounts and Security | |
| User private key | The private key of the Kafka user | Connects to Kafka with mutual TLS | Accounts and Security |
Table: Kafka connection setup using mutual TLS authentication
Using SASL SSL Plain#
| Item | Description | Usage | Planning and installation guide | Checked |
|---|---|---|---|---|
| Kafka Username | The username of the Kafka user | Connects to Kafka with SASL SSL Plain | Accounts and Security | |
| Kafka Password | The password of the Kafka user | Connects to Kafka with SASL SSL Plain | Accounts and Security |
Table: Kafka connection setup using SASL SSL Plain
System Topics#
The installation creates the Kafka topics that the system uses, and the Host creates the topics of each IoT Event Definition.
The following table lists the system topics and when the installation creates them:
| Topic | Installed when | Partitions | Retention |
|---|---|---|---|
_<system name>_workqueue_operationevents | Always | 1 | 90 days |
<system name>_dp_shids_ordered | The installation includes Data Platform | 1 | 90 days |
<system name>_dp_replication_shids | The installation includes Data Platform | Setting or 10 | 90 days |
<system name>_dp_replication_events | The installation includes Data Platform | Setting or 10 | 90 days |
<system name>_dp_iotevents_dlq | The installation includes Data Platform | 1 | 90 days |
_<system name>_workqueue_datamanager | The installation includes Data Platform | 10 | 90 days |
_<system name>_workqueue_clickhouse | The installation includes Data Platform | 1 | 90 days |
<system name>_edp_shids_ordered | The license includes Enterprise Data Platform | 1 | 360 days |
<system name>_edp_replication_events | The license includes Enterprise Data Platform | Setting or 10 | 360 days |
<system name>_edp_replication_shids | The license includes Enterprise Data Platform | Setting or 10 | 360 days |
Table: System Kafka topics created by the installation
The system name in the topic names is in lowercase.
Partitions#
The Kafka Topic Max Partitions setting of the Customer Environment sets the maximum number of partitions of the system topics. The setting is optional, and it is available in the Kafka settings with the advanced configuration level.
To set the number of partitions of each topic, the installation compares the value in the table with the setting:
- When the table shows a number and the setting has a value, the installation uses the lower of the two. A topic with one partition keeps one partition.
- When only one of them has a value, the installation uses that value.
- When neither has a value, the installation uses 10.
The setting also caps the number of partitions of the topics that the Host creates for IoT Event Definitions.
Kafka cannot remove partitions. When a topic already has more partitions than the resolved value, the installation logs a warning and keeps them. When it has fewer, the installation adds partitions.
Retention#
The installation sets the retention of each system topic to the value in the table. When the new value is shorter than the current one, the installation logs a warning, because Kafka deletes the messages that are older than the new retention.
If you change the retention of a system topic in Kafka, the next installation sets it back to the value in the table.
Installations That Restore the Databases#
When an installation restores the databases, the system topics must not exist. A topic that already exists holds messages of the previous environment, so the installation fails with the following error:
Kafka topic '<topic name>' already exists but a clean installation (database restore) occurred. Delete the stale topics before installing.
Delete the system topics of the environment before you run the installation again.
Note
The installation creates topics, adds partitions, and changes the retention of topics. The Kafka user of the system needs the permissions listed in Access Control Lists (ACLs).