Kafka Requirements#
For Kafka, the system requirements change depending on several factors, such as:
- Message volume (count and size)
- Throughput
- Number of partitions
- Replication factor
- Data retention policies
Benchmark and monitor Kafka to fine-tune the resource allocation.
Minimum System Requirements#
| Item | Requirement |
|---|---|
| Version | >=4.0.0 |
| Brokers | 3 |
| Memory | 8GB per broker |
| CPU | 2vCPUs per broker |
| Disk space | 1TB per broker |
Table: Minimum system requirements for Kafka.
Recommended System Requirements#
| Item | Requirement |
|---|---|
| Version | >=4.0.0 |
| Brokers | 3 |
| Memory | 16GB per broker |
| CPU | 4vCPUs per broker |
| Disk space | 3TB per broker |
Table: Recommended system requirements for Kafka.
The MES is tested with Apache Kafka 4.3.1 and Confluent Platform 8.3.1.
Note
Kafka 4.0.0 and later versions run only in KRaft mode. If your Kafka cluster uses ZooKeeper, migrate it to KRaft before you upgrade to Kafka 4.0.0. For more information, see the Apache Kafka upgrade documentation ↗.
Data Retention#
Kafka keeps messages for a configurable retention time before deleting them. The recommended value is 90 days.
Size the retention time for the periods when consumers fall behind, such as production peaks or system stoppages. This value defines how long a consumer can stay behind before messages are lost.
Note
A lower retention time is possible if you monitor the consumer lag and confirm that messages are processed as expected. This decision falls under the responsibility of the customer, because messages deleted before consumption cannot be recovered.
Access Control Lists (ACLs)#
The Kafka user of the system needs permissions on topics and consumer groups only. The MES does not require cluster permissions to install, upgrade, or run. Some optional features use cluster permissions. For more information, see Optional Cluster Permissions.
These permissions follow the Apache Kafka authorization model and apply to the Kafka versions listed in Minimum System Requirements. Managed Kafka services, such as Confluent Cloud or Amazon MSK, can use their own authorization model. In that case, map the permissions below to the equivalent permissions of the provider.
Resource Naming#
The Kafka topics that the MES creates follow a standard naming convention. By default, each topic starts with the name of the system in lowercase, followed by an underscore. Internal topics also start with an underscore. For example, if the system name is MESIntegrationEnvironment, the Kafka topics have the prefix mesintegrationenvironment_ or _mesintegrationenvironment_. For the list of system topics, see Kafka Connection Setup.
The consumer groups of the MES also start with the name of the system in lowercase, for example, mesintegrationenvironment_workflowexecutor_group.
Grant the permissions as prefixed ACLs on these prefixes. Prefixed ACLs also cover the topics and consumer groups that the MES creates later, such as the topic of a new IoT Event Definition. Kafka matches prefixes with case sensitivity, so use the system name in lowercase.
When you upgrade from a version earlier than 12.0, the upgrade also uses topics and consumer groups with other prefixes. For more information, see Kafka Topic Migration in the Upgrade to 12.0.
Required Permissions#
| Resource | Operation | Use |
|---|---|---|
| Topic | Create | Creates the system topics when they do not exist. A topic Create ACL that matches the topic name is enough to create it. |
| Topic | Describe | Reads the topic metadata and checks that the system topics exist. |
| Topic | DescribeConfigs | Reads the current configuration of the topics, such as the retention, to validate and reconcile it. |
| Topic | Alter | Adds partitions to the topics. |
| Topic | AlterConfigs | Updates the configuration of the topics, such as the retention. |
| Topic | Delete | Deletes the system topics when you uninstall the environment. |
| Topic | Read | Consumes messages. |
| Topic | Write | Produces messages. |
| Consumer group | Read | Joins the consumer groups and commits the offsets. |
| Consumer group | Describe | Reads the consumer group metadata, such as the offsets and the lag. |
Table: Required Kafka permissions for the MES.
The installation and the upgrade use the topic administration permissions (Create, Describe, DescribeConfigs, Alter, and AlterConfigs) to create, validate, and update the system topics. The MES also uses them while it runs, for example, to create the topic of a new IoT Event Definition, add partitions, or update the retention. Without them, the MES can still produce and consume messages on topics that already exist, but the installation, the upgrade, and the creation of new IoT event topics can fail.
Info
Topic permissions and cluster permissions with the same name are different permissions. For example, topic DescribeConfigs reads the configuration of a topic and is required. Cluster DescribeConfigs reads the configuration of the brokers and is optional.
The MES does not need the cluster IdempotentWrite permission. In the supported Kafka versions, topic Write is enough for idempotent producers. The MES does not use Kafka transactions, so it does not need transactional ID permissions.
Optional Cluster Permissions#
The MES works without cluster permissions. The following cluster permissions are optional. If you grant them, the MES uses them for the features described below.
| Cluster permission | Use | Behavior without the permission |
|---|---|---|
| Describe | Lists every consumer group in the cluster, for the lag monitoring and for the Kafka Topic Migration Tool. | The listing returns only the consumer groups that the consumer group ACLs cover. The lag monitoring shows only these groups, and the Kafka Topic Migration Tool moves only the offsets of these groups. For more information, see the next section. |
| DescribeConfigs | Reads the message.max.bytes setting of the broker, so that the Data Platform services, such as HouseKeeper, know the largest message they can send. | The services log a warning and use the Kafka default limit of 1 MB instead of the limit of the broker. Payloads larger than the limit are stored in external storage, and the Kafka message references them. |
Table: Optional Kafka cluster permissions, their use, and the behavior without them.
If you grant a cluster permission only for an upgrade, remove it after the upgrade.
Kafka Topic Migration in the Upgrade to 12.0#
Version 12.0 changes the names of the IoT event topics. When you upgrade from an earlier version, the Kafka Topic Migration Tool copies the messages of the old topics to the new topics in the same Kafka cluster, with MirrorMaker 2. Then, it moves the offsets of the consumer groups to the new topics. The tool does not need cluster permissions.
Before the upgrade, do the following:
- Grant the topic permissions and the consumer group permissions on the
cm-topic-migration-prefix. The tool uses these topics and consumer groups to track the progress of the migration. - Grant the topic permissions on the
mm2-offset-syncs.cm-topic-migration-prefix. MirrorMaker 2 uses these topics. - If other applications consume the IoT event topics with their own consumer groups, grant consumer group Describe and Read on these groups, or grant cluster Describe for the upgrade. Otherwise, the tool does not see these groups and does not move their offsets.
- Stop the other applications that consume the IoT event topics. The tool moves the offsets of a consumer group only when the group is inactive.
The tool deletes its migration helper topics after a successful migration. It keeps its progress in the consumer groups that start with cm-topic-migration-, so that a later run can resume a partial migration without copying the same messages again.
Kafka keeps the offsets of these consumer groups for the time set in the offsets.retention.minutes setting of the broker, 7 days by default, counted from the last run of the tool. Resume a partial migration within this time. After this time, the progress of the tool expires, and a new run copies the retained messages of the old topics again. This can create duplicate messages in the new topics.
Note
The tool does not delete consumer groups. Deleting them is the responsibility of the customer. After the migration is complete, delete the consumer groups that start with cm-topic-migration-, and the old CDM Builder consumer groups, whose offsets the tool copied to one new consumer group per IoT Event Definition:
<systemname>_cdm_builderand<systemname>_cdm_builder_light.- With the Enterprise Data Platform,
<systemname>_edp_cdm_builder,<systemname>_edp_cdm_builder_light, and<systemname>_edp_cdm_builder_<silo>.
Do not delete the consumer groups of other applications that consumed the old IoT event topics. The tool moves their offsets to the new topics in the same consumer groups, so these applications keep using them.