Kafka Requirements#
For Kafka, the system requirements change depending on several factors, such as:
- Message volume (count and size)
- Throughput
- Number of partitions
- Replication factor
- Data retention policies
Benchmark and monitor Kafka to fine-tune the resource allocation.
Minimum System Requirements#
| Item | Requirement |
|---|---|
| Version | >=4.0.0 |
| Brokers | 3 |
| Memory | 8GB per broker |
| CPU | 2vCPUs per broker |
| Disk space | 1TB per broker |
Table: Minimum system requirements for Kafka.
Recommended System Requirements#
| Item | Requirement |
|---|---|
| Version | >=4.0.0 |
| Brokers | 3 |
| Memory | 16GB per broker |
| CPU | 4vCPUs per broker |
| Disk space | 3TB per broker |
Table: Recommended system requirements for Kafka.
The MES is tested with Apache Kafka 4.3.1 and Confluent Platform 8.3.1.
Note
Kafka 4.0.0 and later versions run only in KRaft mode. If your Kafka cluster uses ZooKeeper, migrate it to KRaft before you upgrade to Kafka 4.0.0. For more information, see the Apache Kafka upgrade documentation ↗.
Data Retention#
Kafka keeps messages for a configurable retention time before deleting them. The recommended value is 90 days.
Size the retention time for the periods when consumers fall behind, such as production peaks or system stoppages. This value defines how long a consumer can stay behind before messages are lost.
Note
A lower retention time is possible if you monitor the consumer lag and confirm that messages are processed as expected. This decision falls under the responsibility of the customer, because messages deleted before consumption cannot be recovered.
Access Control Lists (ACLs)#
The Kafka user of the system needs permissions on topics and consumer groups only. The MES does not require cluster permissions to install or run. Some optional features use cluster permissions. For more information, see Optional Cluster Permissions.
These permissions follow the Apache Kafka authorization model and apply to the Kafka versions listed in Minimum System Requirements. Managed Kafka services, such as Confluent Cloud or Amazon MSK, can use their own authorization model. In that case, map the permissions below to the equivalent permissions of the provider.
Resource Naming#
The Kafka topics that the MES creates follow a standard naming convention. By default, each topic starts with the name of the system in lowercase, followed by an underscore. Internal topics also start with an underscore. For example, if the system name is MESIntegrationEnvironment, the Kafka topics have the prefix mesintegrationenvironment_ or _mesintegrationenvironment_. For the list of system topics, see Kafka Connection Setup.
The consumer groups of the MES also start with the name of the system in lowercase, for example, mesintegrationenvironment_workflowexecutor_group.
Grant the permissions as prefixed ACLs on these prefixes. Prefixed ACLs also cover the topics and consumer groups that the MES creates later, such as the topic of a new IoT Event Definition. Kafka matches prefixes with case sensitivity, so use the system name in lowercase.
Required Permissions#
The permissions that the MES needs depend on the stage and on the license. The following table shows when each permission applies:
| Resource | Operation | Use | Installation and upgrade | Runtime with Data Platform license | Runtime without Data Platform license |
|---|---|---|---|---|---|
| Topic | Create | Creates the system topics and the topics of new IoT Event Definitions. A topic Create ACL that matches the topic name is enough to create it. | Required | Required | Not required |
| Topic | Describe | Reads the topic metadata and checks that the topics exist. | Required | Required | Required |
| Topic | DescribeConfigs | Reads the current configuration of the topics, such as the retention, to validate and reconcile it. | Required | Required | Not required |
| Topic | Alter | Adds partitions to the topics. | Required | Required | Not required |
| Topic | AlterConfigs | Updates the configuration of the topics, such as the retention. | Required | Required | Not required |
| Topic | Read | Consumes messages. | Required | Required | Required |
| Topic | Write | Produces messages. | Required | Required | Required |
| Consumer group | Read | Joins the consumer groups and commits the offsets. | Required | Required | Required |
| Consumer group | Describe | Reads the consumer group metadata, such as the offsets and the lag. | Required | Required | Required |
Table: Required Kafka permissions for the MES, by stage and license.
The installation and the upgrade use the topic administration permissions (Create, DescribeConfigs, Alter, and AlterConfigs) to create, validate, and update the system topics.
With a Data Platform license, the MES also uses these permissions while it runs. It creates the topic of each new IoT Event Definition, adds partitions to it, and updates its retention. Without these permissions, the creation of IoT Event Definitions fails.
Without a Data Platform license, you cannot create IoT Event Definitions, so the MES does not use the topic administration permissions while it runs. If you remove them after the installation, grant them again before the next upgrade.
The MES uses topic Delete only to delete the system topics when you uninstall the environment.
Info
Topic permissions and cluster permissions with the same name are different permissions. For example, topic DescribeConfigs reads the configuration of a topic. Cluster DescribeConfigs reads the configuration of the brokers and is optional.
The MES does not need the cluster IdempotentWrite permission. In the supported Kafka versions, topic Write is enough for idempotent producers. The MES does not use Kafka transactions, so it does not need transactional ID permissions.
Optional Cluster Permissions#
The MES works without cluster permissions. The following cluster permissions are optional. If you grant them, the MES uses them for the features described below.
| Cluster permission | Use | Behavior without the permission |
|---|---|---|
| Describe | Lists every consumer group in the cluster, for the lag monitoring. | The listing returns only the consumer groups that the consumer group ACLs cover, and the lag monitoring shows only these groups. |
| DescribeConfigs | Reads the message.max.bytes setting of the broker, so that the Data Platform services, such as HouseKeeper, know the largest message they can send. | The services log a warning and use the Kafka default limit of 1 MB instead of the limit of the broker. Payloads larger than the limit are stored in external storage, and the Kafka message references them. |
Table: Optional Kafka cluster permissions, their use, and the behavior without them.
For the Kafka permissions and the consumer group cleanup of the upgrade to version 12.0, see the Kafka Topic Migration guide ↗ in the Developer Portal.