Skip to content

Kafka Requirements#

In regards to Kafka, the system requirements might change depending on several key factors such as:

  • message volume (count and size)
  • throughput
  • number of partitions
  • replication factor
  • data retention policies

Proper benchmarking and monitoring of Kafka is essential for fine-tuning resource allocation.

Minimum system requirement#

Item Requirement
Version >=3.8.1 to <4.0.0
Brokers 3
Memory 8GB per broker
CPU 2vCPUs per broker
Disk space 1TB per broker

Table: Minimum system requirements for Kafka

Item Requirement
Version >=3.8.1 to <4.0.0
Brokers 3
Memory 16GB per broker
CPU 4vCPUs per broker
Disk space 3TB per broker

Table: Recommended system requirements for Kafka

Data Retention#

Kafka keeps messages for a configurable retention time before deleting them. The recommended value is 90 days.

Size the retention time for the periods when consumers fall behind, such as production peaks or system stoppages. This value defines how long a consumer can stay behind before messages are lost.

Note

A lower retention time is possible if you monitor the consumer lag and confirm that messages are processed as expected. This decision falls under the responsibility of the customer, because messages deleted before consumption cannot be recovered.

Access Control Lists (ACLs)#

To simplify the setup of ACLs by system administrators, Kafka topics created by MES adhere to a standardized naming convention. For example, if the system name is "MESIntegrationEnvironment", all Kafka topics will be created with the prefix of the system name in lower case. So, in this case, all Kafka topics will have the prefix of "mesintegrationenvironment" or "_mesintegrationenvironment".

The user used by the system must have at least the following permissions:

  • Topic Permissions

    • Alter
    • AlterConfigs
    • Create
    • Delete
    • Describe
    • DescribeConfigs
    • Read
    • Write

  • Consumer Group Permissions

    • Read
    • Delete
    • Describe

  • Cluster Permissions

    • Create
    • Describe
    • DescribeConfigs