Skip to content

Kafka Requirements#

Kafka is an integral part of the MES event management pipeline, serving as the central messaging backbone for data replication, analytics, and IoT integration. Namely for:

  • Replication Events

    • The HouseKeeper component identifies SQL Server Online Database changes and publishes replication events to Kafka.
    • HouseKeeper instances consume these events and write data to ClickHouse ODS (with horizontal scaling support).
    • Events are retained in Kafka for reprocessing when new CDM events are created.

  • CDM Events

    • Canonical Data Model (CDM) Events are a core component of the Critical Manufacturing MES Data Strategy and Data Platform.
    • HouseKeeper produces CDM events based on replication events and publishes them to Kafka.
    • External systems may also emit CDM events to MES.
    • CDM events serve as the data foundation that powers reports and dashboards built with Grafana, Stimulsoft Reports, and CubeJS.
    • CDM events trigger Data Platform workflow execution when configured.

  • IoT Events

    • MES can receive events from equipment and external sources as part of the Data Platform.
    • Events undergo authentication, authorization, and schema validation.
    • Valid IoT events are published and also stored in Kafka.

System Requirements Considerations#

In regards to Kafka, the system requirements might change depending on several key factors such as:

  • message volume (count and size)
  • throughput
  • number of partitions
  • replication factor
  • data retention policies

Proper benchmarking and monitoring of Kafka is essential for fine-tuning resource allocation based on your specific workload.

Minimum system requirement#

Item Requirement
Version >=3.8.1 to <4.0.0
Brokers 3
Memory 8GB per broker
CPU 2vCPUs per broker
Disk space 1TB per broker

Table: Minimum system requirements for Kafka

Item Requirement
Version >=3.8.1 to <4.0.0
Brokers 3
Memory 16GB per broker
CPU 4vCPUs per broker
Disk space 3TB per broker

Table: Recommended system requirements for Kafka

Data Retention#

Kafka keeps messages for a configurable retention time before deleting them. The recommended value is 90 days.

Size the retention time for the periods when consumers fall behind, such as production peaks or system stoppages. This value defines how long a consumer can stay behind before messages are lost.

Note

A lower retention time is possible if you monitor the consumer lag and confirm that messages are processed as expected. This decision falls under the responsibility of the customer, because messages deleted before consumption cannot be recovered.

Access Control Lists (ACLs)#

Topic Naming Convention#

To simplify ACL setup, Kafka topics created by MES follow a standardized naming convention using the system name as a prefix.

Example: If the system name is MESIntegrationEnvironment, all Kafka topics will use the lowercase prefix:

  • mesintegrationenvironment.*
  • _mesintegrationenvironment.*

Required Permissions#

The MES system user must have the following permissions:

  • Topic Permissions

    • Alter
    • AlterConfigs
    • Create
    • Delete
    • Describe
    • DescribeConfigs
    • Read
    • Write

  • Consumer Group Permissions

    • Read
    • Delete
    • Describe

  • Cluster Permissions

    • Create
    • Describe
    • DescribeConfigs