Skip to content

Kafka Requirements#

Kafka is an integral part of the MES event management pipeline, serving as the central messaging backbone for data replication, analytics, and IoT integration. Namely for:

  • Replication Events

    • The HouseKeeper component identifies SQL Server Online Database changes and publishes replication events to Kafka.
    • HouseKeeper instances consume these events and write data to ClickHouse ODS (with horizontal scaling support).
    • Events are retained in Kafka for reprocessing when new CDM events are created.

  • CDM Events

    • Canonical Data Model (CDM) Events are a core component of the Critical Manufacturing MES Data Strategy and Data Platform.
    • HouseKeeper produces CDM events based on replication events and publishes them to Kafka.
    • External systems may also emit CDM events to MES.
    • CDM events serve as the data foundation that powers reports and dashboards built with Grafana, Stimulsoft Reports, and CubeJS.
    • CDM events trigger Data Platform workflow execution when configured.

  • IoT Events

    • MES can receive events from equipment and external sources as part of the Data Platform.
    • Events undergo authentication, authorization, and schema validation.
    • Valid IoT events are published and also stored in Kafka.

System Requirements Considerations#

In regards to Kafka, the system requirements might change depending on several key factors such as:

  • message volume (count and size)
  • throughput
  • number of partitions
  • replication factor
  • data retention policies

Proper benchmarking and monitoring of Kafka is essential for fine-tuning resource allocation based on your specific workload.

Minimum system requirement#

Item Requirement
Version >=3.8.1 to <4.0.0
Brokers 3
Memory 8GB per broker
CPU 2vCPUs per broker
Disk space 1TB per broker

Table: Minimum system requirements for Kafka

Item Requirement
Version >=3.8.1 to <4.0.0
Brokers 3
Memory 16GB per broker
CPU 4vCPUs per broker
Disk space 3TB per broker

Table: Recommended system requirements for Kafka

Access Control Lists (ACLs)#

Topic Naming Convention#

To simplify ACL setup, Kafka topics created by MES follow a standardized naming convention using the system name as a prefix.

Example: If the system name is MESIntegrationEnvironment, all Kafka topics will use the lowercase prefix:

  • mesintegrationenvironment.*
  • _mesintegrationenvironment.*

Required Permissions#

The MES system user must have the following permissions:

  • Topic Permissions

    • Alter
    • AlterConfigs
    • Create
    • Delete
    • Describe
    • DescribeConfigs
    • Read
    • Write

  • Consumer Group Permissions

    • Read
    • Delete
    • Describe

  • Cluster Permissions

    • Create
    • Describe
    • DescribeConfigs