Skip to content

Enterprise Data Platform (EDP) Tutorial#

Estimated time to read: 3 minutes

Overview#

The Enterprise Data Platform (EDP) creates a central CDM Data Lake that aggregates data from all MES installations belonging to a customer.

Local sites replicate their operational data into a shared ClickHouse database, where the EDP House Keeper processes it into a unified CDM model.

At a high level, the movement of replicated data from each MES installation into the EDP occurs through the sequence below:

  1. Local MES - Generates Replication Events and sends to the Local Kafka.
  2. EDP Connector - Consumes local replication topics and forwards events to the EDP API.
  3. EDP Service - Receives the payload and publishes it into the EDP Kafka topics.
  4. EDP House Keeper - Consumes the EDP topics to build the per-installation ODS and the centralized CDM.

EDP architecture

Information Flow#

Local Environment to EDP Connector#

Local MES systems publish two types of messages into Kafka replication topics:

  • dp_replication_events
  • dp_replication_shids (Service History IDs)
  • dp_shids_ordered (Ordered Service History IDs)

The EDP Connector consumes these topics from Kafka and forwards the messages to the EDP. The connector must be configured following the instructions described on the EDP Connector Installation Process.

EDP Connector to EDP Service#

The EDP exposes a dedicated API endpoint. For each payload received, the service it identifies the source installation:

  • Either extracted from message headers added by the connector.
  • Or enriched at the host level on arrival.

The validated payload is then ready to be forwarded into the internal EDP data pipeline.

EDP Service to EDP House Keeper#

After processing the incoming request, the EDP Service publishes the data into the EDP Kafka topics:

  • edp_replication_events
  • edp_replication_shids
  • edp_shids_ordered

These topics are then consumed by the EDP House Keeper, which is responsible for:

  • Replicating ODS data into per-installation ClickHouse databases.
  • Executing the CDM scripts to build the unified Data Lake.
  • Enforcing metadata consistency and schema transformation rules.

This ensures that all incoming data is transformed, normalized, and aligned with the EDP’s unified CDM structure.

EDP House Keeper Responsibilities#

As previously mentioned, the EDP House Keeper performs three main functions:

Replicate ODS Data Per Installation#

Each installation receives its own ClickHouse ODS database. To perform replication correctly, the House Keeper must load all metadata associated with that installation. This metadata is stored across a set of State and History tables, listed below.

State Tables#

  • dbo_T_EntityType
  • dbo_T_EntityTypeProperty
  • dbo_T_SmartTable
  • dbo_T_SmartTableProperty
  • dbo_T_SmartTablePrecedenceKey
  • dbo_T_GenericTable
  • dbo_T_GenericTableProperty
  • dbo_T_ScalarType

History Tables#

  • dbo_T_EntityTypeHst
  • dbo_T_EntityTypePropertyHst
  • dbo_T_SmartTableHst
  • dbo_T_SmartTablePropertyHst
  • dbo_T_SmartTablePrecedenceKeyHst
  • dbo_T_GenericTableHst
  • dbo_T_GenericTablePropertyHst

These metadata tables are stored inside ClickHouse ODS and must be correctly transferred from local systems.

Selection and Execution of CDM Scripts#

To build the final CDM Data Lake:

  • A set of scripts must exist per MES version.
  • Each script set converts that installation’s ODS structure into the EDP-standard CDM structure.

Maintaining ISA-95 Structure#

All ingested data must contain the mandatory ISA-95 hierarchy:

  • Enterprise
  • Site
  • Facility (Area equivalent)

If ISA-95 information is missing in the replicated payload, the EDP does not inject default values automatically. Instead, this information is defined through the Data Silo entity, which customers must configure for each MES installation. The Data Silo provides the authoritative defaults for these properties and ensures that all incoming data is grouped and interpreted correctly within the Enterprise Data Platform. To configure the Data Silo entity, follow the steps described in the EDP Connector Installation Process.