Skip to content
View as markdown Ask ChatGPT Ask Claude

Data Platform Components Monitoring#

Estimated time to read: 4 minutes

Overview#

In this tutorial we describe ways to monitor Data Platforms components that handle data ingestion, processing, aggregation and replication.

System Monitoring#

System Monitoring, in the Administration section, contains eight probes that can be used to monitor Data Platforms components.

In the Status tab there are two probes:

  1. Online Replication Queue tells us how many messages are waiting to be published to Kafka, with information regarding what needs to be done for replicating data and building the Canonical Data Model (CDM) events; these messages are produced whenever a transaction is closed in CM MES.

  2. ODS Replication Queue provides the same information, only for messages created in ODS by the Initial Sync mechanism (which is used to run replication/CDM building for historical data).

    Status Monitoring

In the Database -> Kafka tab there are six more probes:

  1. ClickHouse Work Queue Lag tells us how many ClickHouse migration messages HouseKeeper has not consumed yet, after an upgrade. It shows N/A when there is no migration message to handle, which is the normal state outside an upgrade.

  2. CDM Lag tells us the lag building the CDM documents.

  3. Work Queue Lag shows how many requests for data sets and IoT Event Definitions are still waiting for HouseKeeper.

  4. Replication Lag tells us the lag materializing replication documents in the destination database.

  5. Metadata Replication Lag shows the lag materializing the metadata replication documents, such as the properties of entity types and smart tables, in the destination database.

  6. Ordered Service History IDs Lag shows how many Service History IDs HouseKeeper has not yet added, in order, to ClickHouse ODS. The CDM Builder waits for these IDs before it builds the CDM documents.

    System Monitoring Database Kafka tab with the ClickHouse Work Queue Lag probe highlighted

    The location of the probes is represented in this simplified flow chart, which can be used to help diagnose potential issues with Data Platform components:

    Data Platform pipeline with the System Monitoring probes numbered 1 to 8, in the order of the diagnosis list

A quick way to look at it:

  1. If Online Replication Queue shows a lot of messages waiting to be published, then it is possible that there is an issue with either Kafka or Data Platform component HouseKeeper.
  2. If ODS Replication Queue shows a lot of messages waiting to be published, then the diagnosis is the same, that is, a potential issue with either Kafka or HouseKeeper.
  3. If CDM Lag is too high, then there might be an issue with HouseKeeper, but it is more likely that either the MES SQL Server database is too slow, or more HouseKeeper instances are needed to keep up with the workload being generated by CM MES.
  4. If Replication Lag is too high, then the most likely diagnosis is an issue with ClickHouse or HouseKeeper.
  5. If Metadata Replication Lag is too high, then the diagnosis is the same as for Replication Lag, an issue with ClickHouse or HouseKeeper.
  6. If Work Queue Lag stays above 0, then new data sets and IoT Event Definitions are not ready yet. Check that HouseKeeper is running, and check its logs for errors.
  7. If Ordered Service History IDs Lag is too high, then the CDM Builder waits, and CDM Lag also grows. Check HouseKeeper, the MES SQL Server database, and ClickHouse.
  8. If ClickHouse Work Queue Lag stays above 0 for a long time after an upgrade, or shows N/A while the migration should still be running, then a ClickHouse migration message is probably failing. Check the HouseKeeper logs for migration errors. For more information, see ClickHouse Migration Process ↗.

By looking at the status of these probes it should be easy to determine if the CDM building and the ODS replication are working as expected and without issues.

Data Orchestrator#

Another Data Platform process that can be monitored in CM MES is the aggregation of the Data Warehouse data sets. This can be done by navigating to the Data Orchestrator, in the Administration section:

Data Orchestrator

The Data Orchestrator is Data Platforms job runner. If you navigate to the Runs tab you can see the status of the job that aggregates the data in the Data Warehouse data sets, which goes by the name materialize_dbt_models and runs every 5 minutes:

Data Orchestrator Job Runs

If you select View for a specific run, you can get details on the aggregations that were performed:

Data Orchestrator Run Log