---
pdfexport: true
alias: tutorials-monitoring
timetoread: true
tutorial: full
description: "This documentation details monitoring probes for data platform components and their processes"
---

# Data Platform Components Monitoring

## Overview

In this tutorial we describe ways to monitor **Data Platforms** components that handle data ingestion, processing, aggregation and replication.

## System Monitoring

**System Monitoring**, in the **Administration** section, contains eight probes that can be used to monitor **Data Platforms** components.

In the **Status** tab there are two probes:

1. **Online Replication Queue** tells us how many messages are waiting to be published to Kafka, with information regarding what needs to be done for replicating data and building the Canonical Data Model (CDM) events; these messages are produced whenever a transaction is closed in CM MES.

2. **ODS Replication Queue** provides the same information, only for messages created in ODS by the Initial Sync mechanism (which is used to run replication/CDM building for historical data).

    ![Status Monitoring](../../images/data_platform_monitoring_01.png)

In the **Database -> Kafka** tab there are six more probes:

1. **ClickHouse Work Queue Lag** tells us how many ClickHouse migration messages **HouseKeeper** has not consumed yet, after an upgrade. It shows **N/A** when there is no migration message to handle, which is the normal state outside an upgrade.

2. **CDM Lag** tells us the lag building the CDM documents.

3. **Work Queue Lag** shows how many requests for data sets and **IoT Event Definitions** are still waiting for **HouseKeeper**.

4. **Replication Lag** tells us the lag materializing replication documents in the destination database.

5. **Metadata Replication Lag** shows the lag materializing the metadata replication documents, such as the properties of entity types and smart tables, in the destination database.

6. **Ordered Service History IDs Lag** shows how many Service History IDs **HouseKeeper** has not yet added, in order, to ClickHouse ODS. The CDM Builder waits for these IDs before it builds the CDM documents.

    ![System Monitoring Database Kafka tab with the ClickHouse Work Queue Lag probe highlighted](../../images/data_platform_monitoring_02.drawio.svg)

    The location of the probes is represented in this simplified flow chart, which can be used to help diagnose potential issues with **Data Platform** components:

    ![Data Platform pipeline with the System Monitoring probes numbered 1 to 8, in the order of the diagnosis list](../../images/data_platform_monitoring_03.drawio.svg)

A quick way to look at it:

1. If **Online Replication Queue** shows a lot of messages waiting to be published, then it is possible that there is an issue with either **Kafka** or Data Platform component **HouseKeeper**.
2. If **ODS Replication Queue** shows a lot of messages waiting to be published, then the diagnosis is the same, that is, a potential issue with either **Kafka** or **HouseKeeper**.
3. If **CDM Lag** is too high, then there might be an issue with **HouseKeeper**, but it is more likely that either the MES **SQL Server** database is too slow, or more **HouseKeeper** instances are needed to keep up with the workload being generated by CM MES.
4. If **Replication Lag** is too high, then the most likely diagnosis is an issue with **ClickHouse** or **HouseKeeper**.
5. If **Metadata Replication Lag** is too high, then the diagnosis is the same as for **Replication Lag**, an issue with **ClickHouse** or **HouseKeeper**.
6. If **Work Queue Lag** stays above 0, then new data sets and **IoT Event Definitions** are not ready yet. Check that **HouseKeeper** is running, and check its logs for errors.
7. If **Ordered Service History IDs Lag** is too high, then the CDM Builder waits, and **CDM Lag** also grows. Check **HouseKeeper**, the MES **SQL Server** database, and **ClickHouse**.
8. If **ClickHouse Work Queue Lag** stays above 0 for a long time after an upgrade, or shows **N/A** while the migration should still be running, then a ClickHouse migration message is probably failing. Check the **HouseKeeper** logs for migration errors. For more information, see [ClickHouse Migration Process](https://developer.criticalmanufacturing.com/explore/guides/migration/version-specific/1200/data-platform/clickhouse_migration_process/).

By looking at the status of these probes it should be easy to determine if the CDM building and the ODS replication are working as expected and without issues.

## Data Orchestrator

Another **Data Platform** process that can be monitored in CM MES is the aggregation of the Data Warehouse data sets. This can be done by navigating to the **Data Orchestrator**, in the **Administration** section:

![Data Orchestrator](../../images/data_platform_monitoring_04.png)

The **Data Orchestrator** is **Data Platforms** job runner. If you navigate to the **Runs** tab you can see the status of the job that aggregates the data in the  Data Warehouse data sets, which goes by the name `materialize_dbt_models` and runs every 5 minutes:

![Data Orchestrator Job Runs](../../images/data_platform_monitoring_05.png)

If you select **View** for a specific run, you can get details on the aggregations that were performed:

![Data Orchestrator Run Log](../../images/data_platform_monitoring_06.png)