Customer Environment MES Stack#

This page lists all the containers that are running inside the Critical Manufacturing MES stack, with an explanation of its purpose as well as specific configurations for each container.

graph LR
subgraph Infrastructure Agent
    Squid[Edge Squid Proxy]
    end
subgraph Customer Environment
    Env[Environment Manager]
    UI[UI]
    Host[Host]
    Help[Help]
    Grafana[Grafana]
    SecurityPortal[Security Portal]
    Traefik[Traefik]
    TraefikForwardAuth[Traefik Forward Auth]
    Bus[MessageBus]
    DiscoveryService[Discovery Service]
    SQL[SQL Server]
    ConnectIoTManager[Connect IoT Manager]
    Rasa[Rasa]
    RasaActions[Rasa Actions]
    LBOGenerator[LBO Generator]
    MESScheduler[MES Scheduler]
    end
subgraph DataPlatform
    direction RL
    ClickHouse[ClickHouse]
    Kafka[Kafka]
    RabbitMQ[RabbitMQ]
    S3[S3]
    DataManager[Data Manager]
    EPFAMAT[EPF Alarm Management Action Trigger]
    EPFAMERH[EPF Alarm Management Event Rule Handler]
    EPFMEH[EPF Alarm Management MES Event Handler]
    MLAgent[ML Platform Agent]
    MLPlatformTraining[ML Platform Training]
    Redis[Redis]
    end

    Traefik --> TraefikForwardAuth
    TraefikForwardAuth --> Traefik
    Traefik --> UI
    Traefik --> Host
    Traefik --> Help
    Traefik --> Grafana
    Traefik --> SecurityPortal
    Traefik --> Bus
    Traefik --> DataPlatform
    Traefik --> ConnectIoTManager
    Traefik --> Rasa
    Traefik --> RasaActions
    Traefik --> DiscoveryService
    Traefik --> LBOGenerator
    Traefik --> MESScheduler
    Host -.-> SQL
    SQL -.-> Env
    Host -.-> Env
    UI -.-> Env
    Help -.-> Env
    Grafana -.-> Env
    SecurityPortal -.-> Env
    Bus -.-> Env
    DataPlatform -.-> Env
    ConnectIoTManager -.-> Env
    Rasa -.-> Env
    RasaActions -.-> Env
    DiscoveryService -.-> Env
    LBOGenerator -.-> Env
    MESScheduler -.-> Env
    Env ---> Squid

classDef mermaid_title color:#000, fill:#fafafa, stroke:#fafafa, stroke-width:0x, font-size:100%, font-weight:200;
classDef mermaid_start color:#000, fill:#fafafa, stroke:#fafafa, color:#fafafa, stroke-width:0x, font-size:100%, visibility: hidden;
classDef mermaid_businessdata color:#000, fill:#65CDE8, stroke:#65CDE8, stroke-width:0px, font-size:100%;
classDef mermaid_nonbusinessdata color:#000, fill:#B7DEE8, stroke:#B7DEE8, stroke-width:0px, font-size:100%;
classDef mermaid_entity color:#000, fill:#FB9F53, stroke:#FB9F53, stroke-width:0px, font-size:100%;
classDef mermaid_entitylinked color:#000, fill:#FCD5B5, stroke:#FCD5B5, stroke-width:0px, font-size:100%;
classDef mermaid_context color:#000, fill:#B9CDE5, stroke:#B9CDE5, stroke-width:0px, font-size:100%;
classDef mermaid_optional color:#000, fill:#B7DEE8, stroke:#65CDE8, stroke-width:1px, font-size:100%, stroke-dasharray: 5 5;
classDef mermaid_state color:#000, fill:#d7e4bd, stroke:#000, stroke-width:1px, font-size:100%, font-weight:300;

class Squid,UI,Host,Help,SecurityPortal,ConnectIoTManager,Bus,Grafana,DiscoveryService,Rasa,RasaActions,LBOGenerator,MESScheduler mermaid_businessdata
class Env,Traefik,TraefikForwardAuth mermaid_entitylinked
class User mermaid_context
class ClickHouse,Kafka,RabbitMQ,S3,DataManager,EPFAMAT,EPFAMERH,EPFMEH,MLAgent,MLPlatformTraining,Redis,ReplicateEvents,ReplicateIDs,ReplicateTableSchemas mermaid_nonbusinessdata
class SQL mermaid_optional

click Traefik "#traefik"
click TraefikForwardAuth "#traefik-forward-auth"
click Bus "#mesmessage-bus"
click Host "#mes-host"
click Env "#environment-manager"
click SecurityPortal "#security-portal"
click UI "#mes-gui"
click Help "#mes-documentation"
click HelpReference "#mes-documentation-reference"
click Grafana "#grafana"
click SQL "#sql-server"
click ClickHouse "#clickhouse"
click Kafka "#kafka"
click RabbitMQ "#rabbitmq"
click S3 "#s3"
click ConnectIoTManager "#connect-iot-manager"
click DataManager "#data-manager"
click DiscoveryService "#discovery-service"
click EPFAMAT "#epf-alarm-management-action-trigger"
click EPFAMERH "#epf-alarm-management-event-rule-handler"
click EPFMEH "#epf-alarm-management-mes-event-handler"
click MLAgent "#ml-platform-agent"
click MLPlatformTraining "#ml-platform-training"
click Rasa "#rasa"
click RasaActions "#rasa-actions"
click Redis "#redis"
click ReplicateEvents "#replicate-events"
click ReplicateIDs "#replicate-ids"
click ReplicateTableSchemas "#replicate-table-schemas"

Core Components#

Traefik#

Manages incoming network traffic, acting as a reverse proxy by forwarding requests to services based on routing rules. For more information, see https://doc.traefik.io/traefik/ ⧉.

Each MES environment includes a Traefik instance that is entirely responsible for that same stack. A top-level Ingress Controller is required in order to route incoming traffic to the appropriate MES stack's Traefik. Security must be dealt with at this first level, such as handling certificate validations and ensuring secure connections.

spec:
  containers:
  - args:
    - --entrypoints.web-secure.address=:8443

Example:

• http://<domain_one>/ - the Traefik instance inside the MES stack sends this request to the proper container inside the MES stack for domain_one that handles the / path, which in this case is the UI.
• http://<domain_one>/api - the Traefik instance inside the MES stack sends this request to the proper container inside the MES stack for domain_one that handles the api path, which in this case is the host.
• http://<domain_two>/help - the Traefik instance inside the MES stack sends this request to the proper container inside the MES stack for domain_two that handles the help path, which in this case is the Documentation Portal.

graph LR
subgraph Agent
    Traefik[Traefik]
end
subgraph mes_instance_two
    Traefik3[Traefik]
    ui3["UI"]
    api3["Host"]
    help3["Documentation Portal"]
end
subgraph mes_instance_one
    Traefik2[Traefik]
    ui2["UI"]
    api2["Host"]
    help2["Documentation Portal"]
end
Traefik --->|https://mes_instance_one.domain.local/|Traefik2
Traefik --->|https://mes_instance_one.domain.local/api|Traefik2
Traefik --->|https://mes_instance_two.domain.local/help|Traefik3
Traefik2 -.->|/|ui2
Traefik2 -.->|/api|api2
Traefik3 -.->|/help|help3

classDef mermaid_title color:#000, fill:#fafafa, stroke:#fafafa, stroke-width:0x, font-size:100%, font-weight:200;
classDef mermaid_start color:#000, fill:#fafafa, stroke:#fafafa, color:#fafafa, stroke-width:0x, font-size:100%, visibility: hidden;
classDef mermaid_businessdata color:#000, fill:#65CDE8, stroke:#65CDE8, stroke-width:0px, font-size:100%;
classDef mermaid_nonbusinessdata color:#000, fill:#B7DEE8, stroke:#B7DEE8, stroke-width:0px, font-size:100%;
classDef mermaid_entity color:#000, fill:#FB9F53, stroke:#FB9F53, stroke-width:0px, font-size:100%;
classDef mermaid_entitylinked color:#000, fill:#FCD5B5, stroke:#FCD5B5, stroke-width:0px, font-size:100%;
classDef mermaid_context color:#000, fill:#B9CDE5, stroke:#B9CDE5, stroke-width:0px, font-size:100%;
classDef mermaid_optional color:#000, fill:#B7DEE8, stroke:#65CDE8, stroke-width:1px, font-size:100%, stroke-dasharray: 5 5;
classDef mermaid_state color:#000, fill:#d7e4bd, stroke:#000, stroke-width:1px, font-size:100%, font-weight:300;
class Traefik,Traefik2,Traefik3 mermaid_entity
class api2,api3,help2,help3,ui2,ui3 mermaid_businessdata

Traefik Forward Auth#

Provides support to authentication through Critical Manufacturing Security Portal, equivalent to an IIS module that is installed and running in traditional installations. Every request that Traefik receives gets carried over to Traefik Forward Auth to ascertain whether that request has valid authentication.

Relevant folders in container volumes:

Table: Traefik Forward Auth relevant folders for containerized environments

Environment Manager#

The Environment Manager works as an overseer of the entire MES stack and coordinates as a primary information point from which all other containers depend. It performs operations like the actual installation setup application of the traditional environments, with added functionality, wherein every time the stack is started, all containers have the same entrypoint (a small piece of executable code) that queries the Environment Manager container to see whether the system is installed and running and also if there are any customization packages that need to be installed for any specific container. When the Environment Manager responds to the queries from all other containers to indicate that the system has been installed, the other containers can safely run.

sequenceDiagram
    Container in MES Stack->>Environment Manager: Is MES installed?
    activate Environment Manager
    Environment Manager-->>Container in MES Stack: Not yet, checking
    Environment Manager-->>Container in MES Stack: Yes, it is, you can run safely
    deactivate Environment Manager
    activate Container in MES Stack
    Container in MES Stack->>+Environment Manager: Are there any custom components I should know about?
    deactivate Container in MES Stack
    Environment Manager-->>Container in MES Stack: Yes, here they are
    activate Environment Manager
    Container in MES Stack->>Environment Manager: Ok, installing before running the main application
    deactivate Environment Manager

Customization packages are usually stored in one of the volumes bound by the Environment Manager and are accessible through it.

Relevant folders in container volumes:

Table: Environment Manager relevant folders for containerized environments

Security Portal#

A multi-tenant application that acts as an identity management system. For more information, see Security Portal.

Relevant folders in container volumes:

Table: Critical Manufacturing MES Security Portal relevant folders for containerized environments

MES Host#

The set of services that operate the core of Critical Manufacturing MES services and orchestration, exposing a public API for external access.

Relevant folders in container volumes:

Table: Critical Manufacturing MES Host relevant folders for containerized environments

MES GUI#

Web server used as the main visual entrypoint for the Critical Manufacturing MES.

Relevant folders in container volumes:

Table: Critical Manufacturing MES GUI relevant folders for containerized environments

MES Documentation#

Complete set of user documentation for Critical Manufacturing MES, including this very page.

Relevant folders in container volumes:

Table: Critical Manufacturing MES Documentation relevant folders for containerized environments

MES Reference Documentation#

Reference documentation for Critical Manufacturing MES, including API documentation and Design System.

Relevant folders in container volumes:

Table: Critical Manufacturing MES Reference Documentation relevant folders for containerized environments

MES Message Bus#

A high-performance publish and subscribe message bus. For more information, see Message Bus.

Relevant folders in container volumes:

Table: Critical Manufacturing MES Message Bus relevant folders for containerized environments

Grafana#

Visualization framework that allows you to query, visualize metrics from the Critical Manufacturing MES through widgets and dashboards.

Relevant folders in container volumes:

Table: Grafana relevant folders for containerized environments

SQL Server#

Installed optionally through DevOps Center, this container holds an instance of SQL Server that can be used only with the Online database. It is lacking:

• Reports
• ODS database
• DWH database
• Analysis Services

Relevant folders in container volumes:

Table: SQL Server relevant folder for containerized environments

Discovery Service#

The Discovery Service is a key component of highly-available infrastructures commonly used to improve the performance and availability of web sites, applications, databases and other services by distributing the workload across multiple servers. For more information, see Discovery Service

Relevant folders in container volumes:

Table: Discovery Services relevant folders for containerized environments

Rasa#

From Rasa ⧉:

Rasa Open Source is an open source conversational AI platform that allows you to understand and hold conversations, and connect to messaging channels and third party systems through a set of APIs. It supplies the building blocks for creating virtual (digital) assistants or chatbots.

Relevant folders in container volumes:

Table: Rasa relevant folders for containerized environments

Rasa Actions#

From Rasa ⧉:

When a Rasa assistant calls a custom action, it sends a request to the action server. Rasa only knows about whatever events and responses come back in the request response; it's up to the action server to call the correct code based on the action name that Rasa provides.

Relevant folders in container volumes:

Table: Rasa Actions relevant folders for containerized environments

Connect IoT Manager#

Connect IoT

The Connect IoT Manager is responsible for the entire Automation component processes lifecycle.

Relevant folders in container volumes:

Table: Connect IoT Manager relevant folders for containerized environments

LBO Generator#

LBO Generator

The service responsible for generating and serving LBOs.

Table: LBO Generator relevant folders for containerized environments

MES Scheduler#

MES Scheduler

The service responsible for MES Scheduling operations.

Data Platform#

ClickHouse#

Data Platform

From ClickHouse ⧉:

ClickHouse is a column-oriented database that enables its users to generate powerful analytics, using SQL queries, in real-time.

Relevant folders in container volumes:

Table: ClickHouse relevant folders for containerized environments

Kafka#

Data Platform

From Apache Kafka ⧉:

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

This is the first instance of Kafka for streaming operations.

Relevant folders in container volumes:

Table: Kafka relevant folders for containerized environments

RabbitMQ#

Data Platform

From RabbitMQ ⧉:

RabbitMQ is a reliable and mature messaging and streaming broker.

Relevant folders in container volumes:

Table: RabbitMQ relevant folders for containerized environments

S3#

Data Platform

From MinIO ⧉:

MinIO is an object storage system. It is an API compatible with the Amazon S3 cloud storage service, capable of working with unstructured data such as photos, videos, log files, backups, and container images.

Relevant folders in container volumes:

Table: S3 relevant folders for containerized environments

Data Manager#

Data Platform

The Data Manager acts as the manager of all Data Stores. A Data Store is a physical data repository, typically a database of a certain technology that is abstracted by a Data Store Driver. Data Sets (structured tabular grid of rows (records) and columns (fields)) are written and read from a Data Store. For internally managed Data Sets, the system takes care of the schema creation, space provisioning and overall data set management (e.g.: purging).

The Data Manager provides a central access to Data in the platform, enabling data security and also exposing the Data via an OData interface or by using the Grafana interface.

Relevant folders in container volumes:

Table: Data Manager relevant folders for containerized environments

EPF Alarm Management Action Trigger#

Data Platform

Listens to Kafka topics from Event Rules and executes the associated actions.

Relevant folders in container volumes:

Table: EPF Alarm Management Action Trigger relevant folders for containerized environments

EPF Alarm Management Event Rule Handler#

Data Platform

Receives an EventRule to be executed and instantiates a new Notification based on the properties defined in the EventRule. Alternatively, if the EventRule defines a Rule, it should be executed instead of creating a notification. This allows for almost infinite extensibility options. All inputs received by the TriggerNotification API should be passed as inputs to the executed rule. More information on Notification and Rule

Relevant folders in container volumes:

Table: EPF Alarm Management Event Rule Handler relevant folders for containerized environments

EPF Alarm Management MES Event Handler#

Data Platform

Generating MES events in order to replicate data to an ODS database. It has a mechanism that watches for new Service History IDs in the Online Database, gathers all data that has been updated or inserted in that transaction, and publishes an MES Event to a specific topic in Kafka.

Relevant folders in container volumes:

Table: EPF Alarm Management MES Event Handler relevant folders for containerized environments

ML Platform Agent#

Data Platform

Interacts with the application, providing values extracted from trained models.

Relevant folders in container volumes:

Table: ML Platform Agent relevant folders for containerized environments

ML Platform Training#

Data Platform

Performs model training from events gathered by the Data Platform.

Relevant folders in container volumes:

Table: ML Platform Training relevant folders for containerized environments

Redis#

Data Platform

From Redis ⧉:

The open source, in-memory data store used by millions of developers as a database, cache, streaming engine, and message broker.

Relevant folders in container volumes:

Table: Redis relevant folders for containerized environments

Replicate Events#

Data Platform

Listens to operation IDs on messages with Kafka topics sent by the Replicate IDs application, matching the retrieved IDs with the ones in the database to check for changes. When found, retrieves the information and produces a JSON document which is then sent through a Kafka topic.

Relevant folders in container volumes:

Table: Replicate Events relevant folders for containerized environments

Replicate IDs#

Data Platform

Checks the main database for all generated IDs, takes them and broadcasts them in a Kafka topic.

Relevant folders in container volumes:

Table: Replicate IDs relevant folders for containerized environments

Replicate Table Schemas#

Data Platform

Sends all table structures to Redis.

Relevant folders in container volumes:

Table: Replicate Table Schemas relevant folders for containerized environments