EDP Connector Installation Process#

Enterprise Data Platform - Data Silo#

When installing the EDP Connector, the very first step must be performed on the Enterprise Data Platform (EDP) instance.

The EDP Connector will extract information from an MES and then transfer it to the EDP. To achieve this, you must register the source MES as a Data Silo object in the EDP instance.

Step 1: Create a User#

Before creating a Data Silo, the first step should start with creating a user that will be used exclusively for this purpose. This user must be associated with the MES role (if this role is set as the Primary Role when creating the user, this is a one step operation, but you can still follow the most traditional way of creating the user and then associating it with the role).

Step 2: Generate an Access Token#

The next step is to generate an Access Token.

To create the Access Token, go to the User page and scroll down to the Access Tokens section. Select Create in the top ribbon, enter a name and an expiration date for the token, then select Create on the wizard to generate it.

Step 3: Create a Data Silo#

To create a Data Silo object, several key properties must be configured to fully leverage the Enterprise Data Platform (EDP).

• Name - The Data Silo name is widely referenced across the platform. Choose a name that clearly identifies the source MES to avoid ambiguity.
• Data Version - This field serves as a reference only. Its value is automatically maintained by the replication process and does not require manual updates.
• Enterprise and Site - In line with the ISA-95 model, these values should always be defined, even though current MES versions often treat them as optional. In practice, multiple MES installations may use the same Enterprise and Site identifiers, leading to potential conflicts. To ensure accurate grouping and reporting in the centralized EDP database, the Enterprise and Site values specified in the Data Silo configuration will override those received from the MES. Select identifiers that make it easy to distinguish the origin of the data in downstream reports.
• Is External - Indicates whether the Data Silo corresponds to data that originates from and is stored in a system external to the Critical Manufacturing MES, rather than being managed internally by the platform.
• Keywords - Optional metadata used to tag and categorize the Data Silo. Keywords can improve discoverability and filtering in environments with multiple silos and are especially useful for large-scale or multi-tenant deployments.
• User - This is one of the most critical properties. When the EDP Connector authenticates with EDP, the platform validates the security token, extracts the associated user information, and uses it to link incoming messages to the correct Data Silo. It is essential to follow this rule: each user must be associated with exactly one Data Silo.

After creating the Data Silo, all important details can be verified on the Details page, as shown below.

Create a New Customer Environment - Devops Center#

To start the installation process of the EDP Connector, follow the steps described in the Installation Guide.

1. Create a new Customer Environment, as described in step 1 of the Installation Guide.

2. After completing step 1 of the Installation Guide, select the Deployment Package EDP Connector <version>, with the version depending on your current needs. In the example below, the version configured is 11.2.2:

   

3. The Package Configuration has no options to be set, so it can be skipped.

4. Choose the Target container management platform, where you can choose among OpenShift, Kubernetes or Docker, among others. For more information, see Deployment Targets ⧉.

   

5. Accept the License Agreement.

   

6. In the Dependencies, define the settings that will allow us to connect to the source of the data, which as you saw above, is called a Data Silo. If the source data is being produced by a CM MES with a version below v11, you only need to set the system name.

   If the source data is being produced by an instance of CM MES v11 or higher, set the Kafka and S3 settings.

   

   

7. In the Database step, define the settings to access the database:

   

8. On the EDP Host step, define all the necessary properties to connect to the EDP server instance.

   

   Note

   If these settings are not known, you can find them in the EDP server instance already installed.

   Set the System Name, Tenant Name, and Host Address, which can be found in the General Data section of the Customer Environment of Enterprise Data Platform on the Customer Portal.

   

   The Security Portal Address is a result of the concatenation of the protocol used to access the Enterprise Data Platform UI, http or https, the application public address, ending in /SecurityPortal.

   The Security Portal Client ID can be found in the General Data of the environment of Enterprise Data Platform on the Customer Portal.

   

   The value of the Access Token generated in the Security GUI in the Enterprise Data Platform will be used to set this EDP User Personal Access Token.

9. The Cloudflare Configs step has no options to be set, so it can be skipped.

10. Set the Service Resources based on the expected traffic volume.

    

11. The last installation step must be set according to your needs. The EDP Connector needs to connect to the database of the source MES system and to the EDP instance, so if the MES version is v11 or higher, it also needs to access the Kafka and S3 instances used by the source MES.

    

Replicate Old Information - SQL Server#

After installing the EDP Connector, the information that is created from that point on will be replicated to the EDP infrastructure. To replicate all the data that was generated by CM MES before the EDP Connector installation, you must manually start a procedure on the ODS Database in the SQL Server.

During the installation process of the EDP Connector, a procedure with the name P_ProcessShids is created. This procedure compiles some information from the ODS database to determine which tables were changed for each Service History ID, which will then feed the replication process to the EDP infrastructure.

To start this process, you must first log in to the ODS Database of SQL Server supporting CM MES - the source of the data - and then run the procedure dbo.P_ProcessShids.

/*
@StartDate DATETIME - The start date of the retrieval process.
@EndDate DATETIME - The end date of the retrieval process (default: current time).
@ExecuteReplication BIT - If = 1, the system will create Replication events for the Service History Ids found in the retrieval process.
@ExecuteCDM BIT - If = 1 it will, the system will create CDM events for the Service History Ids found in the retrieval process.
@Debug BIT - Prints extra debug information (default: 0).
@BlockSize INTEGER - Number of valid Service History Ids to be analysed in each batch (default: 10000).
@IgnoreQueuedCommands - If = 1, it will not check if there are commands in the command queue for ODS. This will only be done if this procedure is run against a ODS database (default: 0).
@SelectedTablesForReplication NVARCHAR(MAX) - If set, only the selected tables will be considered for Replication. This should be in a a comma separated format (e.g.: CoreDataModel.T_AutomationDriverInstanceHistory,CoreDataModel.T_CmfTimerHistory).
@SelectedEventsForCDM NVARCHAR(MAX) - If set, only the selected events will be considered for CDM. This should be in a a comma separated format (e.g.: CDM.Material.LossBonus,CDM.Calendar.CalendarDay).
*/

DECLARE @RC int
DECLARE @StartDate datetime
DECLARE @EndDate datetime
DECLARE @ExecuteReplication bit
DECLARE @ExecuteCDM bit
DECLARE @Debug bit
DECLARE @BlockSize int
DECLARE @IgnoreQueuedCommands bit
DECLARE @SelectedTablesForReplication nvarchar(max)
DECLARE @SelectedEventsForCDM nvarchar(max)

-- Set parameter values here.

EXECUTE @RC = [dbo].[P_ProcessSHIDs] 
@StartDate
,@EndDate
,@ExecuteReplication
,@ExecuteCDM
,@Debug
,@BlockSize
,@IgnoreQueuedCommands
,@SelectedTablesForReplication
,@SelectedEventsForCDM

If this is the first time you are running this procedure, it is recommended to set the StartDate parameter with a valid date before the first transaction that occurred in your CM MES.

You can check this date manually on the dbo.T_ServiceHistory table and retrieve the oldest record on this table (from the ODS database) to extract the ServiceEndTime. Alternatively, initialize the variable @StartDate with the oldest ServiceEndTime minus one day by changing the declaration of the StartDate:

DECLARE @StartDate datetime = DATEADD(DAY, -1, (SELECT TOP(1) ServiceEndTime FROM dbo.T_ServiceHistory ORDER BY ServiceEndTime ASC))

or simply replace the comment

-- Set parameter values here.

with

SET @StartDate = DATEADD(DAY, -1, (SELECT TOP(1) ServiceEndTime FROM dbo.T_ServiceHistory ORDER BY ServiceEndTime ASC))