--- alias: operation-guide-connect-iot-runbook tags: - automation - connect iot description: "Operate running Connect IoT instances: what a change in MES reaches, when to restart, reconnect, or replace an instance, and which log lines confirm each step" --- # Connect IoT Runbook A Connect IoT integration lives in MES, and each **Automation Controller** instance runs a copy of it. For each instance, the **Automation Manager** runs one controller process and one driver process per driver. The controller reads its workflows from MES when it starts and follows workflow changes over the message bus while it runs. Everything else, such as the driver instances, the configuration that the Setup workflow sent each driver, and what each driver set up on the equipment, is taken when the processes start. Use this page to decide what a change needs, to restart, reconnect, or replace an instance, and to confirm each step in the logs. ## Instances and Names - One **Automation Controller** runs on several machines as one instance per entity. Each instance holds one **Automation Driver Instance** per driver of the controller, created with the instance. - An entity that is already connected to an active instance, or to a driver of another active instance, cannot be connected again. See [[user-guide-automation-manager-connect#remarks]]. - An instance is named `__`, unless a name generator is set in `/Cmf/System/Configuration/Business/NameGenerators/AutomationControllerInstance/`. A driver instance is named `___`. In both names, `` is the creation time in seconds since 1970. - A driver instance is connected to an entity of the entity type of its **Automation Driver Definition**, which the Connect wizard asks for separately from the entity of the instance. - Keep what differs between machines, such as a folder or an address, in MES per entity, in a **Smart Table** row or an entity attribute. The Setup workflow reads it at `OnInitialize` with **Entity Instance** and **Resolve Table**, before **Equipment Configuration** and **Driver Command** `Connect`. - Give each instance its own folder, port, or topic. Two file drivers that watch one folder race for every file, and the loser fails with `ENOENT`. - A terminated object stays in MES, and its name stays taken. Number the names of objects that you create and retire repeatedly. ## What a Change Reaches A change applied in MES reaches a running instance at once, at the next restart, or only when the instance is replaced. | Change in MES | What the running instance does | What to do | |---|---|---| | A workflow is added. | The controller loads it at once and logs `Notified that workflows were created. Adding ...` and `Adding workflow: Name='', …`. | Check the log for both lines. | | A workflow is changed. | Only the instances that run the changed version, which is in the `Created` state, apply the change. The controller replaces the workflow at once and logs `Notified that workflows were changed. Updating` and `New workflow '' created and running, draining the old one`. | Check the log for both lines. For the Setup workflow, see the next row. | | The Setup workflow, a connection parameter, or a value that Setup reads from MES changes. | A driver that is already `Communicating` keeps the parameters it received. **Equipment Configuration** emits `Success` without sending the new ones. | Restart the instance. | | The properties, events, or commands of a driver definition change. | The driver keeps the definition it received during its setup. | Restart the instance. | | A workflow is removed. | The controller logs `Notified that workflows were deleted. Removing ...` and `Removing workflow ''`. | Check the log for both lines. If they are missing, or the workflow still acts, restart the instance. | | A workflow is moved to the other designer. | The controller runs each workflow in the designer stored with it, so the move is a new workflow and a removed one. | Create the converted workflow as a new workflow in the target designer, remove the old one, and check the log for `Adding workflow` and `Removing workflow`. See [[user-guide-automation-controller-workflow-custom#converting-a-workflow-to-the-other-designer]]. | | A driver is added to the controller. | The running instance has no driver instance for the new driver. | Check the driver instances on the **Connect IoT** page. If the new driver has none, replace the instance. **Update** adds driver instances only when it moves the instance to another version, see [[user-guide-automation-controller-versions]]. | | A new version of the controller is created. | The instance keeps running the version it points to, and changes to the new version do not reach it. | Stop the instance and update it with **Update** on the **Connect IoT** page, which works only on an instance that is not running. See [[user-guide-automation-controller-versions]]. | Table: What a change in MES reaches When the controller log has no line for a change you applied, restart the instance: the new controller reads every workflow from MES as it starts. ## Restart, Reconnect, or Replace ### Restart an Instance Select the instance on the **Connect IoT** page and select **Restart**. For the procedure, see [[user-guide-automation-connect-iot]]. A restart stops the controller process and its driver processes, and starts new ones that know only what MES holds: - Each driver disconnects, and its state moves down to `Ready`. Each process logs `Process is about to be shutdown! Freeing any associated resources`. - The controller drains and stops its engine. - The new controller logs one `Adding workflow` line for each workflow that MES holds, and then `Controller process started with success`. - Each Setup chain runs again from `OnInitialize`, and each driver starts from `Unknown` and moves to `Connecting`. A restart has costs: - Every workflow of the controller starts again, not only the one that changed. - Values stored as `Temporary` are lost. Values stored as `Persistent` remain in the persistency folder of the manager host. See [[operation-guide-connectiotadministration#data-on-the-manager-host]]. - A file driver with `ignoreInitial` set to `false` reports every file still in its folder as new. Restart through MES, so that the controller and its drivers restart together, rather than stopping processes by hand. An ASA action that updates or moves an instance costs the same. See [[user-guide-automation-scheduled-action-action#what-the-action-costs-at-run-time]]. ### Reconnect a Driver A **Driver Command** with `Reconnect` sends `Disconnect` and then `Connect`. The full connection cycle runs again, including the `Setup` phase and the `OnSetup` event. The driver keeps the communication parameters it already holds, so a reconnect does not apply a changed configuration or security setting. Use it for a session that went quiet, for example from a workflow that reads a known property on `OnHeartbeat`. See [[user-guide-automation-task-core-driver-command]]. ### Replace an Instance Replace an instance when it must get a driver instance for a driver added to the version that it runs. **Update** does not accept the version the instance already runs. Terminate the instance, and connect the same entity to the same **Automation Manager** again with [[user-guide-automation-manager-connect]]. The new instance gets one driver instance for each driver of the controller, and a new name and ID. To terminate an instance, open the page of the **Automation Controller Instance** and select **Terminate** in the **More** options of the action bar. MES also terminates the driver instances of the controller instance. The action needs the `AutomationControllerInstance.Terminate` functionality. ## Take Things Out of Service - A workflow - remove it, and check the controller log for `Removing workflow ''`. After the next restart, the log has one `Adding workflow` line for each workflow that MES holds and none for the removed one. - An instance - stop it with **Stop** on the **Connect IoT** page, or terminate it when it is no longer needed. - A whole integration - terminate the objects in the reverse order of their creation: instances, then controllers ([[user-guide-terminate-automation-controller]]), driver definitions ([[user-guide-automation-driver-definition-terminate]]), and protocols ([[user-guide-automation-protocol-terminate]]). ## Log Lines to Look For In the default logging configuration, the processes write their logs under `/ConnectIoT/Logs`, where `` is the temporary folder of the account that runs the **Automation Manager**. For a Windows service, that is the temporary folder of the service account, which is Local System unless you chose another account when you installed the service. In a container, it is the temporary folder of the container (see [[operation-guide-connectiotinstallation-docker]]). - Each instance writes to `/ConnectIoT/Logs/Instances//`, one folder per controller or driver process. Spaces in the entity name are written as `0x20`. - The manager writes to `/ConnectIoT/Logs/AutomationManager/`, and the monitor to `/ConnectIoT/Logs/AutomationMonitor/`. A new file starts every 10 MB, and files are kept 30 days, so open the newest file. The `logging` section of the manager configuration can set other folders. See [[operation-guide-connectiotadministration#data-on-the-manager-host]]. | Log line | Written by | What it tells you | |---|---|---| | `Adding workflow: Name='', Id='', …` | Controller | The workflow was loaded, at start or when it was added. | | `New workflow '' created and running, draining the old one` | Controller | A changed workflow replaced the running one. | | `Removing workflow ''` | Controller | A removed workflow was unloaded. | | `Workflow '' will not be started because it is a subworkflow` | Controller | A Data Flow subworkflow runs only when a **Workflow** task calls it. | | `Requesting package location. Package='', Version=''` | Controller | The package version the controller loads. | | `Controller process started with success` | Controller | The controller finished starting. | | `Process is about to be shutdown! Freeing any associated resources` | Controller, driver | The process is stopping gracefully. | | `Communication state changed -> ` | Driver | Each move of the connection, such as `Setup -> Communicating`. | | `Already communicating, so no further connection attempts will be done` | Controller | A `Connect` reached a driver that is already `Communicating`. | | `Received Event Occurrence: …` | Controller | An equipment event reached the controller. When no workflow acts on it, see [[user-guide-automation-troubleshooting-symptoms]]. | | `[DEPRECATION] The topic '' is deprecated. Please use native Connect IoT interaction mechanisms.` | Driver | A workflow uses a deprecated driver extension topic. | Table: Log lines of a running instance ## Runtime Symptoms When every instance of an **Automation Manager** is down, or the manager does not come online, read the monitor log first, then the manager log. Until the monitor has read the configuration from MES, a manager with a minimal configuration logs only to the console. For a manager created by **Download**, stop the service and start `scripts\StartConsole.bat` to see those lines. See [[installation-guide-connectiotautomationmanagerconfig#where-the-configuration-lives]]. | Log line | Written by | Cause | What to do | |---|---|---|---| | `Error connecting to MES: `, then `Waiting 30s until next attempt` | Monitor | The monitor cannot reach MES or sign in. | Check the `system` section of the configuration file, the [[operation-guide-connectiotadministration#network-connections]], and the [[operation-guide-connectiotadministration#access-tokens]]. For certificate errors, see [[installation-guide-connectiotinstallation#troubleshooting]]. | | `Error identifying Manager: `, then `Waiting 60s until next attempt` | Monitor | MES does not return an **Automation Manager** for the `id` of the configuration file. | Set `id` to the **Automation Manager** ID defined in MES. | | `Monitor process failed to retrieve full config!` | Monitor | The monitor cannot read the configuration of the **Automation Manager** from MES, and stops. | Check the `id` of the configuration file, and the configuration of the **Automation Manager** in MES. | | `Looks like the token has expired and must be renewed!`, about once a minute | Monitor | The access token expired or was revoked. | Replace the token. See [[operation-guide-connectiotadministration#access-tokens]]. | | `Failed to resolve package '' with version '': ` | Monitor | The package version is not in the cache, and the monitor cannot download it from the repository. | Publish that version to the repository, or correct the `repository` settings. | | `Lost communication with process with error ''` | Monitor | A controller or driver process stopped answering. | Read the log of that instance around the same time. | | `Process '' () exceeded the maximum amount of allowed failures. Performing action ''` | Monitor | A controller or driver process did not reply to several heartbeats in a row. With the action `Restart`, the monitor kills it so that it starts again. | Read the log of that instance before the restart. For the thresholds, see [[installation-guide-connectiotautomationmanagerconfig#process-heartbeat-settings]]. | | `Version mismatch! will try to update from to `, then `Version updated failed! from to ` | Monitor | The **Monitor Version** of the **Automation Manager** changed, and the monitor could not update to it. | Publish that monitor version to the repository. A successful update logs `Version updated successfully!`, and the manager logs `Restarting Monitor process due to version update`. | | `Monitor exited unexpectedly (Id='', exitcode=). Restarting...` | Manager | The monitor process stopped. The manager starts it again only when the exit code is not `0`. | Read the monitor log before that line. When the exit code is `0`, restart the manager. | Table: Log lines of a manager that does not run its instances A change of **Monitor Version** restarts only the monitor process. The controller and driver processes belong to the manager process and keep running, and the new monitor logs `Reusing running process '' with pid=''` for each of them. ## Versions - The **Automation Protocol** sets the driver package and its version, the **Automation Controller** sets its controller and task packages, and each instance runs one version of the controller. - In 12.0, the workflow designer loads tasks only from **Automation Tasks Library** entities. Tasks of a bundled task package show the `FALLBACK` label and are not updated by the **Update Controller Versions** wizard. See [[user-guide-automation-tasks-library-index]] and the [bundled task packages migration guide](https://developer.criticalmanufacturing.com/explore/guides/migration/version-specific/1200/connect_iot_bundled_tasks_1200/). - Read the documentation of the version that the controller runs. - To put a new controller version into production and to roll it back, see [[user-guide-automation-controller-versions]]. To roll a version out to many instances, use the ASA **Update Controller Version** action. See [[user-guide-automation-scheduled-action-action]]. - To roll out an MES change that concerns many instances, see [[operation-guide-connect-iot-change-rollout]]. ## Upgrade the Runtime Upgrade the Connect IoT runtime in the order of the [Connect IoT 12.0 upgrade checklist](https://developer.criticalmanufacturing.com/explore/guides/migration/version-specific/1200/connect_iot_upgrade_checklist_1200/#during-the-upgrade): publish the new packages to the repository, change the **Monitor Version** and **Manager Version** of each **Automation Manager** (or run the new container image), then move the instances to the new controller versions, with an ASA for many instances. ## Apply a Patch A fix reaches an instance through a new package version that MES points to: 1. Publish the fixed package to the repository with a new version number. See [[installation-guide-connectiotinstallation#manually-deploy-packages]]. 2. Point MES at the new version: - A driver package comes from the **Automation Protocol**. Create a new version of the protocol with the new package version. - A task package comes from its **Automation Tasks Library**. Synchronize the library. See [[user-guide-synchronize-automation-tasks-library]]. - The controller package is selected in the **Controller Version** step of the **Update Controller Versions** wizard. 3. Create the new version of the controller with the **Update Controller Versions** wizard. See [[user-guide-automation-controller-versions#update-controller-versions-wizard]]. 4. Move the instances to the new version with **Update** on the **Connect IoT** page, or with an ASA **Update Controller Version** action for many instances. See [[user-guide-automation-controller-versions#put-a-new-version-into-production]]. ### A Package Republished Under the Same Version The monitor keeps each package version that it downloads in its cache folder, set by `cache` (`/ConnectIoT/Cache` by default), in a folder named `@`. Packages outside the `@criticalmanufacturing` scope are in a subfolder named for their scope. When a process needs a version that is already in the cache, the monitor uses that folder and does not check the repository. A folder left by an interrupted download has no `.connect_iot_package_done` file, so the monitor deletes it and downloads the package again. A package that you unpublish and publish again under the same version therefore does not reach a manager that already ran that version. On each such manager: 1. Stop the instances that use the package. 2. Delete the `@` folder from the cache folder. 3. Start the instances. The monitor log shows `Downloading package '' with version ''`.