Skip to content
View as markdown Ask ChatGPT Ask Claude

Connect IoT Runbook#

A Connect IoT integration lives in MES, and each Automation Controller instance runs a copy of it. For each instance, the Automation Manager runs one controller process and one driver process per driver. The controller reads its workflows from MES when it starts and follows workflow changes over the message bus while it runs. Everything else, such as the driver instances, the configuration that the Setup workflow sent each driver, and what each driver set up on the equipment, is taken when the processes start.

Use this page to decide what a change needs, to restart, reconnect, or replace an instance, and to confirm each step in the logs.

Instances and Names#

  • One Automation Controller runs on several machines as one instance per entity. Each instance holds one Automation Driver Instance per driver of the controller, created with the instance.
  • An entity that is already connected to an active instance, or to a driver of another active instance, cannot be connected again. See Connect Automation Manager.
  • An instance is named <entity>_<controller>_<n>, unless a name generator is set in /Cmf/System/Configuration/Business/NameGenerators/AutomationControllerInstance/. A driver instance is named <entity>_<driver>_<driver definition>_<n>. In both names, <n> is the creation time in seconds since 1970.
  • A driver instance is connected to an entity of the entity type of its Automation Driver Definition, which the Connect wizard asks for separately from the entity of the instance.
  • Keep what differs between machines, such as a folder or an address, in MES per entity, in a Smart Table row or an entity attribute. The Setup workflow reads it at OnInitialize with Entity Instance and Resolve Table, before Equipment Configuration and Driver Command Connect.
  • Give each instance its own folder, port, or topic. Two file drivers that watch one folder race for every file, and the loser fails with ENOENT.
  • A terminated object stays in MES, and its name stays taken. Number the names of objects that you create and retire repeatedly.

What a Change Reaches#

A change applied in MES reaches a running instance at once, at the next restart, or only when the instance is replaced.

Change in MES What the running instance does What to do
A workflow is added. The controller loads it at once and logs Notified that workflows were created. Adding ... and Adding workflow: Name='<name>', …. Check the log for both lines.
A workflow is changed. Only the instances that run the changed version, which is in the Created state, apply the change. The controller replaces the workflow at once and logs Notified that workflows were changed. Updating and New workflow '<name>' created and running, draining the old one. Check the log for both lines. For the Setup workflow, see the next row.
The Setup workflow, a connection parameter, or a value that Setup reads from MES changes. A driver that is already Communicating keeps the parameters it received. Equipment Configuration emits Success without sending the new ones. Restart the instance.
The properties, events, or commands of a driver definition change. The driver keeps the definition it received during its setup. Restart the instance.
A workflow is removed. The controller logs Notified that workflows were deleted. Removing ... and Removing workflow '<name>'. Check the log for both lines. If they are missing, or the workflow still acts, restart the instance.
A workflow is moved to the other designer. The controller runs each workflow in the designer stored with it, so the move is a new workflow and a removed one. Create the converted workflow as a new workflow in the target designer, remove the old one, and check the log for Adding workflow and Removing workflow. See Create a Workflow.
A driver is added to the controller. The running instance has no driver instance for the new driver. Check the driver instances on the Connect IoT page. If the new driver has none, replace the instance. Update adds driver instances only when it moves the instance to another version, see Change a Controller in Production.
A new version of the controller is created. The instance keeps running the version it points to, and changes to the new version do not reach it. Stop the instance and update it with Update on the Connect IoT page, which works only on an instance that is not running. See Change a Controller in Production.

Table: What a change in MES reaches

When the controller log has no line for a change you applied, restart the instance: the new controller reads every workflow from MES as it starts.

Restart, Reconnect, or Replace#

Restart an Instance#

Select the instance on the Connect IoT page and select Restart. For the procedure, see Connect IoT. A restart stops the controller process and its driver processes, and starts new ones that know only what MES holds:

  • Each driver disconnects, and its state moves down to Ready. Each process logs Process is about to be shutdown! Freeing any associated resources.
  • The controller drains and stops its engine.
  • The new controller logs one Adding workflow line for each workflow that MES holds, and then Controller process started with success.
  • Each Setup chain runs again from OnInitialize, and each driver starts from Unknown and moves to Connecting.

A restart has costs:

  • Every workflow of the controller starts again, not only the one that changed.
  • Values stored as Temporary are lost. Values stored as Persistent remain in the persistency folder of the manager host. See Connect IoT Administration.
  • A file driver with ignoreInitial set to false reports every file still in its folder as new.

Restart through MES, so that the controller and its drivers restart together, rather than stopping processes by hand. An ASA action that updates or moves an instance costs the same. See Actions.

Reconnect a Driver#

A Driver Command with Reconnect sends Disconnect and then Connect. The full connection cycle runs again, including the Setup phase and the OnSetup event. The driver keeps the communication parameters it already holds, so a reconnect does not apply a changed configuration or security setting. Use it for a session that went quiet, for example from a workflow that reads a known property on OnHeartbeat. See Driver Command.

Replace an Instance#

Replace an instance when it must get a driver instance for a driver added to the version that it runs. Update does not accept the version the instance already runs. Terminate the instance, and connect the same entity to the same Automation Manager again with Connect Automation Manager. The new instance gets one driver instance for each driver of the controller, and a new name and ID.

To terminate an instance, open the page of the Automation Controller Instance and select Terminate in the More options of the action bar. MES also terminates the driver instances of the controller instance. The action needs the AutomationControllerInstance.Terminate functionality.

Take Things Out of Service#

  • A workflow - remove it, and check the controller log for Removing workflow '<name>'. After the next restart, the log has one Adding workflow line for each workflow that MES holds and none for the removed one.
  • An instance - stop it with Stop on the Connect IoT page, or terminate it when it is no longer needed.
  • A whole integration - terminate the objects in the reverse order of their creation: instances, then controllers (Terminate Automation Controller), driver definitions (Terminate Automation Driver Definition), and protocols (Terminate Automation Protocol).

Log Lines to Look For#

In the default logging configuration, the processes write their logs under <temp>/ConnectIoT/Logs, where <temp> is the temporary folder of the account that runs the Automation Manager. For a Windows service, that is the temporary folder of the service account, which is Local System unless you chose another account when you installed the service. In a container, it is the temporary folder of the container (see Docker Container Support).

  • Each instance writes to <temp>/ConnectIoT/Logs/Instances/<entity>/<component>, one folder per controller or driver process. Spaces in the entity name are written as 0x20.
  • The manager writes to <temp>/ConnectIoT/Logs/AutomationManager/<automation manager id>, and the monitor to <temp>/ConnectIoT/Logs/AutomationMonitor/<automation manager id>.

A new file starts every 10 MB, and files are kept 30 days, so open the newest file. The logging section of the manager configuration can set other folders. See Connect IoT Administration.

Log line Written by What it tells you
Adding workflow: Name='<name>', Id='<id>', … Controller The workflow was loaded, at start or when it was added.
New workflow '<name>' created and running, draining the old one Controller A changed workflow replaced the running one.
Removing workflow '<name>' Controller A removed workflow was unloaded.
Workflow '<name>' will not be started because it is a subworkflow Controller A Data Flow subworkflow runs only when a Workflow task calls it.
Requesting package location. Package='<package>', Version='<version>' Controller The package version the controller loads.
Controller process started with success Controller The controller finished starting.
Process is about to be shutdown! Freeing any associated resources Controller, driver The process is stopping gracefully.
Communication state changed <old> -> <new> Driver Each move of the connection, such as Setup -> Communicating.
Already communicating, so no further connection attempts will be done Controller A Connect reached a driver that is already Communicating.
Received Event Occurrence: … Controller An equipment event reached the controller. When no workflow acts on it, see Troubleshooting by Symptom.
[DEPRECATION] The topic '<type>' is deprecated. Please use native Connect IoT interaction mechanisms. Driver A workflow uses a deprecated driver extension topic.

Table: Log lines of a running instance

Runtime Symptoms#

When every instance of an Automation Manager is down, or the manager does not come online, read the monitor log first, then the manager log. Until the monitor has read the configuration from MES, a manager with a minimal configuration logs only to the console. For a manager created by Download, stop the service and start scripts\StartConsole.bat to see those lines. See IoT Runtime Components Configuration.

Log line Written by Cause What to do
Error connecting to MES: <error>, then Waiting 30s until next attempt Monitor The monitor cannot reach MES or sign in. Check the system section of the configuration file, the Connect IoT Administration, and the Connect IoT Administration. For certificate errors, see Connect IoT Installation.
Error identifying Manager: <error>, then Waiting 60s until next attempt Monitor MES does not return an Automation Manager for the id of the configuration file. Set id to the Automation Manager ID defined in MES.
Monitor process failed to retrieve full config! Monitor The monitor cannot read the configuration of the Automation Manager from MES, and stops. Check the id of the configuration file, and the configuration of the Automation Manager in MES.
Looks like the token has expired and must be renewed!, about once a minute Monitor The access token expired or was revoked. Replace the token. See Connect IoT Administration.
Failed to resolve package '<package>' with version '<version>': <error> Monitor The package version is not in the cache, and the monitor cannot download it from the repository. Publish that version to the repository, or correct the repository settings.
Lost communication with process with error '<message>' Monitor A controller or driver process stopped answering. Read the log of that instance around the same time.
Process '<id>' (<name>) exceeded the maximum amount of allowed failures. Performing action '<action>' Monitor A controller or driver process did not reply to several heartbeats in a row. With the action Restart, the monitor kills it so that it starts again. Read the log of that instance before the restart. For the thresholds, see IoT Runtime Components Configuration.
Version mismatch! will try to update from <old> to <new>, then Version updated failed! from <old> to <new> Monitor The Monitor Version of the Automation Manager changed, and the monitor could not update to it. Publish that monitor version to the repository. A successful update logs Version updated successfully!, and the manager logs Restarting Monitor process due to version update.
Monitor exited unexpectedly (Id='<id>', exitcode=<code>). Restarting... Manager The monitor process stopped. The manager starts it again only when the exit code is not 0. Read the monitor log before that line. When the exit code is 0, restart the manager.

Table: Log lines of a manager that does not run its instances

A change of Monitor Version restarts only the monitor process. The controller and driver processes belong to the manager process and keep running, and the new monitor logs Reusing running process '<id>' with pid='<pid>' for each of them.

Versions#

  • The Automation Protocol sets the driver package and its version, the Automation Controller sets its controller and task packages, and each instance runs one version of the controller.
  • In 12.0, the workflow designer loads tasks only from Automation Tasks Library entities. Tasks of a bundled task package show the FALLBACK label and are not updated by the Update Controller Versions wizard. See Automation Tasks Library and the bundled task packages migration guide ↗.
  • Read the documentation of the version that the controller runs.
  • To put a new controller version into production and to roll it back, see Change a Controller in Production. To roll a version out to many instances, use the ASA Update Controller Version action. See Actions.
  • To roll out an MES change that concerns many instances, see Rolling Out an MES Change to Many Controllers.

Upgrade the Runtime#

Upgrade the Connect IoT runtime in the order of the Connect IoT 12.0 upgrade checklist ↗: publish the new packages to the repository, change the Monitor Version and Manager Version of each Automation Manager (or run the new container image), then move the instances to the new controller versions, with an ASA for many instances.

Apply a Patch#

A fix reaches an instance through a new package version that MES points to:

  1. Publish the fixed package to the repository with a new version number. See Connect IoT Installation.
  2. Point MES at the new version:
    • A driver package comes from the Automation Protocol. Create a new version of the protocol with the new package version.
    • A task package comes from its Automation Tasks Library. Synchronize the library. See Synchronize Automation Tasks Library.
    • The controller package is selected in the Controller Version step of the Update Controller Versions wizard.
  3. Create the new version of the controller with the Update Controller Versions wizard. See Change a Controller in Production.
  4. Move the instances to the new version with Update on the Connect IoT page, or with an ASA Update Controller Version action for many instances. See Change a Controller in Production.

A Package Republished Under the Same Version#

The monitor keeps each package version that it downloads in its cache folder, set by cache (<temp>/ConnectIoT/Cache by default), in a folder named <package name>@<version>. Packages outside the @criticalmanufacturing scope are in a subfolder named for their scope. When a process needs a version that is already in the cache, the monitor uses that folder and does not check the repository. A folder left by an interrupted download has no .connect_iot_package_done file, so the monitor deletes it and downloads the package again.

A package that you unpublish and publish again under the same version therefore does not reach a manager that already ran that version. On each such manager:

  1. Stop the instances that use the package.
  2. Delete the <package name>@<version> folder from the cache folder.
  3. Start the instances. The monitor log shows Downloading package '<package>' with version '<version>'.