Troubleshooting by Symptom#

Use this page when a Connect IoT integration does not behave as expected. Find the row that matches what you see, check the likely cause, and apply the fix. The tables are grouped by the kind of symptom.

Where to Look First#

A symptom shows in one of three places, and each answers a different question:

Connect IoT page - the Controller State, Driver State, and Communication State of each instance tell you whether the connection came up. See Connect IoT and the states in Communication Drivers.
Controller log - what each task did, under the task name and the execution context it ran in. In Control Flow, a failed branch is logged as Failed to execute the workflow!, followed by the cause.
Driver log - what the driver did with the equipment. A connection that fails before any workflow runs often shows only here.

In the default logging configuration, the Automation Manager writes the logs of each instance under Logs/Instances/<entity>, one folder per process, and starts a new file every 10 MB. For the container paths, see Docker Container Support.

Read the first failure in the log before you change the workflow: later lines are usually its consequences. When a workflow does nothing, look for the line each task writes when it starts, such as OnInitialize event occurred, Executing command '<name>', or Sleep activated. A missing line means the task never ran.

When an instance does not start, or every instance of an Automation Manager stops, see the runtime symptoms in Connect IoT Runbook.

Get More Detail#

The logging configuration of the Automation Manager sets how much each process writes. In the default configuration template, the Drivers file transport writes the driver logs at debug, and the Controllers file transport writes the controller logs at info. At debug, the driver log carries the communication with the equipment:

SECS/GEM - each message received, as a << line, and sent, as a >> line, and the Communication parameters: that the connection runs with, written each time the driver creates the link. Set the communication parameter monitorEnabled to true to add the trace of the underlying SECS/GEM communication, as Monitor: lines.
OPC UA - each read, write, and method call, as a >> line, and its result, as a << line.

To raise the level of a process:

Open the Automation Manager and select Configuration, as described in Configure Automation Manager.
In logging, find the transport whose applications list the process: AutomationController for the controller, and Driver* or the driver name, such as DriverSecsGem or DriverOpcUA, for a driver.
Set the level of that transport to debug, and save.
Restart the instance, as described in Connect IoT Runbook. The processes of an instance read the logging configuration when they start.

When you save, the monitor downloads the new configuration, but a running process keeps the level it started with. Restarting the instance is enough for its controller and driver processes. The manager and monitor logs change level only when the Automation Manager restarts. A manager that runs on a full local configuration file does not use the configuration saved in MES: change logging in that file instead, and restart the instance.

The applications list selects processes by type, not by instance, so the new level applies to every instance of that Automation Manager. A debug log grows much faster: each file still rotates at maxSize, and the default maxFiles keeps 30 days of files, so the log folder grows for 30 days. Set level back to its previous value when the diagnosis ends, and restart the instance again. For every transport option, see IoT Runtime Components Configuration.

The Connection Never Comes Up#

The instance runs, and its Communication State never reaches Communicating.

Symptom	Likely cause	What to check	Fix
The state never leaves its first value.	Nothing sends `Connect` to that driver.	The Setup workflow has a Driver Event on `OnInitialize`, an Equipment Configuration, and a Driver Command `Connect` for that driver.	Add the missing task or link. Bind every task of the chain to the same driver.
Driver Command fails with `Connection with driver instance was not yet established or it was not yet configured`.	`Connect` runs before an Equipment Configuration of the same driver.	The order of the chain and the driver of each task.	Give each driver its own Equipment Configuration and run it before `Connect`.
The driver log shows `Ignoring the SetupComplete action because the state is not 'Setup'`.	`SetupSuccess` is sent outside the `OnSetup` event, for example right after `Connect`.	Where the workflow sends `SetupSuccess`.	Send `SetupSuccess` only from the `OnSetup` handler, or keep Automatic Setup on the `OnInitialize` trigger.
The state reaches `Setup` and then `ConnectingFailed`, on every attempt.	Nothing answers the setup phase within the Setup Timeout.	Automatic Setup on the `OnInitialize` trigger, or the paths of the `OnSetup` handler.	Turn on Automatic Setup, or end every path of the `OnSetup` handler with `SetupSuccess` or `SetupFailure`.
A file driver logs `Unable to find path` and moves from `Connecting` to `Disconnected` again and again.	The watched folder does not exist.	The `path` the Setup workflow sends.	Create the folder, or correct the path or the data in MES that it comes from.
A SECS/GEM HSMS connection fails every few seconds while the equipment also waits.	The equipment's role was copied into `connectionMode`.	`connectionMode` is the role of the driver: `Active` when the equipment listens.	Set the role opposite to the equipment's.
A secure OPC UA server refuses the connection.	The server does not trust the driver's certificate, or the security settings do not match the server's.	The server's rejected certificates and its security policies.	Trust the certificate on the server or correct the settings, then restart the instance. A reconnect does not apply new security settings.
A driver added to the controller never starts.	The running instance has no driver instance for it.	The driver instances of the instance on the Connect IoT page.	Replace the instance, as described in Connect IoT Runbook, or add the driver in a new version and update the instance to it, see Change a Controller in Production.

Table: Symptoms of a connection that does not come up

Communicating, but No Event Arrives#

The driver is Communicating, and no occurrence reaches a workflow.

Symptom	Likely cause	What to check	Fix
A trigger never fires although the equipment reports.	The trigger is not enabled: its Auto Enable or equivalent setting is off, or a value that converts to `false` reached its Activate.	The enable setting of the task and its default, and what is linked into Activate.	Set the enable setting explicitly. On Driver Event and Subscribe In Driver, a value that converts to `true` on Activate starts the task and any other value stops it.
An MQTT broker shows no subscription from the driver.	The topics are not declared in the driver definition, for example because the workflow sends them with Send Notification to Driver.	The properties and events of the Automation Driver Definition.	Declare each topic as a property of an event and use On Equipment Event. For entries known only at run time, use Custom Driver Definition.
An MQTT property whose topic is `#` alone, or starts with `$`, never receives a value.	The driver does not match these topics.	The property's topic.	Use a topic with at least one level, such as `sensors/#`.
An OPC UA value changes and the event never fires.	No property of the event has `isTrigger` set, or the NodeId does not exist.	`isTrigger` on the event's properties. Read the property: a missing node returns an empty value whose `originalValue` carries the quality `Bad` and `BadNodeIdUnknown`.	Set `isTrigger` on the property that must fire the event, and correct the NodeId.
Every REST request is answered `404`.	The event's `deviceId` is not an endpoint path, or the request arrived before the setup completed.	The `deviceId`, such as `/api/trackIn/:id`, and the instance state.	Use the endpoint path as the `deviceId`, and send requests after the driver reaches `Communicating`.
A SECS/GEM workflow started by a state-change event never runs, and the equipment is already in that state.	The state was reached before the host enabled the event, so no event reports it.	The equipment state when the connection reaches `Communicating`.	Also do the work at connection: read the state after the setup and act on it. Keep the event handler for later changes.

Table: Symptoms of a connection that delivers no events

The Workflow Does Nothing#

The connection is up, the occurrence happened, and the workflow does not act.

Symptom	Likely cause	What to check	Fix
A Data Flow task with its values linked never runs.	Nothing reaches its Activate. Log Message, Switch, and XPath Expression Evaluator also run only from Activate.	The link into Activate.	Link the Success of the task that feeds the values into Activate, so values and activation arrive together.
A Control Flow task never runs.	It sits outside every trigger's handler, so it belongs to no branch.	Whether the block is attached to a trigger branch. A detached block is shown faded.	Move it into the handler of the trigger whose occurrences it handles.
The task after a Switch route never runs, while the default route works.	The route has no value, so it emits `undefined`, which activates nothing.	The value of each route.	Give every route a value, for example `true` of type `Boolean`.
A Data Flow workflow with a Workflow Start and a Workflow End never runs, and the log shows `Workflow '<name>' will not be started because it is a subworkflow`.	A subworkflow does not start with the engine. It runs only when a Workflow task calls it.	Which workflow calls it.	Call it from a Workflow task, or move its triggers to a workflow without start and end tasks.
A Workflow call never returns, and each input change logs `Life cycle can't proceed, check settings definitions`.	The called workflow is not on the controller, or was renamed.	The target selected in the caller.	Select the target again in the caller.
A Synchronize never emits, and the log shows `doesn't have any inputs` or `has expired. Lost inputs`.	It is activated from the trigger, before its values arrive.	What is linked into its Activate.	Activate it from the task that produces the last value, or from every producer with Emit when all defined on.
An On Equipment Event in `OneEvent` mode is followed by nothing, and the log shows `Timeout: Event not received after … ms. Deactivating listener`.	The wait timed out and its Error is not linked.	The Error output.	Link Error and decide what a missing event means.
Steps after a Try..Catch inside a function never run.	A Return inside the try ends the function.	Where the Return sits.	Place the Return after the work that must run.
Send Message Bus Message sends nothing, emits neither Success nor Error, and logs nothing.	Its `MessageType` is neither `Notification` nor `Request`.	The value that reaches `MessageType`.	Send `Notification` or `Request`.
A change applied in MES makes no difference to the running instance.	The controller did not load the change, or the change applies only at the next start.	The controller log for the line of the change.	Follow Connect IoT Runbook.

Table: Symptoms of a workflow that does nothing

It Runs on the Wrong Value#

Nothing fails, and the log, the record, or the command carries the wrong value.

Symptom	Likely cause	What to check	Fix
A value from an earlier occurrence appears where this one carried none.	A Data Flow task keeps each input until a new value replaces it.	Which inputs each occurrence feeds, and whether the task clears its inputs after each run.	Feed every input on every activation, or build the handler in Control Flow.
A value belongs to another item now and then, and the log warns `Task '<page>\<task>' input '<input>' will receive value '<new>' instead of '<old>'`.	Occurrences overlap in Data Flow, where each task is a single instance.	How close together the occurrences arrive, and any wait in the chain.	Build the handler in Control Flow, where each occurrence runs in its own branch.
Several items are handled once, with the last item's value.	Values that reach a Data Flow task in the same moment merge into one change.	Inputs fed by a loop or by a Code task that emits several values.	Handle each item in a Control Flow Foreach.
A task receives `<null>`, a default, or nothing on a port.	A link or a row uses a name the task does not read, such as the bare name where the designer adds `In` or `Out`.	The port names as the designer shows them for that task.	Use the port name the task declares in that designer.
A task receives the text of an expression, such as `{{ $onNewFile.FileName }}`.	The expression is written in a setting or a default value, where nothing evaluates it.	Where the expression is written.	Write it on the input row of the same name. In Data Flow, link the value.
A count works in one designer and resets, or keeps climbing, in the other.	The count is kept in a variable of a Code task.	The code.	Keep the count in the data store, with Store Data and Retrieve Data.
A count or total is wrong from its first run.	A `Persistent` value was stored under the same identifier by another or an earlier workflow.	The identifiers used by Store Data.	Name each identifier after its workflow.
Retrieve Data emits `Error while retrieving data: undefined` once, then its default.	The value was stored with an expiration and expired.	The expiration set where the value is stored.	Link Error, and test for the default before you use the value.
A job meant for a local hour runs at another hour.	On Cron Job evaluates its expression in UTC.	The cron expression.	Write the expression in UTC.
MES receives `{}`, or an MES `Id` that lost its last digits, from Send Message Bus Message.	Text that arrives on `Message` together with the activation is parsed as JSON.	What feeds `Message`.	Send an object, with each MES `Id` inside it as text.
MES reports a missing value for a property linked into Execute Service Call.	The input was renamed by hand to `input$<property>`, so MES receives it under that name.	The input names.	Name each input by the bare property, as the designer does.

Table: Symptoms of a workflow that runs on the wrong value

It Runs Twice or Too Often#

Symptom	Likely cause	What to check	Fix
A file is handled again after a restart.	`ignoreInitial` is `false`, and the handled file stayed in the watched folder.	Where the workflow leaves each handled file.	Move or delete each handled file, or set `ignoreInitial` to `true`.
The same command is sent again and again.	The command raises the event whose handler sends it.	The equipment documentation: which events each command raises.	Guard the command with a test of the state, or store a mark when you send it and check it before the next one.
Reads or commands fail each time the connection changes state, for example with `Request timeout error`.	The `OnCommunicationChange` handler works on the equipment in every state.	The state the handler tests.	Test for `Communicating` before any equipment work.
Every MQTT write comes back as a received message.	The written property is also readable, so the driver subscribes to its topic.	The topics of the written and the readable properties.	Write through a property whose topic no readable property subscribes to.
A watchdog raises its alarm about once per interval although items keep arriving, and the log shows `Timeout is already activated. Ignoring new request.`	Every occurrence arms the Timeout, and none disarms it.	What reaches the Timeout.	In Data Flow, disarm and then re-arm the Timeout with each occurrence, as two deliveries.
Unhandled Error reports failures that a Try..Catch handled, `Execution Return invoked`, `Execution Break invoked`, or failures of other workflows.	In Control Flow it receives every error before the catch runs, including Return and Break. With Handler Mode `AllWorkflows`, it hears every workflow of the controller.	Its Handler Mode and Context Aware settings.	Filter those messages, and set Handler Mode to `ThisWorkflow` where it must hear its own workflow only.
A removed workflow still acts.	The running controller has not removed it.	The controller log for `Removing workflow '<name>'`.	Restart the instance.
The same log line repeats without end.	The Log Message is activated by its own Success.	The link into its Activate.	Activate it from the task before it.

Table: Symptoms of a workflow that runs too often

Every Occurrence Fails#

The workflow starts, and every occurrence ends with the same line in the controller log.

Symptom	Likely cause	What to check	Fix
`Failed to parse expression '…': Value expected (char …)`	`&&`, `\\|\\|`, or `!` in an expression.	The expression.	Write `and`, `or`, and `not`. See Control Flow Expressions.
`Undefined symbol $…`	A task is read under a name it does not have, or before it ran in the branch.	The stored name of the task and its position.	Read each task by its stored name, from a task placed after it.
`Undefined function <name>`	The expression engine has no such function. Only eight converters are registered as functions: `stringify`, `stringToBase64`, `stringToEntityInstance`, `epochToDateTime`, `isEmpty`, `jsonata`, `now`, and `toTime`.	The function name.	Use a registered function, or move the step to a Code task.
`Bad escape character \T (char 4)`, or a path with a tab where a folder starts with `t`	A Windows path inside a quoted string of an expression.	The backslashes in the string.	Double each backslash, or use forward slashes.
`… is not valid JSON`	Plain text on a Control Flow row typed `Object`.	The row type.	Type the row `String` or `Any`.
`Last branch must be either 'ElseIf' or 'Else'`	The Condition branches are in an order the engine refuses.	The branch types.	Order the branches `If`, `ElseIf`, `Else`.
`Unsupported data type 'undefined'. Cannot be iterated`	A Foreach receives no list.	The list expression.	Read the list by its stored task name and output.
`EntityInstance was not yet notified about the entity it will represent.`	Entity Instance runs before the controller knows the entity of its instance.	When the task is activated.	Run it from the `OnInitialize` chain or later, and link its Error.
`Found '0' functions with the name '…' within the entire controller`, or `Found '2' functions …`	No function, or two functions, with that name in the controller.	The function names across all workflows.	Give each function a unique name and select it in the caller.
`Unable to find a context that expects this reply to be sent`	A reply task runs where no request waits for it.	The branch of the request and of the reply.	Reply in the branch of the request. See Zones.
`Execution context '…' has expired. Lost inputs: …`	A value and its activation come from two different triggers.	The emitters that feed the task.	Keep one trigger per chain. Store the value and retrieve it when the other trigger fires.
`execution timeout after '…' ms. Stopping context tree`	A Timeout stopped its branch. The line is logged at error level even when the timeout branch handles it.	Whether the timeout branch ran.	Read it as a timeout, not as a failure, when the timeout branch handles it.
`Task '…' failed to execute and there is no handler for the error`	Nothing handles the failure.	The Error output and any Try..Catch around the task.	Link Error, wrap the work in a Try..Catch, or add an Unhandled Error task.
`ENOENT` for new files	Two instances watch the same folder, or a file command parameter is empty.	The `path` of each instance and the parameters of the command.	Give each instance its own folder, and link every parameter.
`[DEPRECATION] The topic '<type>' is deprecated. Please use native Connect IoT interaction mechanisms.`	The workflow uses a legacy extension topic of the driver.	The tasks that send to the driver.	Use the Custom Driver Definition task. See the driver extensions migration guide ↗.

Table: Log lines of a failure on every occurrence

File Operations That Fail#

A file command of the File (raw) or CSV driver fails, or does its work in the wrong place. For ENOENT on new files, see Every Occurrence Fails.

Symptom	Likely cause	What to check	Fix
Every file after the first fails to move with `Cannot move "…" to "…": destination already exists`, and a file named after the target folder appears.	The `destination` folder of the first `Move` did not exist, so the file was moved to a file with the name of the folder. With `overwrite` at `false`, the default, each later move collides with that file. `Copy` does the same.	Whether the `destination` folder exists when the command runs.	Delete the file named after the folder. Run `CreateDirectory` on the folder before each `Move` or `Copy`, in the same branch. `CreateDirectory` also succeeds when the folder exists.
A file that the workflow wrote or moved is not in the expected folder.	The path is relative, such as a bare file name, so it resolves against the working directory of the driver process, not against `path`.	The paths that the command receives.	Give full paths: link `FullPath` for the file that arrived, and start a new path with `${path}` or `${archive}`. See the commands of the File (raw) driver.
After a Code task, the next task runs on empty values, for example a file command fails with `ENOENT` and an empty path.	The code reads or returns the port names, such as `inputs.contentIn`, while the task passes and maps the names you declared, such as `content`.	The names that the code uses.	In the code, use the names declared on the Inputs and Outputs tabs. See Ports You Add.
Only the first block of a CSV file arrives, or `MoreDataRequest` fails with `Unable to fetch more data in file "…"`.	The driver reads the next block only after a `MoreDataRequest` for that file. The request was not sent, its `path` is not the full path of the file, or the file was deleted, which drops its reader.	What activates `MoreDataRequest`, and what reaches its `path`.	Send `MoreDataRequest` after each block is handled, with the full path of the file as `path`. Move the file away only after its last request. See Connect IoT - CSV File Driver.

Table: Symptoms of a file operation that fails

A Call That Never Answers#

A caller waits, or MES receives an answer that the workflow did not mean, and nothing fails. For Life cycle can't proceed, check settings definitions, Found '0' functions …, and Unable to find a context that expects this reply to be sent, see the tables above.

Symptom	Likely cause	What to check	Fix
A Workflow task does not continue for 20 minutes, then fails with `Workflow '<name>' execution createdAt:'…' has now expired`.	No Workflow End of the called workflow received `Success` or `Error` on the path that the call took, so the call ends only at the Execution Expiration Time, `1200000` milliseconds by default.	Whether every path of the called workflow, error paths included, reaches a Workflow End.	Link every path to a Workflow End, and set Execution Expiration Time to the longest time that the call can take. See Workflow.
Part of the work of a sub-workflow never happens, and nothing reports it.	A Workflow End fired before that work finished. The first end to fire ends the call, and tasks still waiting do not continue.	Which task feeds the `Success` or `Error` of each Workflow End.	Feed each Workflow End from the task that finishes the work.
Call Function logs `Unable to parse the results into a valid json object`, and its results carry no value.	The `Data` of the Return is text that is not valid JSON.	The `Data` of each Return of the function.	Write `Data` as a JSON object, such as `{}`, and return each result in an input row of its own. See Control Flow - Return.
A scheduled action never runs, because every ready-state check is answered `false`.	In Control Flow, the branch of Ready State Requested ends without a Return, which answers `false`. In Data Flow, `isReady` reached Send Ready State Reply once: the task clears it after each answer, so later requests receive Default Reply.	How the run of each request produces its answer.	In Control Flow, end each path that means ready with a Return of `true`. In Data Flow, deliver `isReady` from the run of each request, for example from Retrieve Data. See Ready State Requested.

Table: Symptoms of a call that never answers

Info

For designs that cause many of these symptoms, see Workflow Anti-Patterns. For restarts, changes, and log lines of a running instance, see Connect IoT Runbook.