---
alias: user-guide-automation-troubleshooting-symptoms
render_macros: false
tags:
- automation
- connect iot
description: "Find the cause of a Connect IoT problem from what you see in the instance state, the controller and driver logs, and the workflow result"
---

# Troubleshooting by Symptom

Use this page when a Connect IoT integration does not behave as expected. Find the row that matches what you see, check the likely cause, and apply the fix. The tables are grouped by the kind of symptom.

## Where to Look First

A symptom shows in one of three places, and each answers a different question:

- **Connect IoT** page - the *Controller State*, *Driver State*, and *Communication State* of each instance tell you whether the connection came up. See [[user-guide-automation-connect-iot]] and the states in [[user-guide-automation-protocol-protocols-index#communication-states]].
- Controller log - what each task did, under the task name and the execution context it ran in. In Control Flow, a failed branch is logged as `Failed to execute the workflow!`, followed by the cause.
- Driver log - what the driver did with the equipment. A connection that fails before any workflow runs often shows only here.

In the default logging configuration, the **Automation Manager** writes the logs of each instance under `Logs/Instances/<entity>`, one folder per process, and starts a new file every 10 MB. For the container paths, see [[operation-guide-connectiotinstallation-docker#configuration-file-included]].

Read the first failure in the log before you change the workflow: later lines are usually its consequences. When a workflow does nothing, look for the line each task writes when it starts, such as `OnInitialize event occurred`, `Executing command '<name>'`, or `Sleep activated`. A missing line means the task never ran.

When an instance does not start, or every instance of an **Automation Manager** stops, see the runtime symptoms in [[operation-guide-connect-iot-runbook#runtime-symptoms]].

### Get More Detail

The logging configuration of the **Automation Manager** sets how much each process writes. In the default configuration template, the `Drivers` file transport writes the driver logs at `debug`, and the `Controllers` file transport writes the controller logs at `info`. At `debug`, the driver log carries the communication with the equipment:

- SECS/GEM - each message received, as a `<<` line, and sent, as a `>>` line, and the `Communication parameters:` that the connection runs with, written each time the driver creates the link. Set the communication parameter `monitorEnabled` to `true` to add the trace of the underlying SECS/GEM communication, as `Monitor:` lines.
- OPC UA - each read, write, and method call, as a `>>` line, and its result, as a `<<` line.

To raise the level of a process:

1. Open the **Automation Manager** and select **Configuration**, as described in [[user-guide-automation-manager-configure]].
2. In `logging`, find the transport whose `applications` list the process: `AutomationController` for the controller, and `Driver*` or the driver name, such as `DriverSecsGem` or `DriverOpcUA`, for a driver.
3. Set the `level` of that transport to `debug`, and save.
4. Restart the instance, as described in [[operation-guide-connect-iot-runbook#restart-an-instance]]. The processes of an instance read the logging configuration when they start.

When you save, the monitor downloads the new configuration, but a running process keeps the level it started with. Restarting the instance is enough for its controller and driver processes. The manager and monitor logs change level only when the **Automation Manager** restarts. A manager that runs on a full local configuration file does not use the configuration saved in MES: change `logging` in that file instead, and restart the instance.

The `applications` list selects processes by type, not by instance, so the new level applies to every instance of that **Automation Manager**. A `debug` log grows much faster: each file still rotates at `maxSize`, and the default `maxFiles` keeps 30 days of files, so the log folder grows for 30 days. Set `level` back to its previous value when the diagnosis ends, and restart the instance again. For every transport option, see [[installation-guide-connectiotautomationmanagerconfig#logging-structure]].

## The Connection Never Comes Up

The instance runs, and its *Communication State* never reaches `Communicating`.

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| The state never leaves its first value. | Nothing sends `Connect` to that driver. | The Setup workflow has a **Driver Event** on `OnInitialize`, an **Equipment Configuration**, and a **Driver Command** `Connect` for that driver. | Add the missing task or link. Bind every task of the chain to the same driver. |
| **Driver Command** fails with `Connection with driver instance was not yet established or it was not yet configured`. | `Connect` runs before an **Equipment Configuration** of the same driver. | The order of the chain and the driver of each task. | Give each driver its own **Equipment Configuration** and run it before `Connect`. |
| The driver log shows `Ignoring the SetupComplete action because the state is not 'Setup'`. | `SetupSuccess` is sent outside the `OnSetup` event, for example right after `Connect`. | Where the workflow sends `SetupSuccess`. | Send `SetupSuccess` only from the `OnSetup` handler, or keep **Automatic Setup** on the `OnInitialize` trigger. |
| The state reaches `Setup` and then `ConnectingFailed`, on every attempt. | Nothing answers the setup phase within the **Setup Timeout**. | **Automatic Setup** on the `OnInitialize` trigger, or the paths of the `OnSetup` handler. | Turn on **Automatic Setup**, or end every path of the `OnSetup` handler with `SetupSuccess` or `SetupFailure`. |
| A file driver logs `Unable to find path` and moves from `Connecting` to `Disconnected` again and again. | The watched folder does not exist. | The `path` the Setup workflow sends. | Create the folder, or correct the path or the data in MES that it comes from. |
| A SECS/GEM HSMS connection fails every few seconds while the equipment also waits. | The equipment's role was copied into `connectionMode`. | `connectionMode` is the role of the driver: `Active` when the equipment listens. | Set the role opposite to the equipment's. |
| A secure OPC UA server refuses the connection. | The server does not trust the driver's certificate, or the security settings do not match the server's. | The server's rejected certificates and its security policies. | Trust the certificate on the server or correct the settings, then restart the instance. A reconnect does not apply new security settings. |
| A driver added to the controller never starts. | The running instance has no driver instance for it. | The driver instances of the instance on the **Connect IoT** page. | Replace the instance, as described in [[operation-guide-connect-iot-runbook]], or add the driver in a new version and update the instance to it, see [[user-guide-automation-controller-versions]]. |

Table: Symptoms of a connection that does not come up

## Communicating, but No Event Arrives

The driver is `Communicating`, and no occurrence reaches a workflow.

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| A trigger never fires although the equipment reports. | The trigger is not enabled: its **Auto Enable** or equivalent setting is off, or a value that converts to `false` reached its **Activate**. | The enable setting of the task and its default, and what is linked into **Activate**. | Set the enable setting explicitly. On **Driver Event** and **Subscribe In Driver**, a value that converts to `true` on **Activate** starts the task and any other value stops it. |
| An MQTT broker shows no subscription from the driver. | The topics are not declared in the driver definition, for example because the workflow sends them with **Send Notification to Driver**. | The properties and events of the **Automation Driver Definition**. | Declare each topic as a property of an event and use **On Equipment Event**. For entries known only at run time, use **Custom Driver Definition**. |
| An MQTT property whose topic is `#` alone, or starts with `$`, never receives a value. | The driver does not match these topics. | The property's topic. | Use a topic with at least one level, such as `sensors/#`. |
| An OPC UA value changes and the event never fires. | No property of the event has `isTrigger` set, or the NodeId does not exist. | `isTrigger` on the event's properties. Read the property: a missing node returns an empty value whose `originalValue` carries the quality `Bad` and `BadNodeIdUnknown`. | Set `isTrigger` on the property that must fire the event, and correct the NodeId. |
| Every REST request is answered `404`. | The event's `deviceId` is not an endpoint path, or the request arrived before the setup completed. | The `deviceId`, such as `/api/trackIn/:id`, and the instance state. | Use the endpoint path as the `deviceId`, and send requests after the driver reaches `Communicating`. |
| A SECS/GEM workflow started by a state-change event never runs, and the equipment is already in that state. | The state was reached before the host enabled the event, so no event reports it. | The equipment state when the connection reaches `Communicating`. | Also do the work at connection: read the state after the setup and act on it. Keep the event handler for later changes. |

Table: Symptoms of a connection that delivers no events

## The Workflow Does Nothing

The connection is up, the occurrence happened, and the workflow does not act.

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| A Data Flow task with its values linked never runs. | Nothing reaches its **Activate**. **Log Message**, **Switch**, and **XPath Expression Evaluator** also run only from **Activate**. | The link into **Activate**. | Link the **Success** of the task that feeds the values into **Activate**, so values and activation arrive together. |
| A Control Flow task never runs. | It sits outside every trigger's handler, so it belongs to no branch. | Whether the block is attached to a trigger branch. A detached block is shown faded. | Move it into the handler of the trigger whose occurrences it handles. |
| The task after a **Switch** route never runs, while the default route works. | The route has no value, so it emits `undefined`, which activates nothing. | The value of each route. | Give every route a value, for example `true` of type `Boolean`. |
| A Data Flow workflow with a **Workflow Start** and a **Workflow End** never runs, and the log shows `Workflow '<name>' will not be started because it is a subworkflow`. | A subworkflow does not start with the engine. It runs only when a **Workflow** task calls it. | Which workflow calls it. | Call it from a **Workflow** task, or move its triggers to a workflow without start and end tasks. |
| A **Workflow** call never returns, and each input change logs `Life cycle can't proceed, check settings definitions`. | The called workflow is not on the controller, or was renamed. | The target selected in the caller. | Select the target again in the caller. |
| A **Synchronize** never emits, and the log shows `doesn't have any inputs` or `has expired. Lost inputs`. | It is activated from the trigger, before its values arrive. | What is linked into its **Activate**. | Activate it from the task that produces the last value, or from every producer with **Emit when all defined** on. |
| An **On Equipment Event** in `OneEvent` mode is followed by nothing, and the log shows `Timeout: Event not received after … ms. Deactivating listener`. | The wait timed out and its **Error** is not linked. | The **Error** output. | Link **Error** and decide what a missing event means. |
| Steps after a **Try..Catch** inside a function never run. | A **Return** inside the try ends the function. | Where the **Return** sits. | Place the **Return** after the work that must run. |
| **Send Message Bus Message** sends nothing, emits neither **Success** nor **Error**, and logs nothing. | Its `MessageType` is neither `Notification` nor `Request`. | The value that reaches `MessageType`. | Send `Notification` or `Request`. |
| A change applied in MES makes no difference to the running instance. | The controller did not load the change, or the change applies only at the next start. | The controller log for the line of the change. | Follow [[operation-guide-connect-iot-runbook]]. |

Table: Symptoms of a workflow that does nothing

## It Runs on the Wrong Value

Nothing fails, and the log, the record, or the command carries the wrong value.

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| A value from an earlier occurrence appears where this one carried none. | A Data Flow task keeps each input until a new value replaces it. | Which inputs each occurrence feeds, and whether the task clears its inputs after each run. | Feed every input on every activation, or build the handler in Control Flow. |
| A value belongs to another item now and then, and the log warns `Task '<page>\<task>' input '<input>' will receive value '<new>' instead of '<old>'`. | Occurrences overlap in Data Flow, where each task is a single instance. | How close together the occurrences arrive, and any wait in the chain. | Build the handler in Control Flow, where each occurrence runs in its own branch. |
| Several items are handled once, with the last item's value. | Values that reach a Data Flow task in the same moment merge into one change. | Inputs fed by a loop or by a **Code** task that emits several values. | Handle each item in a Control Flow **Foreach**. |
| A task receives `<null>`, a default, or nothing on a port. | A link or a row uses a name the task does not read, such as the bare name where the designer adds `In` or `Out`. | The port names as the designer shows them for that task. | Use the port name the task declares in that designer. |
| A task receives the text of an expression, such as `{{ $onNewFile.FileName }}`. | The expression is written in a setting or a default value, where nothing evaluates it. | Where the expression is written. | Write it on the input row of the same name. In Data Flow, link the value. |
| A count works in one designer and resets, or keeps climbing, in the other. | The count is kept in a variable of a **Code** task. | The code. | Keep the count in the data store, with **Store Data** and **Retrieve Data**. |
| A count or total is wrong from its first run. | A `Persistent` value was stored under the same identifier by another or an earlier workflow. | The identifiers used by **Store Data**. | Name each identifier after its workflow. |
| **Retrieve Data** emits `Error while retrieving data: undefined` once, then its default. | The value was stored with an expiration and expired. | The expiration set where the value is stored. | Link **Error**, and test for the default before you use the value. |
| A job meant for a local hour runs at another hour. | **On Cron Job** evaluates its expression in UTC. | The cron expression. | Write the expression in UTC. |
| MES receives `{}`, or an MES `Id` that lost its last digits, from **Send Message Bus Message**. | Text that arrives on `Message` together with the activation is parsed as JSON. | What feeds `Message`. | Send an object, with each MES `Id` inside it as text. |
| MES reports a missing value for a property linked into **Execute Service Call**. | The input was renamed by hand to `input$<property>`, so MES receives it under that name. | The input names. | Name each input by the bare property, as the designer does. |

Table: Symptoms of a workflow that runs on the wrong value

## It Runs Twice or Too Often

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| A file is handled again after a restart. | `ignoreInitial` is `false`, and the handled file stayed in the watched folder. | Where the workflow leaves each handled file. | Move or delete each handled file, or set `ignoreInitial` to `true`. |
| The same command is sent again and again. | The command raises the event whose handler sends it. | The equipment documentation: which events each command raises. | Guard the command with a test of the state, or store a mark when you send it and check it before the next one. |
| Reads or commands fail each time the connection changes state, for example with `Request timeout error`. | The `OnCommunicationChange` handler works on the equipment in every state. | The state the handler tests. | Test for `Communicating` before any equipment work. |
| Every MQTT write comes back as a received message. | The written property is also readable, so the driver subscribes to its topic. | The topics of the written and the readable properties. | Write through a property whose topic no readable property subscribes to. |
| A watchdog raises its alarm about once per interval although items keep arriving, and the log shows `Timeout is already activated. Ignoring new request.` | Every occurrence arms the **Timeout**, and none disarms it. | What reaches the **Timeout**. | In Data Flow, disarm and then re-arm the **Timeout** with each occurrence, as two deliveries. |
| **Unhandled Error** reports failures that a **Try..Catch** handled, `Execution Return invoked`, `Execution Break invoked`, or failures of other workflows. | In Control Flow it receives every error before the catch runs, including **Return** and **Break**. With **Handler Mode** `AllWorkflows`, it hears every workflow of the controller. | Its **Handler Mode** and **Context Aware** settings. | Filter those messages, and set **Handler Mode** to `ThisWorkflow` where it must hear its own workflow only. |
| A removed workflow still acts. | The running controller has not removed it. | The controller log for `Removing workflow '<name>'`. | Restart the instance. |
| The same log line repeats without end. | The **Log Message** is activated by its own **Success**. | The link into its **Activate**. | Activate it from the task before it. |

Table: Symptoms of a workflow that runs too often

## Every Occurrence Fails

The workflow starts, and every occurrence ends with the same line in the controller log.

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| `Failed to parse expression '…': Value expected (char …)` | `&&`, `\|\|`, or `!` in an expression. | The expression. | Write `and`, `or`, and `not`. See [[user-guide-automation-controller-workflow-controlflow-expressions]]. |
| `Undefined symbol $…` | A task is read under a name it does not have, or before it ran in the branch. | The stored name of the task and its position. | Read each task by its stored name, from a task placed after it. |
| `Undefined function <name>` | The expression engine has no such function. Only eight converters are registered as functions: `stringify`, `stringToBase64`, `stringToEntityInstance`, `epochToDateTime`, `isEmpty`, `jsonata`, `now`, and `toTime`. | The function name. | Use a registered function, or move the step to a **Code** task. |
| `Bad escape character \T (char 4)`, or a path with a tab where a folder starts with `t` | A Windows path inside a quoted string of an expression. | The backslashes in the string. | Double each backslash, or use forward slashes. |
| `… is not valid JSON` | Plain text on a Control Flow row typed `Object`. | The row type. | Type the row `String` or `Any`. |
| `Last branch must be either 'ElseIf' or 'Else'` | The **Condition** branches are in an order the engine refuses. | The branch types. | Order the branches `If`, `ElseIf`, `Else`. |
| `Unsupported data type 'undefined'. Cannot be iterated` | A **Foreach** receives no list. | The list expression. | Read the list by its stored task name and output. |
| `EntityInstance was not yet notified about the entity it will represent.` | **Entity Instance** runs before the controller knows the entity of its instance. | When the task is activated. | Run it from the `OnInitialize` chain or later, and link its **Error**. |
| `Found '0' functions with the name '…' within the entire controller`, or `Found '2' functions …` | No function, or two functions, with that name in the controller. | The function names across all workflows. | Give each function a unique name and select it in the caller. |
| `Unable to find a context that expects this reply to be sent` | A reply task runs where no request waits for it. | The branch of the request and of the reply. | Reply in the branch of the request. See [[user-guide-automation-zones]]. |
| `Execution context '…' has expired. Lost inputs: …` | A value and its activation come from two different triggers. | The emitters that feed the task. | Keep one trigger per chain. Store the value and retrieve it when the other trigger fires. |
| `execution timeout after '…' ms. Stopping context tree` | A **Timeout** stopped its branch. The line is logged at error level even when the timeout branch handles it. | Whether the timeout branch ran. | Read it as a timeout, not as a failure, when the timeout branch handles it. |
| `Task '…' failed to execute and there is no handler for the error` | Nothing handles the failure. | The **Error** output and any **Try..Catch** around the task. | Link **Error**, wrap the work in a **Try..Catch**, or add an **Unhandled Error** task. |
| `ENOENT` for new files | Two instances watch the same folder, or a file command parameter is empty. | The `path` of each instance and the parameters of the command. | Give each instance its own folder, and link every parameter. |
| `[DEPRECATION] The topic '<type>' is deprecated. Please use native Connect IoT interaction mechanisms.` | The workflow uses a legacy extension topic of the driver. | The tasks that send to the driver. | Use the **Custom Driver Definition** task. See the [driver extensions migration guide](https://developer.criticalmanufacturing.com/explore/guides/migration/version-specific/1200/connect_iot_driver_extensions_1200/). |

Table: Log lines of a failure on every occurrence

## File Operations That Fail

A file command of the File (raw) or CSV driver fails, or does its work in the wrong place. For `ENOENT` on new files, see [Every Occurrence Fails](#every-occurrence-fails).

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| Every file after the first fails to move with `Cannot move "…" to "…": destination already exists`, and a file named after the target folder appears. | The `destination` folder of the first `Move` did not exist, so the file was moved to a file with the name of the folder. With `overwrite` at `false`, the default, each later move collides with that file. `Copy` does the same. | Whether the `destination` folder exists when the command runs. | Delete the file named after the folder. Run `CreateDirectory` on the folder before each `Move` or `Copy`, in the same branch. `CreateDirectory` also succeeds when the folder exists. |
| A file that the workflow wrote or moved is not in the expected folder. | The path is relative, such as a bare file name, so it resolves against the working directory of the driver process, not against `path`. | The paths that the command receives. | Give full paths: link `FullPath` for the file that arrived, and start a new path with `${path}` or `${archive}`. See the commands of the [File (raw) driver](../definition/automation-protocol/automation-protocol-protocols/driver_fileraw.md#commands). |
| After a **Code** task, the next task runs on empty values, for example a file command fails with `ENOENT` and an empty path. | The code reads or returns the port names, such as `inputs.contentIn`, while the task passes and maps the names you declared, such as `content`. | The names that the code uses. | In the code, use the names declared on the **Inputs** and **Outputs** tabs. See [Ports You Add](../reference/tasks/index.md#ports-you-add). |
| Only the first block of a CSV file arrives, or `MoreDataRequest` fails with `Unable to fetch more data in file "…"`. | The driver reads the next block only after a `MoreDataRequest` for that file. The request was not sent, its `path` is not the full path of the file, or the file was deleted, which drops its reader. | What activates `MoreDataRequest`, and what reaches its `path`. | Send `MoreDataRequest` after each block is handled, with the full path of the file as `path`. Move the file away only after its last request. See [[user-guide-driver-csv]]. |

Table: Symptoms of a file operation that fails

## A Call That Never Answers

A caller waits, or MES receives an answer that the workflow did not mean, and nothing fails. For `Life cycle can't proceed, check settings definitions`, `Found '0' functions …`, and `Unable to find a context that expects this reply to be sent`, see the tables above.

| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| A **Workflow** task does not continue for 20 minutes, then fails with `Workflow '<name>' execution createdAt:'…' has now expired`. | No **Workflow End** of the called workflow received `Success` or `Error` on the path that the call took, so the call ends only at the **Execution Expiration Time**, `1200000` milliseconds by default. | Whether every path of the called workflow, error paths included, reaches a **Workflow End**. | Link every path to a **Workflow End**, and set **Execution Expiration Time** to the longest time that the call can take. See [[user-guide-automation-task-core-workflow]]. |
| Part of the work of a sub-workflow never happens, and nothing reports it. | A **Workflow End** fired before that work finished. The first end to fire ends the call, and tasks still waiting do not continue. | Which task feeds the `Success` or `Error` of each **Workflow End**. | Feed each **Workflow End** from the task that finishes the work. |
| **Call Function** logs `Unable to parse the results into a valid json object`, and its results carry no value. | The `Data` of the **Return** is text that is not valid JSON. | The `Data` of each **Return** of the function. | Write `Data` as a JSON object, such as `{}`, and return each result in an input row of its own. See [[user-guide-automation-task-core-return]]. |
| A scheduled action never runs, because every ready-state check is answered `false`. | In Control Flow, the branch of **Ready State Requested** ends without a **Return**, which answers `false`. In Data Flow, `isReady` reached **Send Ready State Reply** once: the task clears it after each answer, so later requests receive **Default Reply**. | How the run of each request produces its answer. | In Control Flow, end each path that means ready with a **Return** of `true`. In Data Flow, deliver `isReady` from the run of each request, for example from **Retrieve Data**. See [[user-guide-automation-task-core-ready-state-request]]. |

Table: Symptoms of a call that never answers

!!! info
    For designs that cause many of these symptoms, see [[user-guide-automation-workflow-antipatterns]]. For restarts, changes, and log lines of a running instance, see [[operation-guide-connect-iot-runbook]].