Connect IoT - Raw File Driver#

This driver function is to allow monitor the lifecycle of any kind of file type in a directory. The directory can be local or remote (network). The driver also provides basic operating system operations that can be executed.

Data Types#

These are the supported specific data types:

Name	Description
String	String Value
Boolean	Boolean Value
Integer	Integer value
Decimal	Decimal value
Object	Structured value

Protocol Parameters#

The protocol supports the following parameters, used define the communication parameters, as well as the expected behavior of the driver:

Name	Type	Possible Values	Default Value	Description
path	String		""	Path of the location to monitor for changes on the files. If no value is provided, no watcher is created, so no events will be triggered.
fileMask	String		"*/"	Glob pattern to use for the watcher to identify the files to handle. Use a tool like https://globster.xyz/ ⧉ to try a valid value to use.
archivePath	String		""	Directory planned to use for archiving the files after processing. This value can later be used as a variable in the available operations to execute.
watcherType	Enum	Chokidar NSFW	Chokidar	Type of watcher to use. Depending on the selection, different settings are necessary. Chokidar is the best overall option, but CPU and Memory heavy and slower to start when many files are present. NSFW is better other scenarios. Test both to determine the best option.
ignoreInitial	Boolean		false	Flag to define if, during startup/restart, the files that were created, changed, deleted should be processed or ignored.
alwaysStat	Boolean		true	Always get additional attributes (size, timestamps, etc) of the file that was identified by the watcher. Will require additional operating system resources.
depth	Integer		0	Number of sub directories to watch. The higher the number, more memory/cpu/time will be required for watchers.
awaitWriteFinish	Boolean		true	Trigger watcher events only when the file finishes writing.
awaitWriteFinishStabilityThreshold	Integer		2000	Amount of time in milliseconds for a file size to remain constant before emitting its watcher event.

Note

Current NSFW is not supported for Linux

Protocol Parameters specific for Chokidar watcherType#

Chokidar watcher type has some settings that are only known to it. If using the other type(s), these settings will not have any effect.

Name	Type	Possible Values	Default Value	Description
watcherMode	Enum	FileSystemEvents Polling	Polling	To successfully watch files over a network (and in other non-standard situations), it is typically necessary to use Polling, however it could lead to high CPU utilization. FileSystemEvents is the most efficient method for monitoring local files.
pollingBinaryInterval	Integer		300	Interval of file system polling for binary files (in milliseconds)
awaitWriteFinishPollInterval	Integer		100	File size polling interval
atomic	Integer		100	Automatically filters out artifacts that occur within the defined value (in milliseconds) when using editors that use 'atomic writes' instead of writing directly to the source file. If defined to 0, this function is disabled.

Extended Parameters#

As with other protocol drivers, some extra parameters are necessary to provide more context to the behavior expected.

Property#

Name	Type	Possible Values	Default Value	Description
propertyType	Enum	FileName RelativePath FullPath Size IsFile IsDirectory AccessTime CreationTime UpdateTime	FileName	Provides a meaning of the property, and will receive the value depending of it when specific events occur. `FileName`Name of the file being handled `RelativePath` Relative path of the file being handled `FullPath` Full path of the file being handled `Size` Size of the file `IsFile` Flag indicating if the handle is a file `IsDirectory` Flag indicating if the handle is a directory `AccessTime` Date and Time when the file was last accessed `CreationTime` Date and Time when the file was created `UpdateTime` Date and Time when the file was last updated

Event#

Name	Type	Possible Values	Default Value	Description
EventTrigger	Enum	NewFile DeletedFile ChangedFile	NewFile	Indicates the Operating System action that will trigger the event. `NewFile` When a new file is detected in the watch directory `DeletedFile` When a file present in the watched directory is deleted `ChangedFile` When a file that was already present in the the watched directory is changed. This event occurs can only occur after the `NewFile`.

Command#

Name	Type	Possible Values	Default Value	Description
commandType	Enum	Exists Copy Rename Delete Move CreateFile ReadFile CreateDirectory EmptyDirectory Mount	Exists	Indicated the type of command that will be executed. `Exists` Checks if a file or directory exists `Copy` Copies a file or entire directory from one location into another `Rename` Renames a file or directory `Delete` Deletes a file or directory `Move` Moves a file or directory `CreateFile` Create or append data into a file `ReadFile` Reads the entire content of an existing file `CreateDirectory` Creates a new directory `EmptyDirectory` Deletes all files and subdirectories of a directory `Mount` Mounts a network drive location into a local path `ListFiles` List all files in a directory

Internal Driver Interface#

This protocol driver provides a set of interfaces ready to be used to ease the usability within the Controller Workflows. This section is intended to detail them. Although most if not all of these interfaces have specifically designed tasks to use them, the user is free to redefine them in the Driver Definition entity.

Properties#

Name	PropertyType	Data Type	Description
FileName	`FileName`	String	Name of the file identified in the watched directory. Will include the extension, but not the directory path.
RelativePath	`RelativePath`	String	File name and relative path to the watched directory.
FullPath	`FullPath`	String	File name and the full path where it is located in the local system.
Size	`Size`	Integer	Size of the file
IsFile	`IsFile`	Boolean	Flag indicating if the detected entry is a file
IsDirectory	`IsDirectory`	Boolean	Flag indicating if the detected entry is a directory
AccessTime	`AccessTime`	DateTime	Date and Time when the file was last accessed
CreationTime	`CreationTime`	DateTime	Date and Time when the file was created
UpdateTime	`UpdateTime`	DateTime	Date and Time when the file was last updated/changed

Events#

Name	EventTrigger	Properties updated	Description
OnNewFile	`NewFile`	`FileName` `RelativePath` `FullPath` * `Size` * `IsFile` * `IsDirectory` * `AccessTime` * `CreationTime` * `UpdateTime` * These properties may only be updated if the `alwaysStat` setting is set to `true`	When the watcher detects a file that was not previously identified appears, will raise this event. The timing when it will be triggered may depend on some settings, like `watcherMode`, `awaitWriteFinish`, `awaitWriteFinishStabilityThreshold` and others.
OnFileChanged	`ChangedFile`	`FileName` `RelativePath` `FullPath` * `Size` * `IsFile` * `IsDirectory` * `AccessTime` * `CreationTime` * `UpdateTime` * These properties may only be updated if the `alwaysStat` setting is set to `true`	When the watcher detects a file that was already identified (and triggered the `OnNewFile`event), and it was changed in some way, will raise this event. The timing when it will be triggered may depend on some settings, like `watcherMode`, `awaitWriteFinish`, `awaitWriteFinishStabilityThreshold` and others.
OnFileDeleted	`DeletedFile`	`FileName` `RelativePath` `FullPath`	When the watcher detects a file that was previously identified and was now deleted (or moved somewhere), will raise this event.

Commands#

All paths of the commands listed in this section can use the following tokens to simplify usability:

Name	Description	Example
`${path}`	Replace the token with the `path` setting defined during setup	`${path}\file.csv` becomes `c:\watchedDirectory\file.csv`
`${archive}`	Replace the token with the `archive` setting defined during setup	`${archive}\newFile.csv` becomes `c:\archive\file.csv`
`${temp}`	Replaces the token with the user temporary location (depends on the operating system)	`${temp}\file.csv` becomes `c:\Users\JohnDoe\AppData\Local\Temp\file.csv`
`${id}`	Replaces the token to the Id of the Driver (with "_" instead of "/")	`${temp}\${id}\file.csv` becomes `c:\Users\JohnDoe\AppData\Local\Temp\DriverInstance_1234567890\file.csv`

Name	CommandType	Parameters / Type / Default Value	Description
Exists	`Exists`	`path` / String / "" / `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Checks the `path` location exists. If it doesn't, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
Copy	`Copy`	`source` / String / "" `destination` / String / "" `overwrite` / Boolean / false `preserveTimestamps` / Boolean / false `recursive` / Boolean / true `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to copy the `source` location (file or directory) into the `destination` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time. If it is a directory and `recursive` is set to true, all subdirectories will also be copied.
Rename	`Rename`	`source` / String / "" `destination` / String / "" `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to rename the `source` location (file or directory) into the `destination` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
Move	`Move`	`source` / String / "" `destination` / String / "" `overwrite` / Boolean / false `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to move the `source` location (file or directory) into the `destination` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
Delete	`Delete`	`path` / String / "" / `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Deleted the `path` location (file or directory). If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
CreateFile	`CreateFile`	`path` / String / "" `content` / String / "" `encoding` / String / "utf8" `mode` / Integer / 0o666 `flag` / String / "w" `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to write into a file located in the `path` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time. The `mode` will define the file attributes / permissions using unix format, but in decimal. The `flag` determines how the file will be accessed (write, rewrite, append, etc). The supported flags are described in https://nodejs.org/api/fs.html#fs_file_system_flags. ⧉
ReadFile	`ReadFile`	`path` / String / "" `encoding` / String / "utf8" `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to read the contents of the file in the `path` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
CreateDirectory	`CreateDirectory`	`path` / String / "" `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to create a new directory in the `path` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
EmptyDirectory	`EmptyDirectory`	`path` / String / "" `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to delete all files and subdirectories of `path` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.
Mount	`Mount`	`path` / String / "" `username` / String / "" `password` / String / "" `localPath` / String / "" `failIfLocalPathExists` / Boolean / false	Tries to mount the network path provided in the `path`parameter, using (optionally) the `username` and `password`. It will map it into the `localPath` location using a symbolic Link. Requires administrative rights. Check the Remarks section for more information.
ListFiles	`ListFiles`	`path` / String / "" `attempts` / Integer / 1 `sleepBetweenAttempts` / Integer / 1000	Tries to list all files in the `path` location. If it fails, will retry `attempts` times, waiting `sleepBetweenAttempts` milliseconds each time.

Driver Extensions#

This protocol driver provides a set of extension that allow the customization to perform actions that are independent of the Driver Definitions associated. This means you can, using customization, get/set node values without needing to describe them.

This is particularly useful when accessing equipment data that is shared/too complex to access using the regular mechanisms. Also useful when needing to access data whose naming convention may vary from device to device.

Objects#

The objects are structures that are either passed as parameters or returned as result values.

RegisterHandler#

Name	Type	Description
type	String	Id of the event to register a notification. Example: `connect.iot.driver.fileBased.event.ChangedFile_1234567891011`

UnregisterHandler#

Name	Type	Description
type	String	Id of the event to unregister notifications. Example: `connect.iot.driver.fileBased.event.ChangedFile_1234567891011`

ExecuteCommand#

Name	Type	Description
command	Json object	Includes the command name, device id and parameters types
parameters	Json object	Parameter names and values.

Methods#

Register Event Notification#

Notifies the protocol driver to raise the occurrences of a specific event when it occurs.

void connect.iot.driver.fileBased.registerEvent(data: RegisterHandler)

Note : The events will be published to the listeners connect.iot.driver.fileBased. appended with the name of the event registered and a random id. Example: connect.iot.driver.fileBased.event.ChangedFile_1234567891011

Unregister Event Notification#

Notifies the protocol driver to stop raising the occurrences of a specific event when they occur.

void connect.iot.driver.fileBased.unregisterEvent(data: UnregisterHandler)

Example#

Notify connect.iot.driver.fileBased.unregisterEvent

{
  "name": "ChangedFile_1234567891011"
}

Execute Command#

Executes a custom command.

any connect.iot.driver.fileBased.executeCommand(data: ExecuteCommand)

Example#

SendRequest connect.iot.driver.fileBased.executeCommand

{
    "command": {
        "name": "FileOrDirectoryExists",
        "deviceId": "Exists",
        "extendedData": { "commandType": "Exists" },
        "parameters": [
            { "name": "path", "dataType": "String", "deviceType": "String" },
        ],
    },
    "parameters": {
        "path": "c:/temp/file.csv",
}

Remarks/Behavior#

Events/Commands#

Although you can define your own events and commands using the Driver Definition, it is highly advisable to use the File Drivers specific tasks in the Workflows. They are already pre-configured with all possible properties/parameters and will highly decrease the development time.

Network Shares#

This driver can only work with local paths. If intended to be used with network shares, they must be previously mounted. This can be accomplished by:

Mount the share as a network disk/share that is managed by the operating system at start-up
Use the Mount command provided by this driver (only supported in Windows Operating Systems)

Mount command#

The Mount command provided will create a symbolic link between the network share and the computer. Due to Microsoft restrictions, this can only be performed by and administrator or by a pre-defined set of users.

Run the Automation Manager/Driver as administrator
Alternatively change in Local Security Policy (secpol.msc):
- Navigate to: Local Policies > User Rights Assignment
- Double click: Create Symbolic Links
- Add your username to the list, click OK
- Log off