Skip to content

Connect IoT - CSV File Driver#

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. A CSV file stores tabular data (numbers and text) in plain text. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format.

Data Types#

These are the supported specific data types:

Name Description
String String Value
Boolean Boolean Value
Integer Integer value
Decimal Decimal value
Object Structured value

Protocol Parameters#

The protocol supports the following parameters, used define the communication parameters, as well as the expected behavior of the driver:

Name Type Possible Values Default Value Description
path String "" Path of the location to monitor for changes on the files. If no value is provided, no watcher is created, so no events will be triggered.
fileMask String "*/.csv" Glob pattern to use for the watcher to identify the files to handle. Use a tool like https://globster.xyz/ ⧉ to try a valid value to use.
hasHeader Boolean True Do the csv files have header?
delimiter String "," What is the separator of the fields?
initialLinesToIgnore Integer 0 Number of lines to ignore from the beginning of the file
commentChar String "" If set, all text after the character will not be handled
numberOfLines Integer 1 Number of lines of each file to read per round
skipEmptyLines Boolean True Should the parser ignore empty lines?
archivePath String "" Directory planned to use for archiving the files after processing. This value can later be used as a variable in the available operations to execute.
watcherType Enum Chokidar
NSFW		 Chokidar Type of watcher to use. Depending on the selection, different settings are necessary. Chokidar is the best overall option, but CPU and Memory heavy and slower to start when many files are present. NSFW is better other scenarios. Test both to determine the best option.
ignoreInitial Boolean False Flag to define if, during startup/restart, the files that were created, changed, deleted should be processed or ignored
pollingInterval Integer 100 Interval of file system polling (in milliseconds)
alwaysStat Boolean True Always get additional attributes (size, timestamps, etc) of the file that was identified by the watcher. Will require additional operating system resources.
depth Integer 0 Number of sub directories to watch. The higher the number, more memory/cpu/time will be required for watchers.
awaitWriteFinish Boolean True Trigger watcher events only when the file finishes writing.
awaitWriteFinishStabilityThreshold Integer 2000 Amount of time in milliseconds for a file size to remain constant before emitting its event.

Protocol Parameters specific for Chokidar watcherType#

Chokidar watcher type has some settings that are only known to it. If using the other type(s), these settings will not have any effect.

Name Type Possible Values Default Value Description
watcherMode Enum FileSystemEvents
Polling		 Polling To successfully watch files over a network (and in other non-standard situations), it is typically necessary to use Polling, however it could lead to high CPU utilization. FileSystemEvents is the most efficient method for monitoring local files.
pollingBinaryInterval Integer 300 Interval of file system polling for binary files (in milliseconds)
awaitWriteFinishPollInterval Integer 100 File size polling interval
atomic Integer 100 Automatically filters out artifacts that occur within the defined value (in milliseconds) when using editors that use 'atomic writes' instead of writing directly to the source file. If defined to 0, this function is disabled.

Extended Parameters#

As with other protocol drivers, some extra parameters are necessary to provide more context to the behavior expected.

Property#

Name Type Possible Values Default Value Description
propertyType Enum FileContent
FileName
RelativePath
FullPath
Size
IsFile
IsDirectory
AccessTime
CreationTime
UpdateTime
EndOfFile		 FileContent Provides a meaning of the property, and will receive the value depending of it when specific events occur.
FileContent Representation of a field in the file
FileName Name of the file being handled
RelativePath Relative path of the file being handled
FullPath Full path of the file being handled
Size Size of the file
IsFile Flag indicating if the handle is a file
IsDirectory Flag indicating if the handle is a directory
AccessTime Date and Time when the file was last accessed
CreationTime Date and Time when the file was created
UpdateTime Date and Time when the file was last updated
EndOfFile Flag indicating the end of file was reached
identifierType Enum Name
Index
None		 Name Way to identify the field data (defined in the Property Device Id. Only applicable when propertyType is FileContent

Event#

Name Type Possible Values Default Value Description
EventTrigger Enum NewFile
DeletedFile
ChangedFile
NewData		 NewData Indicates the Operating System action that will trigger the event.
NewFile When a new file is detected
DeletedFile When a file is deleted
ChangedFile When a file is changed
NewData When the file is created/changed, trigger an event with a number of lines (configurable)

Command#

Name Type Possible Values Default Value Description
commandType Enum Exists
Copy
Rename
Delete
Move
CreateFile
ReadFile
CreateDirectory
EmptyDirectory
Mount
MoreDataRequest		 MoreDataRequest Indicated the type of command that will be executed.
Exists Checks if a file or directory exists
Copy Copies a file or entire directory from one location into another
Rename Renames a file or directory
Delete Deletes a file or directory
Move Moves a file or directory
CreateFile Create or append data into a file
ReadFile Reads the entire content of an existing file
CreateDirectory Creates a new directory
EmptyDirectory Deletes all files and subdirectories of a directory
Mount Mounts a network drive location into a local path
MoreDataRequest After Data is processed, request for the next chunk.
ListFiles List all files in a directory.

Internal Driver Interface#

This protocol driver provides a set of interfaces ready to be used to ease the usability within the Controller Workflows. This section is intended to detail them. Although most if not all of these interfaces have specifically designed tasks to use them, the user is free to redefine them in the Driver Definition entity.

Properties#

Name PropertyType Data Type Description
FileName FileName String Name of the file identified in the watched directory. Will include the extension, but not the directory path.
FullPath FullPath String File name and the full path where it is located in the local system.
FileSize Size Integer Size of the file
EndOfFile EndOfFile Boolean End of file reached
ExampleField FileContent String Example of a field of the Csv File identified by Header name
ExampleField2 FileContent String DExample of a field of the Csv File identified by index (2nd)

Events#

Name EventTrigger Properties updated Description
OnNewFile NewFile FileName
FullPath
FileSize
 When the watcher detects a file that was not previously identified appears, will raise this event.
The timing when it will be triggered may depend on some settings, like watcherMode, awaitWriteFinish, awaitWriteFinishStabilityThreshold and others.
OnFileChanged ChangedFile FileName
FullPath
FileSize
 When the watcher detects a file that was already identified (and triggered the OnNewFileevent), and it was changed in some way, will raise this event.
The timing when it will be triggered may depend on some settings, like watcherMode, awaitWriteFinish, awaitWriteFinishStabilityThreshold and others.
OnFileDeleted DeletedFile FileName
FullPath
FileSize
 When the watcher detects a file that was previously identified and was now deleted (or moved somewhere), will raise this event.
OnDataAvailable NewData FullPath
ExampleField
ExampleField2
EndOfFile		 Triggered when data inside the file was read and parsed

Commands#

All paths of the commands listed in this section can use the following tokens to simplify usability:

Name Description Example
${Path} Replace the token with the path setting defined during setup ${path}\file.csv becomes c:\watchedDirectory\file.csv
${Archive} Replace the token with the archive setting defined during setup ${archive}\newFile.csv becomes c:\archive\file.csv
Name CommandType Parameters / Type / Default Value Description
MoreDataRequest MoreDataRequest path / String / "" After Data is processed, request for the next chunk
FileOrDirectoryExists Exists path / String / "" Checks the path location exists
CopyFileOrDirectory Copy source / String / ""
destination / String / ""
overwrite / Boolean / false
preserveTimestamps / Boolean / false
recursive / Boolean / true
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to copy the source location (file or directory) into the destination location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
If it is a directory and recursive is set to true, all subdirectories will also be copied.
RenameFileOrDirectory Rename source / String / ""
destination / String / ""
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to rename the source location (file or directory) into the destination location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
MoveFileOrDirectory Move source / String / ""
destination / String / ""
overwrite / Boolean / false
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to move the source location (file or directory) into the destination location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
DeleteFileOrDirectory Delete path / String / "" /
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Deleted the path location (file or directory). If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
CreateFile CreateFile path / String / ""
content / String / ""
encoding / String / "utf8"
mode / Integer / 0o666
flag / String / "w"
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to write into a file located in the path location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.

The mode will define the file attributes / permissions using unix format, but in decimal.
The flag determines how the file will be accessed (write, rewrite, append, etc). The supported flags are described in https://nodejs.org/api/fs.html#fs_file_system_flags. ⧉
ReadFileContent ReadFile path / String / ""
encoding / String / "utf8"
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to read the contents of the file in the path location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
CreateDirectory CreateDirectory path / String / ""
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to create a new directory in the path location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
EmptyDirectory EmptyDirectory path / String / ""
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to delete all files and subdirectories of path location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.
ListFiles ListFiles path / String / ""
attempts / Integer / 1
sleepBetweenAttempts / Integer / 1000		 Tries to list all files in the path location. If it fails, will retry attempts times, waiting sleepBetweenAttempts milliseconds each time.

Driver Extensions#

This protocol driver provides a set of extension that allow the customization to perform actions that are independent of the Driver Definitions associated. This means you can, using customization, get/set node values without needing to describe them.

This is particularly useful when accessing equipment data that is shared/too complex to access using the regular mechanisms. Also useful when needing to access data whose naming convention may vary from device to device.

Objects#

The objects are structures that are either passed as parameters or returned as result values.

RegisterHandler#

Name Type Description
type String Id of the event to register a notification. Example: connect.iot.driver.fileBased.event.ChangedFile_1234567891011

UnregisterHandler#

Name Type Description
type String Id of the event to unregister notifications. Example: connect.iot.driver.fileBased.event.ChangedFile_1234567891011

ExecuteCommand#

Name Type Description
command Json object Includes the command name, device id and parameters types
parameters Json object Parameter names and values.

Methods#

Register Event Notification#

Notifies the protocol driver to raise the occurrences of a specific event when it occurs.

void connect.iot.driver.fileBased.registerEvent(data: RegisterHandler)

Note

The events will be published to the listeners connect.iot.driver.fileBased. appended with the name of the event registered and a random id. Example: connect.iot.driver.fileBased.event.ChangedFile_1234567891011

Unregister Event Notification#

Notifies the protocol driver to stop raising the occurrences of a specific event when they occur.

void connect.iot.driver.fileBased.unregisterEvent(data: UnregisterHandler)

Example#

Notify connect.iot.driver.fileBased.unregisterEvent

{
  "name": "ChangedFile_1234567891011"
}

Execute Command#

Executes a custom command.

any connect.iot.driver.fileBased.executeCommand(data: ExecuteCommand)

Example#

SendRequest connect.iot.driver.fileBased.executeCommand

{
    "command": {
        "name": "FileOrDirectoryExists",
        "deviceId": "Exists",
        "extendedData": { "commandType": "Exists" },
        "parameters": [
            { "name": "path", "dataType": "String", "deviceType": "String" },
        ],
    },
    "parameters": {
        "path": "c:/temp/file.csv",
}