Event Ingestion through IoT Data Platform#

Estimated time to read: 4 minutes

Event ingestion is the process of obtaining and importing events for the immediate use or storage in a database. Events can be streamed in real time, where each event item is imported as the source emits it, or ingested in batches.

This document will provide a quick guide for the configuration of a mechanism of event ingestion using the IoT Data Platform capabilities of Critical Manufacturing.

Overview#

In this example we will define an event that receives values from different sensors on a shop floor. This event will have a specific structure and so we shall define it, make the system aware of it and able to parse the same event when received as an API call to the system. To create the appropriate structure to receive and adequately ingest and handle the events on Critical Manufacturing, a number of steps will have to be followed:

1. Create an IoT Event Definition to specify the event structure.
2. Create an IoT Consumer Definition to define which type of consumer will receive the event.
3. Ensure that the system ingests and operates on the event through an IoT Consumer.

The steps detailed above will create the objects in the system that will handle the events and the payload contained therein. This way, you can start from there:

Creating an IoT Event Definition#

In the Business Data section of the main menu, navigate to the IoT Event Definition tile and create a new entry. Provide a name for the IoT Event Definition. Bear in mind that the name will identify the Event throughout the Data Platform infrastructure so it should be a meaningful name for later reference. An example of the values can be seen below:

The event to be created has the following structure with sample values:

AppProperties is a field included by the Kafka host that is running and receives the event, while Data is the actual event payload.

Adding properties to the event definition can be done manually or by importing a JSON structure such as the one shown above. The system will infer the data types from the actual values and generate a property for each of the fields.

Several properties are essential for the proper functioning of the event ingestion, so specify for each property:

• Property name – name that matches the property in the Event JSON document.
• Array – identifies this property as an array of objects.
• Mandatory – identifies if the property is mandatory. If true, this information will be used to validate all the arriving events.
• Indexed – if this value is defined, the fields will be automatically selected in the creation of the IoT Consumer.
• DataType – the data type of the property. Mandatory, an enumeration. Will be mapped to the following JSON types (in parentheses).
• Default Value – specifies the default value of the property. If not present in the received event, this value will be set by the event validation process.
• Shared – a specific IoT Schema can be selected to be matched to a JSON structure and later shared as a separate object in the system.

Importing from a sample JSON structure can be simple as well:

After creating the properties manually or importing from a JSON file, the properties will be generated and the IoT Event Definition can be created.

Upon original import, and if the Shared option is not defined above, a new IoT Schema is also created and associated in a one-to-one relationship with this particular event definition, and it could then be used in other definitions.