Data integration options with Cumulocity IoT

7 min readNov 15, 2022

Introduction

The core of Cumulocity IoT is device integration but with pure device data only a few use cases can be implemented such as Device Management or Condition Monitoring. The more comprehensive use cases rely on additional data which resides in other databases, systems or services. Only when combining device data with other data like master data or production data the real value of IoT can be unleashed!

In this article I will give you an overview about the options you have to integrate different kind of data with Cumulocity IoT to implement high value IoT solutions.
For each option I will give an overview about the architecture and use cases which can be implemented.

Let’s get started!

The power of (IoT) data

When building an IoT solution the most earliest and mandatory step is the device connectivity to retrieve device data. There are two main needs and therefor different kind of data in IoT solutions:

The need to get near real-time insights of what’s going on or when you have to act → live device data (hot storage)
The need to detect patterns and explore correlation of device data to adapt the solution accordingly → long-term device data (cold storage)

While for the live device data you need dashboard like visualization and streaming analytics capabilities, you need different capabilities for long-term device data. Most likely they are data exploration tools, pattern recognition and training of machine learning models.

Both device data variants are supported by the capabilities of Cumulocity IoT.

Again, having access to device data is a very good starting point for an IoT solution but there is most likely more data in your (or other) companies that needs to be integrated in your IoT solution.

To name the most common used data:

Master Data — Any kind of materials, customers or suppliers you have in your systems and must be correlated to device data.
Transactional Data — Any kind of processes, production orders or customer orders you have in your systems and must be correlated to device data.
Other Data — Any kind of 3rd Party data which could be environmental data or geolocation data which should be correlated to device data.

What they have in common is that they reside in different kind of systems or databases most likely offering different kind of APIs you need to integrate. The way the data flows is also very use case dependent. Sometimes the device data is integrated to the 3rd Party Systems (e.g. Ticketing solutions) and sometimes it is the other way around (e.g. Production Order data which should be visualized in Cumulocity IoT). Also sometimes it makes sense to replicate data sometimes you just need additional data for visualization or analytics/calculations. All ways are valid and must be supported!

Being most flexible and fast in regards of data integration is key for an IoT solution. You don’t want to spend much time building complex integrations or even workarounds.

Let’s have a view about the options & different kind of architectures you have!

Data Integration Options

Device Integration

Let’s start with the obvious one from an IoT standpoint: Device Integration.

Supporting device integration by an IoT platform is mandatory. With Cumulocity IoT you have multiple ways how you can integrate devices. On a high level perspective there are three options:

Connecting a device directly to one of the core protocols LWM2M, REST or MQTT. Meaning the device firmware contains the connectivity and mapping part to Cumulocity endpoints and domain model, called device agent. Those devices are called “smart” or “connected” devices which directly can communicate with an IP-Protocol like TCP (MQTT or REST). For LWM2M this is not Cumulocity specific as the data model is part of the protocol (instead of MQTT or REST) and can be used across all other systems supporting LWM2M not only Cumulocity IoT.
Connecting a device directly to a server side agent. Meaning the device can handle the IP-Protocol like TCP but supports any proprietary data models which is not supported out of the box by Cumulocity IoT and therefor must be mapped to the domain model. The server side agent can run inside or outside of Cumulocity IoT which is called in the picture above a Backend Agent (outside) or Server Side Agent (inside). This option is often used for network providers like LPWAN etc.
Connecting a device using any kind of gateway. Meaning the device either don’t support any IP-Protocol (BLE, ZigBee etc.) or is behind a firewall (production network etc.) with no access to Cumulocity IoT running on an edge or the cloud. Also the data model most likely must be mapped to the domain model of Cumulocity IoT.
For that purpose Cumulocity IoT offers the thin-edge.io which is an edge software running on such limited hardware addressing the need of connecting other devices and mapping data to Cumulocity IoT.

Device Integration is a complex topic and of course this is just a rough overview to integrate devices. I will address this need in another blog post where I will describe in detail what are the steps & process to get your device integrated

Supported use cases

With device data only you can already implement a lot of common IoT use cases. Here is a list of some examples with not intention of completeness:

Device Management — Manage your devices including Software, Firmware & Configuration Management
Device Connectivity Management — Manage the availability of your device and act if something suspicious is happening
Condition Monitoring — Retrieve time series data and events to monitor the condition of devices or machines the devices are attached to.
Overall Equipment Effectiveness (when machine data is provided and existing) — Monitor and optimize your fleet of machines and production processes.

Data Contextualization

Let’s continue with the most simple data integration after device integration: Data Contextualization

In this option data reside in a 3rd Party System or database and should be written to Cumulocity IoT Database to extend the device data with master or transactional data. The interface to do this is REST. In this option no logic is running at Cumulocity IoT but the 3rd Party System or any other component implements the REST interface of Cumulocity IoT to add context data. In such use cases the context data is replicated in the Cumulocity IoT database and most likely to be used when the frontend of the solution is running within Cumulocity IoT.

It is not commonly used as the data replication of contextual data increases the operational costs of the IoT platform and keeping the data synchronous is quite complex.

A better pattern would be to use Service Integration

Supported use cases

The use case here depend totally on the data that is ingested to Cumulocity IoT. To summarize a few use cases:

Asset Management — Manage your assets and correlate them to device data
Master Data Management Harmonization — Combine multiple master data and device data to harmonize the access and view on data
Production Data Management — Correlate production data and device data to see how efficient your production is.

Data Push Integration

With the data push integration, we enable 3rd Party Systems or Message Broker to work on near real-time device data.

With Cumulocity IoT you have 3 options to enable this pattern:

Using Apama Streaming Analytics you can implement a model that listens on device data and pushes the data in any format to a HTTP endpoint
With the pretty new Notification API 2.0 you can implement a client that subscribes on specific kind of device data and get the information reliable pushed when any data is forwarded. Reliable means if the client is down for any reason the messages are queued and when it comes online it gets the messages which were subscribed before. The protocols used between the client and the Notification API is HTTPS WebSocket.
Very similar to option 2 also using the Notification API 2.0 but the client subscribing is implemented within a microservice which forwards the information to any protocol of the broker or 3rd Party System. The microservice can use any outgoing protocol to forward the data like AMQP, REST or MQTT.

Supported use cases

By pushing data we can enable the following use cases:

Alarming — Keep informed via push notification on your mobile
Incident Management — Create incidents in any incident management system
Data Forwarding — Forward data reliable which should not reside for long at Cumulocity IoT.
Data Replication — Copy data in target systems as a duplicate.
Data Synchronization — Keep the data synchronous across systems & services.
High Priority Processes — Any process that listens on a event of device data e.g. maintenance process, shipping process etc.

Data Pull Integration

Please read the full article on the Software AG Tech Community Knowledge Base to discover more about Cumulocity IoT’s data integration options: