How to integrate with AWS IoT Core

The Business Goal

Did you know the Datacake platform can integrate with AWS Cloud using the MQTT protocol, enabling it to receive MQTT messages originating from AWS-managed devices? This capability opens up various possibilities, allowing users to leverage Datacake's intuitive interface to explore and make data actionable.

Benefits of Datacake and AWS Cloud Integration:

1. Real-time Data Ingestion:

   • Datacake's integration with AWS Cloud enables real-time ingestion of MQTT messages from AWS-managed devices.

   • This capability is particularly valuable for applications where timely access to data is crucial, such as IoT monitoring, industrial automation, and real-time analytics.

2. Simplified Data Exploration:

   • Datacake's user-friendly interface simplifies data exploration, allowing users to quickly visualize, analyze, and interpret data received from AWS-managed devices.

   • This eliminates the need for complex data engineering or programming skills, making it accessible to users of all technical backgrounds.

3. Actionable Insights:

   • Datacake empowers users to derive actionable insights from the data received from AWS-managed devices.

   • Customizable dashboards and visualizations enable users to monitor key metrics, identify trends, and make informed decisions based on real-time data.

In this article, I will explain the integration between the Datacake platform and the AWS Cloud using the MQTT protocol. That is the first step to unlocking the full potential of IoT data, gaining actionable insights, and driving data-driven decision-making to optimize operations and achieve business outcomes.

You can experiment with Datacake's capabilities without disrupting your operations since no device-level configuration changes are needed.

I will guide you through the AWS and Datacake setup. Together, we will create the end-to-end working solution using a simulated device. I will explain every step, so no AWS, Datacake, or MQTT knowledge is required.

Solution Overview

This is the overview of the end-to-end solution:

Datacake is a multi-purpose, low-code IoT platform that requires no programming skills and minimal time to create custom IoT applications.

AWS IoT can support billions of devices and trillions of messages. It can process and route those messages reliably and securely.

Combining those platforms creates a highly scalable yet easy-to-use solution.

Datacake Setup

Let’s start by configuring the Datacake platform.

Log into your Datacake account and create a Workspace. I called mine “AWS Workspace,” but you can use any name relevant to your use case.

Our objective is to seamlessly integrate Datacake with the AWS IoT platform. To accomplish this, we will utilize the Generic API device. This device supports the MQTT protocol, essential for establishing connectivity between the AWS Cloud and Datacake.

MQTT (Message Queuing Telemetry Transport) is a specialized messaging protocol tailored for the Internet of Things (IoT) domain. Developed and managed by the OASIS (Organization for the Advancement of Structured Information Standards), MQTT offers a lightweight publish/subscribe messaging transport.

Use the web interface to add a new Generic API device:

Datacake leverages Products to manage a fleet of connected devices effectively. Let’s create a new Product for our use case.

Fill in the attributes characterizing your first device. Provide the Serial Number and Name to describe your equipment.

To conclude this step, select a billing plan. Datacake offers a Free Plan, allowing users to explore the platform without financial commitment. This option provides all the necessary features to implement our use case.

Congratulations! You successfully added your first device to the Datacake platform!

Let’s review our progress: We created the Workspace, Product, and Device.

But how can we link that Device to our physical equipment connected to the AWS Cloud? To achieve that goal, we will utilize the MQTT protocol.

MQTT configuration

Select your Device in the Datacake platform and scroll down to the MQTT configuration.

As the hist suggests, the MQTT integration enables connection to an external MQTT broker. In the presented scenario, we will utilize the MQTT broker exposed by AWS IoT Core.

Let’s start by clicking the Add new MQTT Broker button to check the required configuration.

To ensure data confidentiality during communication between AWS and Datacake, we will leverage the MQTTS protocol. It provides an additional layer of security by encrypting the communication using Transport Layer Security (TLS). This encryption ensures that our data remains protected and confidential throughout the communication.

AWS IoT Core uses port 8883 for MQTT, so that is the port number we need to define.

To establish trust between AWS and Datacake, we will use X.509 Certificates. Certificates allow both parties to confirm identities and securely communicate with each other. I will show you how to obtain those certificates and the hostname from your AWS Account in the next step.

AWS Setup

Switch to the AWS Console to prepare AWS IoT Core for integration with Datacake.

In the AWS Console, search for the IoT Core service.

On the AWS IoT page, go to settings (bottom left part of the screen) and copy the Endpoint value. We will use it as the Hostname of the MQTT Broker.

To obtain the X.509 Certificate, go to the Security → Certificates and click on the “Create certificate”.

Leave the Auto-generated certificates option and change the Certificate status to Active so that Datacake can use this certificate to communicate with AWS.

Download the following files:

• Device certificate

• Private key file

• Amazon Root CA1

We use the X.509 Certificates for authentication - to confirm the identity of a device or application.

Additionally, we need to authorize Datacake to communicate with the AWS IoT Core. To provide that permission, we use the AWS IoT Policies.

Go to Security → Policies and hit “Create”.

I will call my Policy a “DatacakePolicy”, but you can use whatever name suits you. As a good design practice, the name should represent the intent of a given Policy.

Then select “Policy examples”. Search for “ticket”, pick “Publish/Subscribe - Subscribe to and receive messages on a ticket”, and hit “Add to policy”.

Remove the first row and switch to the JSON view:

We need to adjust the IoT Policy to our needs. I propose the following configuration:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:eu-west-1:123123123:client/*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": "arn:aws:iot:eu-west-1:123123123:topicfilter/aws-workspace/aws-integration/*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Receive",
      "Resource": "arn:aws:iot:eu-west-1:123123123:topic/aws-workspace/aws-integration/*"
    }
  ]
}

Let me break it down for you:

123123123 - specifies the AWS Account ID, this value is unique for every AWS Account. Do not change the integer provided by the sample policy!

"Action": "iot:Connect" - permits connecting to the AWS IoT Core.

"Action": "iot:Subscribe" - permits subscribing to a specified MQTT Topic (aws-workspace/aws-integration/* in our case).

"Action": "iot:Receive" - permits receiving MQTT messages from a specified MQTT Topic.

When defining the MQTT Topic Structure, we should start from a generic level and follow with more specific parts. I used the aws-workspace representing my Datacake Workspace and the Datacake Product name: aws-integration.

That approach will ensure the separation of data managed by distinct workspaces and products. I suggest it to avoid conflicts and maintain data confidentiality for every Product.

Update the Policy document and create the IoT Policy.

To establish the integration between AWS IoT and Datacake, we must link the X.509 Certificate with the IoT Policy. Currently, there is no such connection. Without an attached Policy, the X.509 Certificate does not allow any action.

To resolve this, navigate to the Security → Certificates. Select the checkbox near the certificate and choose "Attach Policy" from the Actions menu.

Pick the IoT Policy we created and hit Attach.

Datacake Setup

Return to the Datacake platform to complete the configuration.

Go to Devices → Configuration, find the “MQTT Configuration” section, and hit “Add New MQTT Server”. Fill the form with data gathered in the previous steps.

Use any text editor to view the content of the Private Key and X.509 Certificates.

This is how the completed form looks on my end. You will have different values, but the structure should be similar.

Hit the “Test Connection” button to verify the connectivity between AWS IoT Core and Datacake.

You should get the “Connection successful” pop-up. Click on the “Add MQTT Server” to save your configuration.

Congratulations! We successfully established the MQTT integration between AWS and Datacake. AWS will forward MQTT Messages sent to the aws-workspace/aws-integration/# MQTT Topic to Datacake.

Now, we have to enable Datacake’s Devices to receive and store those MQTT Messages.

Go to the “Temp Sensor 0001” Configuration and scroll down to the Fields section:

Fields describe the attributes stored by devices. In our example, we use a simulated temperature sensor, so add a Field describing temperature readings:

Now, we need to create the MQTT Uplink Payload Decoder. Uplink decoders subscribe to MQTT Topics, process incoming MQTT Messages, and store them.

Find the MQTT Uplink Decoders section and click on “Add Uplink Decoder”.

We want to enable our devices to receive MQTT Messages from the aws-workspace/aws-integration/<deviceid> MQTT Topic, where the <deviceid> represents the ID of a specific device. To achieve that functionality, our Uplink Decoder has to subscribe to the aws-workspace/aws-integration/+ MQTT Topic.

The "+" is a single-level wildcard; in our case, it will match any Device ID.

Use the following code as the Decoder function:

function Decoder(topic, payload) {
    
    // Extract the Device ID from the MQTT Topic.
    dev = topic.split("/")[2];
    
    // Parse the payload of the MQTT Message.
    msg = JSON.parse(payload);
    
    // Obtain the temperature reading from the MQTT Message.
    temperature = msg.temp
    
    return [{
        device: dev,
        field: "TEMP",
        value: parseFloat(temperature),
        timestamp: Date.now()
    }];
}

The picture below presents a complete definition of the Uplink Decoder.

End-to-end Test

It is time for the end-to-end test!

Go to the AWS IoT Console and find the MQTT test client:

We will use the MQTT test client to publish MQTT Messages simulating a device connected to the AWS IoT Core. Switch to the “Publish to a topic” tab and define the Topic name as:

aws-workspace/aws-integration/tempsensor-0001

And the Message payload as:

{
  "temp": 21.0
}

Once you are ready, hit “Publish”! It looks as if nothing has happened… Let’s switch to the Datacake platform and go to the Debug tab:

There is the MQTT Message!

Summary

Let’s summarize what we accomplished:

1. AWS IoT Infrastructure Setup for Datacake Integration:

   • Configuration of AWS IoT resources, including an IoT Core, X.509 Certificates, and IoT Policies.

   • Establishing secure MQTT communication between the AWS IoT platform and Datacake.

2. Datacake Setup:

   • Creation of a Datacake product describing our use case, expected properties, and data format.

   • Configuring External MQTT Server to facilitate communication with AWS Cloud.

3. End-to-End Test using a Simulated Device:

   • Connect the simulated device to AWS IoT using the MQTT protocol.

   • Publishing sensor data from the simulated device to AWS IoT.

   • Verifying successful data transfer between AWS IoT and Datacake.

By completing these steps, we have successfully established a secure and reliable data integration pipeline between AWS IoT and Datacake. This integration enables the seamless flow of device data from AWS IoT to Datacake's platform, allowing for real-time data analysis, visualization, and actionable insights.

Congratulations! Your dedication and hard work have paid off. You are now prepared to delve deeper into the capabilities of the Datacake platform.