Incoming Webhooks

How to connect Devices to Datacake using Webhook (from Third-Party IoT Platform such as Pycom PyBytes, Actinius I/O, Blues Wireless Notehub / Notecard - or any other source that can send data via Web)

Last updated 2 years ago

Introduction

With our new Webhook integration, anything that can relay messages via HTTPS call can be connected to Datacake. This is also called Webhook.

The smart thing about this is that we don't require a structure, but instead provide tools that you can use to accept any format and "decode" it appropriately.

We do this by providing Payload Decoders - a concept well known on LoRaWAN ecosystems.

Add new Device

To add a new Device that can receive data through our Webhook Integration please press on "Add Device" in the fleet overview of your workspace.

This will bring up a modal where you have to select the "API" Device type.

Datacake Product

Now chose "New Product" and provide a new for this product, something like "My Temperature Product" or "My Sensor Template".

What is a product?

Each device on Datacake belongs to a product, which defines:

The dashboard
The fields in the database
Payload Decoder
Routing of the data to the devices

If you add another device and add it to the product we are creating right now, this new device will inherit the dashboard, decoder and database fields from the product.

The data is stored individually.
Dashboard, Decoder and the number of fields in the database will be shared between all devices belonging to the same product.
If you change settings on the product (dashboard, add/remove or change fields) then these changes immediately affect all devices that belong to the product.

Add Device(s)

In the next step you are asked for the devices that are being created and added to the product you defined in the step before.

Right now we only want to create a single device but this dialog would allow us to create multiple devices and add them all to the same product.

Serial Number

Serial numbers are used to identify devices when ingesting data via Webhook. Datacake auto-generates a Serial Number for you which you at any time later can change to a custom one.

We leave this set to "Auto-Generate" for now.

Name

Enter a descriptive name for this device here.

Plan

Before we can continue we need to select a plan this device (or multiple devices) will belong to. We do support many different plans and your first two devices are free.

So in case you are new to Datacake, select "Free". This will let you continue directly.

For more Information on Pricing please visit the following page on our website:

Create Database Fields

Now that we have created our first device, we open it up by clicking on the List-Entry and use the Navigation Tab-Bar to access its Configuration-View:

Next step is creating a few fields in the database section of this device. So we now scroll down a bit until we reach the "Fields" section:

Here we press on "Add Device".

Using this Modal / View we create the following fields:

Temperature
Humidity
Battery Voltage

So that in the end the Database Section would look like the following:

Product-wide Setting

Please note that the Database Fields are Product-wide Settings!

Even if it seems that we have only attached the fields of the database to the respective device, the fields are stored in the database in the definition of the product.

This means: If you now add another device to the existing product, it will automatically receive the same database fields (but can store its own data).

Forward Data

Now that we have the Database Section set up we can move on and get to the part where the data is being forwarded.

Webhook URL

Each product on Datacake has its own custom API endpoint.

This URL can be found in the Configuration under the Payload Decoder Settings section.

And it looks a bit like this:

https://api.datacake.co/integrations/api/2de1c5ade79bcef8/

Route Data via Paths

You can add custom paths to your Webhook URL if you want to divide the data of the payload into separate API calls, for example:

https://api.datacake.co/integrations/api/2de1c5ade79bcef8/data/
https://api.datacake.co/integrations/api/2de1c5ade79bcef8/status/
https://api.datacake.co/integrations/api/2de1c5ade79bcef8/diagnostics/

Send Payload

Form Data

If your Content Type is application/x-www-form-urlencoded - You can send the payload as a

JSON

You can send JSON and parse that into an object.

Query Parameters

You can also send data directly as Query Parameters (in combination with custom Paths and User Payload Data):

https://api.datacake.co/integrations/api/2de1c5ade79bcef8/?device=123&check=true
https://api.datacake.co/integrations/api/2de1c5ade79bcef8/diagnostics/?foo=bar

Set up Payload Decoder

Purpose and Features

With the Payload Decoder you have the possibility with the help of a small piece of JavaScript code to customize the data of the webhook so that you can forward it to Datacake's database.

In summary, a payload decoder has the following features:

Code snippet that is executed when the webhook URL is called.
Access to measured values of all devices of the product.
Routing into the respective device or a group of devices.

Data Flow

The Payload Decoder is connected to the Webhook URL and is executed whenever a message / HTTP call is received via this URL.

Product-Wide

Payload Decoder are product-wide settings!

Since each product on Datacake has its own webhook URL, so is the payload decoder tied to that product.

To forward data from the payload decoder to the respective device, a corresponding ID (serial number) must be passed. We will show you how to do this in the following steps.

Basic Example

Let's get together a very quick example. Imagine your Webhook contains device data encapsulated as a JSON String. So this could look like this:

curl --header "Content-Type: application/json" -X POST
--data '{"device":"MySerial01","data":{"temp":23.01,"bat":3.21}}'
https://api.datacake.co/integrations/api/2de1c5ade79bcef8/

Decoder

To extract this data the decoder on Datacake would look like this:

function Decoder(request) {

    // Parse JSON into Object
    var payload = JSON.parse(request.body);
    
    // Extract Serial for Routing into Device
    var device = payload.device;
    
    // Load Data from JSON
    var data = payload.data;
    var temperature = data.temp;
    var humidity = data.hum;
    var battery = data.bat;
    
    // Forward Data into Device using Serial
    return [
        {
            device: device,
            field: "TEMPERATURE",
            value: temperature
        },
        {
            device: device,
            field: "HUMIDITY",
            value: humidity
        },        
        {
            device: device,
            field: "BATTERY_VOLTAGE",
            value: battery
        },
    ];
}

Debugging

You can test the Decoder using the built in tools that you also find in the Payload Decoder section in the Device Configuration View.

Simply paste in the JSON as a text into the Body-Field (marked red) and press on "Try Decoder". You can see the Output of the Decoder in the Debug-Fields (marked blue).

If the payload decoder was able to successfully forward data into a device (given correct serial number) you can see the mapping under the "Recognized Measurements" section:

Real Test

You can use CURL or any other URL Calling tool to simulate the API Webhook and test the Decoder under real conditions.

curl --header "Content-Type: application/json" -X POST 
--data '{"device":"MySerial002","data":{"temp":23.01,"bat":3.21,"hum":54.22}}' 
https://api.datacake.co/integrations/api/25f1de28-2630-bea1-be33-c5adc79bef68/

Database

After Data has come in you can see this on the Database Fields in the Fields-Section.

Logging

You can see all incoming Webhooks in the Logs-Section.

Simply click on "Show Logs" which will open a Logging-View showing the last recent Webhook calls.

Device Identification

The following is important. Please read carefully!

Decoders are bound to the URL and the Product the URL belongs to. They are not per Device. So you need to return a routing information so that Datacake can route your data into the actual device.

We do this by providing an additional Key/Value pair when returning data from the payload decoder. You might have already seen that all Payload Decoder Snippets above return a device-Key/Value in their return type, so like:

return [
    {
        device: deviceID, // Serial Number or Device ID
        field: "TEMPERATURE",
        value: temperature
    }
];

Serial Number

When creating a new API Device Datacake does autogenerate a Serial Number for you. However you can override this Serial Number and paste in Serial Number or any other Identification Info that is also present in the payload of your Webhook.

So imagine your Webhook does contain the following ID:

{"id":"MySerial001","temperature":22.01}

The important part here is the: "id":"MySerial001" - We can now override the Serial Number of the Device on Datacake and set it to MySerial001.

So in the Configuration-View on the Device in Datacake you find an API Configuration Section that shows the current Serial Number and a Button "Change".

If you click on that you can override the Serial. In the case of this example here we override this to MySerial001.

So once more. Imagine you have the following Webhook Payload:

{"id":"MySerial001","temperature":22.01}

The Decoder with correct device routing information would look like this.

function Decoder(request) {

    var payload = JSON.parse(request.body);

    var deviceID = payload.id;
    var temperature = payload.temperature;
    
    return [
        {
            device: deviceID, // Serial Number or Device ID
            field: "TEMPERATURE",
            value: temperature
        }
    ];
    
}

Hard-coded Identification

What if your Webhook does not contain device identification like a serial number or any other? You can of course hard-code the Device ID in your Decoder, so this would look like this:

return [
    {
        device: "MyHardcodedSerial",
        field: "TEMPERATURE",
        value: temperature
    }
];

Not on native Integrations

You may have noticed that the concept of Device-ID Identification is not present on the native integrations for LoRaWAN or Particle.io.

This is correct, because here the routing is automated in the backend. This works because the webhook structure is fixed and Datacake always knows where to find the ID (namely DevEUI or Particle Device-ID -Which is also the reason why the serial number for LoRaWAN or Particle is the serial of the respective integration).

Payload Decoder Programming Guide

Decode JSON

Let's assume you are sending JSON as payload:

curl --header "Content-Type: application/json" -X POST
--data '{"device":"MySerial01","data":{"temp":23.01,"bat":3.21}}'
https://api.datacake.co/integrations/api/123456/

If you want to decode JSON you can turn

// Parse JSON into Object
var payload = JSON.parse(request.body);

// Extract Serial for Routing into Device
var device = payload.device;

// Load Data from JSON
var data = payload.data;
var temperature = data.temp;
var battery = data.bat;

// Forward Data into Device using Serial
return [
    {
        device: device,
        field: "TEMPERATURE",
        value: temperature
    },
    {
        device: device,
        field: "BATTERY",
        value: battery
    },
];

Decode Form Data and Requests

if your Content Type is: application/x-www-form-urlencoded - you can send a body like this:

deviceid=MyDeviceID123&temperature=23.01&battery=3.43

The request-Object in your Payload Decoder will hold already parsed Objects of this payload which you can access in the following way:

function Decoder(request) {

    var payload = request["POST"];

    var deviceID = payload.deviceid[0];
    var battery = payload.battery[0];
    var temperature = payload.temperature[0];
    
    return [
        {
            device: deviceID, // Serial Number or Device ID
            field: "TEMPERATURE",
            value: temperature
        },
        {
            device: deviceID, // Serial Number or Device ID
            field: "BATTERY",
            value: battery
        }
    ];
    
}

Decode Query Parameters

If you are Sending Data as Query Parameters like this:

https://api.datacake.co/integrations/api/2de1c5ade79bcef8/?device=123&check=true

The following snippet shows you how to access those in the Decoder:

function Decoder(request) {

    var payload = request["GET"];

    var deviceID = payload.device[0];
    var temperature = payload.temperature[0];
    
    return [
        {
            device: deviceID, // Serial Number or Device ID
            field: "TEMPERATURE",
            value: temperature
        }
    ];
    
}

Combining Query & Payload

If you want to combine URL Query Parameters with Form Encoded Data or any other Payload you can do this using the following snippet:

function Decoder(request) {

    // query is always GET - even on POST
    var queryParameters = request["GET"];
    
    // payload is on POST only
    var payloadData = request["POST"];

    ...
    
}

Decode Methods

You can query info about the current Method and use this as a routing info for your payload decoder:

function Decoder(request) {

    if (request.method === "POST") {
    
        // Do Stuff when "POST"
        
    } else if (request.method === "GET") {
    
        // Do Stuff when "GET"
    }
    
    ...
}

Route URL Paths

You can use Custom URL Endings on your API Webhook URL to separate Actions inside the Payload decoder. So let's assume you are using the following URL Endings:

https://api.datacake.co/integrations/api/2de1c5ade79bcef8/data/
https://api.datacake.co/integrations/api/2de1c5ade79bcef8/status/
https://api.datacake.co/integrations/api/2de1c5ade79bcef8/diagnostics/

Here is how you use these in a Payload Decoder:

function Decoder(request) {
    
    var data = JSON.parse(request.body);
    
    var urlElements = request.path.split('/');
    var lastURLSegment = urlElements[urlElements.length-2];
    
    if (lastURLSegment === "data") {
    
    } else if (lastURLSegment === "status") {
    
    } else if (lastURLSegment === "diagnostics") {
    
    }
    
}

Return Multiple Devices

If your Webhook Payload contains data for multiple devices you can return them all in a single call.

Webhook

Imagine your Webhook is sending JSON Data that contains payload for multiple devices, structured something like this.

[
    {"id":"MySerial001","temperature":22.01},
    {"id":"MySerial002","temperature":22.01},
    {"id":"MySerial003","temperature":22.01}
]

Decoder

In this case the Decoder would look something like this:

function Decoder(request) {

    var devices = JSON.parse(request.body);
    
    for (var i = 0; i < devices.length; i++) {
    
        var device = devices[i];
        
        datacakeFields.push({
            device: device.id, // Serial Number or Device ID
            field: "TEMPERATURE",
            value: device.temperature        
        });
        
    }
    
    return datacakeFields;
}

Device Identification

Please make sure that the JSON-Data contains information that can be used to identify the devices in the payload decoder (eg. Serial Number).

Recording Historical Data

Webhook

Lets assume you have a temperature sensing device that collects four individual measurments (over a time of 15 Minutes) and then sends them as a combined message (each hour).

The payload would look like this:

{"id":"MySerial001","temperatures":[21.23, 22.34, 45.43, 56.40]}

Decoder

Our Decoders can return timestamps so you can backfill the Database Fields. This would look something like the following snippet:

function Decoder(request) {

    var devices = JSON.parse(request.body);

    var deviceID = payload.id;
    var temperatures = payload.temperatures;
    
    // current Unix Timestamp
    var timestamp = Math.floor(Date.now() / 1000);
    
    for (var i = 0; i < temperatures.length; i++) {
    
        var temperature = temperatures[i];
        
        datacakeFields.push({
            device: deviceID, // Serial Number or Device ID
            field: "TEMPERATURE",
            value: temperature
            timestamp: timestamp
        });
        
        // decrease Timestamp 15 Minutes (go back in time)
        timestamp = timestamp - (15 * 60);
        
    }
    
    return datacakeFields;
}

Access Request Object

Let's assume you are calling the following Webhook:

curl -X POST -d 'post=parameter' 
https://api.datacake.co/integrations/api/123456/optionalsubpath?foo=bar

In your Payload Decoder you have the access to the following parameters of your HTTP Webhook everytime there is a Webhook call happening:

{
    "method": "POST",
    "GET": {
        "foo": ["bar"]
    },
    "POST": {
        "post": ["parameter"]
    },
    "body": "post=parameter",
    "headers": {
        "Content-Length": "14",
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "path": "/integrations/api/123456/optionalsubpath"
}

Working with Geolocation

If your device has GPS coordinates and you want to store them to a geolocation field type, you have to return the latitude and longitude in the form of a tuple. See the following example for an explanation.

function Decoder(request) {

    var payload = JSON.parse(request.body);

    var deviceID = payload.serial;
    var lat = payload.latitude;
    var lon = payload.longitude;
    
    var position = "(" + lat + "," + lon + ")";
    
    return [
        {
            device: deviceID,
            field: "LOCATION",
            value: location
        }
    ];
    
}