Efento NB-IoT sensors – integration with a Python CoAP server

This tutorial will show you how to set up a simple CoAP server, which receives the data from Efento NB-IoT sensors and saves it to a PostgreSQL database. In this tutorial we are using Python and PostgreSQL database, but the same idea can be easily implemented in other programming languages / with different databases. Should you have any issues or questions, feel free to drop us a line at help.efento.io

How does it work?

Efento NB-IoT sensors send the data in Protobuf format using CoAP protocol. This guarantees fast transmissions and small size of data, which results in up to 10 years battery life time. Moreover, as both CoAP and Protobuf are popular standards, it’s easy to integrate Efento wireless sensors with any cloud platform or custom application. To learn more about CoAP and Protobuf, please visit our Knowledge Library.

The Python Script we are going to write sets up the CoAP server. The server is constantly listening for data sent by Efento NB-IoT sensors. Once a new message arrives, the server parses the data, saves it in the PostgreSQL database and responds to the sensor with confirmation that the message has been received (code  2.01 “CREATED”). This means that the message has been successfully parsed and saved in the database. If anything goes wrong (e.g. database is down), the sensor will receive a response with code 5.00 “INTERNAL_SERVER_ERROR” . In that case, the NB-IoT sensor will retry to send the same data after a while.

Important! The response time from the server to the sensor should be as short as possible. In real-life integrations, the data received by the server should be pushed to a queue, and server should reply with 2.01 CREATED as quickly as possible. Saving the data to the database before responding to the sensor decreases the sensor’s battery life time.

Before you start

Before you start this, you will need to install and configure: 

  • Pycharm or any Python 3 IDE
  • PostgreSQL server

You will also need:

PostgreSQL database 

After downloading and installing PostgreSQL you will need to create the first database. This is one of the steps during the PostgreSQL installation. By default, the database will be created with the following credentials:

DATABASE_HOST = ‘localhost’;

DATABASE_USER = ‘postgres’;

DATABASE_PASSWORD = ‘Your password’;

DATABASE_NAME = ‘postgres’;

If you want to, you can change the names / credentials. Write them down, as they will be needed in the next steps. If you want to check database credentials, open pgAdmin in the PostgreSQL folder. Next open Object -> Properties -> General

Create table

To save the measurements coming from Efento Gateway in your database, you need to create a table. In this example, we are creating a very simple table to store all the data from the sensors, no matter what the sensor type. The table will have 5 columns, all of them of “text” type. Please note that this architecture of the database is only for the demonstration purposes. Database structure should be selected according to your project requirements.

You can create the table manually, using pgAdmin’s interface or using a SQL query. In pgAdmin select your database, open Tools menu: Tools -> Query Tools. Copy the request below into the Query Editor and click Execute (▶) :

CREATE TABLE  measurements (
    measured_at text ,
    serial_number text ,
    battery_ok  text ,
    type text,
    value text);

CREATE TABLE will create a new, initially empty table in the current database. The table will be owned by the user issuing the command.

CoAP Server

Before you start

The script uses a bunch of libraries. Before you start, you will need to download and install the following libraries:

  • psycopg2 – Psycopg is the most popular PostgreSQL database adapter for the Python programming language. If you want to know more check project’s website
  • aiocoap – The aiocoap package is an implementation of CoAP, the Constrained Application Protocol. It is written in Python 3 using its native asyncio methods to facilitate concurrent operations while maintaining an easy to use interface. If you want to know more about aiocoap check project’s website
  • asyncio – asyncio is a library to write concurrent code using the async/await syntax. asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. If you want to know more about asyncio check project’s website  
  • protobuf – is a free and open source cross-platform library used to serialize structured data. It is useful in developing programs to communicate with each other over a network or for storing data.
  • base64 – This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. It provides encoding and decoding functions for the encodings specified in RFC 3548, which defines the Base16, Base32, and Base64 algorithms, and for the de-facto standard Ascii85 and Base85 encodings.
  • datetime – The datetime module supplies classes for manipulating dates and times.

Compiling Protocol Buffers

Protocol buffers (or Protobuf) are a method of serializing the data that can be transmitted between microservices, applications or used in communication between IoT devices and servers. Protocol Buffers have the same function as well known and widely used JSON and XML. Like JSON and XML, the Protobufs are language-neutral and platform-neutral. Moreover, Protobuf is optimised to be fast and use as little network bandwidth as possible by minimizing the size of transmitted data. This makes Protobuf a perfect choice for serializing the data sent by the battery powered IoT devices.

Unlike JSON and XML, the data and context are separated in Protobuf. Context is defined in the configuration files called proto files (.proto). These files contain field names along with their types and identifiers (eg. string first_name = 1; string surname = 2;), based on the configuration fields Protobuf data is being serialized. The proto files can be compiled to generate code in the user’s selected programming language – a class with setters and getters for all the fields defined in the proto file. Currently, Google provides a code generator for multiple languages including C++, C#, Dart, Go, Java and Python under an open source license.

In order to compile the proto file to a Python class, you will require a protobuf compiler. If you don’t have the compiler yet, download the compiler from its official Github repository: https://github.com/protocolbuffers/protobuf/releases/tag/v3.17.1. You need to download the compiler dedicated for the operating system you are using (eg. for Windows  download protoc-3.17.1-win64.zip and unzip it).

Once you have the protbuf compiler, download the proto files, unzip the folder and place the files in the same directory as the protobuf compiler (if you are using Windows …/protoc-3.17.1-win64/bin/). Run the protocol buffer compiler protoc on your .proto files – open a terminal window and enter:

protoc --python_out=.. proto_measurement_types.proto
protoc --python_out=.. proto_measurements.proto

This will generate:

  • proto_measurement_types_pb2.py”,
  • “proto_measurements_pb2.py”,

in your specified destination directory. Move the files to your project directory (your_python_project/protos/protos_files). Note! The repository with the sample code already contains classes resulting from compiling the proto files. If you use the sample code, you can skip this step.

NB-IoT sensor configuration 

Configuration of Efento sensors is done with a free mobile application for Android. Application can be downloaded from Google Play. Once you download and install the application select “Nearby sensors” mode and unlock the power user mode: open the application menu and quickly tap the Efento logo five times.

Before you start the configuration, make sure the sensor is able to register in the NB-IoT network and the APN settings are right and the APN you use allows the device to connect to the server. Detailed user manual of Efento NB-IoT sensors and Efento mobile application can be found in the support section of our website.

Using the mobile application, connect to the sensor -> click on the menu (three dots in the upper right corner) -> Cellular network status. Mare sure that the field “Registration status” value is either “REGISTERED” or “REGISTERED_ROAMING”

Set the sensor to send the data to your server. Connect to the sensor -> click on the menu (three dots in the upper right corner) -> Power user -> Server configuration. Select “Other” and fill in the IP address and the port of the server. Note! The IP address used for setting up the CoAP server must be a static, public IP. If you are running the script on your computer, make sure you set the port forwarding on your router right.

Python code

The code below can be downloaded from Efento’s GitHub repository

import base64
import datetime
import asyncio
import os.path
​
import aiocoap.resource as resource
import aiocoap
from protobuf import proto_measurements_pb2
from google.protobuf.json_format import MessageToDict
import psycopg2
​
# Enter your database host, database user, database password and database name
DATABASE_HOST = 'host_name';
DATABASE_USER = 'database_user';
DATABASE_PASSWORD = 'database_password';
DATABASE_NAME = 'database_name';
​
#  Initial database connection:
conn = psycopg2.connect(
    dbname=DATABASE_NAME,
    user=DATABASE_USER,
    host=DATABASE_HOST,
    password=DATABASE_PASSWORD
)
​
​
class Measurements(resource.Resource):
​
    def __init__(self):
        super().__init__()
​
    async def render_post(self, request):
​
        # Creating a dictionary from a message received from a sensor
        data = [MessageToDict(proto_measurements_pb2.ProtoMeasurements().FromString(request.payload))]
        record = []
      
        # iterating through 'data'
        for measurement in data:
​
            for param in measurement['channels']:
                # iterating through lists data/measurement/channels/sampleOffsets and creating a list of sensor parameters (measured_at,serial_number, battery_status) and measurements
                try:
                    for sampleOffset in param['sampleOffsets']:
                        record.extend([(datetime.datetime.fromtimestamp(param['timestamp']),
                                        base64.b64decode((measurement['serialNum'])).hex(),
                                        measurement['batteryStatus'],
                                        param['type'], (param['startPoint'] + sampleOffset) / 10)])
                except:
                    print("")
​
                measurements = "INSERT INTO measurements(measured_at, serial_number, battery_ok, type, value) VALUES (%s, %s, %s, %s, %s)"
                with conn.cursor() as cur:
                    try:
                        # inserting a list of sensor parameters and measurement into the table in PostgresSQL database
                        cur.executemany(measurements, record)
                        conn.commit()
                        cur.close()
                    except (Exception, psycopg2.DatabaseError) as error:
                        print(error)
        # returning ACK to the sensor
        return aiocoap.Message(mtype=aiocoap.ACK, code=aiocoap.Code.CREATED,
                               token=request.token, payload="")
​
​
def main():
    # Resource tree creation
    root = resource.Site()
    # Set up "m" endpoint, which will be receiving the data sent by Efento NB-IoT sensor using POST method.
    root.add_resource(["m"], Measurements())
    # Starting the application on set IP address and port.
    asyncio.Task(aiocoap.Context.create_server_context(root, ("192.168.120.132", 5683)))
    # Getting the current event loop and running until stop() is called.
    asyncio.get_event_loop().run_forever()
​
​
if __name__ == '__main__':
    # Getting the current event loop and running until complete main()
    asyncio.get_event_loop().run_until_complete(main())

Results

When you run the script, all the data coming from the sensor will be saved in the database. To view the measurements open pgAdmin 4,  select your database, then open Tools > Query Tools.

Enter the request below into the Query Editor and select Execute (▶) :

SELECT * FROM measurements;

Data coming from Efento NB-IoT sensor is saved in the database

Moving on – adding two way communication

All parameters of Efento NB-IoT sensors can be changed remotely, from a server. The new configurations or server requests are sent in the responses (ACK) to the messages sent by the sensor.  

We are going to modify the script from the example above to add a “device info” request sent from the server to the sensor. Device info is used by the sensor to send detailed information about the sensor’s operations and radio-related statistics to the server. In order to increase the battery lifetime, this information is sent by the sensor only at the server’s request. The server can request “device info” by sending a proper flag in the response to any of the sensor’s confirmable message.Once the server receives the device info frame from the sensor, it will parse it and log to a file.

Device info message sent by Efento sensor contains information about:

  • serial number, 
  • software version and commit ID,
  • runtime information:
    •     uptime,
    •     number of messages sent,
    •     processor temperature,
    •     the lowest battery voltage,
    •     processor temperature during the lowest battery voltage,
    •     battery reset timestamp,
    •     max processor temperature,
    •     min processor temperature,
    •     runtime errors,
  • modem information:
    • modemy type and version,
    • network / signal related parameters (CMSA, CMSA, NCMSA, MSS, SP, TP, TxP, C.ID, ECL, SNR, EARFCN, PCI, RSRQ, UPLINK, DOWNLINK),
  • memory usage statistics,

How does it work?

The CoAP server is constantly listening for data sent by Efento NB-IoT sensors. Once a new message arrives, the server parses the data, saves it in the PostgreSQL database and responds to the sensor with request device info and confirmation that the message has been received (code  2.01 “CREATED”). The response sent by the server will also contain a “Device info flag” – once the sensor receives it, it will send a second message with the detailed statistics (device info). The same approach can be applied to change the sensor’s settings.

Compiling Protocol Buffers

On top of the protobufs used in the first example, we will need two additional proto files: “proto_device_info.proto” – used to deserialize the device info frames and “proto_proto_config.proto” – used to send the server’s request to the sensor. Run the protocol buffer compiler protoc on your .proto files – open a terminal window and enter:

protoc --python_out=.. proto_device_info.proto
protoc --python_out=.. proto_proto_config.proto

This will generate:

  • ”proto_device_info_pb2.py”,
  •  and “proto_config_pb2.py”

in your specified destination directory. Move the files to your project directory (your_python_project/protos/protos_files). Note! The repository with the sample code already contains classes resulting from compiling the proto files. If you use the sample code, you can skip this step.

Python code

The code below can be downloaded from Efento’s GitHub repository

import base64
import datetime
import asyncio
import os.path
​
import aiocoap.resource as resource
import aiocoap
from protobuf import proto_measurements_pb2
from protobuf import proto_device_info_pb2
from protobuf import proto_config_pb2
from google.protobuf.json_format import MessageToDict
import psycopg2
​
# Enter your database host, database user, database password and database name
DATABASE_HOST = 'host_name';
DATABASE_USER = 'database_user';
DATABASE_PASSWORD = 'database_password';
DATABASE_NAME = 'database_name';
​
# Making the initial connection:
conn = psycopg2.connect(
    dbname=DATABASE_NAME,
    user=DATABASE_USER,
    host=DATABASE_HOST,
    password=DATABASE_PASSWORD
)
​
​
class Measurements(resource.Resource):
​
    def __init__(self):
        super().__init__()
​
    async def render_post(self, request):
​
        # Creating a dictionary from a message received from a sensor
        data = [MessageToDict(proto_measurements_pb2.ProtoMeasurements().FromString(request.payload))]
        record = []
        # Set request_device_info to true
        device_config = proto_config_pb2.ProtoConfig()
        device_config.request_device_info = True
        # Serializing device config.
        response_payload = device_config.SerializeToString()
​
        # iterating through 'data'
        for measurement in data:
​
            for param in measurement['channels']:
                # iterating through lists data/measurement/channels/sampleOffsets and creating a list of sensor parameters (measured_at,serial_number, battery_status) and measurements
                try:
                    for sampleOffset in param['sampleOffsets']:
                        record.extend([(datetime.datetime.fromtimestamp(param['timestamp']),
                                        base64.b64decode((measurement['serialNum'])).hex(),
                                        measurement['batteryStatus'],
                                        param['type'], (param['startPoint'] + sampleOffset) / 10)])
                except:
                    print("")
​
                measurements = "INSERT INTO measurements(measured_at, serial_number, battery_ok, type, value) VALUES (%s, %s, %s, %s, %s)"
                with conn.cursor() as cur:
                    try:
                        # inserting a list of sensor parameters and measurement to table in PostgresSQL
                        cur.executemany(measurements, record)
                        conn.commit()
                        cur.close()
                    except (Exception, psycopg2.DatabaseError) as error:
                        print(error)
        # returning "ACK" and response payload to the sensor
        return aiocoap.Message(mtype=aiocoap.ACK, code=aiocoap.Code.CREATED,
                               token=request.token, payload=response_payload)
​
​
class DeviceInfo(resource.Resource):
​
    def __init__(self):
        super().__init__()
​
    async def render_post(self, request):
        # Creating a dictionary from a message received from a sensor
        data = MessageToDict(proto_device_info_pb2.ProtoDeviceInfo().FromString(request.payload))
        data["serialNum"] = base64.b64decode((data['serialNum'])).hex()
        # Create the file "Deviceinfo.txt" and save the data in this file
        if not os.path.isfile("Deviceinfo.txt"):
            file = open("Deviceinfo.txt", 'x')
        else:
            file = open("Deviceinfo.txt", 'w')
        file.write(str(data))
        file.close()
        # returning "ACK" to the sensor
        return aiocoap.Message(mtype=aiocoap.ACK, code=aiocoap.Code.CREATED,
                               token=request.token, payload="")
​
​
def main():
    # Resource tree creation
    root = resource.Site()
    # Set up "m" endpoint, which will be receiving the data sent by Efento NB-IoT sensor using POST method.
    root.add_resource(["m"], Measurements())
    # Set up "i" endpoint, which will be receiving the data sent by Efento NB-IoT sensor using POST method.
    root.add_resource(["i"], DeviceInfo())
    # Starting the application on set IP address and port.
    asyncio.Task(aiocoap.Context.create_server_context(root, ("192.168.120.132", 5683)))
    # Getting the current event loop and  running until stop() is called.
    asyncio.get_event_loop().run_forever()
​
​
if __name__ == '__main__':
    # Getting the current event loop and running until complete main()
    asyncio.get_event_loop().run_until_complete(main())

Results

When you run the script, all the data coming from the sensor will be saved in the database. To view the measurements open pgAdmin 4,  select your database, then open Tools > Query Tools.

Enter the request below into the Query Editor and select Execute (▶) :

SELECT * FROM measurements;

On top of that, after each confirmable message, the sensor will send a “Device Info” message which will be saved in the “Deviceinfo.txt” file


By continuing to use the site, you agree to the use of cookies. more

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close