Bobbie Smulders Creative Software Developer

Serverless IoT with CouchDB and the ESP8266

WEMOS D1 mini Pro and DHT Shield

I’ve recently dipped into IoT and was overwhelmed with the options from companies such as Amazon, IBM and Microsoft. They have great solutions for big corporate projects but seem to need a lot of configuration and knowledge setting everything up. With my recent experience in CouchDB and serverless applications I decided to try something new: Serverless IoT. Is it possible to capture sensor data and plot the data on a chart with just an instance of a CouchDB database running?

Hardware

The hardware used for this project is the WEMOS D1 mini Pro with a WEMOS DHT Shield. The D1 mini Pro is based on an ESP-8266EX, the DHT shield is based on an DHT11. To enable the device from waking itself up from deep sleep, the GP16 pin (D0 on the WEMOS device) is connected to the RST pin. The final piece is a battery connected to the device so that everything can work wireless.

Environment sensor prototype

The firmware used for this device is NodeMCU. Building a firmware image can be done by using the NodeMCU custom builds tool. I included the following modules in my build:

dht, file, http, net, node, tmr, wifi, tls

TLS can be included by using the TLS option in the builder. It is necessary for certain CouchDB providers such as Cloudant.

The ESP8266 can be flashed using esptool.py. I used the following command to do so:

esptool.py --port /dev/cu.SLAB_USBtoUART erase_flash
esptool.py --port /dev/cu.SLAB_USBtoUART write_flash --flash_mode dio --flash_size 16m 0x00000 ../builds/nodemcu-master-8-modules-2017-06-11-12-54-27-integer.bin  

Using esplorer we can now upload scripts to the ESP8266. I prepared a Lua script that has the following flow:

  • Connect to WiFi
  • If successful, read DHT sensor
  • If successful, do HTTP POST request with sensor data to server
  • Sleep

The script does not have a while loop or repeat flow. It ends after sending the data. After waking up, the EPS8266 is reset and we automatically start at the beginning of the script.

CouchDB

To store the data, we need an instance of CouchDB. I use Cloudant as I prefer a cloud-based solution but it is also possible to run your own CouchDB instance from a Docker container.

Safety first, so use HTTPS where possible and create three user accounts:

  • admin, for administrative purposes (write and read permissions)
  • sensor, for storing sensor data (write permission only)
  • client, for reading sensor data (read permission only, doesn’t necessarily need a password)

Create a database named measurement. You would figure that saving the sensor data is as easy as posting it directly to this database doing an HTTP POST request from the ESP8266 but that does have a little problem.

The problem being that we need a timestamp on every sensor value. The EPS8266 could connect to an NTP server to get the current time but for a battery powered device that would be a very costly operation. Another approach is to use an update handler and let CouchDB append a timestamp to each document inserted in to the database.

To do so we add a Design Document with an Update Function to the measurement database. I have prepared one that adds a timestamp onto the request and saves the document to the database.

{
  "_id": "_design/update_handlers",
  "updates": {
    "add_with_timestamp": "function (doc, req) { var measurement = JSON.parse(req.body); measurement._id = new Date(); return [measurement, 'created']; }"
  }
}

By doing an HTTP POST request to the following URL we can add a document:

https://couchdb/measurement/_design/update_handlers/_update/add_with_timestamp

An example of a document we can post:

{
    "node": "living room",
    "temperature": 22,
    "humidity": 47
}

Client

For the client, I prepared an HTML/JavaScript page that uses PouchDB to query a remote CouchDB instance and plot it on a time-based line chart using Chart.js. We can use PouchDB to connect to the remote CouchDB instance directly as we don’t need any of the replication functionality. Doing this we still get data pushed to our client if a sensor posts data. In this example, we limit the dataset to the last 200 documents (readings) but we could also write a map function to get all data from the last 12 hours.

Screenshot of measurement chart

As stated earlier, I only want to run a CouchDB instance to manage the project. Luckily, we can serve the client from CouchDB by inserting the HTML page as an attachment. Yes, serving HTML straight from the database! As Nolan Lawson mentioned:

But for all its developers’ humility, CouchDB is a really exciting technology. When you step back and look at it, it’s a daring, crazy proposition, a bold statement about how awesome web development would be if we could just let it be the web. It’s a raving streetside lunatic, grabbing random people by the shoulders and screaming at them with frantic urgency: “We don’t need the server anymore! We only need the database! The database is the server!”

To serve the client as HTML from the database, create a design document in the measurement database using the provided JSON. After creating the document, edit it and add the client HTML file as an attachment.

{
  "_id": "_design/client"
}

The client is now accessible from the following URL:

https://couchdb/measurement/_design/client/chart.html

The last thing we need is a rewrite rule to make the URL prettier. CouchDB uses HTTP Rewrite Handler for redirecting requests to make it easier for the end user to type the URL of your application. Edit the design document we just created and add the rewrites property. Your final document should look like what is shown below.

{
  "_id": "_design/client",
  "rewrites": [
    {
      "from": "",
      "to": "chart.html",
      "method": "GET",
      "query": {}
    },
    {
      "from": "/measurement/*",
      "to": "../../../measurement/*"
    }
  ],
  "_attachments": {
    "chart.html": {
      "content_type": "text/html",
      "revpos": 32,
      "digest": "md5-sEJsbTcqnZIUqoleTnfpqg==",
      "length": 5042,
      "stub": true
    }
  }
}

The client is now available on:

https://couchdb/measurement/_design/client/_rewrite

The final URL is still a bit ugly, so to finish everything off we use Virtual Hosts. Add a virtual host for the URL above to forward requests. In my case I used a subdomain and added a CNAME record towards my Cloudant account.

Screenshot of Cloudant Virtual Host configuration

The client is now available at:

https://subdomain.website.com

The database API is available at:

https://subdomain.website.com/measurement

Wrapping it up

We ended up with a solution that uses CouchDB and CouchDB only for storing sensor data, retrieving data and displaying it on an HTML page. We used Design Documents, Update Functions, HTTP Rewrite and Virtual Hosts to achieve those tasks. Instead of the route of Client » Application Server » Database Server we go straight from the client to the database and put the business logic inside of it. We also serve the HTML from the database instead of a traditional HTTP server.

It’s a different way of thinking, very suitable for some projects but not for all.

A couple of interesting articles for further reading:

Attachments

init.lua

-- Wifi Settings
wifi_config = {}
wifi_config.ssid = "networkname"
wifi_config.pwd = "networkpassword"
wifi_config.auto = false

-- HTTP Settings
http_url = "https://couchdb/measurement/_design/update_handlers/_update/add_with_timestamp"
http_auth = "base64_encoded_credentials_here"

-- Application settings
sleep_seconds = 300
node_name = "living room"
dhtPin = 4

function connectWifi ()
    wifi.setmode(wifi.STATION)
    wifi.sta.config(wifi_config)

    -- Read sensor after receiving an IP, sleep otherwise
    wifi.eventmon.register(wifi.eventmon.STA_GOT_IP, function()
        readDHT()
    end)
    wifi.eventmon.register(wifi.eventmon.STA_DHCP_TIMEOUT, function()
        sleep()
    end)

    -- Connect to wifi
    wifi.sta.connect()
end

function readDHT()
    status, temp, humi = dht.read11(dhtPin)

    -- Publish sensor value if sensor read was succesful, sleep otherwise
    if (status == dht.OK) then
        publishSensorValues(temp, humi)
    else
        sleep()
    end
end

function publishSensorValues(temperature, humidity)
    -- Sleep after two seconds, prevents requests taking too long
    tmr.alarm(1, 2000, tmr.ALARM_SINGLE, function()
        sleep();
    end)
    
    -- Post sensor data to HTTP server, sleep when finished
    http.post(http_url,
        "Content-Type: application/json\r\n" ..
        "Authorization: Basic " .. http_auth .. "\r\n",
        [[
        {
            "node": "]] .. node_name .. [[",
            "temperature": ]] .. temperature .. [[,
            "humidity": ]] .. humidity .. [[
        }
        ]],
        function(code, data)
            sleep()
        end
    )
end

function sleep()
    -- Disconnect from wifi
    wifi.sta.disconnect()
    
    -- Go to deep sleep for the specified time
    node.dsleep(sleep_seconds*1000000)
end

connectWifi()

chart.html

<!DOCTYPE html>
<html style="height: 100%">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="initial-scale=1.0">
    <title>Environment measurements</title>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/pouchdb/6.2.0/pouchdb.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.18.1/moment.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.6.0/Chart.min.js"></script>
</head>
<body style="height: 100%; margin: 0">

<div style="height: calc(100% - 20px); padding: 10px; background-color: #FFF">
    <canvas id="temperature-chart"></canvas>
</div>

<script charset="utf-8">
    // Initialize PouchDB with remote instance
    var db = new PouchDB(window.location.href + '/measurement');

    // Initialize chart for Chart.js
    var ctx = document.getElementById('temperature-chart').getContext('2d');
    var chart = new Chart(ctx, {
        type: 'line',
        options: {
            responsive: true,
            animation: false,
            maintainAspectRatio: false,
            legend: {
                position: 'right'
            },
            scales: {
                xAxes: [{
                    type: "time",
                    time: {
                        unit: "hour",
                        displayFormats: {
                            hour: 'HH:mm'
                        }
                    },
                    gridLines: {
                        display: false,
                    }
                }],
                yAxes: [{
                    position: "left",
                    id: "y-axis-temperature",
                    gridLines: {
                        drawBorder: false
                    },
                    ticks: {
                        stepSize: 2,
                        min: 12,
                        max: 32,
                        callback: function(value) {
                            return value + ' \u2103'
                        }
                    }
                }, {
                    position: "right",
                    id: "y-axis-humidity",
                    gridLines: {
                        drawBorder: false
                    },
                    ticks: {
                        min: 0,
                        max: 100,
                        callback: function(value) {
                            return value + '%'
                        }
                    }
                }]
            }
        }
    });

    // Change handler
    var changes = db.changes({
        live: true,
        since: 'now'
    }).on('change', function(change) {
        updateDocs();
    });

    // Handler for updating the table
    var updateDocs = function() {
        console.log('update docs');
        db.allDocs({
            include_docs: true,
            descending: true,
            limit: 200
        }, updateChart);
    }

    // Build chart data
    var updateChart = function(err, result) {
        if (result.rows.length > 0) {
            var datasets = new Object();
            resetColor();

            for (var key in result.rows) {
                var item = result.rows[key];
                var doc = item.doc;
                var id = doc['_id'];
                var date = new Date(id);

                if (!id.startsWith("_design/")) {
                    getDataset(datasets, doc['node'], 'temperature').data.push({
                        x: date,
                        y: doc['temperature']
                    });
                    getDataset(datasets, doc['node'], 'humidity').data.push({
                        x: date,
                        y: doc['humidity']
                    });
                }
            }

            chart.data.datasets = Object.values(datasets);
            chart.update();
        }
    }

    // Get dataset, create one if it doesn't exist
    var getDataset = function(datasets, node, type) {
        var key = node + ' ' + type;

        if (datasets[key] == undefined) {
            datasets[key] = createDataset(node, type);
        }

        return datasets[key];
    }

    // Create a dataset using default values
    var createDataset = function(node, type) {
        var color = getColor();

        return {
            label: node + ' ' + type,
            yAxisID: 'y-axis-' + type,
            fill: 'false',
            backgroundColor: color,
            borderColor: color,
            pointRadius: 1,
            data: []
        };
    }

    // Color generator, palette by Stephen Few
    var colors = ["#4D4D4D", "#5DA5DA", "#FAA43A", "#60BD68", "#F17CB0", "#B2912F", "#B276B2", "#DECF3F", "#F15854"];
    var colorCounter = 0;

    var getColor = function() {
        colorCounter = (colorCounter == colors.length - 1) ? 0 : colorCounter + 1;
        return colors[colorCounter];
    }

    var resetColor = function() {
        colorCounter = 0;
    }

    // Initial update after loading the page
    updateDocs();

</script>

</body>
</html>