InfluxData Blog - David G. Simmons

Integrating Particle.io with InfluxDB Cloud

David G. Simmons (InfluxData) — Fri, 28 Feb 2020 07:00:56 -0700

If you followed the tutorial I presented a couple of years ago about integrating Particle.io with InfluxDB and were unhappy, or simply couldn’t get it working, have I got a treat for you! Integrating Particle.io with InfluxDB Cloud is very straightforward and requires no outside services or customizations outside of what Particle Cloud already offers. Here are the steps to get it all working:

Go to your Particle.io Console and click on the Integrations tab.
Click on New Integration, and choose Webhook.
Fill out the form! You need to know your event name, and it needs to be spelled exactly the same as it is in your on-device code!
You'll then enter the URL of your InfluxDB Cloud instance. It might be different depending on the region you signed up for, which can be found on Load Data > Client Libraries Page: https://<your region>.<aws or gcp>.cloud2.influxdata.com/api/v2/write?org=YOUR_ORG&bucket=YOUR_BUCKET&precision=s

You will need to know the name of your Organization and Bucket as well, which you can grab from the InfluxDB Cloud management interface.

Next you’ll want to choose ‘Custom Body’ as your request format, and then expand the ‘Advanced’ setting as well. In there, you will insert the following:

You can set whatever tags, if any, that you’d like, and then use the ‘mustache’ notation to reference the Event Name and the Event Value, which will be used as the value name and value in InfluxDB.

Finally, you will need to add the Authorization Header exactly as below. The Authorization Header should have the value ‘Token long-token-string-from-InfluxDBCloud’ in it. This is the exact same header needed when accessing any of the InfluxDB Cloud APIs. You can then save your integration, and you’re all set.

Once you save your integration, you will see a page that tells you all the details of it:

And below that, it even gives you some sample on-device code that will trigger the integration:

And that’s it! So simple! Try it out and let us know what you think in our Community Slack.

The World's Smallest InfluxDB Server

David G. Simmons (InfluxData) — Thu, 13 Feb 2020 11:19:10 -0700

I’ve built a lot of InfluxDB servers in my time here, and I’ve built some pretty esoteric ones at that, but I think I’ve finally pulled off what can only be described as the World’s Smallest InfluxDB Server! Back in the summer of 2019, I saw a project on CrowdSupply.com for something called the ‘Giant Board’. It looked really, really cool! A complete Single Board Computer (SBC) that ran Linux, all in a Feather form factor. I immediately backed it! Then, thinking about it for a second, I backed it again! So I got 2 of these things. Total accident, I swear.

Some specs. Here’s what the Giant Board actually is:

Giant Board Specs:

Processor: Microchip SAMA5D2 ARM® Cortex®-A5 Processor 500 MHz
Memory: 128 MB DDR2 RAM
Storage: microSD card
Sensing: 6 x 12-bit ADC with 3.3 V reference and external trigger
Actuation: 4 x 16-bit PWM with external trigger
Connectivity: 1 x I²C, 1 x SPI, 1 x UART, more with Flexcom
Power: via USB, with support for LiPo batteries
Operating System: mainline Linux kernel

All in this tiny form-factor that I usually use for Microcontrollers!

Well, a few weeks ago, they arrived! So, some unboxing pictures:

Wait, the whole thing fits in that one little bag? Yes. Not only that — there are multiple parts in that bag! And I realize (now) that there’s no scale to that bag in the picture, but that’s a Mac Mouse next to it. I’d retake the picture, but I already threw the bags away!

That’s right: I ordered the WiFi Feather Wing add-on because, after all, what’s an IoT board — and especially what’s an InfluxDB server — without networking!

And just for scale, that’s a US Quarter in between them. So that’s going to be my server and my network interface! For something called the ‘Giant Board’ it sure is small!

After a little soldering, I had it all put together and, with the addition of a MicroSD card, it booted up!

Woah! This tiny little thing runs Debian Linux? Why yes, yes it does! So that makes installing InfluxDB super simple since we already ship ARMv7 binaries for InfluxDB and all the rest of the TICK Stack!

Ok, sure, it’s currently hammering this tiny CPU, but it is also running!

And as you can see, the dashboard runs in Chronograf to give me a view into how the whole thing is running!

Now, would I run this as a production system? Absolutely not! As you can see, it is using far too many system resources on such a small device. Would I run it as an edge collection and forwarding device? Quite possibly. Would I run it as an embedded Telegraf agent device? 100% absolutely. Especially since I can also do this with it:

That’s right, I can run it on a battery! A rechargeable LiPo battery at that (and the recharging circuitry is built in to the board!). And for completeness, I’ll provide a pin-out here so you can see what else I can add to this thing — sensors, actuators, etc., — to make it both a data node and a sensor node:

Next I will build a version of InfluxDB 2.0 for ARMv7 and see if that runs either better or worse on this device, so watch this space to see what happens!

Using Flux to Get IoT Sensor Metadata from MySQL

David G. Simmons (InfluxData) — Thu, 05 Dec 2019 07:00:49 -0700

If you’ve deployed an IoT solution, you’ve had to decide where, and how, to store all your data. At least from my perspective, the best and easiest place to store the sensor data is, of course, InfluxDB. My saying that can’t come as a surprise to you. But what about the other data you need to store? The data about the sensors? Things like the sensor manufacturer, the date it was placed into service, the customer ID, what kind of platform it’s running on. You know, all the sensor metadata.

One solution, of course, is to simply add all that stuff as tags to your sensor data in InfluxDB and go on about your day. But do you really want to store all your sensor data with each datapoint? Lots of things seem like a good idea at the time, but then rapidly devolve into a terrible idea when reality hits. Since most of this metadata doesn’t change often, and may also be associated with customer information, the best place for it is very likely in a traditional RDBMS. Most likely you already have an RDBMS with customer data in it, so why not just continue to leverage that investment? As I’ve said repeatedly, this is not the best place for your sensor data. So now you’ve got your IoT data in two different databases. How do you access it and merge it into one place where you can see it all?

Flux is the answer

Tell me you saw that coming. You had to have seen that coming. Ok, to be fair, you may have because, after all, how are you going to get your SQL-based data via Flux? That’s the beauty of Flux: it’s extensible! So we now have an extension that allows you to read data from either MySQL, MariaDB or Postgres via Flux. When I heard that this SQL connector was ready to go, I just had to try it. I’ll show you what I built, and how.

Build a customer database

The first thing to do was build a MySQL database with some Customer Information. I created a new database called IoTMeta into which I put a table with some sensor metadata. I also added another table with customer information about those sensors.

Pretty basic tables, really. The Sensor_ID field I populated with data corresponding to the Sensor_id tag in my InfluxDB instance. I bet you can see where I’m going with this already. I added a bunch of data:

So now my sensor metadata database has some information about each sensor I’m running here, as well as some ‘customer data’ about who owns the sensors. Now it’s time to pull this all into something useful.

Query the data with Flux

First, I built a query in Flux to get some of my sensor data, but I wasn’t actually interested in the sensor data itself. I was looking for an identifying Tag value: Sensor_id. This query will look a little strange, but it will make sense in the end, I promise.

temperature = from(bucket: "telegraf") 
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop) 
  |> filter(fn: (r) => r._measurement == "temperature" and (r._field == "temp_c")) 
  |> last() 
  |> map(fn: (r) => { 
       return { query: r.Sensor_id } 
       }) |> tableFind(fn: (key) => true) |> getRecord(idx: 0)

It returns a table of one row, and then pulls out the Sensor_id tag, and that’s where you’re probably saying “Whaaaat?” Remember: Flux returns everything in tables. What I need is essentially a scalar value out of that table. In this case, it’s a string value for the Tag in question. That’s how you do that.

Next, I’m going to get the username and password for my MySQL database, which is conveniently stored in the InfluxDB Secrets Store.

uname = secrets.get(key: "SQL_USER") 
pass = secrets.get(key: "SQL_PASSW")

Wait, how did I get those values into this Secrets Store anyway? Right, let’s back up a minute.

curl -XPATCH http://localhost:9999/api/v2/orgs/<org-id>/secrets -H 'Authorization: Token <token>' -H 'Content-type: application/json' --data '{ "SQL_USER": "<username>" }'

One thing to note is that you get the <org-id> out of your URL. It’s not the actual name of your organization in InfluxDB. Then you do the same thing for the SQL_PASSW secret. You can call them anything you want, really. Now you don’t have to put your username/password in plain text in your query.

Next, I’m going to use all of that to build my SQL Query:

sq = sql.from(
  driverName: "mysql",
  dataSourceName: "${uname}:${pass}@tcp(localhost:3306)/IoTMeta",
  query: "SELECT * FROM Sensor_data, Customer_Data WHERE  Sensor_data.Sensor_ID = ${"\""+temperature.query+"\"  AND Sensor_data.measurement = \"temperature\" AND Sensor_data.CustomerID = Customer_Data.Customer_ID"}" //"SELECT * FROM Sensor_data WHERE Sensor_ID = ${"\""+temperature.query+"\" AND measurement = \"temperature\""}" //q //  humidity.query //"SELECT * FROM Sensor_Data WHERE Sensor_ID = \"THPL001\""// humidity.query 
)

You’ll see that I’m using the value from my first Flux query in the SQL query. Cool! You might also notice that I am performing a join in that SQL query so that I can get data from both tables in the database. How cool is that? Next, I’ll format the resulting table to have just the columns I want to display:

fin = sq
|> map(fn: (r) => ({Sensor_id: r.Sensor_ID, Owner: r._Sensor_owner, Manufacturer: r.Sensor_mfg, MCU_Class: r.MCU_class, MCU_Vendor: r.MCU_vendor, Customer: r.Customer, Address: r.Address, Phone: r.phone}))

I’ve now got a table that contains all the metadata about my sensor, as well as all the customer contact data about that sensor. It’s time for some magic:

What is this sorcery? I have a table that has all the metadata about the sensor, some customer data, and the sensor readings too? Yep. I do. And here’s the really magic thing: Since you can get data from both SQL databases and InfluxDB buckets, you can also join that data together into a single table.

Here’s how I did that:

temp = from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "temperature" and (r._field == "temp_c"))

Gets me a table of the sensor data. I already have a table of the metadata from SQL, so …

j1 = join(tables: {temp: temp, fin: fin}, on: ["Sensor_id"] )
|> map(fn: (r) => ({_value: r._value, _time: r._time, Owner: r.Owner, Manufacturer: r.Manufacturer, MCU_Class: r.MCU_Class, MCU_Vendor: r.MCU_Vendor, Customer: r.Customer, Address: r.Address, Phone: r.phone}))
|> yield()

I just join those two tables on a common element (the Sensor_id field) and I have a table that has everything in one place!

There are any number of ways that you can use this ability to merge data from different sources. I’d love to hear how you would implement something like this to better understand your sensor deployments.

I’ve done all this using the Alpha18 build of InfluxDB 2.0, which is what I run — actually I custom-build my version from the master because I have some additions to Flux that I use, but that’s a whole other post. For this stuff, the Alpha builds of OSS InfluxDB 2.0 work just fine. You should absolutely give it a try!

Connecting The Things Network to InfluxDB

David G. Simmons (InfluxData) — Thu, 10 Oct 2019 09:25:17 -0700

There are many ways to connect your sensors to the network in the IoT. For short-range connections, there is Bluetooth LE, or Zigbee, or 802.15.4, or ZWave. For longer distances (though still fairly short) there’s always WiFi. But when you need longer distances, sometimes very long distances, there’s LoRaWAN. It’s a sub-gigahertz set of frequencies that are available for small bits of data. These are typically only a few bytes of data but can be sent over much longer distances — up to 2 km or more in some instances! They are very low-power, so they are great for remote-sensing applications.

In order to test out some LoRaWAN data transmission, and see how hard it might be to get that data into InfluxDB, I decided to move one of my sensors, a temperature/humidity/pressure sensor, to The Things Network (TTN), a community-based LoRaWAN provider. I wasn’t sure how hard, or easy, this transition might be, but I was able to complete it in less than a day! So here’s how you can do it too.

The hardware

First of all, you will need to make sure that either you have a TTN Gateway yourself, or that there is one in your area. As you can see, there are a lot of gateways available.

There wasn’t one close enough, so I put up my own (Tip: These are not cheap to buy — mine cost me >$200) so you can see me on the map now:

There are lots of tutorials on how to set up a gateway, so I’m not going to cover that here.

Next, you’ll need a LoRA radio for your sensor. I happened to have an Adafruit Feather M0 board lying around (I have a lot of random pieces of hardware just ‘lying around’), so I got an Adafruit LorRa Featherwing for it and put it on. Lastly, I used a BME280 breakout board (again from Adafruit. They really ought to sponsor me!) to collect the data and I was ready to go.

Wiring everything up takes a minute, so I’ll give you the details on how I wired mine. The first thing to note is that with the LoRaWAN Featherwing, you must do additional soldering. You can see below how I had to solder jumpers in from IRQ, CS, RST, DIO1 and DIO2. These then map to pins on the M0 Feather, which we will see in the software section. If you wire these jumpers up differently, you will need to adjust the pin settings in your software accordingly.

You can also see some small red wires coming in from off-screen (I love this ceramic-coated wire, even if it is a pain to solder). Those come from the BME280 Breakout board and go to the I2C pins and the 3v/ground on the board to power the sensor. Once all that is wired up, it’s down to software!

The software

Software for this took me a minute to get working, but most of that had to do with the difference in how you send data over LoRaWAN. I’m used to using BLE or WiFi, so the size of the data packets really doesn’t matter that much. With LoRaWAN, the size of the data packets reigns supreme.

The first thing you’ll want to do is install the right library for your Arduino. I used the MCCI LoRaWAN LMIC Library. It seemed the easiest for integrating with TTN. Some of the documentation on this library was a bit less than clear (at least for me) so I’ll give you the specifics of what I did to get this running on an M0 Feather. From there, I started with the ttn-otaa-feather-us915 sample program. Now, to fill in the pieces. You’ll need to go to your TTN Console and create a new Application.

Once you’ve registered that application, you will need to get the Application EUIS (ID) to paste into your Arduino Sketch. It’s important to note that, by default, the TTN console gives you your EUIS with most significant bit first (big-endian) whereas the Arduino sketch expects it in little-endian. Luckily, the TTN console makes all of that easy:

As you can see, it even makes copying it into a byte-array simple.

static const u1_t PROGMEM APPEUI[8] = { 0xB2, 0x38, 0x02, 0xD0, 0x7E, 0xD5, 0xB3, 0x70 };
void os_getArtEui (u1_t* buf) {
  memcpy_P(buf, APPEUI, 8);
}

So, that’s the APP ID. Next you’ll do the same thing for your Device ID and your App Key. Click to register a new device, and you can get access to all the needed information from there:

// This should also be in little endian format, see above.
static const u1_t PROGMEM DEVEUI[8] = { <insert Dev Key };
void os_getDevEui (u1_t* buf) {
  memcpy_P(buf, DEVEUI, 8);
}

// This key should be in big endian format (or, since it is not really a
// number but a block of memory, endianness does not really apply). In
// practice, a key taken from the TTN console can be copied as-is.
static const u1_t PROGMEM APPKEY[16] = { <insert Program Key };
void os_getDevKey (u1_t* buf) {
  memcpy_P(buf, APPKEY, 16);
}

And don’t worry about me publishing those IDs. I created that dummy application and device just for this blog post, and they are long-gone now. But as a reminder, never publish your IDs or keys like this.

You will need to adjust this data buffer for your data, but here’s what I used: unsigned char mydata[11]; Remember, I said that the data transmitted was intentionally kept very low, so I’m packing a whole lot of data into this 11 bytes! We’ll see how I do that in a bit.

Next comes the pins. Remember from the hardware section? If you wired your LoRaWAN feather exactly like mine, this should work for you.

#if defined(ARDUINO_SAMD_FEATHER_M0) || defined(ADAFRUIT_FEATHER_M0)
// Pin mapping for Adafruit Feather M0 LoRa, etc.
const lmic_pinmap lmic_pins = {
  .nss = 5,
  .rxtx = LMIC_UNUSED_PIN,
  .rst = 6,
  .dio = {9, 10, 11},
  .rxtx_rx_active = 0,
  .rssi_cal = 8,              // LBT cal for the Adafruit Feather M0 LoRa, in dB
  .spi_freq = 8000000,
};

Those were a little hard to figure out, and in the sample code it leaves a couple of those dio pins as LMIC_UNUSED_PIN but mine wouldn’t work until I defined them all.

For the rest of my code, I used some boiler-plate I have for the BME280:

Adafruit_BME280 bme;
double temperature = 0.00;
double pressure = 0.00;
double altitude = 0.00;
double humidity = 0.00;
bool bme_config = true;

// this goes in the setup() function:
int tryInit = 0;
  while (!bme.begin()) {
    Serial.println("Could not find a valid BME280 sensor, check wiring!");
    delay(3000);
    if (++tryInit > 9) {
      bme_config = false;
      break;
    }
  }

// a function to get readings:
void getReadings() {
  if (bme_config) {
    temperature = bme.readTemperature();
    pressure = bme.readPressure() / 100.0F;
    altitude = bme.readAltitude(SEALEVELPRESSURE_HPA);
    humidity = bme.readHumidity();
    Serial.print("Temp:     "); Serial.println(temperature);
    Serial.print("Humidity: "); Serial.println(humidity);
    Serial.print("Pressure: "); Serial.println(pressure);
    Serial.print("Altitude: "); Serial.println(altitude);
  }
}

Now, to get the data stuffed into 11 bytes! You’ll notice that in my boilerplate BME280 code I defined all the measurements as double which was fine for high-bandwidth applications, but it just won’t do for LoRaWAN. So I’m going to whittle them down to 2-bytes each (except the pressure measurement, which will stay at 4-bytes).

getReadings();
    uint16_t ft = (uint16_t)(temperature * 100);
    uint16_t fh = (uint16_t)(humidity * 100);
    uint32_t fp = (uint32_t)(pressure * 100);
    uint16_t fa = (uint16_t)(altitude * 100);

If I have readings like:

Temp: 25.04 Humidity: 54.60 Pressure: 1006.38 Altitude: 57.34

Then I’ll end up with:

Temp: 2502 Humidity: 5460 Pressure 100638 Altitude: 5734

All 16- and 32-bit integers. Now, to stuff them all into my data array:

mydata[0] = ft >> 8;
    mydata[1] = ft & 0xFF;
    mydata[2] = fh >> 8;
    mydata[3] = fh & 0xFF;
    mydata[4] = fp & 0xFF; 
    mydata[5] = (fp >> 8) & 0xFF;
    mydata[6] = (fp >> 16) & 0xFF;
    mydata[7] = (fp >> 24) & 0xFF;
    mydata[8] = fa >> 8;
    mydata[9] = fa & 0xFF;

If you’re not familiar with bit-wise data manipulation, basically I’m just moving each byte of each number into a spot in my byte-array. Since the pressure number is a 4-byte value, I have to do extra shifting. I can then send that off to TTN via LoRaWAN and my data transmission is completed.

Sadly though, we’re not done yet!. TTN will politely send all the data I send to it out to an MQTT broker for me so that I can subscribe to it and do with it as I please. (Spoiler alert: I’m going to put it into InfluxDB!)

Getting the data

My data is now coming into TTN via LoRaWAN, and is being written out to an MQTT broker for me, but how do I get at it? Well, the first thing to do is subscribe! I use an app called MQTT Box on my Mac to subscribe to various MQTT brokers to see data from different inputs. It allows me to define multiple brokers, and to subscribe to any number of topics from those brokers to see my data. To subscribe to the broker you need 3 pieces of information: The name/address of the broker, the username and the password to connect. For those of us in the US, the address of the broker is us-west.thethings.network . For your username, you will use the name of your application. In the example above, we’d use my-temp-app as the username. For the password, you’ll go to your application on the TTN Console, and look for the Application Key at the bottom of the page. Copy/paste that in to the password field for the broker and you should connect.

If I look at my data coming out to my MQTT broker, I immediately notice a problem: It’s just a random-looking string of characters. It’s actually not random at all — it’s your data buffer, base-64 encoded.

{ "app_id":"my-temp-app",
  "dev_id":"my-device",
  "hardware_serial":"009E9BA30C869232",
  "port":1,
  "counter":17,
  "payload_raw":"CdgVLRGJAQAWzg==",
}

Not super helpful. I did find a super helpful website that would help me translate that into something more meaningful. Go Crypti.com and paste in your raw, base-64-encoded data, and it will … translate it to Hexadecimal. Hmm … still not what I want to see. It turns out, in order to get the data in a usable form, you have to go back to your TTN console and click on the Payload Formats tab. From here, we will decode the Hex into something we can actually use.

Remember, we send an array of bytes. I pasted the base-64-encoded message into that website above, and got the following:

It decoded it into a series of bytes. Cool! Now to decode those bytes! (We’re almost there, I promise!)

On your TTN Consoles Payload Formats tab, we will enter the following function (it’s javascript!)

function Decoder(bytes, port) {
  var decoded = {};

  var cInt = (bytes[0] << 8) | bytes[1]; // temperature ºC
  var rem =(bytes[2] << 8) | bytes[3]; // humidity %
  var pre = (bytes[4]) + // pressure is a 4-byte value
  ((bytes[5]) << 8) 
      + ((bytes[6]) << 16) 
  + ((bytes[7]) << 24) ;
  var alt = (bytes[8] << 8) + bytes[9];
  
  

  // Decode integer to float
  decoded.temp_c = cInt / 100;
  decoded.humidity = rem / 100;
  decoded.pressure = pre / 100;
  decoded.altitude = alt / 100;

  return decoded;
}

I’ll step through what we’re doing here. If we look back at the data buffer we sent, you’ll remember that the first 2 bytes were the temperature. So we strip off those 2 bytes and store those in a temperature variable. We strip off the next 2 and that’s our humidity. We then need to grab the 4 bytes of the pressure, and finally the last 2 bytes for the altitude. Finally, we decode those back into their original floating-point state, and we’re done! Now if we look at what’s coming out of our MQTT broker, we will see:

{ "app_id":"my-temp-app",
  "dev_id":"my-device",
  "hardware_serial":"009E9BA30C869232",
  "port":1,
  "counter":28,
  "payload_raw":"CeEVJg6JAQAW7A==",
  "payload_fields":{
      "altitude":58.68,
      "humidity":54.14,
      "pressure":1006.22,
      "temp_c":25.29
  },
}

Which is a proper JSON object, with our data in a usable form! Now for the final bit: getting it all into InfluxDB!

Getting it into InfluxDB

Luckily, this is the easiest part of the whole thing thanks to Telegraf! On your telegraf host, edit your telegraf.conf file. Look for the section entitled Read metrics from MQTT topic(s) and add the following:

[[inputs.mqtt_consumer]]
  servers = ["tcp://us-west.thethings.network:1883"]
  qos = 0
  connection_timeout = "30s"
  topics = [ "+/devices/+/up" ]
  client_id = "ttn"
  username = "APP_NAMEr"
  password = "APPKEY"
  data_format = "json"

Then restart telegraf, and, like magic, you should be getting data into InfluxDB! If I go look in my data explorer in Chronograf, I should see a new measurement called mqtt_consumer and in there … Whoa!! A lot of data fields! It turns out that TTN provides a bunch of additional data about how the device connected and sent its data, and that is all preserved by the Telegraf plugin.

Your sensor data will have payload_fields_ prepended to it. All the rest is data about your data.

As usual, the easiest part of almost any deployment I do is the InfluxDB part. Once I had the data coming out of the MQTT broker in the proper format, having it stored in InfluxDB took just a few lines of configuration. I can now build a dashboard of my temperature, humidity, pressure and altitude data in Chronograf.

Hiding Complexity with Custom Functions: Calculating Heat Index

David G. Simmons (InfluxData) — Fri, 23 Aug 2019 07:00:22 -0700

Have you ever watched the local weather — or The Weather Channel — and heard them announce the temperature, and then the “Heat Index”? Or how hot it feels outside? At least here in the United States, and especially in the South, it’s part of every weather broadcast in the summer. But have you ever wondered how they actually calculate the Heat Index? Since I collect temperature and humidity data, I decided one day I’d go and figure it out. After all, how hard could it be really? Hint: never ask yourself that question!

The Heat Index calculation

As soon as I went looking for what I was sure was a simple calculation to convert a temperature and a humidity reading into a heat index (HI) I was sorry I’d headed down this road. I easily found a page where the National Oceanic and Atmospheric Administration (NOAA) lays out the formula. It’s something called the Rothfusz Regression equation, and it goes like this:

HI = -42.379 + 2.04901523T + 10.14333127RH - .22475541TRH - .00683783TT - .05481717RHRH + .00122874TTRH + .00085282TRHRH - .00000199TTRHRH

Where T is temperature and RH is relative humidity. Simple, right? Not so fast! It turns out there are a bunch of caveats to that (simple) equation. For instance, if the relative humidity is less than 13% and the temperature is between 80ºF and 112ºF, then you have to adjust the heat index by subtracting the following Adjustment from it:

ADJUSTMENT = [(13-RH)/4]*SQRT{[17-ABS(T-95.)]/17}

Ok, fine. But wait, there’s more! If the relative humidity is above 85%, and the temperature is between 80ºF and 112ºF, then you need to add the following Adjustment to it:

ADJUSTMENT = [(RH-85)/10] * [(87-T)/5]

Outstanding! We’re finally there … wait, no we’re not there yet. If the heat index is below 80ºF, then you don’t need to use that calculation to determine the heat index. You can use a more simple Steadman equation:

HI = 0.5 * {T + 61.0 + [(T-68.0)1.2] + (RH0.094)}

So first you calculate the heat index using that ‘simple’ equation – and, according to NOAA, you average that with the temperature reading. If that is less than 80ºF, you have your answer. If not, you need to proceed with the Rothfusz regression and add (or subtract) any adjustments, if required.

See? Simple, really.

Doing it in Flux

Here’s where some complicated Flux comes into play. You see, my temperature and humidity readings are in different measurements, so first there’s the task of building the tables, and joining them (yay for the beauty of cross-measurement math in Flux!).

humidity = from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "environment" and (r._field == "humidity"))
  |> aggregateWindow(every: 30s, fn: mean)
  |> fill(column: "_value", usePrevious: true)
  |> keep(columns: ["_value", "_time"])

temperature = from(bucket: "telegraf")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "temperature" and (r._field == "temp_f"))
  |> aggregateWindow(every: 30s, fn: mean)
  |> fill(column: "_value", usePrevious: true)
  |> keep(columns: ["_value", "_time"])
  |> yield(name: "Temperature")

That gets me my 2 tables of values. (Side note: I yield the Temperature table so that my graph will show both the temperature and the heat index on the same graph.)

Next, I join the 2 tables:

first_join = join(tables: {temperature: temperature, humidity: humidity}, on: ["_time"])
  |>map(fn: (r) => ({temperature: r._value_temperature, humidity:r._value_humidity, _time: r._time}))
  |> keep(columns: ["_time", "humidity", "temperature"])

So now I have a single table, aligned on time, with columns for temperature and humidity. All that’s left is to do the calculation:

|> map(fn: (r) => ({t: r.temperature, h: r.humidity}))
  |> map(fn: (r) => ({
     r with heatIndex:
        if ((0.5 * (r.t + 61.0 + ((r.t-68.0)*1.2) + (r.h*0.094)))/2.0) < 80.0 then (0.5 * (r.t + 61.0 + ((r.t - 68.0)*1.2) + (r.h*0.094)))
        else if ( r.h < 13.0 and r.t > 80.0) then ((-42.379 + 2.04901523*r.t + 10.14333127*r.h - .22475541*r.t*r.h - .00683783*r.t*r.h - .05481717*r.t*r.h + .00122874*r.t*r.t*r.h + .00085282*r.t*r.h*r.h - .00000199*r.t*r.t*r.h*r.h - (((13.0-r.h)/4.0)*math.sqrt(x: ((17.0-math.abs(x: (r.t-95.0))/17.0))))))
        else if r.h > 85.0 and r.t >= 80.0 and r.t <= 87.0 then ((-42.379 + 2.04901523*r.t + 10.14333127*r.h - .22475541*r.t*r.h - .00683783*r.t*r.h - .05481717*r.t*r.h + .00122874*r.t*r.t*r.h + .00085282*r.t*r.h*r.h - .00000199*r.t*r.t*r.h*r.h) + (( r.h-85.0 )/10.0) *((87.0-r.t)/5.0))
        else (-42.379 + 2.04901523*r.t + 10.14333127*r.h - .22475541*r.t*r.h - .00683783*r.t*r.h - .05481717*r.t*r.h + .00122874*r.t*r.t*r.h + .00085282*r.t*r.h*r.h - .00000199*r.t*r.t*r.h*r.h)
        })
  )
  |> map(fn: (r) => ({_value: r.heatIndex, _time: r._time}))
  |> yield(name: "HeatIndex")

I told you it was complicated Flux!

But in the end, you can then have a graph that shows the actual temperature and the calculated Heat Index in the same table cell:

And that’s a pretty cool thing!

Next step

Since that’s a lot of complicated Flux. The next step is to pull that all into Flux functions so that you can simply call |> HeatIndex() on your table with temperature and humidity columns and get back the proper table with the resulting heat indexes all calculated for you. So I’m in the process of adding the HeatIndex() function to Flux so you can have pretty Flux when you calculate your heat indexes.

I’m planning to add in the calculation for Ideal Gas Law (an easy one) and Wind Chill as well (another ugly one!) to this future package as well as converters from ºF to ºC as well, so if you have any other ideas for things that you would use for environmental calculations, please reach out and let me know!

How to Output Data from Flux to MQTT Natively

David G. Simmons (InfluxData) — Fri, 16 Aug 2019 07:00:58 -0700

Writing data from InfluxDB to MQTT using Flux

I started using the Open Source (OSS) version of InfluxDB v2.0 very early on in the Alpha releases. Even in the early releases, I was very enamored with the way things were shaping up. But as you know, I do a lot of IoT builds, and use InfluxDB for all of it, so there were a few things I needed it to do that it just didn’t, yet.

One of the things I have all my IoT Demos do is to write out alerts to an MQTT broker. I have other IoT Devices that read from that broker, and take actions based on what messages they receive. But InfluxDB 2.0 Alpha had no real output capabilities.

Note: There is an alerting framework that is coming to InfluxDB 2.0 very soon, but a) it wasn’t available then and b) I needed it now.

What to do? Well, Flux is an extensible language, so I decided to extend the language to write to MQTT. First, it’s important to note that Flux has 2 language constructs for reading and writing data: from() and to(). If you’ve written any Flux at all, you’ll recognize the from() syntax as being how you get data back from InfluxDB. The to() business is a bit harder. Built into the language is the ability to write back to InfluxDB, using the to() syntax. I also found a to() extension for http that allows you to write the results of your Flux query out to an http end-point. At least I now had a starting place!

Adding MQTT to Flux

I began poking around in the Flux code to see how the http to() method was implemented and quickly saw that it would be almost trivial to use this same framework for MQTT, so I copied all the code fro the http to() output and began working to move it over to MQTT. As with all of these things, it was a little less ‘trivial’ than I at first thought, but after a few weeks of on-again off-again work, I had a working output to MQTT from Flux!

First, I had to define what options the MQTT output would need, and I settled on a sort of default minimum set of options:

golang
type ToMQTTOpSpec struct {
    Broker string `json:"broker"`
    Name string `json:"name"`
    Topic string `json:"topic"`
    Message string `json:"message"`
    ClientID string `json:"clientid"`
    Username string `json:"username"`
    Password string `json:"password"`
    QoS int `json:"qos"`
    NameColumn string `json:"nameColumn"` // either name or name_column must be set, if none is set try to use the "_measurement" column.
    Timeout time.Duration `json:"timeout"` // default to something reasonable if zero
    NoKeepAlive bool `json:"noKeepAlive"`
    TimeColumn string `json:"timeColumn"`
    TagColumns []string `json:"tagColumns"`
    ValueColumns []string `json:"valueColumns"`
}

Of course, not all of those are required, but I’ll go through the ones that are.

First, of course, you need to define a Broker. This is the URL of the MQTT broker that you want to use. In your URL your broker should be identified as either tcp, ws or tls so tcp://mqtt.mybroker.com:1883 would be what it’s looking for. Most of the rest are, by and large, optional to a degree. If you supply a Username then you must also supply a password. You can’t have one without the other! Also, if you don’t supply a Topic then one will be created for you by stringing together all of the tags returned from your query. I’d advise giving a topic, as a topic of /tag1/tag_2/tag_3/... would be less than ideal in a lot of situations.

How to use this new thing?

I’m glad you asked! First, it’s not actually part of Flux just yet. I have submitted a PR, it has been accepted, but (as of this writing) has not been merged. If you want to build your own version of Flux in order to get your hands on this now then you’ll need to pull the branch and build from source. See the MQTT PR and go from there.

Once you’ve done that, the Flux code to start writing to an MQTT broker is actually trivial! You’ll want to create a Task in the InfluxDB 2.0 UI, and then you can paste in the following code:

import "mqtt"
from(bucket: "telegraf")
    |> range(start: -task.every)
    |> filter(fn: (r) =>
        (r._measurement == "cpu"))
    |> filter(fn: (r) =>
        (r._field == "usage_system"))
    |> filter(fn: (r) =>
        (r.cpu == "cpu-total"))
    |> last()
    |> mqtt.to(
        broker: "tcp://davidgs.com:8883",
        topic: "cpu",
        clientid: "cpu-flux",
        valueColumns: ["_value"],
        tagColumns: ["cpu", "host"],
    )

This will write the last CPU usage_system value to your MQTT broker. Using the UI, you can decide how frequently you want this data written.

A few things to note

It’s important to realize that Flux returns all of its data from queries as tables. The reason the above Task uses the last() function is to limit the returned value to a table with exactly one row. The MQTT to() function will write the whole table to the broker, as line-protocol. If your query returns a very large table, be prepared for your MQTT broker to get a very large table as the message payload.

In addition, if your query returns multiple tables, the MQTT to() function will write one message per table with each message containing an entire table. If this is not the behavior you want, you should think about how to craft your query such that it returns a single (preferably small) table as the results.

You may also have noticed an optional field Message above. If what you’d like is to send a pre-defined message instead of a table of results, you can define the message parameter in your call to to() and that message will be sent. So far, I’ve been using this for about 2 months with fantastic results! I’m able to control some IoT devices based on the readings from other IoT devices and it works great!

Update: The PR for this has now been merged into the master branch so it should be showing up in a release of Flux soon!

Publishing Data to InfluxDB from Swift

David G. Simmons (InfluxData) — Thu, 28 Mar 2019 13:32:26 -0700

I’ve been a very busy man. It was only a few days ago that I wrote about a new InfluxDB library for writing data from Arduino devices to InfluxDB v2 and here I am again, writing about a new library for writing data to InfluxDB. This time, it’s in Swift. Now your native Apple apps can write data directly to InfluxDB v2.0 with ease.

It’s a really simple library to use, and you can download the entire Xcode project for it from my GitHub. You can use it to write single data points to the DB, or to do bulk writes of any size. Here’s a quick tutorial on how to use it.

let influxdb = InfluxData()

That gets you an instance of the InfluxData class. Once you have that, you’ll need to set some configuration parameters for it.

influxdb.setConfig(server: "serverName", port: 9999, org: "myOrganization", bucket: "myBucket", token: "myToken")

You will, of course, need to set all those values according to your InfluxDB v2.0 server’s settings. You can also set the time precision with

let myPrecision = DataPrecision.ms // for Milliseconds, 'us' for microseconds, and 's' for seconds
influxdb.setPrecision(precision: myPrecision)

At this point, you’re ready to start collecting data and sending it to InfluxDB v2.0! For each data point you collect and want to store, you will create a new Influx object to hold the tags and data.

let point: Influx = Influx(measurement: "myMeasurement")
point.addTag(name: "location", value: "home")
point.addTag(name: "server", value: "home-server")
if !point.addValue(name: "value", value: 100.01) {
    print("Unknown value type!\n)
}
if !point.addValue(name: "value", value: 55) {
    print("Unknown value type!\n)
}
if !point.addValue(name: "value", value: true) {
    print("Unknown value type!\n)
}
if !point.addValue(name: "value", value: "String Value" {
    print("Unknown value type!\n)
}

As you can see, it accepts Integers, floating point values, Booleans and strings. If it cannot determine the data type, it will return the Boolean false so it’s always a good idea to check the return value.

For best performance, we recommend writing data in batches to InfluxDB, so you’ll need to prepare the data to go into a batch. This is easy to do with a call to

influxdb.prepare(point: point)

And when it’s time to write the batch, just call

if influxdb.writeBatch() {
    print("Batch written successfully!\n)
}

Again, writeBatch() returns a Boolean on success or failure, so it’s a good idea to check those values.

If you want to write each data point as it comes in, just take the data point you created above and call

influxdb.writeSingle(dataPoint: point)

You can write data to multiple measurements simultaneously as each data point is initialized with its measurement, and you can add as many tags and fields as you’d like.

This is really the first pass at the InfluxDB v2.0 Swift library as I’ll be adding the ability to query, create buckets, and a lot of other features of the Flux language to the library in the future, but since what most people want to do right away is write data to the database, I thought I’d get this out there.

I hope this is helpful! I know it has been for me! You see, I have lately been just using my Mac laptop to grab data off of my Bluetooth CO2 sensor that I built. In order to do that, I built a small BLE application that connects to the sensor, subscribes to the data ID, and constantly writes the data to InfluxDB. Needless to say, I used this library and have been scraping this data and storing it happily.

I’d love to hear what you plan to do with a Swift Library for 2.0 so be sure to follow me on twitter and let me know what you’re doing!

Writing Data from Arduino to InfluxDB v2

David G. Simmons (InfluxData) — Tue, 26 Mar 2019 09:01:46 -0700

As InfluxData moves ever closer to releasing v2.0, it’s becoming increasingly important to be able to get data into InfluxDBv2, of course. Makes sense, right? Since the vast majority (like, indistinguishable from 100%) of my data comes from IoT devices, I decided it was time to start making those devices InfluxDB v2-capable.

I’m happy to say that the first step in that direction is now complete! One of my favorite sensors is a particulate matter sensor that measures the amount of very small particulate in the air (from 2.5µM to 100µM in diameter). This stuff, it turns out, is really, really bad for you. So knowing how much is in the air is a good idea. To that end, I ordered one of these sensors from Adafruit.

It’s small and easy to hook up to pretty much anything since it just spews data out via UART. Since I have a giant pile of ESP8266 boards lying around (I typically order them by the dozens since they are so cheap and easy to deal with), I hooked it up to one of those. The code was simple, thanks to Adafruit providing it, and there was a handle InfluxDB library to write data with, but it only supported InfluxDB v1.x. The first thing I did (because I was in a hurry) was to grab the 1.x library and just re-write it for 2.x. This took me about 1/2 hour or less, and it worked great! (you can use that version here if you’d like). That really wasn’t the right solution though. So today I went back and created a proper fork of the original repository, and updated it to support either version 1.x or version 2.x of InfluxDB. I’ve of course submitted a proper Pull Request against the original library and hope that it will be accepted/merged soon.

Let’s walk through what it takes to use this new library then. It’s dead simple, really. At least with Arduino, all you have to do is add the Library, then include it in your sketch:

#include <InfluxDb.h>
//#include <InfluxDataV2.h> // if you want to use the other library I built and that's in my GitHub 
#define INFLUXDB_HOST "myhost.com"
Influxdb influx(INFLUXDB_HOST);

That gets you started. Next, you’re going to need some specific information from your InfluxDB v2.0 (alpha still!) installation. Notably, you will need the organization, bucket, and token that are associated with your account. You can find these by pointing your web browser at your InfluxDB server, port 9999, entering your username and password, and going to the Configuration Page.

You can then enter them into the Arduino Sketch:

influx.setBucket("myBucket");
influx.setVersion(2);
influx.setOrg("myOrg");
influx.setPort(9999);
influx.setToken("myToken");

Once you’ve done that, in your setup() function, you can start writing data to your v2.0 Influx server!

void loop() {
    loopCount++;
    InfluxData row("temperature");row.addTag("device", "alpha");
    row.addTag("sensor", "one");
    row.addTag("mode", "pwm");
    row.addValue("loopCount", loopCount);
    row.addValue("value", random(10, 40));
    influx.write(row);delay(5000);
}

See? I told you it was easy!

Practical Uses of Cross-Measurement Math in Flux

David G. Simmons (InfluxData) — Fri, 22 Mar 2019 08:00:18 -0700

I’ve been spending a lot of time lately with the 2.0 Alpha releases and I’m here to tell you: some of the new things coming are really, really cool! Especially for IoT. The one I’ve been using lately has been the ability to do math across measurements, which is really a game-changer for IoT data in InfluxDB.

Let’s look at why here for a minute. As you probably know, I’ve been building out a bunch of IoT sensors that stream data to various instances of InfluxDB. One of the main sensors I use is a SenseAir K-30 CO2 sensor. It’s a fabulously accurate sensor, though it’s not cheap. I’ve been using it for a year or so and it’s extremely reliable and accurate, but I was wondering about how the temperature and the pressure might affect the CO2 readings. What I found was, of course, these things do, in fact, matter.

I read up on it here, but essentially the temperature and atmospheric pressure affect the amount of gas inside the measuring chamber, not the concentration, so one should compensate for that. It’s a fairly simple calculation to do using some reference values like absolute zero and the pressure at sea level.

We’ll use the Ideal Gas Law formula:

ppm CO2 corrected = ppm CO2 measured * ((Temp measured * Pressure Reference ) / (Pressure measured * Temp Reference))

Great. Easy! Almost. I have a temperature/pressure sensor that stores values in an ‘environment’ measurement in my InfluxDB instance. My CO2 sensor stores its readings in a “k30_reader” measurement. If you’re not using Flux, you already see the problem here: These values live in different measurements, so I either can’t do this calculation or I have to do a fair amount of gymnastics to somehow rewrite all these values into a common measurement first. Neither is a really viable answer, is it?

Enter Flux and cross-measurement maths! So let’s walk through how I accomplished this in Flux (with a ton of help from fellow DevRel Anais, among others). A little refresher on how Flux returns values first. Let’s remember that when I submit a query in Flux, I get back a table of values. This becomes important as we walk through how the Flux query is built, so keep that in mind.

Tref = 298.15
Pref = 1013.25

Tmeas = from(bucket: "telegraf")
   |> range(start: v.timeRangeStart)
   |> filter(fn: (r) => r._measurement == "environment" and (r._field == "temp_c"))
   |> fill(column: "_value", usePrevious: true)
   |> aggregateWindow(every: 30s, fn: mean)
   |> keep(columns: ["_value", "_time"])

First I’ve defined my reference temperature and pressure values. The reference temperature is usually 25ºC, converted to Kelvin, and the reference Pressure is sea level. I then query values of the Measured Temperature Values and store it in a table called ‘Tmeas’. If I yield() on this table, I’ll see my temperature values:

I then repeat this for the CO2 and Pressure values:

CO2meas = from(bucket: "telegraf")
   |> range(start: v.timeRangeStart)
   |> filter(fn: (r) => r._measurement == "k30_reader" and (r._field == "co2"))
   |> fill(column: "_value", usePrevious: true)
   |> aggregateWindow(every: 30s, fn: mean)
   |> keep(columns: ["_value", "_time"])

Pmeas = from(bucket: "telegraf")
   |> range(start: v.timeRangeStart)
   |> filter(fn: (r) => r._measurement == "environment" and (r._field == "pressure"))
   |> fill(column: "_value", usePrevious: true)
   |> aggregateWindow(every: 30s, fn: mean)
   |> keep(columns: ["_value", "_time"])

Great! I now have 3 tables with all the values I need to do my calculations. All I have to do now is pull them all together, and I’ll do that through a series of join() statements:

first_join = join(tables: {CO2meas: CO2meas, Tmeas: Tmeas}, on: ["_time"])
    |> fill(column: "_value_CO2meas", usePrevious: true)
    |>fill(column: "_value_CO2meas", value: 400.00)
    |> fill(column: "_value_Tmeas",usePrevious: true)
    |>fill(column: "_value_Tmeas", value: 20.0)

This first join() gets me a table that contains the CO2 and temperature values, so I’m part way there.

second_join = join(tables: {first_join: first_join, Pmeas: Pmeas}, on: ["_time"])
   |>fill(column: "_value", usePrevious: true)
   |>fill(column: "_value", value: 1013.25)
   |>map(fn: (r) => ({_time: r._time, _Pmeas: r._value, _CO2meas:r._value_CO2meas, _Tmeas:r._value_Tmeas}))

This second join() gets me a table with all the measured values in it! You’d think I’d be done, but in order to do the final calculation, since I’m joining on time, I have to build a final table where I fill in the reference values for each row in the table.

final = second_join
    |>map(fn: (r) => ({Pmeas: r._Pmeas, CO2meas:r._CO2meas, Tmeas:r._Tmeas, Pref: Pref, Tref: Tref, _time: r._time,}))
    |> keep(columns: ["_time", "CO2meas", "Pmeas", "Tmeas", "Pref", "Tref"])

So now I can create a final table and calculate the compensated CO2 value, for each row (time).

CO2corr = final
    |> map(fn: (r) => ({"_time": r._time, "CO2-Measured": r.CO2meas, "CO2-Adjusted": r.CO2meas * (((r.Tmeas + 273.15) * r.Pref) / (r.Pmeas * r.Tref))}))
   |> yield()

Since this is my final table, I call yield() at the end so that the values will get displayed.

And now I have a graph that shows the raw, measured CO2 value and the compensated CO2 value! All done on-the-fly using Flux’s cross-measurement math!

I’ve actually been wanting to do this sensor compensation for a long time and I’m thrilled that I can now do it quickly and easily using Flux! So what will you do with cross-measurement math in Flux? Feel free to tweet me your ideas, solutions, etc. @davidgsIoT!