InfluxData Blog - Ben Tasker

Improving Our Solar Battery Savings

Ben Tasker (InfluxData) — Tue, 18 Feb 2025 07:00:00 +0000

This blog was originally posted on www.bentasker.co.uk

This project was built using a previous version of InfluxDB. InfluxDB 3 moved away from Flux and a built-in task engine. Users can use external tools, like Python-based Quix, to create tasks in InfluxDB 3. Last month, I analyzed the performance of our solar install. I found that, while our solar battery was generating daily savings, it would likely never save enough to offset its purchase cost.

There were a number of factors involved, but one of the bigger ones was the battery’s not always being sufficiently charged, with our average max daily charge level being 72%. The battery gets a full charge on sunny days, but British weather being what it is, there are plenty of days dragging that average down.

As I noted at the time, solar energy isn’t the only source from which the battery can be charged: there’s also the option of charging from the grid (ideally when prices are low).

So, in the month since, that’s exactly what I’ve done.

In this post, I will describe how I’ve configured things, analyze the impact of the change, and talk about some changes that my electricity supplier (Octopus) has recently made. Energy consumption monitoring is a time series use case so I write the metrics that I collect into InfluxDB. It’s designed specifically for time series, which allows me to easily visualize data in real-time as well as look at historic data.

Configuration

It was my intention to build automation that consumes pricing information and dynamically controls when the battery should charge. There’s a wealth of information out there on how to control a Solis Inverter from HomeAssistant.

However, as soon as I tried to set this up, I ran into a fundamental issue: all the setup guides and tools rely on older data sticks.

My inverter (a Solis RHI-3.6K-48ES-EG) came with a third-generation stick (model S3-WIFI-ST), which, unlike older counterparts, does not expose Modbus locally. As a result, tools like solismod and pysolarman cannot work with them.

There is actually a guide to getting things working using a rs485 splitter, but I’ve not tried it yet (in fact, I only found it while double-checking the links above as part of drafting this post).

However, Soliscloud has an interface that allows you to set the registers on your inverter manually. It’s not enabled by default, so access needs to be requested. Once granted, there’s no API access (update: there is now), but it does what it says on the tin.

The navigation interface is quite simplistic: Solis still seems to be working on improving it, as several things have moved around/changed since I first started playing with the interface.

The default mode is Self-Consumption (basically, it prioritizes household demand over export revenue). Within that menu there are some additional options: The Charge and Discharge button opens a side pane that allows charge times and rates to be configured. It’s worth noting here that, to a certain extent, it doesn’t really matter what you enter as a charge rate (as long as it’s not too low): the Battery Management System (BMS) will not allow the battery to charge faster than it’s capable of. Still, it is better to err on the side of caution in case that ever changes.

The changes made here, though, will have absolutely no impact until the schedule is enabled.

On the Self-Use menu is a button labeled Time of Use Switch. Anything ending in Switch is essentially a boolean—you turn it on or off. In this case, the switch defines whether or not the Charge/Discharge schedule is observed.

Adjusting time of use

In my previous post, one of the mitigations that I suggested was adjusting settings so that energy would only be drawn from the battery when most valuable (i.e., at peak times). The idea was that any energy remaining after peak would be carried forward into the next day’s peak rather than trickling away overnight.

I did look into doing this, but unfortunately, the inverter’s settings don’t quite work that way:

If you set the Discharge Current field to 0 and specify a time range, the battery will not discharge during that time (good!)
However, it will also consider it a discharge period and will, therefore, not charge the battery (boooo!)

So, although you can technically set the battery to only discharge at certain times, it probably won’t have any charge to discharge in the first place.

In fiddling around to get this working, I learned a few things about the Solis Inverter’s scheduling:

If you set a charge period, the battery will charge at that rate, taking energy from the grid to top-up whatever the panels are delivering.
If you set a discharge period, the battery will constantly discharge at the specified rate, exporting energy to the grid if there’s insufficient household demand to consume it (this took me by surprise, resulting in me accidentally dumping a battery load to the grid shortly before peak time started).
There isn’t a way (other than setting a discharge period), within Self-Use, to tell the inverter not to charge the battery from the panels.
If you switch to Grid Feedback Priority the meaning of the schedule changes. The battery will not be charged from the panels unless there’s a charge period defined.

In effect, Self-Use mode means the tooling only really allows you to influence how the battery interacts with the grid: When should it charge from it, and when should it discharge to it? This is actually quite logical, given that the mode is supposed to try and ensure that consumption primarily uses locally generated energy, but it can be a little restrictive.

Routine schedule

With no ability to dynamically shift the charging schedule based on price, I needed to choose a time range that would generally see low prices.

The obvious choice for this, of course, was in the small hours of the night, so I set an appropriate schedule: The next night, the battery charged as it should. Except, of course, that charge was long gone by the time we reached peak (in fact, it was gone before I even got out of bed).

I messed around with settings for a bit but ultimately found that there isn’t a way to configure things so that the charge is maintained until peak (at least, not if we want the panels to top it up).

Helpfully, it’s not just the morning that prices are lower: there are a good few hours in the afternoon (where Solar farms are presumably picking up the slack) with lower pricing, too: So, I updated the configuration so that we charge the battery from 12 pm to just before peak time starts: This carries an additional benefit: the battery will have had an entire morning of Solar charging, so we’ll only use the grid for whatever’s necessary to top it up (of course, if import prices drop below export value, we’ll need to re-assess that).

Impact on charge levels

The change didn’t have as significant an impact on the average charge level as I’d been expecting:

from(bucket: "Systemstats/rp_720d")
|> range(start: 2023-08-14T00:00:00Z)
|> filter(fn: (r) => r["_measurement"] == "solar_inverter")
|> filter(fn: (r) => r["_field"] == "batteryPowerPerc")
|> keep(columns: ["_time", "_field", "_value"])
|> aggregateWindow(every: 1d, fn: max, createEmpty: false)
|> mean()

However, on closer inspection, this average is pulled down by a handful of particularly overcast days on which the only charge that the battery got was from the grid.

If we instead look at the proportion of days where the battery was charged to at least 65% (i.e., enough to see us through even a busier peak):

from(bucket: "Systemstats/rp_720d")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "solar_inverter")
|> filter(fn: (r) => r["_field"] == "batteryPowerPerc")
|> keep(columns: ["_time", "_field", "_value"])
|> aggregateWindow(every: 1d, fn: max, createEmpty: false)
|> map(fn: (r) => ({r with 
       _field: if r._value > 65.0 then "full" else "partial",
       _value: 1
       }))
|> sum()

The difference is quite pronounced: Prior to the change, nearly 31% of days didn’t result in the battery reaching a 65% charge level (yay, British weather…). Since the change was made, that number has dropped to just 6%.

The difference is even more pronounced if we use a 45% charge threshold: partial charges account for 20% before and 1% after the change.

Impact on savings

The introduction of the afternoon charge schedule led to a fairly significant change in daily savings rates (the change was made at the red line): (If you’re wondering about those earlier multi-pound savings, they’re explained here and, unfortunately, are not repeatable).

The change had two outcomes:

Savings, when realized, tended to be higher than before.
Negative savings have occurred more frequently.

Negative savings occur when the total financial value (i.e., the cost of import from the grid or lost export cost) of the power that we store in the battery is higher than the equivalent import value of the energy when it’s discharged.

In the time covered, one of the primary determining factors seems to be whether or not we use the oven in the evening: If we cook with the air fryer (or a gas hob), we don’t draw enough energy at peak time. The battery’s charge is then released gradually over the course of the night when prices are lower, leading to negative savings.

So, just as in my last analysis, there is a perverse incentive to use less energy-efficient appliances in the evening to help drive the numbers up.

Of course, no one actually benefits from improving those figures in that way: using extra energy unnecessarily is still a net loss. This is an example of the sort of thinking that ultimately leads to trying to burn tires in order to improve the efficiency of burning coal to generate Cryptocurrency. Utterly crazy.

Anyway, despite those days with negative savings, the average daily saving has increased to £0.152 (from £0.125). Although that change is only a few pence, it still represents a 21% increase in the average daily saving.

At that rate, the battery will have paid itself off in… yeah, let’s not go there.

If we can eliminate negative-savings days (don’t you just love a magic wand?), then the average savings becomes 0.29 a day. That roughly halves the break-even time (but still leaves it many times the battery’s rated lifetime).

Getting free electricity

Our energy tariff is Octopus Agile, which sometimes sees prices drop very low (or even become negative). Unfortunately, that’s not really happened since I updated the charge schedule.

However, we’re now enrolled in a new Octopus scheme: Power-Ups. With power-ups, we occasionally get periods where our electricity is free despite the Agile price not being 0. The scheme aims to show local network operators that excess renewable energy can be consumed and discourage them from switching off green generation when surpluses are expected.

With the electricity being free, it obviously makes sense to charge the battery from the grid rather than from solar, helping drive up that day’s savings.

In order to account for these in my stats, I created a new field—override_price—which allows me to override the reported agile unit cost with a simple curl.

curl \
-d 'octopus_pricing,tariff_direction=import,charge_type=usage-charge,payment_method=None,reason=misc/solar#6 override_price=0.0 1692365400000000000' \
"http://192.168.3.96:8086/write?db=Systemstats"`

This sets the cost to 0.0p/kWh for that 30-minute slot.

The following query allows us to look at when there have been power-ups and how much time they covered:

SELECT
   count("override_price") * 30 as "minutes"  
FROM "Systemstats"."autogen"."octopus_pricing" 
WHERE $timeFilter 
GROUP BY time(1d) fill(null)

As you might expect, the impact that power-up periods have had on battery savings has been pretty positive (relevant days are marked with red dots): A number of factors have played into savings performance so far, but the primary one is striking a balance between solar and grid charging.

The maximum rate at which the battery can charge is limited, so it’s simply not possible to take the battery from empty to full in 2 hours. This means that it’s important to let the battery charge from solar during the morning because otherwise, there likely won’t be enough charge to cover the peak.

Conversely, it also doesn’t make sense to put too much solar energy into the battery because then we won’t be able to fully utilize the free grid power.

To complicate matters, the battery’s maximum charge rate is influenced by things like temperature, so there is an unavoidable element of guesswork in scheduling charges.

I found that the key lies in not relying solely on the Soliscloud schedules and instead switching things around manually:

Set a schedule to charge from the grid during the power-up (so far, that’s been 1400 - 1600).
In the morning, let the battery charge from Solar until it’s at about 47% (on some days, we reached that by 1030).
Toggle off the Time of Use Switch and then switch the inverter to Feedback Grid Priority mode (so that it’ll start exporting rather than charging the battery)
Just before the start of the power-up, switch back to Self-Use and toggle Time of Use back on so that the battery charges from the grid.

This approach, of course, relies on me being available at the right times to switch things around. But, it also leads to much higher export volumes (and therefore revenue). In effect, each exported kWh becomes worth double what it would otherwise have been.

Rather than storing 1 kWh, we export it (export revenue makes balance +15p).
During the power-up period, we replace it using free grid power stored in the battery (balance remains +15p).
We later discharge and consume it at the prevailing import price (assuming 30p/kWh, the balance becomes +45p).

The power-up allows us to claim the export value, avoiding loss of opportunity cost (which, as I wrote last time, is a major contributor to the low savings rate), and still realize a full kWh’s worth of savings during peak times.

Although that level of gain couldn’t exist without the battery, it isn’t fully accounted for in the battery savings figures—to see the true benefit of the approach, we need to look at the total system savings figure instead: As a rule, the days with power-ups outperformed their adjacent days. The exception is the 18th of August: not only was the weather crap, but it was also my first spin at configuring the inverter to take advantage of a power-up, and I ballsed it up a bit.

It’s worth noting, too, that the power-ups also delivered non-solar savings that aren’t accounted for in these graphs: we shifted some of our load to the power-up window, and so we were able to run the washing machine, etc., for free.

The result is that our consumption costs were reduced by at least sixty pence a day: The grid, for its part, gets an additional supply of green energy (our exports) at times of lower supply, without needing to spin down green generation capacity.

Further improvements

A couple of days ago, I made an additional change to the schedule.

Although much lower than the evening peak, there is a pricing peak each morning (coinciding with people getting ready for work, etc.). At that time in the morning, our panels are active but far from operating at their best, so it seems sensible to take a small overnight charge to help shave the morning peak. The savings, if any, will likely be a few pence at most, but that’s still a reasonable percentage improvement in the battery’s daily value. It will also likely be higher in winter when our panels start generating later in the day.

The outcome

Battery break-even is still decades away. But given that there is no way to unpurchase the battery, any additional savings that the battery can generate are still welcome, not least because it helps lower our energy bills. If I were being smart, I could probably find a way to invest those savings so that their growth could contribute further to achieving break-even.

Because our day-to-day activities vary, charging the battery (whether from the grid or solar) carries an element of risk: the battery’s ability to unlock savings is quite reliant on whether or not we demand sufficient energy at peak times.

That could potentially be mitigated by limiting battery discharge to certain periods, but it’s not something that’s currently possible with this inverter (or at least, not without significant trade-offs).

Somewhat unsurprisingly, the best way to unlock additional battery savings is to use the battery more. Ensuring the battery has at least enough charge to survive peak usage has helped increase savings by about 20%.

The introduction of Octopus Power-Ups has also proven beneficial, allowing the system to derive a reasonable amount of additional value. The benefit will, very likely, be less during winter as we won’t be able to export anywhere near the same amount of energy. Conversely though, our batteries will likely be in more need of a grid charge.

My work on this, clearly, isn’t done.

First, I need to see what benefit (if any) we gain from shaving that morning peak.

Once I’ve got meaningful stats on that, the next step is likely going to be adding a rs485 switch into the mix so that I can potentially start automating.

With automation in place, I’ll be more able to:

Avoid running up negative savings: On days when we routinely don’t use the oven, the automation can disable (or perhaps just reduce) the scheduled grid charge.
Ensure a sensible charge level on cloudy days: Weather forecasting can be used to define whether we’re likely to need a prolonged grid charge to cope with the peak.
Automate switching between Self-Use and Grid-Priority, allowing finer control over when the battery charges and discharges.

I don’t believe we’ll ever reach the point where buying the battery won’t have been a financial mistake, but given that we’ve got it, I think there’s still some scope for additional savings—particularly as the rest of the system is over-performing enough to offset the cost of the battery relatively quickly.

Is an Air Fryer More Energy Efficient Than an Oven

Ben Tasker (InfluxData) — Wed, 27 Nov 2024 07:00:00 +0000

This blog was originally posted on www.bentasker.co.uk

At first glance, the question I’m asking in this post almost seems redundant. An air fryer heats a much smaller area and uses a smaller heating element, so of course, it should be more energy efficient. However, that’s not guaranteed to be the case. Although an oven uses a larger heating element, because it’s better insulated, it’s possible that it might do a better job of keeping heat in and so have to consume less energy replacing lost heat. If, due to these losses, the air fryer’s element is on for a longer duration of the cook, it’s plausible that an air fryer might end up consuming more energy than the oven.

If (as seems likely) the air fryer is more energy efficient, the question becomes:

How much more efficient?
When does it amortize (i.e., at what point do the energy savings outweigh the initial purchase cost)?

That second question will also help answer whether it’s worth investing in an air fryer to try to counter high energy prices. To see how they compare, I cook myself some chips and compare the resulting energy usage using:

A Rangemaster Professional oven (3.2 kWh)
An air fryer (1700w 5.5L air fryer)

The cook

Both the oven and the air fryer cooked the same thing: Co-op’s brand of frozen oven chips. The appliances are situated in the same kitchen, but at opposite ends (so the air fryer does not benefit from heat radiating from the oven warming the air it draws in).

Although they’re cooking the same food, there are some differences:

The oven requires 25 minutes, and the air fryer takes 15
The oven needs to be set to 180°c (per the packaging), while the air fryer manual says to use 200°c

Unlike the oven, the air fryer doesn’t need to warm-up first, cutting 15-20 minutes off the run time (depending on how long it takes you to come back and put food into the oven).

The oven started first. When the air fryer finished, the oven was turned off and the food was removed from both.

As with my previous forays into measuring energy consumption, I collected and wrote energy usage readings using InfluxDB so I could query statistics back out with Flux.

InfluxDB is ideal for monitoring energy consumption because it’s purpose-built to handle time series data. Time series data is the type of data I’m collecting from both the oven and air fryer. InfluxDB ingests the data in real-time so I can see my visualizations immediately.

Retrieving results

Because of how it’s wired in (into its own circuit, with no convenient access to the wiring for a clamp meter), calculating the oven’s usage requires a slightly more complex query than is normally required for individually monitored appliances.

The way usage is calculated is:

Retrieve usage for all individually monitored devices.
Retrieve usage for power meter.
Subtract the sum of device usage from the power meter (find details on how I collect this here).
Subtract a known base load figure (to account for items not monitored) to arrive at the Oven’s usage.

There is a small margin for error involved in the subtraction of the base load, but it is quite small: I’ve been doing a lot of work recently to understand our base load (to find where it can be reduced).

The air fryer was plugged into a Tapo P110 Smart Socket, so its usage is easy to collect and query.

I wrote a Flux query to retrieve and combine usage data for each of the appliances into a single output stream:

// Start/Stop times
start = 2022-09-04T16:40:00Z 
stop = 2022-09-04T17:30:00Z 

// Group into 30s  time windows 
// helps ensure meter and appliance readings appear in the same window
period = 30s

// get appliance usage
known = from(bucket: "Systemstats")
  |> range(start: start, stop: stop)
  |> filter(fn: (r) => r._measurement == "power_watts" and 
                       r._field == "consumption")
  |> filter(fn: (r) => r.host != "power-meter")
  |> aggregateWindow(every: period, fn: mean)
  // Combine all appliances into a single table
  |> group()
  // Calculate the combined consumption per window period
  |> aggregateWindow(every: period, fn: sum, createEmpty: true)

// Get total usage measured at the meter
pm = from(bucket: "Systemstats")
  |> range(start: start, stop: stop)
  |> filter(fn: (r) => r._measurement == "power_watts" and 
                       r._field == "consumption")
  |> filter(fn: (r) => r.host == "power-meter")
  |> aggregateWindow(every: period, fn: mean)

// Join the two
oven = join(tables: {t1: pm, t2: known}, on: ["_time"])
  // subtract appliances + base load to get oven usage
  |> map(fn: (r) => ({
      _time: r._time,
      _field: "Oven",
      _value: r._value_t1 - r._value_t2 - 400.0
  })) 

// Get the air fryer's usage over the same period
airfry = from(bucket: "Systemstats")
  |> range(start: start, stop: stop)
  |> filter(fn: (r) => r._measurement == "power_watts" and r._field == "consumption")
  |> filter(fn: (r) => r.host == "air fryer")
  |> aggregateWindow(every: period, fn: mean)

// Union the tables so we can display a graph with 2 series
union(tables: [oven, airfry])

This gives us the following graph. It’s already fairly apparent which appliance is going to be responsible for most usage.

All subsequent statistics in this post are retrieved by appending to the union() statement in the query above.

Mean usage

If we look at the mean draw across the cook, we see that the oven’s average usage is significantly higher:

start = 2022-09-04T16:40:00Z 
stop = 2022-09-04T17:30:00Z 

...

// Union the tables so we can display a graph with 2 series
union(tables: [oven, airfry])
|> mean()
|> group()

Total usage

We know the average draw, but let’s look at the total power consumption for each appliance.

Calculating consumption from draw is fairly easy: If we draw n watts for an hour, then the usage is n Wh (with 1000 Wh being 1 kWh). If we draw n watts for 1/xth of an hour, then we need to divide n by x to have the calculated usage represent that fraction of time. For example:

A 30-second 3kW draw consumes (3000 / (60 * 2)): 25-watt hours.

Our data is windowed into 30-second periods (1/120th of an hour), so each window consumes 1/120th of the calculated usage.

So, to implement this, we add a simple conversion to the end of the query, taking the draw for that period and dividing it by 120:

// Union the tables so we can display a graph with 2 series
union(tables: [oven, airfry])
  |> map(fn: (r) => ({ r with
      // data is grouped into 30s windows, so if we're pulling 2kW 
      // in that window we're consuming
      //
      // usage / (mins-in-hour \* 2)
      // 2000 / 120
      //
      _value: r._value / 120.0
  }))
  // Calculate totals
  |> sum()
  |> group()

Giving the following results: The oven consumed more than double the energy demanded by the air fryer. Unsurprisingly, the proportions here are very similar to the proportion between the two appliance’s mean draw (the two figures being quite well correlated).

However, the oven did need to warm-up first. What happens if we only look at the time spent cooking (i.e., when food was in there)?

To do that, we adjust the start time at the head of the query so that the query starts after the oven has completed its warm-up:

start = 2022-09-04T16:59:00Z

The result is a large reduction in the reported consumption: The oven has a significant edge, managing to outperform the air fryer (which, even if it is able to do so while cooking, had to warm-up during that time).

Warmups

Of course, you may be thinking that this statistic isn’t particularly useful: you can’t reliably cook food without warming the oven.

However, the oven’s post-warm-up consumption is lower than that of the air fryer, which suggests that the oven might be able to outperform the air fryer when cooking over longer periods.

To understand the usage, we start by looking a little closer at the warmups:

How long does warm-up take?
What power is consumed during that time?
What’s the consumption with that subtracted?

Because both devices have thermostatic control, we can easily tell when they reached temperature: there’s a corresponding drop in power consumption when the thermostat turns the heating element off.

You can see the warm-up periods highlighted in the graph below (red: oven, purple: air fryer). The warm-up timings were:

Oven: (17:45 - 17:53) 8 minutes
Air fryer: (18:08 - 18:15) 7 minutes

Of course, in practice, the oven didn’t cook anything for longer than 8 minutes because I wandered off to do other things and took a little too long to come back. Although that’s representative of real usage (or, certainly, representative of my usage), we won’t penalize the oven for my mistake when we do our calculations.

By changing the start and stop values on the query, we get consumption for each of the warm-up windows before taking the usage from the warm-up periods and subtracting them from each of the total usage figures:

| Device    | Warm Up Usage | Remaining Usage |
|---------------------------------------------|
| Oven      | 137 Wh        | 206 Wh          |
| Air fryer | 96.4 Wh       | 96.6 Wh         |
-----------------------------------------------

Although interestingly, they don’t tell us much on their own. We’ll use these figures later.

Heating element activity

One thing that stands out in the graph is how long the air fryer’s element stays active after the initial warm-up.

It’s a horrible visualization, but look at how much longer the air fryer element is active (the purple boxes) per instance than the oven (red boxes): The oven mainly has 30-second bursts (though there are a couple of 1-minute periods mixed in). Conversely, although the air fryer has a couple of 30s bursts, and there’s a 3- minute period where the element is constantly active.

This would seem to support the idea that the Oven is able to maintain temperature much more easily than the air fryer.

We can check how long the element in each device spent turned on:

**union**(tables: [oven, airfry])
  // Set a state to group by later
**|> map**(fn: (**r**) => ({r with
    state: **if** **r**._value > 1100 then
        "ON"
    else
      "OFF"
  }))

  // Calculate the total stateduration
**|> elapsed**(unit: 1s)
**|> filter**(fn: (**r**) => **r**.state == "ON")
**|> sum**(column: "elapsed")
**|> group**()

By changing the start and end times, we can see how long the element was active during each cooking stage.

| Device    | Total   | After Oven Warmup | After Air Fryer Warmup |
|------------------------------------------------------------------|
| Oven      | 22 mins | 14 Mins           | 1 Minute               |
| air fryer | 12 mins | 12 Mins           | 6 Minutes              |
--------------------------------------------------------------------

50% of the air fryer’s activity occurred after warmup, while 63% of the oven’s heating time occurred post-warmup.

But the oven was also on for longer (making its post-warmup time disproportionately larger than the air fryers), so we need to make the comparison proportional by taking the total cook time into account:

| Device    | Cook Time (exc warmup) | Element Time (exc warmup) | Element Time |
|-------------------------------------------------------------------------------|
| Oven      | 32 Mins                | 14 Mins                   | 43.75%       |
| air fryer | 10 Mins                | 6 Mins                    | 60.00%       |
---------------------------------------------------------------------------------

The air fryer spent significantly more of its non-warmup time with the element sucking power. So, it’s certainly possible that the oven could prove to be more energy-efficient after a period.

Can the oven be more efficient?

We can work out roughly where the break-even falls. If we reduce the post-warmup average consumption to a per-minute average, we can compare them to predict how many minutes of cooking time it would take for the oven to outperform the air fryer.

We collected the figures we need above:

| Device    | Cook Time (exc warmup) | Power Consumed (exc warmup) | Power consumed (Warmup) |
|--------------------------------------------------------------------------------------------|
| Oven      | 32 Mins                | 206 Wh                      | 137 Wh                  |
| air fryer | 10 Mins                | 96.6 Wh                     | 96.6 Wh                 |
----------------------------------------------------------------------------------------------

To calculate the average Wh consumed per minute, we do:

Consumed / time = average minutely consumption Giving the following figures:

Oven: 206 Wh / 32 mins = 6.4375 Wh/min
Air fryer: 96.6 Wh / 10 mins = 9.66 Wh/min

The oven is significantly more efficient per minute.

But before that efficiency advantage can translate into an overall energy saving, the oven must cancel out the additional energy it consumed during its warmup period.

To calculate when it’ll reach that point, we calculate the difference in warmup consumption and divide it by the difference in per-minute consumption.

    137 - 96.6 = 40.4 Wh
-------------------------------   = 12.54 minutes
   9.66 - 6.4375 = 3.2225 Wh

So, all things being equal, the oven would break even after ~13 minutes.

But not all is equal: The air fryer also requires less cooking time (10 minutes less for chips), so the oven consumes an extra 10 minutes of energy.

Adding this into our calculation moves the break-even point quite considerably:

    (137 - 96.6)
  + (10 * 6.4375) = 104.757 Wh
--------------------------------   = 32.51 minutes
   9.66 - 6.4375 = 3.2225 Wh

At nearly 33 minutes, this is more than double the time the air fryer requires to cook the chips.

Spuds

Fifteen minutes is quite a short cooking time, though—do these results hold for something that requires longer?

If we take the timings from two baked potato recipes (oven and air fryer), we should be able to work out which is more energy-efficient:

Oven potatoes require 75-90 mins (we’ll split the difference and call it 83).
Fryer potatoes require 50 mins.

The oven potatoes need 33 mins more, so assuming that warmup and per-min consumption doesn’t change:

    (137 - 96.6) 
  + (33 * 6.4375) = 252.84 Wh
--------------------------------   = 78 minutes
   9.66 - 6.4375 = 3.2225 Wh

The break-even is at 78 minutes, which (again) is longer than the air fryer needs.

Technically, the break-even will be even further out than that because the oven recipe requires a higher temperature than our chips, so the warm-up and maintenance consumption would be higher.

To summarize, an oven uses less energy per minute post-warmup than an air fryer, but because it needs substantially less time to cook food, it still uses less energy overall.

Purchase break-even

Given the energy crisis looming in the UK, it’d be remiss not to also examine the financial side of this.

If you’re reading this, it’s possible you haven’t yet bought an air fryer and are considering doing so to reduce your energy bills. Since you probably already have an oven, the upfront cost comparison is £0 vs. $airfryer_cost.

To determine whether it’s worth buying an air fryer, we need to consider how long it will take for the energy savings to break even with the purchase cost (known as amortization).

If we consider the following:

The air fryer I tested costs £72.99 (although you can get an Air Fryer for around £40).
Including warm-ups (let’s be realistic, you’re never going to consistently leap on the oven as soon as it’s warmed up), the air fryer consumed 150 Wh less than the oven when cooking chips.
In October, the cost of electricity will (for many) rise to £0.52/kWh

We can conclude that:

The air fryer saved (52/1000 * 150) £0.078 in energy when cooking my chips
At £0.078 per cook, break-even would happen after (72.99 / 0.078) 936 similar cooks

At one cook a day, the energy savings would take two and a half years to offset the £72.99 purchase price. Assuming a £40.00 air fryer can deliver the same energy savings, it would still take nearly 18 months to break even.

To put that into perspective:

If I cook chips once daily for 31 days, I would reduce my energy bills by just 4.65 kWh / £2.41 a month.
If I spread the cost of the air fryer, interest-free, over 18 months, I would pay £4.05/month
Despite the energy saving, I’d therefore lose £1.64 a month

It’s (hopefully) unlikely that you’re going to be living off chips alone, so let’s also project usage for baked potatoes:

     Oven: 137 + (83 * 6.4375) = 671.31 Wh
  -  air fryer: 96.6 + (50 * 9.66) = 579.6 Wh
    -------------------------------------------
                  91.71 Wh

The saving on baked potatoes is actually lower, and saves (52/1000 * 91.71) £0.0477 per cook (giving a 4 year amortization period).

This means that, even if you manage to find an air fryer at half the price I paid, the break-even is still years away.

Conclusion

Despite needing to achieve a higher temperature, the air fryer used significantly less power cooking my chips.

The fact that it doesn’t need to be warmed up first contributes quite significantly to this: not just because it’s not spending time heating rather than cooking, but also because it removes the opportunity for a human to get side-tracked and forget to put the food in (guilty, your honor).

The oven consumes significantly less energy maintaining temperature than the air fryer (likely because of better insulation). However, the air fryer’s faster cooking time mitigates this, lowering overall energy consumption.

So, to answer our questions:

Is an air fryer more energy efficient than an oven?

Yes, but only if you use it properly and adjust cooking times.

If you air-fry food for the same time as you would in an oven, it may take as little as 13 minutes for the oven to become the less expensive option (and your food will probably be horribly burnt).

When will I break even?

Even at the insane prices we’re all going to be paying, the savings per meal are very small. It’ll likely take years for the energy savings to break even with the up-front cost of purchasing an air fryer. In some cases, it may take longer than the useful life of the air fryer.

Should I invest in an air fryer to avoid high energy prices?

The long amortization period means that if you’re concerned about how you will afford energy bills this winter, purchasing an air fryer isn’t a good way to address it. The air fryer will reduce your energy consumption, but only very slightly, and the up-front purchase cost will make you worse off overall.

Unless you can find a deeply discounted air fryer (or be gifted one, or pay off the loan interest-free over an extremely extended period), it’s unlikely to be the right decision from a purely financial point of view.

Obviously, if you already have an air fryer, then you incurred the capital cost, so using it where you can will slightly reduce your energy bill, with each saving bringing the break-even point slightly closer. And they do make some very nice chips.

To start monitoring the energy of your appliances, sign up for a free cloud account now. To add InfluxDB to larger projects, contact our sales team for a custom POC.

Update: cooking at 180

After I published this post, a Twitter user raised an interesting point. The oven’s manual doesn’t provide any guidance, but I could have tried cooking in both at 180c, demanding lower consumption from the air fryer and increasing its overall advantage. I expected that it wouldn’t make much material difference to the conclusion of this post, but I thought I’d give it a try nonetheless.

This evening, I cooked some chips using the following:

Temperature: 180c
Time: 15 minutes (as before)

The chips came out cooked, but, honestly, they weren’t that great—they hadn’t crisped very much and had that unpleasant wet oven-cook feel to them. They either needed longer or the manual’s recommended temperature. But what we’re interested in is the power consumption.

I graphed consumption using the same query as before:

period = 30s
from(bucket: "Systemstats")
  |> range(start: 2022-09-24T16:45:00Z, stop: 2022-09-24T17:05:00Z)
  |> filter(fn: (r) => r._measurement == "power_watts" and r._field == "consumption")
  |> filter(fn: (r) => r.host == "air-fryer")
  |> aggregateWindow(every: period, fn: mean)

Giving the following graph: Using the same queries as in the earlier part of this post, we can see that the total consumption for this cook was 174 Wh, just 19 Wh less than the cook at 200 Celsius. So it saved 169 Wh when compared to the oven cook.

If we re-run the break-even calculations using this figure, we can see that

At 180c it saves (52/1000 * 169) £0.088 in energy.
At £0.088 per cook, break even will take (72.99 / 0.088) 829 similar cooks.
At one cook a day, break-even is around two years, 3 months, and 8 days.
A £40 fryer achieving the same savings would take 1 year, 2 months, and 27 days to break even.
31 days of chips once a day would reduce energy bills by just 5.24 kWh / £2.72.
Assuming interest-free repayment of the purchase at £4.05/month, we’d be £1.33 worse off a month.

There’s also the issue of the quality of the chips we get. Realistically, you’re only likely to eat them like that once, and then on subsequent cooks, you’ll either increase the temperature (back to 200) or the time.

So, let’s work out the impact of adding time. W. We’re interested in the average power usage while the air fryer maintains temperature (the bulk of energy usage is in the initial warm-up).

The graph shows that the warm-up finished at 17:53:30, the cook finished at 18:02:30, and the maintenance period was 9 minutes.

During that time it consumed 76.6 Wh, so the average consumption per minute was (76.6/9) 8.5Wh.

Assuming the chips required an additional 5 minutes (in practice, that’s probably still not enough), we’d likely consume another 42.5 Wh. The advantage gained by dropping to 180c was only 19 Wh, so we’re now worse off by around 23.5 Wh (although still ahead of the oven).

So, although cooking at 180c does reduce the energy consumption a little

It doesn’t make a material difference to the conclusion: it’s still a poor solution to energy prices unless you already own one
You’ll save around £0.009 versus cooking at 200c
The chips you get from those cooks won’t be nearly as good as those done at 200c.
Adjusting the cook time to account for the lower temperature results in higher energy usage than the shorter, high-temperature cook

One additional concern I’ve developed since originally writing the post is realistic usage patterns.

The air fryer will save you a little money if you use it instead of the oven. However, because it makes such good chips, it’s easy to slip into the habit of using it as well as the oven, increasing your overall power consumption. Having a dual-zone air fryer can help with this a little, but it becomes quite hard to put chips in the oven knowing how much better they’ll be from the fryer.

Building My Own Streaming TV Station

Ben Tasker (InfluxData) — Tue, 08 Oct 2024 08:00:00 +0000

Recently a colleague mentioned having seen a YouTuber set up their own cable TV channel at home.

This struck me as quite a good idea: I sometimes want to have the TV on in the background (the background noise helps the dogs settle) but often struggle to pick something from the myriad of available options (which include a ripped copy of most of our DVDs).

Live TV and Prime Video aren’t really viable options: the aim is to have some background entertainment, not to get distracted by annoying ads for stuff that I’m never going to buy.

Having an ad-free channel that only plays stuff that I like would make it a lot easier to just slap something on; it’s not like I’m usually intending to watch it.

We’ve got a range of devices in the house, so I wanted my implementation to use commonly supported protocols (HLS and/or RTMP) and, ideally, to be able to run on my Kubernetes cluster.

In this post, I’ll talk about some of the more interesting aspects of the container image that I built in order to run my own streaming TV station using my collection of ripped DVDs.

Overview

The system is simply a docker container which randomly selects media from a local library, making it available to players via RTMP and HLS.

Just like a TV channel, it plays its own schedule.

Inevitably, I got a little carried away, so the feature set also includes

Allow and Block lists for media: constraining what the channel can stream
Ability to set a broadcast window: constraining when the channel can stream
Dynamic broadcast triggering: reducing use of compute if no one’s actually watching
Writing play history and other stats into InfluxDB
An API endpoint to skip to the next episode (in case of playback issues or “I’ve seen this”)

I chose to use InfluxDB rather than (for example) writing a textual log because writing into a time-series database allows easier exploration and analysis of the play history. It also allows the history to easily be correlated against [performance and usage stats] (for example, to identify whether certain TV series tend to result in an increased frame-drop rate).

Other InfluxDB features (like support for bucket retention periods also allow easy management of that data’s life cycle (for example, by expiring stuff out when it’s old enough that it’s not likely to be of interest).

Running

Invocation via docker is simple:

Inevitably, the YAML to run on Kubernetes is a little more verbose but still pretty simple as K8s config goes:

RTMP & HLS

Thanks to some of my previous jobs, I’ve got quite a bit of prior experience with video delivery, including RTMP.

In fact, in a previous role, I owned and operated a private fork of nginx-rtmp, that had to be heavily customized (in other words, I did some unspeakable things to it) in pursuit of extremely diverse customer needs.

This experience is relevant because one of the things that you get from owning, operating and supporting a custom RTMP application (particularly at CDN scale), is the wisdom to know that it’s something that you only ever want to do once (if that…).

So, although nginx-rtmp sits at the core of this build, I’ve not customized it at all and have instead built tooling around it.

Although I’ve built HLS tooling in the past, there was no need for that this time: nginx-rtmp has HLS support built into it (it supports DASH too, but I’m not currently using it).

Video Processing

The project involves reading in arbitrary video files, so, obviously, I reached for the Swiss army knife of video conversion: FFmpeg

Using ffmpeg to stream files to a RTMP server is a well-solved problem:

The crucial flag in this command is -re , it instructs ffmpeg to read the file at its native frame rate, ensuring that the file is streamed in real time rather than all at once.

Publishing Script

I wrote the automated publishing in BASH.

At the job with the RTMP stack, I had a co-worker who would occasionally jokingly leave a code review comment on shell scripts: “Line count suggests this should have had a .py extension instead.”

I imagine that the publishing script would attract a similar review, and…. he’d not be wrong. I started to regret choosing BASH at about the time that I realised I needed to iterate through lines in a file and then do the equivalent of '|'.join() on them.

But, I was already invested… Maybe I’ll rewrite it at some point.

Video Selection

The publishing script chooses a random series followed by a random episode, filtering each with a regular expression constructed from the blocklist.

The filter is applied at both stages so that I can include filters that target episodes as well as series.

For example, if I wanted to ensure that specials aren’t included, I might filter out any file containing the string s00e.

Dynamic Publishing Trigger

The first build of the container was pretty simple:

Launch nginx-rtmp
Loop, publishing random episodes into nginx-rtmp

The result was an IPTV channel that was always on.

But…. It was always on. Even if I wasn’t watching, ffmpeg was dutifully whizzing away, tying up precious cores and sapping electricity:

So, I decided to change it so that the publishing script only starts a new episode if a player is connected.

The nginx-rtmp module has a range of event hooks, including a play event. Each hook places a simple HTTP request to an arbitrary endpoint.

So, with a single line of config:

Nginx sends details of the play request onto the destination:

I cobbled together a small control server (in Python—I wasn’t repeating the mistake I made with the entry point), allowing for a somewhat janky but functional flow:

Player connects to nginx-rtmp
nginx-rtmp sends play event to control server
Control server writes play (and player count) to control files
Publishing script detects a change in state and starts ffmpeg
Playback starts

There’s a similar workflow if a player later disconnects:

Player disconnects
nginx-rtmp sends play_done event to control server
Control server subtracts 1 from player count
If player count is < 1, writes stop state
ffmpeg continues current stream but does not start next while state is stop

I decided not to stop ffmpeg on disconnect because I might want to reconnect (perhaps as a result of changing rooms).

In the logs, it looks like this:

Technically, this process extends video startup times, but to help ensure smooth playback, RTMP players tend to sit and fill a buffer first, so the additional delay is not all that noticeable.

Introducing this delivered a couple of benefits, including:

CPU & electricity aren’t wasted processing something that no one is watching
I don’t miss the beginning of episodes because the system won’t start the stream til someone’s watching

The one real drawback is that it only works with RTMP streaming. HLS is served over a standard HTTP connection, so there’s no simple equivalent (there are ways around it, but they’ve all got tradeoffs that I didn’t want to have to make).

Broadcast Window

If I’m honest, implementing this was much more about nostalgia than efficiency.

The broadcast window states that the system should only broadcast video between set hours:

Outside of those times, rather than starting a new episode, players will receive an old BBC test card.

It does differ a little from the original, though, in that the card is streamed in silence. Nostalgia only goes so far, and I didn’t really fancy reliving the experience of being woken by the beep kicking in.

Avoiding Freezes Between Episodes

The system’s design means that ffmpeg has to consume a range of arbitrary video files. Having been ripped over the course of quite a few years, there’s quite a lot of variation between them, including:

Video and audio codecs
Framerate
Resolution
Pixel Format (rarer, but still)

nginx-rtmp pretty much just forwards what’s being piped into it, so when an episode changes, the downstream player might end up receiving frames that are completely different to those that came before.

Some players (like ffplay) handle this ok. Most, though, end up freezing in response.

To avoid these unexpected changes, I adjusted the ffmpeg invocation so that it would normalize it’s output:

In order to ensure a common resolution, a scale filter is used to target 720p. If the input file has a lower resolution, then letterboxing is used.

With that change made, the transition between episodes became a lot more reliable.

I also implemented an optional mechanism that could be used to redirect players away from the stream (and back again) at episode change-over. However, Kodi’s Simple IPTV Player plugin doesn’t seem to like that very much.

Play History and Stats

We’ve amassed a fair collection of episodes over the years, enough that I sometimes may not recognise what we’re watching (or perhaps I’m just getting old).

Originally, I implemented a small API endpoint which could be called to check what’s currently playing.

But what if I’m absorbed and forget to check until after it’s finished? Then I’ll never know what that great show was!

Instead, I decided to have the container write a play history into InfluxDB.

Conclusion

At its heart, this was just a fun project to play around with and keep myself out of trouble for a couple of days.

But, it also meets its target use case quite well, even if it doesn’t always succeed in just being in the background: the other evening I sat down, flicked it on and was greeted by an episode of Bucky O’Hare!

Best of all, there are no ad breaks. Admittedly, I did consider adding one so that I could sue Musk for “witholding billlions of dollars in advertising revenue” (that’s how it works now, right?).

The pod’s resource demands vary a bit depending on the episode being streamed: processing higher resolution videos obviously demands more resources than lower res ones. I’ve currently got the pod capped at 2 cores and that’s been fine for the majority of media.

Not that I’m going to be adding that many players, but the majority of the compute expense is in the initial publishing—the actual stream delivery is incredibly cheap (I’d likely hit network contention before CPU became an issue on that front).

Deploying InfluxDB and Telegraf to Monitor Kubernetes

Ben Tasker (InfluxData) — Tue, 17 Sep 2024 08:00:00 +0000

I run a small Kubernetes cluster at home, which I originally set up as somewhere to experiment.

Because it started as a playground, I never bothered to set up monitoring. However, as time passed, I’ve ended up dropping more production-esque workloads onto it, so I decided I should probably put some observability in place.

Not having visibility into the cluster was actually a little odd, considering that even my fish tank can page me. I don’t need (or want) the cluster to be able to generate pages, but I do still want the underlying metrics, if only for capacity planning.

In this post I talk about deploying and automatically pre-configuring an InfluxDB OSS 2.x instance (with optional EDR) and Telegraf to collect metrics for visualization with Grafana.

The plan

First, a quick overview of the plan:

Create a dedicated namespace (called monitoring)
Deploy and preconfigure an InfluxDB OSS 2.x instance to write metrics into
Deploy Telegraf as a DaemonSet to run on every node using both the kubernetes and kube_inventory plugins
Graph out metrics in Grafana (I already had a Grafana instance—see here if you need to roll one out)

Because this is a home lab, I’m going to commit an ops-sin and deploy InfluxDB into the same cluster that I’m using it to monitor.

If you’re doing this in production, you should ensure that metrics are available during an outage by deploying InfluxDB elsewhere, using InfluxDB Cloud, or configuring Edge Data Replication (EDR) so that the data is replicated out of the cluster.

In the steps below, I’ve made it easy to configure EDR (no changes are needed if you don’t want it; just don’t create that secret).

Throughout this doc, I’m going to include snippets of YAML. The assumption is that you’ll append them to a file (mine’s called influxdb-kubemonitoring.yml) ready for feeding into kubectl at various points.

For those in a hurry, there’s also a copy of the manifests below in my article_scripts repo.

Why InfluxDB?

Using InfluxDB and Telegraf offers a number of benefits:

Easy off-site replication (via EDR)
Ease of setup: Telegraf has everything rolled in
Metrics can be queried with InfluxQL (and/or SQL if replicated into v3)

Because Telegraf can buffer data and InfluxDB can accept writes into the past, if something were to happen to the cluster, there’s the potential to gain retrospective visibility into its state when later analysing the cause of an incident.

And, of course, there’s familiarity: all of my metrics go into InfluxDB, so it makes most sense to use the solution I’m most comfortable with.

Namespace

Everything will be deployed into a namespace called monitoring, so first we need to create that:

Secret creation

We’re going to need to define a secret containing several items, including:

Admin credentials for the InfluxDB instance
A token to create in the InfluxDB instance
The name of the InfluxDB org
Credentials for two non-privileged users

These will be used to create the accounts within InfluxDB.

I use gen_passwd to generate passwords but use whatever suits you.

In order to create a random token, I used:

Create a secret called influxdb-info:

Storage

InfluxDB needs some persistent storage so our metrics remain available even after a pod is rolled.

Generally, I use NFS to access my NAS, so I defined a NFS backed physical volume and an associated claim:

Optional: Edge Data Replication

As we’re deploying InfluxDB into the monitored cluster, to help ensure continuity, we might want to replicate the data onward (to InfluxDB cloud or another OSS instance) so that it also exists outside the cluster.

OSS 2.X has a feature called Edge Data Replication—this involves defining a remote (i.e., where we want to replicate to) and then specifying which bucket should be replicated.

To do this, you will need:

The URL of your remote instance
An authentication token for the remote instance
The org ID to use with the remote instance
The ID of the remote bucket you want to write into

If you decide not to configure EDR and want to add it later, you can always enable it manually.

To enable EDR, we’re going to define a secret to hold the information necessary to configure it:

The manifests we create later in this post will reference these values (they’re optional, so nothing should error if the secret isn’t defined).

Preconfiguring InfluxDB Auth

The InfluxDB container supports pre-configuration of credentials via environment variables, so we’re going to pass in references to the secret that we created earlier.

However, there is still a catch: Although the image allows us to pass a known token value (via DOCKER_INFLUXDB_INIT_ADMIN_TOKEN) it’s used to create the operator token.

The operator token is an extremely privileged credential, and not something that we really want to be passing into Grafana and Telegraf.

We need to pre-populate some less privileged credentials, but neither the image nor InfluxDB provides a means to create a (non-operator) token with a known value. We could write a script to call the API and mint a pair of tokens before writing them into a k8s secret, but we’d then need to give something the ability to update secrets, which is somewhat less than ideal.

Instead, we can create a username/password pair that can then be used with the v1 API.

The InfluxDB image can run arbitrary init scripts (but only triggers them if Influx’s data files exist, so there’s no risk of accidental re-runs).

We’re going to define a ConfigMap to store a small script that will create a DBRP policy and a pair of non-privileged users (one read-only, one write-only). The script also configures EDR if we’ve chosen to enable it.

Deploying InfluxDB

Next, we need to deploy InfluxDB itself.

Things to note about this definition are:

Variables DOCKER_INFLUXDB_INIT_BUCKET and DOCKER_INFLUXDB_INIT_RETENTION tell InfluxDB to create a bucket called telegraf with a retention policy of 90 days.
My NFS share is configured to squash permissions, so I included a securityContext to use the appropriate UID.
We make the various values from our secret available via the environment variable.
As well as the PVC, we mount the configmap containing the init script into the container.

The Deployment will need to be fronted by a service, so we define that next, passing port 8086 through:

If we apply our work so far:

We should now be able to retrieve service details:

It should now also be possible to use the cluster IP to run a query against InfluxDB (replace $INFLUX_TOKEN with the token you generated when first defining the secret).

We should also be able to use the read-only creds with the v1 query API:

Deploying Telegraf

Deploying Telegraf into Kubernetes is really easy. However, authorizing it to fetch information from the Kubernetes APIs involves interacting with the k8s auth model, which is a bit more complex.

First, we’re going to define a service account called telegraf and then authorize that to talk to both kubelet and cluster APIs.

Define the service account:

Define a pair of cluster roles:

Note: Technically, you could probably combine those, but as they give access to different things, I prefer to keep them separate.

Finally, create the role bindings to link both back to the Service Account:

With that unpleasantness out of the way, we’re ready to configure and deploy Telegraf.

As the Telegraf config is simple, we drop it into a ConfigMap.

Note: There’s no need to replace the variables by hand; we’ll be setting them in the container config.

Then, we’re ready to define our DaemonSet.

There are a couple of important points here:

We’re exposing the node IP via env var HOSTIP
We’re setting the env var HOSTNAME to the node Hostname via spec.nodeName
We’re passing in credentials from the secret we created earlier
We’re mapping directories from the host into the container
We’re mounting our configmap as config
We set MONITOR_HOST to use the name of the service we created for InfluxDB (http://influxdb:8086).

If we apply the updated config:

Telegraf should spring to life and start collecting metrics from the Kubernetes APIs.

We can check by tailing Telegraf’s logs:

If it’s not logging errors, then everything is working.

Checking for metrics

We can confirm that metrics are arriving in InfluxDB by logging into its web interface.

Assuming that you deployed into the cluster, grab the cluster IP by running the following:

Then, visit http://[cluster ip]:8086 in a browser.

You should be able to log in with the credentials you created at the beginning of this process.

If you browse to the Data Explorer, you should see a bunch of Kubernetes-related measurements.

Dashboarding

With metrics coming into InfluxDB, we now just need a dashboard to help visualize things. I do all of my dashboarding in Grafana so that I only have to go to one place to visualize metrics from a wide variety of sources.

We’ll need to tell Grafana how to speak to our new InfluxDB instance, so:

Burger Menu
Connections
Connect Data
Search for InfluxDB

When adding the new InfluxDB datasource in Grafana, you’ll need to provide the following:

Language: InfluxQL (because we’re using the v1 API)
URL: If InfluxDB and Grafana are running in the same cluster, you can simply use the InfluxDB service name. Otherwise you’ll need to provide an IP or domain name

Use the following settings in Grafana:

Database: telegraf
User: kubestatsro
Password: the password we defined earlier

Once the datasource has been created, we can start to pull stats out with queries like:

Ultimately, we are building a single dashboard that allows us to view resource usage at node, namespace, and pod levels.

For example, we can see that my Atuin sync server has been assigned far more resources than it’s actually using (potentially preventing other workloads from being scheduled on that node): An importable copy of the dashboard can be found on Github. Because it uses InfluxQL, it should be compatible with InfluxDB 1.x, 2.x and 3.x.

Conclusion

If you’ve been following along, you should now have an InfluxDB instance receiving metrics about each resource in your Kubernetes cluster. If you enabled EDR, these metrics will be replicated to an external instance in order to help ensure continuity of monitoring during outages.

I now have the means to see a bit more about what’s going on inside my cluster, including the information I need to ensure that resource requests aren’t being set too high, unnecessarily limiting capacity in the process.

Although YAML’s verbosity makes it seem like a lot more than it is, it didn’t take much to get InfluxDB and Telegraf up and running in the cluster.

The auto-provisioning of credentials and configuration means that if I ever want to start over, I can simply wipe the PVCs and roll the pods—the image will recreate the credentials, and everything will come back up.

To get started on this or another project on InfluxDB, sign up for a free cloud account now.

Monitoring an Aquarium with InfluxDB and Grafana

Ben Tasker (InfluxData) — Mon, 10 Apr 2023 07:00:00 +0000

This article was originally posted on Ben Tasker’s blog and is reposted here with permission.

I’ve been setting up a new tropical fish tank and wanted to add some monitoring and alerting because, well, why not?

The key questions that I was interested in answering were:

Is the filter running properly?
Is the temperature within acceptable bounds?
Are scheduled things (like the surface skimmer and lights) actually happening?
Are both heaters working?

The plan is to also add monitoring for PH levels, but the probe that I need for that hasn’t arrived yet.

In this post I’ll talk about the aquarium monitoring and alerting system that I’ve built using a Raspberry Pi, InfluxDB, Telegraf and Grafana.

Metrics sources

Some elements of tank health can be monitored via the metrics exposed by my existing energy usage tracking implementation which allow us to see which components are drawing power (and how much/when).

All I needed to do for this was plug things like heaters into smart sockets and add them to the config consumed by my docker image:

kasa:
    devices:
        # ... etc ...     
        -
            name: "aquarium-light"
            ip: 192.168.5.165

        -
            name: "aquarium-surface-skimmer"
            ip: 192.168.5.166
        # ... etc ...

The use of smart sockets also means that HomeAssistant can be used as a timer to schedule runs of things like the surface skimmer and the aquarium light:

Watching power consumption on its own isn’t enough because some of the metrics that we care about require physical measurements to be taken.

For example, watching the energy consumption of the pump is only part of the solution: if the impeller were to fail, the motor would continue drawing power whilst spinning uselessly. There are many failure modes for an aquarium filter, and motor failure is just one, what we actually care about is the flow rate that the pump is achieving.

So, we need some sensors:

The sensors used are:

A DS18B20 based temperature probe (a 1-wire device)
A Digitem FL-408 Water Flow Meter

Obviously, the future PH monitoring project will also require a physical probe of its own.

A note on temperature sensors

If you’re planning on building something similar, it’s worth noting here that it’s very important to buy your temperature probe from a reputable vendor (like RS Components). There are a lot of fake DS18B20 chips about and they tend to misbehave in annoying and unpredictable ways.

The temperature sensor shown in the image above was ordered from Amazon and seemed to work fine at first. However, after about 6 hours of operation it developed a habit of dropping off the bus entirely, not re-appearing unless the Pi was fully power-cycled.

This behavior, apparently, is not uncommon amongst dodgy chips. They’re also known to return some very suspect readings:

That’s right, if the probe is to be believed, then (in a matter of moments) my tank went from ~20c to 3x water’s critical temperature…

Once I became aware of the issue, I ordered a replacement probe from DSRobot. It even managed to be cheaper than the original: knock-offs don’t just exist at the bottom end of the price range.

Raspberry Pi build

I had a spare Raspberry Pi 4 that had sat unused since it’s SD card failed (I migrated stuff elsewhere at the time), so I decided to use that. In theory though, any GPIO equipped single board computer should work.

For convenience’s sake, I ordered a GPIO break-out board with screw terminals.

It turned out, though, that this was not actually that convenient a route, because what turned up was a board and a set of connectors with not a drop of solder in sight.

The manufacturer presumably had had a conversation along the following lines:

Pers 1: Should we provide these pre-assembled?

Pers 2: It’ll mean using solder, that costs money.

Pers 1: Customers will expect it to work though and they probably have a project they want to get moving on.

Pers 2: No, these people are hobbyists they love soldering. We’ll be giving them more of what they enjoy and they’ll love us for it

So, with some grumbling, I took my “convenient” board and soldered the connectors onto it. This didn’t go particularly well because I’d had a conversation of my own:

Me: Where the F@!# is my good soldering iron?

Me: the gas powered one will have to do

Me: Oh, it’s only got a fat tip… better than nothing

Me: Which **** didn’t order replacement solder the last time I nearly finished it all up?*

It turns out (surprising no-one) that delicate soldering can be a bit of a challenge when you’re having to compete with an unweildy soldering iron which is heating a big fat tip and blowing hot exhaust out over fingers that are trying to hold onto an unnecessarily short length of solder.

It might just be the worst bit of soldering I’ve ever done, but solder it I did:

(No, I’m not being self-deprecating, you haven’t seen the other side)

With my newly soldered breakout board in hand, it was time to connect the sensors.

The temperature probe came first, wiring as follows

VCC (red) to pin 1 (3.3v supply)
Signal (yellow) to pin 7 (GPIO 4)
Ground (black) to pin 6 (GND)
4.7k resistor between Pin 1 and 7

Next was the water flow meter:

VCC (red) to pin 17 (3.3v supply)
Signal (yellow) to pin 29 (GPIO 5)
Ground (black) to pin 39 (GND)
4.7k resistor between Pin 17 and 29

With the hardware connected, I moved onto kicking together some code to collect and submit readings.

Collecting readings

The two sensors are accessible in different ways. I could have built a more graceful implementation, using a single script to collect readings from both, but opted not to — in part because I wanted to poll at entirely different intervals.

Full copies of the scripts referred to below are available on GitHub.

Temperature sensor

The temperature sensor presents a 1-wire bus device which can simply be read from:

$ cat /sys/bus/w1/devices/28-3c6df6482d24/w1_slave
5a 01 55 05 7f a5 a5 66 c3 : crc=c3 YES
5a 01 55 05 7f a5 a5 66 c3 t=21625

In the output above, the sensor is reporting 21.63 degrees celsius.

28-3c6df6482d24 is the serial number of the chip in the probe (with the benefit of hindsight, it also tells us that there’s a good chance that the chip is a knock-off).

Because the probe can be read using cat, collecting and submitting readings from it is very simple

#!/bin/bash
#
# Read temperature from a 1-wire temperature sensor
#
# example: ./read_temperature.sh "dev name"
#
# Copyright (c) 2023 B Tasker
# Released under BSD 3-Clause
# https://www.bentasker.co.uk/pages/licenses/bsd-3-clause.html
#
DEVNAME=${1:-"28-3c6df6482d24"}
LOCATION=${LOCATION:-"diningroom"}

# Read the sensor
ts=`date +'%s'`
tempread=`cat /sys/bus/w1/devices/$DEVNAME/w1_slave`
temp=`echo "scale=2; "\`echo ${tempread##*=}\`" / 1000" | bc`

# Build line protocol
lp="aquarium,location=$LOCATION water_temperature=$temp $ts"

# Write the stats out
curl -s -d "$lp" "http://127.0.0.1:8086/write?db=telegraf&precision=s"

I copied the script over to the Pi and added a crontab entry to call it every minute.

* * * * * pi /home/pi/tank-monitoring-scripts/app/read_temperature.sh

Water-flow meter

The water-flow meter required a little more effort, because the OS doesn’t helpfully expose an interface for us to cat.

The sensor itself is quite simple, consisting of a turbine (turned by the water passing through) and a hall-effect sensor (to pick up the turbine’s rotation).

As the turbine spins, the sensor sends pulses down the wire to the GPIO pin, so we need to count those pulses and then convert them into a flow-rate (e.g. liters per hour).

At its simplest, the counting could look something like this:

#!/usr/bin/env python3
#
# Copyright (c) 2023 B Tasker
# Released under BSD 3-Clause
# https://www.bentasker.co.uk/pages/licenses/bsd-3-clause.html
#
import RPi.GPIO as GPIO

def countPulse(channel):
    ''' Callback function for the GPIO event
    Increment the counter, assuming the counter is active
    '''
    global counter
    if start_counter == 1:
        counter += 1

global counter
counter = 0

GPIO.setmode(GPIO.BCM)
GPIO.setup(GPIO_NUM, GPIO.IN, pull_up_down = GPIO.PUD_UP)
GPIO.add_event_detect(GPIO_NUM, GPIO.FALLING, callback=countPulse)

Obviously we need something to read the value of counter. But, it also needs to convert the count over to a flowrate.

The label on the sensor tells us how to perform that conversion:

The annotation F=7.5*Q(L/min) says that every 7.5 blips is equivalent to a rate of 1L/min.

This means that the math that we need to perform is quite simple: count pulses for a second and then divide the count by 7.5 to get a rate in liters per minute.

# We use this list to track rates between writes
flowrates = []

while True:
    try:
        # Capture some pulses
        start_counter = 1
        time.sleep(1)
        start_counter = 0

        # Calculate the flow rate
        # Pulse frequency (Hz) = 7.5Q, Q is flow rate in L/min.
        flow = (counter / 7.5)

        # Add to the list of observed rates
        flowrates.append(flow)

        counter = 0
        time.sleep(5)

Every iteration a flow-rate is calculated and pushed into the list flowrates.

5 seconds is a far more regular a reading than I really need to submit, but I wanted to take those regular readings and periodically aggregate them before writing out to InfluxDB:

# We use this list to track rates between writes
flowrates = []

while True:
    try:
        # Capture some pulses
        start_counter = 1
        time.sleep(1)
        start_counter = 0

        # Calculate the flow rate
        flow = (counter / 7.5) # Pulse frequency (Hz) = 7.5Q, Q is flow rate in L/min.

        # Add to the list of rates
        flowrates.append(flow)

        # Write out
        if len(flowrates) >= 4:
            # Calculate bounds and average
            stats = {
                "min" : min(flowrates),
                "max" : max(flowrates),
                "mean" : sum(flowrates) / len(flowrates)
            }
            writeStat(stats, session)
            flowrates = []        

        counter = 0
        time.sleep(5)

Now, the script periodically writes out the mean, min and max flowrate for that sampling interval.

After tidying the script up a bit, I was ready to deploy it, so copied it over to the Pi and created a unit file:

[Unit]
Description=FlowRate Monitor
After=multi-user.target

[Service]
Type=simple
ExecStart=/home/pi/tank-monitoring-scripts/app/read_flowrate.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

I then installed, enabled and started it.

sudo -s
cp flowrate.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable flowrate
systemctl start flowrate

Collecting with Telegraf

Whilst both scripts could write directly into my InfluxDB instance, neither script implements a write buffer, so if my Influx instance were to be unavailable (whether because it’s down or because the tank Pi had Wifi problems) those writes would be lost. Probably not the end of the world, but also far from ideal.

Adding a buffer to the Python script would be easy, the InfluxDB Python Client library supports batched writes, however adding it to the BASH script wouldn’t be nearly so straightforward. In either case, if the script had to be restarted, buffered writes would be lost — not ideal considering how little effort I’ve really put into writing them.

So, I decided to have the scripts instead write into Telegraf’s InfluxDB_Listener Input Plugin so that it could provide a buffer against downstream failures. As an additional positive benefit, it also allows collection of system metrics to help monitor the Pi itself.

I’m running Telegraf in docker (because I’d originally intended to dockerize the collection scripts, but have since decided they’re too simple to really be worth the effort).

Telegraf is launched from the following docker-compose.yml:

version: '3.1'

services:
    telegraf:
         image: telegraf
         restart: always
         user: telegraf:995
         container_name: telegraf
         network_mode: "host"
         environment:
            HOST_ETC: /hostfs/etc
            HOST_PROC: /hostfs/proc
            HOST_SYS: /hostfs/sys
            HOST_VAR: /hostfs/var
            HOST_RUN: /hostfs/run
            HOST_MOUNT_PREFIX: /hostfs
         ports:
            - 8086:8086
         volumes:
            - /home/pi/tank-monitoring-scripts/config/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf
            - /var/run/docker.sock:/var/run/docker.sock
            - /:/hostfs:ro

Telegraf’s config file contains the following:

[agent]
  interval = "1m"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = false
  quiet = true
  hostname = "tankmonitor"
  omit_hostname = false

[[inputs.cpu]]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]

# Add the listener  
[[inputs.influxdb_listener]]
  service_address = "127.0.0.1:8086"

# Send it all to the local InfluxDB instance
[[outputs.influxdb]]
  urls = ["http://192.168.3.84:8086"]
  database = "telegraf"

Starting Telegraf is as simple as:

docker-compose up -d

The monitoring scripts can then simply write line protocol into http://127.0.0.1:8086/write as if they were talking to an InfluxDB 1.x instance.

Hardware installation

With the sensors up and working, it was time to connect it all to the tank.

The water flow meter needed to be connected into the filter circuit. It needs to be plumbed into the return flow pipe and not the feed: not only might it get gummed up by unfiltered crud, but we also want to know that the water is actually getting back to the tank rather than (say) leaking from the cannister.

The flow-meter has G1/2 screw threads, so I used a pair of 16mm barb to G1/2 connectors.

I ran into a couple of small issues here:

The connectors arrived without washers.
The tubing for my filter, despite being 20mm externally has a 15mm inner bore (not the more standard 16mm).

The first was a matter of either patience (ordering some o-rings) or sealant (which I had on hand), the second simply required a bit of elbow grease.

Although seemingly well connected and water-tight, it’s still a join and so gets to live in a bucket until I learn to trust it (or, at least, until I next need the bucket).

Next up was mounting the Pi itself: I screwed the base of its case to the tank’s cabinet, making sure it was positioned under an overhang (to help protect against wayward drips).

Finally, the temperature sensor needed to be put into place. I used a wire tie (like those you might use on a freezer bag) to attach it to suckered hook and then installed it in the tank.

I’ve since ordered some suckers with clips intended to carry aquarium air lines — the clip is the perfect diameter for the sensor and looks much tidier than a wire tie.

With everything connected and in place, I plugged the Pi’s power supply in and within a minute, readings started arriving in InfluxDB.

Graphing

With readings coming into the database, the next thing to do was to build a dashboard in Grafana to allow visualization of that data:

The gap at the top right will contain PH information once the probe arrives.

There’s nothing particularly special about the underlying queries, the state indicators at the top simply query consumption and assess whether it’s over a threshold.

from(bucket: "Systemstats/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "power_watts")
  |> filter(fn: (r) => r._field == "consumption")
  |> filter(fn: (r) => r.host == "aquarium-filter-pump")
  |> aggregateWindow(every: v.windowPeriod, fn: last)
  |> map(fn: (r) => ({Device: r.host, 
     _state: if r._value > 9 then
        1
      else
        0
  }))

The pump only uses 10 watts when active, so we check if it’s consuming more than 9 to confirm that it’s actually turned on.

The flow rate indicator just pulls the mean over time:

from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "aquarium")
  |> filter(fn: (r) => r._field == "flow_mean")
  |> aggregateWindow(every: v.windowPeriod, fn: last)

There’s a graph further down the page which shows how the flow-rate bounds vary over time.

The relative consistency of the differences between them is probably reflective of a sampling issue rather than real fluctuations in the pump rate. What we really care about, though, is that water is being pumped at a relatively consistent rate.

Alerting

I wanted to play around with Grafana’s inbuilt alerting, so after adding SMTP details to grafana.ini I added myself as a Contact Point and then proceeded to create a simple alert.

The query used for the alert is:

from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "aquarium")
  |> filter(fn: (r) => r._field == "water_temperature")
  |> mean()

The alert simply calculates the average reported water temperature over the last 5 minutes and sends an notification if it’s not between 22 and 26 (many tropical fish will actually tolerate 20-28C, but these tighter thresholds allow time to respond before the fish get too stressed).

When the alert fires, an email is sent.

If there’s an interruption in data being received (for example because someone accidentally bought a counterfeit temperature probe which has dropped off the bus…) an alert mail goes out to notify of the deadman.

Of course, it’s not just temperature that we want to alert on: because we collect filter flow rate we can alert if it drops below a threshold (indicating either pump issues or a clog in the filter or its intake).

I also have a separate alert with an upper threshold — if the tank suddenly starts pumping very quickly it might indicate that we’re pumping water all over the floor somewhere beyond the meter.

There are also alerts based on power consumption to check that scheduled tasks (like the daily surface skimmer run) actually happened. These are driven by (more or less) the same query used for the status cells in the dashboards.

Once I have the bits for it, I’ll create a PH level based alert too.

Conclusion

It’s fairly straightforward to automate collection of metrics from an aquarium.

Most of the components involved are also relatively inexpensive with the most expensive sensor being the water-flow meter (at £10). Obviously, the project’s cost would have been higher if I hadn’t already had a raspberry pi lying around, although it should be possible to do all of this with a cheap Raspberry Pi Zero or an Arduino Nano.

As with anything involving water and piping, there is a little risk involved in fitting a water-flow meter: any join carries some risk of leakage. If you really didn’t mind the look, that could be mitigated by fitting the meter inside the tank (obviously above the waterline) so that any leak remained inside the tank.

Now that I’ve got the base system setup, it’s not just the forthcoming PH sensor that can be fitted, there are various other probes on the market which could help monitor water quality — for example a Total Dissolved Solids (TDS) sensor could probably be used to predict the levels of ammonium nitrate present in the water column.

The real challenge, as I found with my temperature sensor, is in finding sensor hardware that’s genuine, reliable, and affordable.

As well as providing automated monitoring/alerting, getting this all set up also helps to kill a little bit of time whilst waiting for the tank to cycle.