InfluxData Blog - Philip O'Toole

Make your mark on the InfluxDB source

Philip O'Toole (InfluxData) — Thu, 05 Nov 2015 08:00:46 -0700

InfluxDB is an open-source project, and we rely heavily on the community when it comes to development. Bug reports, performance numbers, and helping each other, as we iterate on the software, are all very helpful. In fact, it is in these ways that the community makes its biggest impact.

Of course, there is another way to contribute to InfluxDB, and that is to actually improve the source code. Knowing that code you wrote will be run by thousands of developers and companies worldwide can be a real source of pride.

Help Wanted!

To help you make your mark on the InfluxDB source code, we on the Core team recently selected a set of issues that we think are the most approachable. All the issues are logged on Github and are marked with the help-wanted label. We have tried to add some introductory notes to each issue to help you get started, and will make a particular effort to answer questions about these issues as you proceed. Some issues are more difficult than others, and have been labelled as such.

Before you start working with the source, be sure to check out the CONTRIBUTING guide to learn how to configure your build enviroment. You will also need to sign the CLA before any of your changes can be merged.

Get your t-shirt

As an added-incentive, anyone whose PR is merged to master will get an InfluxDB t-shirt. Just remind us in the Issue when the change is merged, and we’ll contact you for shipping details. Wear the shirt with pride, knowing your code is running across the world!

So start coding!

We know it can be difficult to get up to speed on a complex code base like InfluxDB, so we hope this will help make it fun. Learning how a time-series database works can be a fascinating experience, and there is no better way to do so than getting into the code. Hopefully these particular issues will help you get going.

Testing InfluxDB Storage Engines

Philip O'Toole (InfluxData) — Tue, 20 Oct 2015 08:00:34 -0700

When you decide to build a database, you set yourself a particular software engineering challenge. As infrastructure software it must work. If people are going to rely on your system for reliably storing their data, you need to be sure it does just that.

InfluxDB is tested in various ways. We use CircleCI for unit-level testing, as well as some basic integration-level testing. We find CircleCI to be easy to use, well-designed, and responsive. But as we approach the 1.0 release our testing is becoming more sophisticated and thorough.

Correct software is obviously critical – every data point received must be indexed, and every query must return the correct results. But resource usage – CPU, disk IO, and RAM – are just as important. We want a system that is stable when running, and monitoring resource usage during testing can flag issues. Memory leaks become apparent, and excessive disk IO can indicate a sub-optimal design or implementation. Bugs too, may be exposed – a pegged CPU may indicate a problem in the code. So in this post we discuss how we monitor resource usage on systems under test.

Test Infrastructure

A high level view of our test infrastructure is shown below.

On each node under test we run collectd, configured to output in Graphite format, and Telegraf. Data from these agents is then sent to another InfluxDB system, which we run specifically to store metric data from our test systems. This setup achieves two objectives – it allows us to analyze the test results and means we are also dogfooding our own software.

While an InfluxDB system is under load we record various metrics about the host machine during the entire test run. Most importantly we record disk IO, memory usage, CPU load, and the resident set size of the InfluxDB process. During and after a test the results are shared with the team, and with the help of our Grafana dashboards, we study the results for problems, and (hopefully!) confirm our software is working as expected.

Testing the TSM1 Engine

We recently starting testing the new tsm1 storage engine. A recent test ran for about 8 hours and involved writing billions points to a single InfluxDB node, across 1000s of different series. The target retention policy also had a duration of 1 hour, so we could test those code paths too – since old data would be deleted hourly as new data was being indexed. The Grafana dashboard, showing results of the complete test run, is shown below.

Some interesting features are present. Write load is steady, as is CPU load. Disk usage reached a steady-state as the retention enforcement deleted data at about the rate incoming data was being indexed. The sawtooth pattern in disk usage is a mixture of compaction performed by the tsm1 engine, as well as retention enforcement. Interestingly InfluxDB RSS correlates closely with disk usage, which makes sense to us. The software memory-maps data on disk, so as data on disk is deleted, memory usage declines. But memory usage is generally steady over the long run, at about 30% total physical RAM on the host machine. This tells us this software does not suffer any detectable memory leaks. (The short spike at the very left of each graph is an aborted run.)

By comparison, the bz1 engine consumes resources in a more irregular manner, as shown below.

Test Environments

We run this kind of testing on a mixture of systems – AWS EC2 instances, Digital Ocean droplets, and physical machines. Physical machines are a particularly important part of our test infrastructure as they allow us to focus on our software – after all nothing is changing between test runs except our code. We don’t have to worry about noisy neighbours or busy networks – running on physical hardware allows us to rule out all factors except the ones we control. But, of course, testing with cloud environments is very important since so many of our users run InfluxDB systems in such environments.

We also use Ansible for our test system deployment and configuration management.

Test Strategies

We recently started building race-detection enabled builds alongside our nightly builds. This allows us to run the test suite against these binaries, which helps us detect race conditions in our code. We also differentiate between stress testing – loading a system to the extreme until it falls over – versus moderate-to-high load testing, which we expect to run for days without any problems. We call this latter testing “burn-in”.

Plans for the Future

We have much more to do as we ramp up testing. Unit and basic integration testing can only take you so far, and it’s important to run tests that last for hours and days. Other key features – such as clustering – are still Beta so as new features come online testing in those areas will increase. Query performance is another key area, which will undergo significant work and testing in the near future.

As Richard Feynman once said: “For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” And our testing makes sure we don’t fool ourselves.

How to Use the SHOW STATS Command and the _internal Database to Monitor InfluxDB

Philip O'Toole (InfluxData) — Tue, 22 Sep 2015 08:00:50 -0700

InfluxDB supports statistics and diagnostics meta queries which allows developers and system administrators to make better use of their InfluxDB system, diagnose problems, and troubleshoot issues.

This post outlines some of statistics and diagnostics currently gathered by InfluxDB, and some advice on how to work with this information.

Who watches the watchers?

A common use of InfluxDB is the monitoring and analysis of IT infrastructure. And to run a successful InfluxDB system, the database itself must be monitored. The command SHOW STATS allows you to do just that. SHOW STATS returns information about the various components of the system, for the node receiving the query. Each module exporting statistics exports a Measurement named after the module, and various series are associated with the Measurement. (The fact that it is a Measurement is important, as will be seen shortly.)

Let’s take a look at the runtime statistics, which capture details about the Go runtime.

> show stats

name: runtime
-------------
Alloc   Frees   HeapAlloc       HeapIdle        HeapInUse       HeapObjects     HeapReleased    HeapSys Lookups Mallocs NumGC   NumGoroutine    PauseTotalNs    Sys             TotalAlloc
4056352 15134   4056352         1712128         4874240         7001            0               6586368 71      22135   4       51              1573952         10918136        13093576

In this case SHOW STATS gives you an overview of memory usage by the InfluxDB system, within the Go runtime. Many Go developers will recognize the importance of these numbers.

Another key statistic is the httpd module:

name: httpd
tags: bind=:8086
query_req       query_resp_bytes        req
---------       ----------------        ---
2               418                     2

This output shows the number of queries received (query_req), by this node, since the system started – 2 in this example – and the number of bytes returned to the client, 418 in this case (this system just started!).

Most inputs, such as Graphite and openTSDB, also have detailed statistics available. This can be particularly useful when working with these systems. We get plenty of questions about performance of these inputs, so this statistical information can be really useful.

Here are example statistics for the Graphite input:

name: graphite
tags: bind=:2003, proto=tcp
batches_tx      bytes_rx        connections_active      connections_handled     points_rx       points_tx
----------      --------        ------------------      -------------------     ---------       ---------
62              1658490         6                       6                       69006           62000

This shows the number of points received by the Graphite service on port 2003 (points_rx), for the TCP protocol. It also shows the number of points sent to the database (points_tx). If you notice points_rx is greater than points_tx. This shows that the Graphite input is buffering points internally, as it batches writes into the database for maximum throughput.

These are just a few quick examples of what SHOW STATS can do. Keep in mind that depending on what services are enabled, and what code paths execute within the database, you may see statistics from other components.

The internal database

All this statistical information is very useful, but is reset when the system restarts. What if we want to analyze the performance of our system over time? Of course, InfluxDB is a time series database, built especially for storing this kind of data. So the system periodically writes all statistical data to a special database called _internal, which allows you to use the full power of InfluxQL to analyze the system itself.

Some examples may help.

If you have questions about how InfluxDB is using the Go heap, it’s easy to see how usage changes over time. For example using the influx CLI, issue the following queries to see Go heap usage every 10 seconds.

> USE _internal
Using database _internal
> SHOW MEASUREMENTS
name: measurements
------------------
name
graphite
httpd
runtime
shard
write

> SELECT HeapAlloc FROM runtime LIMIT 5
name: runtime
-------------
time                            HeapAlloc
2015-09-18T18:40:04.199587653Z  548536
2015-09-18T18:40:14.199761008Z  3895536
2015-09-18T18:40:24.199791989Z  2057504
2015-09-18T18:40:34.19971719Z   2111680
2015-09-18T18:40:44.199490569Z  2169848

Even better, when coupled with a tool like Chronograf, you can visualize all this data.

The next example of a query, also visualized using Chronograf, shows a derivative query of the total garbage collection (GC) pause time of the Go runtime. Since this graph shows rate-of-change, the spikes in the graph show when a GC pause took place.

Cluster-level statistics

Because every node in your cluster writes these statistics to the _internal database, queries against _internal return data for the whole cluster, which can be very useful. However, all data is tagged with the hostname and node ID, so analysis of a specific node is always possible. Shown below is points_rx for the Graphite service on just the node with hostname malthus.

> SHOW TAG VALUES WITH key=hostname
name: hostnameTagValues
---------------------
hostname
malthus
> SELECT points_rx FROM graphite WHERE hostname='malthus' LIMIT 5
name: graphite
--------------
time                            points_rx
2015-09-18T18:40:54.199425753Z  141001
2015-09-18T18:41:04.19947468Z   315608
2015-09-18T18:41:14.1993757Z    476001
2015-09-18T18:41:24.199438213Z  641001
2015-09-18T18:41:34.199454694Z  802001

But remember, the commands SHOW STATS and SHOW DIAGNOSTICS only ever return data for the node on which the query executes.

Expvar support

All statistics data are available in standard expvar format, if you wish to use external tools to monitor InfluxDB. This information is available at the endpoint /debug/vars.

Diagnostics

Diagnostic information is treated a little differently within the InfluxDB system. It’s mostly information about the system that is not necessarily numerical in format. It is important to note that diagnostic information is not stored in the _internal database.

Example data is the build version of your InfluxDB and its uptime. This information is particularly useful to InfluxDB Support, so be sure to include the output of this query anytime you file a Support ticket or GitHub issue.

Example output is shown below.

> SHOW DIAGNOSTICS
name: system
------------
PID     currentTime                     started                         uptime
7299    2015-09-18T20:32:22.219545782Z  2015-09-18T19:54:04.069260449Z  38m18.150285438s


name: build
-----------
Branch  Commit                                   Version
master  d81618c57fae135d9b1c1a8fb3403722ceb29354 0.9.4


name: runtime
-------------
GOARCH  GOMAXPROCS      GOOS    version
amd64   8               linux   go1.5


name: network
-------------
hostname
malthus

More to come

As always, we encourage open-source developers to add statistics and diagnostics to any code they contribute, if it makes sense. We hope you find this information useful, as you work with InfluxDB. If you have any questions or concerns, come join us in our Community Slack.