InfluxData Blog - Bharat Bhat

October Monthly Product Update – InfluxDB New Engine and More!

Bharat Bhat (InfluxData) — Tue, 01 Nov 2022 07:00:00 +0000

We love to write and ship code to help developers bring their ideas and projects to life. That’s why we’re constantly working on improving our product in sync with developer needs to ensure their happiness and accelerate Time To Awesome.

This month is very special. We now have a new engine that significantly increases the “horsepower and torque” for InfluxDB.

We also expanded the Data Explorer UI, made it easier for you to apply custom Data Retention to your organization, and added an onboarding wizard for the Arduino platform – along with fixing bugs, of course.

1 - New engine – unbounded cardinality, SQL, and real-time analytics

InfluxDB Cloud customers using the new storage engine have their cardinality limits lifted completely. Users can write any kind of event data with infinite cardinality and slice-and-dice data on any dimension without sacrificing performance. This opens up use cases such as events, tracing, and all sorts of ephemeral unbounded cardinality data.

The new engine also brings blazing fast queries. Queries that touch a large number of time series are orders of magnitude faster with the new engine than in our previous versions of InfluxDB. Querying across 10 series or 1 million series is all the same. This makes performing real-time analytics – in the way you want – a reality.

The new engine supports SQL natively and our cloud customers will soon be able to connect using Postgres-compatible clients like psql, Grafana’s Postgres data source, and BI tools like PowerBI and Tableau. We will also be rolling out Apache Arrow FlightSQL as the standard matures, giving users high-performance access to millions of rows of time series data.

Here’s what it looks like under the hood:

The new storage engine represents an advanced core that we intend to build many new features upon. Bulk data ingest, bulk data export, and integrations with other third-party systems are all planned for the near future. We’ll also be launching dedicated cloud tiers and a new on-premises Enterprise product based on these new engine capabilities.

To take advantage of all these advancements, please sign up here.

2 - Other key features shipped in October

Along with the new engine, we also released major new features relating to Data Explorer, Custom Data Retention, and onboarding to Arduino. Please take a look at my blog posts below for more information.

Data Explorer - Significant improvements in Raw Data Mode for parsing and display. You also now have the capability to search for multiple measurements at the same time.
Custom Data Retention - With InfluxDB being able to handle hundreds of thousands of data points per second, you might need a way to manage expiring data that is no longer valuable. InfluxDB offers retention policies that automate the process of expiring old data.
Arduino Onboarding - The new Arduino onboarding wizard allows you to write and query data to and from InfluxDB within your first 5 minutes of logging in.

3 - Bug fixes

This month, we fixed a number of outstanding bugs to improve the user experience with InfluxDB:

Improved styling of graphs and single stat visualizations during rendering
InfluxDB University is now accessible from the “Useful Links” section on the homepage
Resolved table header rendering issues for simple table
Prevented decimal place customizations from crashing visualizations

Please reach out to us if you have any specific design or implementation questions. We are always here to help.

Product Update – Faster Data Explorer in Raw Data Mode and Improved Search for Multiple Measurements

Bharat Bhat (InfluxData) — Thu, 20 Oct 2022 07:00:00 +0000

We love to write and ship code to help developers bring their ideas and projects to life. That’s why we’re constantly working on improving our product to meet developers where they are, to ensure their happiness, and accelerate Time to Awesome.

This week, we are covering a product release that we think will help you use and govern your InfluxDB queries more efficiently.

Significant improvements in parsing and display time

When using the InfluxDB UI to perform queries – especially with large datasets or high cardinality – the queries would sometimes take a long time when trying to view the raw data or view in a simple table display. To solve this, we created a faster way to run queries by separating the actual query from the “display” of the query.

We now have the capability to run super fast raw data mode queries. In one of our test cases we reduced a 30 second parsing/display time down to 4 seconds. Enjoy the faster queries!

Multiple measurements

We made a change to the InfluxDB Cloud Data Explorer to enhance the capability to search for multiple measurements at the same time.

In the InfluxDB data model, you can think of measurements kind of like you would think of tables in SQL. Searching for multiple measurements at once or searching all measurements – especially when working with large datasets – can result in very large queries. We found that some users were not getting the performance they expected from these queries. However, for most users – and most use cases – if you were looking for something in a very specific measurement and if the measurement was selected first, instead of a field or tag, the queries were extremely fast.

Unfortunately, while this helped the majority of users, it didn’t help if you didn’t know your measurements before filtering on a field or tag, or you really needed to be able to select more than one measurement at once.

What’s changed?

To better address this use case, we have made two changes to the InfluxDB Cloud Data Explorer:

You can select more than one measurement.
If you need to search a field or tag value first, you can select “All Measurements” from the bottom of the list of measurements — this will then let you select a field or a tag.

Best practices

The new Script Editor still requires a single measurement selection to start your query. We are working to add multi-measurement support in the coming months. In the interim, you can simply comment out the filter statement for the measurement in the displayed Flux.

Despite making these changes to help existing users, it is still recommended to start your queries with a single measurement whenever possible. That is going to be your best performing query. As always, please remember to keep the InfluxDB hierarchy (Fields and Tags within Measurements, and Measurements within Buckets) in mind when designing your schemas.

Please reach out to us if you have any specific design or implementation questions. We are always here to help.

Product Update – Custom Data Retention Periods for Buckets Made Easy

Bharat Bhat (InfluxData) — Fri, 14 Oct 2022 07:00:00 +0000

This week, we are covering a product release that we think will save you time and effort when using InfluxDB with data retention requirements.

What is Data Retention and how does it work with InfluxDB buckets?

InfluxDB can handle hundreds of thousands of data points per second. Working with that much data over a long period of time can create storage concerns. A natural solution is to downsample the data: keep the high precision raw data for only a limited time, and store the lower precision, summarized data for much longer. And when creating a Data Recovery plan, you may want to consider using shorter retention times for the high precision data.

InfluxDB offers retention policies that automate the process of expiring old data. The InfluxDB Cloud data retention enforcement service checks for and removes data with timestamps beyond the defined retention period of the bucket the data is stored in. This service is designed to automatically delete “expired” data and optimize disk usage without any user intervention.

What’s changed?

As a PAYG or Annual Contract user, when creating a TSM bucket, you can now set a custom data retention period. Users are no longer limited to the retention period options displayed in the dropdown. Note: Free users can only retain data up to 30 days.

How do I use the custom data retention options?

Make sure to sign up for a free InfluxDB Cloud account. From there, just follow the steps below. For additional details on Retention Policy configuration, please take a look at our docs.

Of course, please reach out to us if you have any specific design or implementation questions. We are always here to help.

Product Update – Arduino Onboarding Made Easy

Bharat Bhat (InfluxData) — Tue, 11 Oct 2022 07:00:00 +0000

This week, we are covering a product release that we think will save you time and effort when onboarding to time series and InfluxDB using Arduino.

What is Arduino?

The Arduino platform is easy to use for beginners, yet flexible enough for advanced users. It runs on Mac, Windows, and Linux.

Arduino is an open-source platform based on user-friendly hardware and software. Arduino boards are able to read inputs — light on a sensor, a finger on a button, or a Twitter message — and turn them into outputs. The platform can interact with motors, LEDs, buttons, speakers, GPS units, cameras, your smartphone, and the internet. It can also be used for real-time control systems and programmable timing, such as traffic lights.

Quickly onboard with Arduino

The new Arduino onboarding wizard allows users to write and query data to and from InfluxDB using Arduino within their first 5 minutes of logging in. Most notably, the Arduino onboarding wizard eliminates the need for users to spend hours heads-down in documentation to learn how to get started.

Regardless of your operating system or environment setup, this new wizard simplifies the learning experience to get you up and running with InfluxDB.

How do I get started?

Make sure to sign up for a free InfluxDB Cloud account. From there, follow the step-by-step guide to help you get onboarded with Arduino.

Of course, please reach out to us if you have any specific design or implementation questions. We are always here to help.

September Monthly Product Update – Multi-orgs, New Enterprise Features, Telegraf Upgrades, and Adaptive Zoom

Bharat Bhat (InfluxData) — Fri, 30 Sep 2022 07:00:00 +0000

This month, we launched new features and capabilities that span org management, data ingestion, core Enterprise features, and UI improvements.

1 - Multi-orgs - Switch between your organizations

Users that belong to multiple organizations within an account can now quickly navigate between those accounts directly in the InfluxDB Cloud user interface. These users can also set their preferred default account and organization in the UI.

2 - Additional key features shipped in September

Along with Multi-org functionality improvements, we also released major new features relating to Adaptive Zoom, Data Ingestion, and our Enterprise version of InfluxDB. Please take a look at my blog posts below for more information.

InfluxDB Enterprise 1.10 – InfluxDB Enterprise 1.10 provides enhanced support, retention policies, APIs, v2 buckets, and Flux.
Adaptive Zoom (aka Dynamic Query Zoom) – You now have additional granularity on your graphs. InfluxDB Cloud UI users can now also trigger an auto-requery of data with this feature.
Data Ingestion with Telegraf – Improved AWS integration and support for PostgreSQL plugins.

3 - Bug fixes

This month, we also fixed a number of outstanding bugs to improve the user experience with InfluxDB Cloud.

The placement of the chart value is bottom-centered on single-stat charts
Telegraf no longer throws the following error when attempting to create an additional telegraf configuration: Warning: Cannot update a component (Wizard) while rendering a different component
Function behind shaded graphs ensures that the shading renders properly
Fixed the styling of the Telegraf configuration styling overlay to correctly position of content of the page
The new globalheader renders well with the rest of the UI, without page-cut-offs or notebook issues.

Product Update – Adaptive Zoom now live

Bharat Bhat (InfluxData) — Fri, 23 Sep 2022 07:00:00 +0000

This week, we are covering a product release that helps all InfluxDB Cloud UI users get more from their graphs.

Get granular on your graphs

Adaptive Zoom – also referred to as Dynamic Query Zoom – allows InfluxDB Cloud UI users the ability to trigger an auto-requery of data. If you create line graphs or single stat graphs using Query-Builder – or explicitly with the v.windowPeriod variable – this functionality can be very useful.

Upon zooming in, you will see more granular information in your graphs. That is, the window period of the graph will dynamically update upon zoom and you can view the updated window period at the top right of the graph cell. If you prefer to zoom without triggering a re-query, you have the option to toggle-off Adaptive Zoom in the customizable graph panel. If you choose to statically assign a constant value to the window period via the script editor – a.k.a. hard code the value – we recommend that you toggle-off Adaptive Zoom.

By default, Adaptive Zoom is turned on for all existing cells and any new cells. Lastly, to visualize the window period of a cell, look to the top right corner of the cell. Here are some helpful definitions while playing around with Adaptive Zoom and the Cloud UI:

Window Period - Time slices within a window in which an aggregate function has been applied.
Window - Timed boundary of the data returned in a query.
Refresh Rate - Rate at which a query will be auto-refreshed (re-running the query).

Please reach out to us if you have any questions on using this new feature or InfluxDB. We are always here to help.

August Monthly Product Update – Improved Tasks, CLI Onboarding, and a New OSS Distribution

Bharat Bhat (InfluxData) — Fri, 02 Sep 2022 07:00:00 +0000

This month is special. We have many features that we think you will love when onboarding or continuing to use InfluxDB.

We also made it easier for you to get started with any client language. Our new CLI Onboarding tools enable you to get set up and start coding right away.

1 - Key features shipped in August

We released major new features relating to Tasks, Onboarding, and our Open Source version of InfluxDB. Please take a look at my blog posts below for more information.

Invokable Tasks - Tasks can now invoke Scripts and pass variables to these Scripts, greatly increasing your ability to manage, reuse, and scale your code and processes.
CLI Onboarding - The new CLI onboarding wizard allows you to interact with InfluxDB’s command line interface to read and write data within the first 5 minutes of logging in.
New OSS Release - InfluxDB OSS 2.4 comes with new core features, Flux enhancements, and additions to the API that support features of the CLI.

2 - Bug fixes

This month, we also fixed a number of outstanding bugs to improve the user experience with InfluxDB Cloud.

Ensured that the raw data view toggle stops freezing the browser with large datasets.
Ensured that dropdowns with predictive text search bars can successfully handle large lists.
Improved usability of sorting table columns on the Data Explorer page.
Reordered the tabs in the left side account navigation to match the new header navigation drop-down options to drive consistency.
Ensured that CSV file downloads from the Data Explorer page properly replace spaces with underscores.
Updated URLs to contain the same naming as the UI components to drive a consistent user experience.
Ensured that labels are no longer required on buckets in order to save and use buckets.

Product Update – New InfluxDB OSS 2.4 Now Available With Significant Improvements

Bharat Bhat (InfluxData) — Fri, 26 Aug 2022 07:00:00 +0000

This week, we are releasing a new InfluxDB OSS version which contains multiple new features, security updates, and bug fixes. Here are the full Release Notes.

InfluxDB OSS 2.4 gains improved user model and CLI additions

This version of InfluxDB comes with new core features, Flux enhancements, and additions to the API that support features of the CLI.

User model and CLI

Operator Model changed to allow Admin user instance-level permissions without operator token.
Added influx remote command.
Added influx replication command.
Enhanced error messaging for InfluxDB and OSS specific commands.
Added api/v2/config endpoint to display the runtime configuration.
Full CLI additions Release Notes
Added back the v1 InfluxQL shell/REPL. More details in the v1 Shell blog.
Supported v1 dbrp names as bucket IDs in replications API, making the creation of replications easier and adding support for replicating to an Enterprise cluster’s v2 /write endpoint.

Flux enhancements highlights

Flux upgraded from 0.171.0 to 0.179.0
Added multiple new join functions to the join package such as join.full()
Promoted experimental.to() to influxdata.influxdb.wideTo()
Added built-in function time to the date package to convert any timetable into datetime
Promoted the following functions from experimental.array into the array package: array.concat(), array.filter(), array.map().
Promoted the following functions from experimental.http.requests into the http.requests: http.requests.do(), http.requests.get(), http.requests.peek(), http.requests.post()
Promoted experimental.bitwise into the bitwise package
Supported conditional expressions, constants, and literals in vectorized map()

Security updates

An issue in the gopkg.in/yaml.v3 package import that could lead to a DoS in the templates service.
An issue in the github.com/buger/jsonparser package import that could potentially lead to a DoS in storage authorization.
Cumulative security fixes for Go 1.17.8-1.18.3 are included in this release. These fixes resolve the following InfluxDB issues:

An issue with processing large PEM files that could lead to a DoS in the templates service or Flux connections using to()
An issue in TLSv1.3 and a lack of ticket randomness
A minor issue with filepath.Clean() on Windows

Bug fixes

Fixed race condition with remote replicating deadlocks that the remote writer prevented from writing to the target.
Resolved error when creating v1 authorization (v1 auth) with a nonexistent bucket ID.
Added fields to the _tasks bucket to match schema of the same bucket in InfluxDB Cloud – this provides consistency for clients accessing both.
Fixed rare case where measurement cardinality reported less than zero.
Resolve panic on cleaning up failed iterators.

Product Update – CLI Onboarding Wizard Now Available

Bharat Bhat (InfluxData) — Fri, 19 Aug 2022 07:00:00 +0000

Thanks to Amy Luckette for being a key contributor to this article.

This week, we are covering a featured product release that we think will save you time and effort when onboarding to time series and InfluxDB.

Quickly onboard with InfluxDB Command Line Interface (CLI)

The new CLI onboarding wizard allows you to interact with InfluxDB’s command line interface to read and write data within the first 5 minutes of logging in, improving your Time to Awesome!

Most notably, the CLI onboarding wizard eliminates the need for you to spend hours heads-down researching in the documentation to learn how to use the command line interface. Regardless of your operating system or environment setup, this new wizard simplifies the learning experience to get you up and running quickly with InfluxDB.

Why use a CLI?

CLIs in general have some great advantages:

Less resources
High precision
Repetitive-task friendly
Powerful commands and processes
For both average users and experts

With InfluxDB’s CLI, you can manage many aspects of your InfluxDB environment and use influx help to see all the commands.

How do I get started?

Make sure to sign up for a free InfluxDB Cloud account. From there, follow the step-by-step guide here to help you get onboarded with the InfluxDB CLI.