InfluxData Blog - Balaji Palani

Announcing the General Availability of Our New High-Performance Time Series Engine in InfluxDB Cloud

Balaji Palani (InfluxData) — Wed, 01 Feb 2023 07:00:00 +0000

Back in October 2022, our Founder and CTO Paul Dix announced the limited release of InfluxDB IOx, our new database engine. After several months of beta testing, we’re excited to announce the next phase of our database engine: general availability. As of today, InfluxDB IOx releases to the rest of the world as the new and improved InfluxDB Cloud.

All time series workloads and all the performance

There are several improvements and updates that we’re thrilled to bring to a global user base. While InfluxDB has always excelled at handling metrics, the new engine can handle metrics, events, and traces with dramatic improvements in performance and efficiency.

What exactly does this mean? Let us go a little deeper.

Metrics are essentially time series data polled at a regular interval from the same source. They are mostly quantitative and useful to understand the availability or performance of the specific service or device being monitored.
Events are state changes triggered by some observed condition(s), e.g., deployed code, container crashes, HTTP 5XX errors, etc. Events can be semi-structured and typically contain more context as tags so they can be correlated with metrics to gain additional insights.
Traces capture information to show the request propagation in a distributed system. Depending on the complexity of the application or service, traces can contain many spans which could be conceptually similar to a wide table.

Currently, there are no other time series datastores available that are optimized to index, store, and analyze metric, event, and trace data together in one place.

InfluxDB Cloud, with the new high-performance time series engine, can ingest, store, and analyze high volume and high velocity data in real-time, and derive metrics on the fly from high granularity events. This expands the number and variety of use cases that InfluxDB can handle, such as observability and distributed tracing, which rely on high cardinality data. Thanks to InfluxDB Cloud’s new columnar structure and compression abilities, the database can handle tag values of unbounded data with no drop in performance. Taken together, these improvements lead to real-time, in-depth insights. This is a huge development and we’re thrilled about all the benefits users will get from it.

Lower cost of ownership

Speaking of data compression, InfluxDB Cloud now uses Apache Parquet as its persistence format. Not only does this contribute to the drastic improvements in compression, but it also enables users to extend the value of their time series data by transmitting it to other systems and ecosystems that utilize Parquet. Perhaps most significant though is that Parquet’s compression means that users can save more data using less space on cloud object storage, like Amazon S3, which is very inexpensive compared to SSD or Elastic Storage Block, reducing storage costs.

Optimized for low latency queries

Ingesting and storing data in InfluxDB is only half the story. Being able to query and use all that data effectively is just as important. We optimized the new engine for performance thanks to several under-the-hood updates. While concepts like parallelism, pushdowns, and in-memory caching are critical aspects driving low latency query performance, what really matters for users is that fast, reliable queries improve user experiences.

Improving developer efficiency

At InfluxData, we talk a lot about how we meet developers where they are. That same sentiment continues with the new InfluxDB Cloud. The most significant development in this regard is the addition of native SQL queries, thanks to Apache DataFusion. Users can still use InfluxQL and Flux language for more advanced data processing, too. But the bottom line is that you can choose the query language that’s right for your needs and maximizes your productivity.

The fine print

With so much great stuff becoming available today, the big question is how to get it. The rollout of the new database engine will occur in stages. Today marks the first stage and it’s available on two AWS regions – Virginia (us-east-1) and Frankfurt (eu-central-1). As of today’s GA launch, everyone who signs up for InfluxDB Cloud in either of these regions will automatically have all the functionality of the new database engine.

When thinking about interoperability with other tools, we have native integrations ready to go at launch. These include Apache SuperSet, Grafana for data visualization, and Flight SQL for interoperability with compatible tools.

More coming soon

Again, this is the first phase of a much larger rollout. Keep an eye out in the coming weeks and months for expanded offerings with the new database engine. We plan to expand the number of regions and cloud providers running it and to update the new engine across our product suite, including InfluxDB Enterprise. We will announce additional regions and cloud provider support as they occur. This includes more AWS regions, as well as Microsoft Azure and Google Cloud.

Our current plan is to provide support for business intelligence tools such as Tableau and PowerBI. We also plan to provide migration tools for those who want to move from the TSM engine to the new database engine.

As of today, you can sign up for the new InfluxDB Cloud powered by IOx here.

New API Tokens UI

Balaji Palani (InfluxData) — Tue, 11 Jan 2022 04:00:42 -0700

InfluxDB Cloud allows users to create API tokens that are used for authentication and authorization to sets of resources when interacting with our API. We recently made changes to the user interface so that after generating a token, you will need to immediately store it in a secure vault of your choosing for safekeeping.

We made this change to conform to industry best practices around both token generation and retrieval.

After initial viewing, the token value itself will become hidden, and you will no longer be able to view it later through the InfluxDB Cloud user interface. Users will still be able to view all other information about a token such as when it was created, who created it, when it was last used and what permissions it grants.

What do I do if I can't retrieve my token?

Because you can no longer retrieve a token via the UI, you should generate a new token with the same permissions using the “Clone” option (described below). Take care to store this new token in a secure vault of your choosing.

What other changes have been made?

Users can now create an all access API token, or custom API tokens by selecting any combination of individual buckets, Telegraf configurations and the many other resources within the InfluxDB Cloud user interface. Previously, the only permissions exposed were the read/write access for all or individual buckets.

You can also filter across buckets, Telegraf Configurations and all other resources using the filter search box on the creation overlay as well as the edit overlay.

If you don’t provide a description for a token, one will automatically be generated for you. But it is good practice to name a token based on things like the resource(s) it is for, the level of access, whether it is read or write etc. This will make it easier to remember what a token is for in the future. And also remember to name each token similarly when you store it in the safe vault of your choosing.

Can I disable a token?

While viewing the token itself, you can update the status to be active or inactive as well as update the description of the token.

Can I create a new token with the same permissions as an existing token?

You now have the ability to clone a token. This will create a new token with all of the permission settings from the original token selected. This displays a new token value that can be copied and stored in a secure vault of your choosing. You can also rename this new token, so you can tell them apart.

Where can I learn more about InfluxDB API tokens?

This past blog post has a lot of great information and tech tips on using API tokens, though it does not mention the new features of this post.

Release Announcement: InfluxDB Cloud Now Available in Europe

Balaji Palani (InfluxData) — Thu, 14 Nov 2019 00:05:55 -0700

Since the launch of our new InfluxDB Cloud platform in September, we have been busy working on the features that our CTO Paul Dix talked about in his blog. For example, a few weeks ago, we launched a beta for AWS Europe support and received great feedback from our community. Today, we are excited to announce that this support for AWS EU (Frankfurt) is now Generally Available for all InfluxDB Cloud customers and users.

To run InfluxDB on AWS Frankfurt, choose EU Frankfurt at the time of signup, like this:

Here are a few reasons you might want to do this:

Legal: Your organization might be required to store data in the European Union due to EU data residency laws, or have a preference for doing so.
Latency: If your applications are already running on the AWS region eu-central-1, and you'd like to keep your InfluxDB time series data in the same region as those apps to minimize any latency issues.
Costs: Amazon charges to transfer data between AWS regions. If you used InfluxDB cloud on US West, and wanted to copy some of that data to AWS Frankfurt (or vice-versa), you'd be charged for data transfer. By keeping everything in one AWS region, you eliminate those charges.
Redundancy: Telegraf makes it easy to send data to multiple instances of InfluxDB. You can configure Telegraf to send data to InfluxDB Cloud on both AWS West and AWS Frankfurt. If one of those AWS regions goes down, you still have the other AWS region to fall back on.

With this release, InfluxDB Cloud is now running on AWS in US-West-2 (Oregon) and EU-Central-1 (Frankfurt) regions. We’ll be rolling out to other AWS regions and making the service available on Google Cloud and Azure in the coming months. If you’d like to be notified when other regions and cloud provider support launch, you can sign up here.

Additionally, this release includes the following capabilities:

Customers who want to remove their Free Plan rate limits can simply switch to usage-based pricing and pay only for what they use.
Secure secret store to help you secure your sensitive credentials, use InfluxDB's integrated secret store and Flux's new secrets library.
Developer-friendly and powerful client libraries for users to send their custom metrics from their application directly into InfluxDB Cloud — including support for C#, Go, Java, JavaScript/Node.js, and Python.
Additional dashboard templates to help reduce time to awesome — like the updated System dashboard below:

Start your InfluxDB Cloud journey today by signing up for your own free account.

And if you’re at AWS Reinvent next month, come by booth #713 for a demo of InfluxDB Cloud running on AWS Frankfurt.