InfluxData Blog - Caitlin Croft

Webinar Highlight: Introducing InfluxDB’s New Time Series Database Engine

Caitlin Croft (InfluxData) — Wed, 01 Mar 2023 07:00:00 +0000

As part of the InfluxDB Cloud, powered by IOx launch, Paul Dix and Balaji Palani provided an InfluxDB Cloud overview and demo. In case you missed it, this blog is a quick 5 minute read summarizing the webinar. We shared the recording and the slides from the presentation for everyone to review and watch at your leisure.

TL;DR:

InfluxDB Cloud, powered by IOx is currently available in two AWS regions - Frankfurt, Germany or Virginia, USA
It is a columnar database written in Rust using the Apache Arrow ecosystem that supports SQL natively
Includes native InfluxQL support & enables unlimited cardinality

Key timestamps:

:36 | Paul Dix provides an overview and roadmap update for InfluxDB Cloud, powered by IOx
22:47 | Balaji Palani demos InfluxDB Cloud and Apache Superset visualizations
38:40 | Q&A time

Webinar highlights

Overview

InfluxDB Cloud, powered by IOx is a columnar database built on Apache Arrow with cloud environments in mind where the object storage and scalable compute layer are separate. The database uses object storage for persistence and Apache Parquet is the persistence format. Developers can use InfluxDB Cloud to query data in real time before it lands in object storage. It supports SQL natively by including a SQL query, parser, planner, and execution engine. InfluxDB Cloud uses Apache DataFusion as the query engine. Paul dives deeper into the differences between the two different storage engines (TSM and IOx), the benefits of using the Apache ecosystem, and schema design tips and tricks.

“InfluxDB Cloud, powered by IOx is the cloud columnar database that is optimized for time series, including workloads and queries.” - Paul Dix | Founder and CTO, InfluxData

Unlimited cardinality!

Paul Dix discussed the cardinality problem, which occurs when having a high number of time series affects system performance and can impact the cost of data ingestion. Queries would slow down, especially if you wanted to run computations across millions of individual unique time series. InfluxDB Cloud, powered by IOx’s underlying design and architecture removes this limitation.

IOx’s capabilities enable our team to develop a bunch of features in the coming years. Community members will be able to scale the compute layer up and down dynamically without much manual work. It will be faster for developers to upload historical data in bulk. We addressed the needs of those collecting high precision data at the edge who don’t need their data in a centralized store for real-time querying, but need it for historical analysis:

“Our goal is to provide a place for developers to send all of the data, metrics, events, traces and log data….InfluxDB Cloud, powered by IOx is the ideal place to ingest data in real time and build automation systems on top of it, and be able to do historical analysis on it.” - Paul Dix | Founder and CTO, InfluxData

Why Rust?

Paul Dix believes Rust is essential for the future of system software. The Rust compiler safeguards against data erasures and other potential bugs. InfluxData’s developers are looking forward to taking advantage of how embeddable Rust is into other systems and languages.

“It gives you fine-grain control over memory, the safety of a higher-level language, and has a great model for concurrent applications.” - Paul Dix | Founder and CTO, InfluxData

Apache Arrow

The team sees Apache Arrow Flight SQL as the new standard for database systems to transfer data between clients and servers. Andrew Lamb is part of our engineering team and is the Chair of Apache Arrow’s Project Management Committee (PMC). Apache Arrow has been quickly adopted by data scientists and by data warehousing and big data engineers. Paul dives into the importance of using Apache Parquet as the persistent format and Apache Arrow Flight SQL. We even built a Flight SQL plugin for Grafana that enables users to build reports and dashboards common in traditional BI tools. There are many more integrations coming soon to InfluxDB Cloud!

“Our goal is to make Flight SQL the standard for larger database ecosystem vendors and make it easy for third-party tool developers to adopt.” - Paul Dix | Founder and CTO, InfluxData

Demo time!

Watch Balaji Palani, VP of Product Marketing, provide a demo of InfluxDB Cloud, powered by IOx. Balaji demoed the new Data Explorer where developers can write SQL queries and review schemas. Balaji shows how to visualize your time-stamped data in Apache Superset and how you can combine metrics and traces to create dashboards. Developers can now collect metrics, events, and traces in a single data store with InfluxDB. Check out the InfluxDB Observability repo.

Q&A

Attendees had lots of questions. Here are some of the most asked questions and their answers.

Question: What are the benefits to using InfluxQL over SQL?

Paul: Whatever is in SQL will be a superset of what’s available in InfluxQL, as InfluxQL uses the same underlying engine. We’ve heard from people over the years that for some basic time series queries, they find InfluxQL easier to use. The primary benefit for existing users is that InfluxDB Cloud, powered by IOx will support InfluxQL natively; there will be a translation layer that will expose the InfluxDB V. 1 query API layer. You’ll be able to submit a query as though it were in InfluxDB V. 1 with InfluxQL. It will execute and return your results in the same format that InfluxQL 1 did, which means, if you’re using third-party tools like Grafana, you’ll just be able to interact with it as though it’s an InfluxDB V1 database without having to rewrite your dashboards.

Our goal is to make it as close to the original InfluxQL as possible, while also taking advantage of the database’s new performance benefits. Many queries will be orders of magnitude faster on IOx, than they will be on the traditional InfluxQL engine. And obviously, requests for better performance have been ongoing for the entirety of the project.

Question: Is Flux supported in InfluxDB Cloud, powered by IOx?

Paul: Flux is enabled in InfluxDB Cloud’s API, but it is not in the user interface. For people who want to use Flux, they can do so in the API. We’re currently pushing SQL in the UI. Once native InfluxQL support is added, we will likely enable it in the UI, too. Flux isn’t just a query language — it’s an entire scripting language and we currently don’t have the bandwidth to add it into a Rust-native implementation. Unfortunately, it means a lot of the query optimizations that happen within influxDB Cloud, powered by IOx are not available in Flux. Flux is essentially acting as a scripting client that’s pulling back a bunch of data and doing some things. For the best performance, we’re encouraging people to use SQL or, when it comes out, InfluxQL.

Question: Now that it’s written in Rust, are there any changes to bucket creation or data ingestion?

Paul: The API for creating buckets is the same. If you signed up for InfluxDB Cloud prior to the launch, you will be using TSM, the previous storage system. Within your account, you can create a new organization which will be IOx-based. In the coming months, we will be migrating existing InfluxDB Cloud accounts from TSM to IOx. We aim to have all users upgraded to InfluxDB Cloud, powered by IOx by the end of year.

Question: Will it be possible to write InfluxDB tasks in SQL?

Paul: Yes, that’s part of the plan. We want to enable people to write tasks in SQL. We’re looking at adding support for Python or JavaScript for scripting within the database, too. But this is currently TBD. Having the ability to write tasks in SQL and InfluxQL is definitely part of the roadmap.

Question: What is the product timeline for InfluxDB OSS and InfluxDB Enterprise?

Paul: The team is primarily focused on building more InfluxDB Cloud features. We’re planning to have cloud-dedicated clusters available in late April. By August, we aim to update InfluxDB OSS and InfluxDB Enterprise. Stay tuned for more updates from the team.

Helpful links:

Sign up for InfluxDB Cloud here
InfluxDB Cloud, powered by IOx GitHub repo

To check out the full webinar and listen to the rest of the Q&A, click here!

How to Use Time-Stamped Data to Reduce Network Downtime

Caitlin Croft (InfluxData) — Fri, 10 Feb 2023 07:00:00 +0000

This article was originally published in The New Stack and is reposted here with permission.

Telecommunication organizations need to ensure they have the necessary resources and technology to maintain service uptime SLAs.

Increased regulations and emerging technologies forced telecommunications companies to evolve quickly in recent years. These organizations’ engineers and site reliability engineering (SRE) teams must use technology to improve performance, reliability and service uptime. Learn how WideOpenWest uses a time series platform to monitor its entire service delivery network.

Trends in the telecommunications industry

Telecommunication companies are facing challenges that vary depending on where the company is in their life cycle. Across the industry, businesses must modernize their infrastructure while also maintaining legacy systems. At the same time, new regulations at both the local and federal levels increase the competition within the industry, and new businesses challenge the status quo set by current industry leaders.

In recent years, the surge in people working from home requires a more reliable internet connection to handle their increased network bandwidth needs. The increased popularity of smartphones and other devices means there are more devices requiring network connectivity — all without a reduction in network speeds. Latency issues or poor uptime lead to unhappy customers, who then become flight risks. Add to this situation more frequent security breaches, which then requires all businesses to monitor their networks to detect potential breaches faster.

Challenges to modernizing networks

Founded in 1996 in Denver, Colorado, WideOpenWest (WOW) provides internet, video and voice services in various markets across the United States. Over the years, WOW acquired various telecommunication organizations, and as its network expanded, it needed a better network monitoring tool to address a growing list of challenges. For instance, WOW engineers wanted to be able to analyze an individual customer’s cable modem, determine the health of a node and understand the overall state of the network. However, several roadblocks prevented the company from doing so. WideOpenWest already used multiple monitoring platforms internally, and the cost of purchasing hardware that aids in monitoring individual nodes was too expensive. It already had a basic process in place to collect telemetry data from specific modems, but there was no single source of truth to tie everything together.

Using time series data to reduce network latency

A few years ago, WideOpenWest decided to replace its legacy time series database, and after considering other solutions, it chose InfluxDB, the purpose-built time series database. It now has a four-node cluster of InfluxDB Enterprise in production and a two-node cluster running on OpenStack for testing. The team uses Ansible to automate cluster setup and installation.

The primary motivations for using InfluxDB are to improve overall observability of the entire network and to implement better alerting. The WOW engineers use Telegraf for data collection whenever possible because it integrates easily with all the other systems. Some legacy hardware requires them to use Filebeats, custom scripts and vendor APIs.

They make extensive use of Simple Network Management Protocol (SNMP) polling and traps in the data collection process because that remains an industry standard, despite its age. Specifically, they use SNMP to collect metrics from cable modems and Telegraf to collect time-stamped data from their virtual machines and containers. Using InfluxDB provided the team with the necessary flexibility to work around restrictions from vendor-managed systems, and they now collect data from all desired sources.

Next they stream the data to Kafka to better control data input and output. Kafka also allows them to easily consume or move data into different regions or systems, if necessary. From the Kafka cluster, they use Telegraf to send data to their InfluxDB Enterprise cluster.

WOW’s team aggregates various metrics from the fiber-to-the-node network, such as:

Telemetry metrics, like usage and uptime, from over 650,000 cable modems on a five-minute polling cycle.
Status of all television channels upstream and downstream, including audio and visual signal strength and outages.
Average signal, port and power levels.
Signal-to-noise ratio (SNR) — used to ensure the highest level of wireless functionality.
Modulation error ratio (MER) — another measurement used to understand signal quality that factors in the amount of interference occurring on the transmission channel.

The WOW team uses all this data to gain insights from real-time analytics to create visualizations and to trigger alerts and troubleshoot processes. Once the data is in InfluxDB, they use Grafana for all their visualizations. They also leverage InfluxDB’s alerting frameworks to send alerts via ServiceNow, Slack and email. Adopting InfluxDB allowed the WOW team to implement an Infrastructure-as-Code (IaC) system, so instead of spending time manually managing their infrastructure, they can write config files to simplify processes.

WideOpenWest’s InfluxDB implementation

WideOpenWest’s next big project is to implement a full CI/CD pipeline with automated code promotions. With this, they hope to improve automated testing. WOW also wants to streamline all monitoring across the organization and increase the level of infrastructure monitoring.

Customer Highlight: How Rune Labs is Improving Parkinson’s Patients’ Quality of Life Using Sensor Data Collected with InfluxDB

Caitlin Croft (InfluxData) — Tue, 20 Dec 2022 07:00:00 +0000

I recently chatted with one of our InfluxDB Cloud customers, Rune Labs, to discuss how they’re using this purpose-built time series platform. Every customer has a unique story — I love sharing their stories as well as their Telegraf, InfluxDB, and Flux tips and tricks. Keep reading to learn about Rune Labs’ approach to precision neurology, and learn from Engineering Manager Carolyn Ranti how they are using InfluxDB to collect sensor data.

Caitlin: Tell us a little about yourself and Rune Labs.

Carolyn: I studied neuroscience in college — while there I did computational research and learned to write code in an academic context. After college, I worked in the biology and neuroscience space, then joined a few startups as a software engineer. In 2019 I was looking for a new job and discovered Rune Labs. At the time, it was a small company and a perfect combination of two of my passions: neuroscience and software engineering.

Rune Labs is a software and data analytics company for precision neurology, supporting care delivery and therapy development. We are collecting and processing data from various sources including a mobile app, sensors implanted inside of patients, and from patients’ wearables. We are primarily working in the Parkinson’s medical research and treatment fields. We were lucky to gain these partnerships early on; there’s a lot of potential because of the deep brain stimulation (DBS) devices used to treat the symptoms and side effects of Parkinson’s. Our StrivePD platform is our care delivery ecosystem for Parkinson’s disease, enabling patients and clinicians to better manage Parkinson’s by providing access to curated dashboards summarizing a range of patient data sources, and by connecting patients to clinical trials.

Deep brain stimulation isn’t new technology, but there have been a lot of recent developments; the newer ones are able to stimulate the brain and record neural activity. You can think of them like electrodes that are implanted into the brain. With advances in recording capabilities, there’s a lot of academic research and clinical potential.

Caitlin: You’ve been at Rune Labs for over three years — how has your career developed since joining?

Carolyn: Over a year ago, I moved into an engineering management role; while I’m not in the technical weeds as much, engineering managers at Rune Labs are still pretty technical. I write less code these days, but I’m kept busy with strategic technical planning, engineering, and architecture design.

The team I currently manage is focused on our ETL data ingestion pipeline — from the point the data is uploaded to the way it’s processed, and the query API’s. A lot of the sensor data coming in is numerical time series data, and this is why we turned to InfluxDB. We call it numerical time series data because the data has a timestamp and at least one numerical value (there may be multiple values). Once the data is cleaned up and extracted from the raw data, it’s used in our platform and stored in InfluxDB. My team is in charge of the query APIs that expose the data to our users.

Caitlin: How is Rune Labs helping Parkinson’s patients?

Carolyn: Our platform collects real-time data that helps patients better understand the day-to-day lives of those with Parkinson’s. The data is ingested from DBS devices as well as our iOS app, and Apple Watch integration. Rune Labs partners with device manufacturers to develop software that helps them make better use of their data. We have a web portal for clinicians where they can view the data that their patients have asked us to share with them.

Rune Labs’ goals include:

Improve patient treatment plans - present patient data that helps improve data-driven healthcare decisions.
Better understand patients’ symptoms - Parkinson’s has a huge range of symptoms that are complicated to track.
Become data-driven using real patient data - reduce the burden of tracking how patients are doing, and simplify tracking how environmental and lifestyle (i.e. sleep and diet) choices influence their state.
Streamline medical care - Create useful data visualizations to make doctor appointments smoother and more effective.

The DBS devices require major brain surgery and not all Parkinson’s patients elect or qualify for the procedure. The devices are very effective at treating the symptoms of Parkinson’s: they don’t cure Parkinson’s, but they can greatly improve patients’ quality of life. Once implanted, it takes time to tinker with the settings to ensure the patient is getting the most benefit. It takes a team of specialists time to dial in the best settings for each patient; there’s a lot of back and forth between the patient and their clinical team.

Deep brain stimulators are electrodes implanted directly on patient’s brains. They’re somewhat similar to pacemakers. Pacemakers are closed-loop systems where most detect what’s happening and stimulate the heart based on the electrical activity. However, approved DBS devices are not closed loop systems; they are either on or off. You can adjust the settings, but they can’t (yet) detect what’s happening in the brain and stimulate the brain appropriately. Making them adaptive is the cutting edge of research right now.

Rune Labs helps facilitate the research side and make it easier to develop the algorithms needed to make DBS devices closed loop systems. The data is currently stored on the DBS devices until the patient is in their physician’s office; there, a special tablet is used to extract the metrics from the device. An advantage of the devices is that patients can share with their doctors not just their current state while in the clinical office, but also a longitudinal snapshot of the patient’s state as recorded by the patient.

Caitlin: Tell us about Rune Labs’ InfluxDB implementation.

Carolyn: We needed a tool that could collect time-stamped data and query it. We’ve been using InfluxDB OSS for a while and after considering other tools, we ultimately picked InfluxDB Cloud. We needed to upgrade to InfluxDB Cloud as we knew we were going to need to scale as we prepared to analyze more patient data. We are using InfluxDB to provide customers with visibility into sensor data collected from DBS and wearables.

We are using InfluxDB for all of our production data. We collect numerical time series data from a variety of devices, including electrical activity from deep brain stimulation devices. We take this raw data and clean it, normalize it, and store it in InfluxDB. We’re using the Go Client Library for read and write. After writing directly from a Go app, the data is used in the dashboards and visualizations that our users see. We want to have pre-built visualizations based on what we know is going to be most useful for clinicians. We don’t want to expose or force them to learn a query language. Eventually our data is sent to AWS for cloud storage.

Internally, we use InfluxDB’s UI and Notebooks to develop new queries. We’re looking at the raw values as that’s the most useful for development. Typically, we’re in the UI looking at output tables and ensuring they look the way they should.

Rune Labs takes security and privacy seriously. This is reflected in the way we store data in InfluxDB. Our data schema is quite simple: time series are tagged with one opaque ID, which is not directly linked to the patient’s identity. In a separate database in our platform, we store the metadata that links the patient to their time series data. This allows us to control the flow of information throughout our architecture: we are able to identify where in our platform there is sensitive information, so that we can be particularly cautious about the way that part of the architecture is accessed and adhere to privacy best practices.

Often with time series projects, data granularity becomes less important as time goes on. However, our internal team of neuroscientists wants to have a rich historical dataset, with as much granularity as possible. They often want all of the raw data metrics; they don’t want averages over time. As a result latency is an ongoing important project for us as we need to make sure clinicians are able to access the portal and graphs.

“We love that we can rely on InfluxDB Cloud because it’s horizontally scalable. It’s self-hosted and we don’t have to worry about our constantly growing data set.”

Carolyn Ranti - Engineering Manager, Rune Labs

Caitlin: What’s next for Rune Labs?

Carolyn: We’d like to help enable patients to be able to look at their data themselves, rather than needing to go into a doctor’s office to learn more. We know that not everyone is interested in looking at their state, but there are some who are really interested in the data collected by the devices and what’s happening with them.

Currently our platform is very tailored to Parkinson’s, but there’s a lot of opportunity to expand from a software engineering standpoint. Just this year we announced that we are partnering with Coastal Research Institute to help identify biomarkers (i.e. demyelination) of Multiple Sclerosis lesions in patients who have spinal cord stimulators. Demyelination is a standard symptom of MS when the patient’s immune system starts attacking nerve fibers which causes brain and spine lesions.

From an InfluxDB standpoint, we want to be able to make use of InfluxDB’s profiling capabilities as we want to ensure the queries we’re writing are performant.

It’s That Time Again — Join Us for InfluxDays 2022!

Caitlin Croft (InfluxData) — Wed, 05 Oct 2022 06:00:00 +0000

We’re excited to announce that InfluxDays is back! Save the date: November 2-3, 2022. The two-day conference is dedicated to sharing knowledge and the roadmap for the InfluxDB platform, and discussing the latest innovations spanning IoT, monitoring, real-time analytics, and developer toolsets. In addition to the conference, InfluxDB University will be hosting in-person and virtual training sessions, and there will be in-person watch parties in San Francisco and London.

Sessions

InfluxDays 2022 will showcase the next evolution of InfluxDB as the smart data platform for developers building applications with time series data and how its latest capabilities support developers at every stage of application development.

It’s already been an exciting year for InfluxData – this year we’ve launched:

InfluxDB University: Free on-demand and instructor-led training on InfluxDB, Telegraf, Flux, and more.
InfluxDB Edge Data Replication: Process data at the edge and automatically replicate it to the cloud

There will be presentations from InfluxData’s executives, DevRels, product managers, and engineers who have been busy developing the latest InfluxDB features.

A few of the sessions include:

Building Toward a Sustainable Future – Evan Kaplan & Colin Breck, Cloud Platforms Lead Tesla Energy Products
The Journey of InfluxDB – Paul Dix, Founder and CTO, InfluxData
Build, Ship, Repeat: Milestones for InfluxDB – Barbara Nelson, Vice President of Engineering, InfluxData
Additional sessions from InfluxData’s product managers and engineers on Revving up the Time Series Engine, Collecting Data with InfluxDB, and the Scripting Languages of InfluxDB

Join us in November to learn what else we’ve been working on behind the scenes! Check out the full InfluxDays agenda here.

InfluxDB University at InfluxDays

Get a head start with InfluxDB University’s Taming the Tiger: Tips and Tricks for Using Telegraf with InfluxDB Training. This instructor-led course will be held virtually on November 1, 2022 from 9am - 12pm PT. The syllabus includes a hands-on lab setting where developers will be provided with an introduction to using Telegraf and InfluxDB. Students will gain experience through a series of lab exercises and will be guided through the examples with remote proctors. Seats are limited — sign up for free here.

On November 8-9, InfluxDB University will be hosting the Hands-on Advanced Flux Training. This course will be held in person from 9am - 5pm GMT in London, UK. During the course, the instructors will dive into advanced uses of Fluxlang and will cover topics including Data Engineering to Data Science using Flux to manipulate time series data. Seats are limited — sign up for £500 here.

Annual Community Awards

We’re also announcing winners of our annual Community Awards, which recognize developers who have made significant contributions to InfluxData’s open source products and community this past year. Community Awards will be given out in the following areas: InfluxDB, Telegraf, Flux Language, and Community. Nominations will close on October 21 — submit nominations here.

In-person watch parties

We’ve missed hanging out with the awesome InfluxDB community! If you live in San Francisco or London, join us for in-person watch parties on November 2. Come mingle with other developers, grab a drink, and snag exclusive swag! To find out more about the locations, click here.

Join us November 2-3 for InfluxDays 2022. RSVP for free today.

InfluxData Community Awards Announced at InfluxDays North America 2021

Caitlin Croft (InfluxData) — Fri, 05 Nov 2021 04:00:00 -0700

We’ve just wrapped up InfluxDays North America 2021 as a virtual experience. Throughout the event, we got to hang out with our community and our engineers shared insights into their work and #InfluxDB best practices. InfluxDB users shared their expertise from implementing Telegraf and the InfluxDB platform within their organizations. Attendees also got to hone their skills during the InfluxDB Challenge. One of the best parts of our conference is to highlight some of our amazing community members.

“We couldn’t create the software we build and support all of the users of the software, without the commitment and contributions from the InfluxDB Community. These awards represent the gratitude and appreciation of the entire community, not just InfluxData, to those individuals who have done so much to benefit all of us.”

– Michael Hall, Community Manager, InfluxData

The InfluxData Community awards aim to recognize developers who have gone above and beyond by sharing their knowledge around the world - through blogs, community forums, InfluxDB community Slack, webinars, and other virtual events. The award categories are:

InfluxDB: Overall contribution to InfluxDB
Telegraf: By contributing bug fixes, new features, plugins, documentation or tutorials
Flux: Through helping with Fluxlang design, submitting bug fixes, and creating documentation and tutorials
Community: Helping developers worldwide by providing support, speaking at events, and other activities
Founder's Choice: An all-encompassing award whose winner was hand-chosen by Paul Dix

InfluxDB winner: Tristan Su

Tristan has provided 30+ code contributions to InfluxDB within the last two years. Thank you, Tristan, for always providing your invaluable feedback and GitHub submissions.

Telegraf winner: Sven Rebhan

Sven has been a valuable member of the InfluxDB community by contributing PR’s and bugs to the Telegraf project. He is also a Telegraf maintainer who meets regularly with InfluxData’s engineering team to help with plugin development. He joined InfluxDays North America 2021 and was super helpful by helping answer other attendees’ questions.

“Thank you guys for the amazing award! It’s such a pleasure to work with you guys!”

– Sven Rebhan

Flux winner: Desmond Hokin

Desmond is an important member of the community by helping manage and continuously answers questions in our community forums. While he helps out with InfluxDB, Telegraf and Flux questions, he has answered more Fluxlang questions than anyone else in the forums!

Community winner: Jorge de la Cruz

Jorge has shared his InfluxDB expertise over the years by speaking at various webinars and events. He is always energetically willing to share his expertise across the tech industry. His tech blog has 50,000 + monthly visitors.

Founder’s Choice winner: Jamie Milton

Jamie has a familiar name to those who have attended the last few InfluxDays. Since joining Flux Training last spring, he has been a fixture in the InfluxDB community. He has provided answers to fellow developers’ questions and become a Flux SME for the community.

Congratulations to this year’s winners! Thank you for making the global InfluxDB community amazing! We invite you to join the Built on InfluxDB program to share your InfluxDB projects and solutions. If you missed any session from InfluxDays North America 2021, check out all sessions here.

Event Highlights: InfluxDays North America 2021

Caitlin Croft (InfluxData) — Thu, 28 Oct 2021 04:00:45 -0700

Roadmap revealed, insight gained, connections made, and InfluxDB Challenge swag won — that’s a wrap for InfluxDays North America 2021! The conference, which brought together the #InfluxDB community to exchange time series acumen and use cases, is always a reminder of why and how we’ve grown. It included subject-matter-expert led courses, on-demand videos and live sessions. Check out session recordings and presentations here. Recordings of the live sessions will be available November 5, 2021.

In less than a two-week span, we hosted the InfluxDB Challenge, two trainings and a 2-day conference dedicated to everything InfluxDB and time series data! Influxers and community members had a great time networking and learning from each other during our Meet the Experts series. Here’s a recap of the event.

InfluxData and the #InfluxDB Community

Trainings

Flux Training: InfluxDays North America 2021 kicked off on October 11-12, 2021 with our popular Hands-On Flux Training. Flux is InfluxData’s standalone data scripting and query language that increases productivity and code reuse. Over the course of two days, students learned everything from Fluxlang core concepts to advanced topics like User Defined Functions, passing by basic queries to manipulate time series data. Check out the awesome dashboards students made using the InfluxDB UI!

Telegraf Training: On October 25, 2021, members of the InfluxDB community came together for a virtual Telegraf Training. This course provided everyone with a fantastic introduction to Telegraf – the open source server used to collect metrics and events from your stacks, sensors and systems. It is InfluxDB's native data collector with nearly 300 plugins. As Alan Pope, the engineering manager for Telegraf pointed out during the training, Telegraf builds nicely on the "super new M1 Macs"!

Day 1 highlights

Paul Dix opened the conference with an update on the future of InfluxDB. Thank you to everyone who asked Paul tons of questions about InfluxDB.

Tim Hall reviewed the InfluxDB Roadmap. Tim demonstrated where InfluxDB is today and where the product is headed in the coming months and years.

Bhagvan Kommadi covered a real-life experience related to building a DevOps Delivery-powered AI platform for doctors’ community and telehealth support for patients during COVID-19 lockdown.

Vijeta Hingorani, Milos Pavkovic, Mladen Maric, Marina Svicevic profiled two projects. The first (transforming email reports into self-serving dashboards) involved generating query-based reports for the sales team replacing email reports. The second (detecting anomalies in-stream) involved detecting strange behaviors in graphs of data streams that are internally considered as anomalies.

Russ Savage showed how to get up and running by creating a Python application built on the InfluxDB platform; how to connect to InfluxDB from a client library; and best practices for user and data management for your time series application.

Tanny Ng and Nadeem Syed showed how WP Engine transformed their monitoring solution into an Observability Platform with InfluxDB, gaining better visibility and reducing infrastructure downtime. It was great learning about their journey that led them to InfluxDB - they are writing 5.5 million points to InfluxDB per minute!

Alan Pope and Sebastian Spaink covered Telegraf plugins and InfluxDB client libraries as well as early access to release candidates. They highlighted recent updates to Telegraf - including the new JSON and XML parsers. Special shout-out to Sebastian for creating awesome Telegraf cartoons!

Scott Anderson closed out Day 1 by demonstrating how to generate and send notifications in raw Flux and familiarized the audience with patterns and functions built into Flux for sending alerts. He was worried about showing too much code - but we know our community loves it!

Day 2 highlights

Ryan Betts provided an engineering update on InfluxDB.

Martin Moucka discussed how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better. Naturally, Red Hat appreciates the open source core of InfluxDB and the enterprise support. Martin demonstrated how they are monitoring 60+ sites, ~1,6k devices and ~14k interfaces with 5 collectors. Martin loves Flux as he finds it powerful and was impressed that it met his needs.

Timothy Spann from StreamNative discussed their edge applications, which classify images and sensor readings real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.

Jacob Marble showed how to record metrics, logs, and traces with one library - OpenTelemetry - and how to store them in one open source database - InfluxDB/IOx.

Anais Dotis-Georgiou demonstrated how to learn Flux by example, using the NOAA dataset, enabling the audience to follow along as she went over some of the most common Flux questions from community members.

Brian Gilmore outlined the key components of architecture for capturing and analyzing IoT data at ANY scale and showcased how he has implemented these recommendations in his own lab. I personally enjoyed his references to aquariums and marine biology.

Michael Hall provided an update on the worldwide InfluxDB community. During this session, we learned that InfluxDB is running on all 7 continents. He also announced this year’s Community Award winners! Congrats to this year’s winners!

Hari Prasad Sudharshan and Ezhilmathy Shanmugam Veeramani showed how quickly, reliably and cost-effectively you can do data-driven software development by writing custom code. Hari discussed how he started mapping software development with a time series DB and created an IoT within a software development tool. He explains why he chose Flux over Python to determine mean, median, mode, and quantiles with Flux’s built-in functions. Hari did a nice job using the Slack channel to make this session interactive.

Ana-Maria Calin demonstrated running InfluxDB OSS on a self-hosted cloud platform on Kubernetes and the process of migrating from OSS to the Usage-Based InfluxDB Cloud platform. Ana provided an amazing demo as part of her session!

Our community loves their #InfluxDB swag! There are so many different styles of t-shirts, hoodies, socks and onesies in the wild!

InfluxDB Challenge

Attendees impressed us with their InfluxDB knowledge by participating in the InfluxDB Challenge. Congrats to the InfluxDB Challenge Extra Credit winner: Egor Morozovxx!

If you missed out on the InfluxDB Challenge — don’t worry, you can still compete! Check out the contest details here.

InfluxDB Challenge extra credit winner: Egor Morozov shared this Data Transformation screenshot:

Thank you to the entire #InfluxDB community for attending and speaking at InfluxDays North America 2021. All sessions’ recordings and slides will be available here by November 5 2021. It was a blast chatting with everyone on Slack, learning more about our community of developers and sharing with you the latest from InfluxData. Stay tuned to learn more about InfluxDays 2022!

See You Tomorrow at InfluxDays North America 2021

Caitlin Croft (InfluxData) — Mon, 25 Oct 2021 04:00:31 -0700

We can’t wait to see everyone tomorrow for the start of InfluxDays North America 2021, our virtual conference bringing together the entire #InfluxDB community for the latest on time series data and related technologies.

The conference spans two half-days tomorrow and Wednesday - October 26-27, 2021 (Pacific Time). Registration is free and attendees will gain first-hand knowledge about the InfluxDB roadmap, time series applications, innovative use cases across industries, and product tips shared by developers for developers.

Speaker lineup – listen to fellow developers

The conference agenda features:

Speakers from InfluxData including members of our engineering teams, product management, with keynotes by InfluxData founder and CTO Paul Dix, and VP of Products Tim Hall
Live sessions from community members and customers who will share real-world use cases on building time series solutions
On-demand sessions that become available on October 26: Going Underground with InfluxDB, MQTT – Machine Data Collection, and How Not to Build an SLO Platform
Get your questions answered by InfluxDB engineers during our Meet the Experts series on both conference days

InfluxData Community Awards 2021 – the winners

During the conference, InfluxData Community Awards 2021 winners will be announced. These awards recognize the InfluxDB open-source community and the many outstanding contributions of engineers working on open-source projects.

InfluxDB Challenge – a chance to win cool swag

Participate in our InfluxDB Challenge for a chance to show us your InfluxDB knowledge! Sign up for a free account before or during the event and follow a few steps to get cool InfluxData swag.

Join us on Slack

Be sure to join our Community Slack before the event to network and get timely InfluxDays updates. And during the event, join the dedicated Slack channel #influxdays-virtual where you can chat and ask questions.

Don’t miss out. InfluxDays is a unique opportunity for developers, solution builders, data scientists, and engineers to share expertise, gain insight, and network with peers.

InfluxDays North America 2021 starts tomorrow and you can still register now for free to save your spot. See you there!

Celebrating Oktoberfest: Making Better Beer With InfluxDB

Caitlin Croft (InfluxData) — Thu, 30 Sep 2021 04:00:46 -0700

Oktoberfest festivities may look a little differently this year, but that doesn’t mean the InfluxDB community can’t celebrate at home. Speaking of Oktoberfest (by default associated with beers!), Luke Bond uses Telegraf and InfluxDB to monitor fermentation temperatures as he makes beer at home. So can you! ????

Oktoberfest Fun Fact #1: It wasn't originally a beer festival — it was originally a celebration to commemorate Bavarian Crown Prince Ludwig's marriage in 1810.

Who is Luke Bond?

Luke Bond is an engineer at InfluxData — he previously worked on the Deployments team and now works in the InfluxDB IOx group. Prior to joining InfluxData, he was a Kubernetes developer; in fact, he founded a container security consulting company. He has over 20 years of developer experience. Discover how he uses the InfluxDB platform to gain observability into his beer yeast by using InfluxDB.

Oktoberfest Fun Fact #2: Attendees at the annual festival can't start drinking until the Mayor of Munich opens the first keg. The mayor will proclaim "O' zapft is" which means "It's tapped!'.

Fermentation basics

At a high level, there is always a sugary liquid needed for fermentation. When making beer, this solution is malted barley. The yeast will consume the sugar and convert it into alcohol and carbon dioxide. The solution starts off as a flat, sweet and dense liquid and eventually turns into a lighter fizzy alcohol liquid. Craft beers are becoming more popular today, and when making these beers, people are using grains beyond just barley.

Oktoberfest Fun Fact #3: In 2019, Oktoberfest attendees, from 55 nations, consumed around 7.3 million liters of beer! The amount of beer drank at the festival could fill more than three Olympic-sized swimming pools!

How is beer made?

Comparing the commercial and home brewing process

As seen above, brewers at home are generally able to skip the milling step and go straight to the mash tun. Usually homebrewers will purchase grain that has already been malted and milled. Luke’s setup is even easier as the mash tun and kettle are the same machine. It is common for homebrewing setups to include a machine that does both. Filtering is less common at home, as it requires more machines. Packaging at home means bottling or pressure-transferring it into a keg.

Oktoberfest Fun Fact #4: Beer consumed at Oktoberfest must be brewed in Munich. The beer must be made within the city walls and come from one of the six special historical breweries.

Tasty beer needs InfluxDB

Fluctuations in yeast temperature can result in undesirable flavors; this is usually the cause of the common “homebrew smell”. Most home brewing yeast needs to be at 18-21ºC (64-70ºF). It’s more important to have the temperature consistent. As Luke points out, it is better to be consistently at 20ºC rather than fluctuating between 18-20ºC.

Luke needed to be able to monitor the ambient temperature in his home to ensure the temperature was ideal and consistent for his beer-making projects. There are various ways to control yeast temperature including: pure luck, affordable DIY solutions and expensive solutions. Space and financial resources are the biggest limiting factors.

Prior to using InfluxDB, Telegraf and Raspberry Pis to monitor everything, Luke had to check the temperature manually. He needed to be able to remotely monitor and tweak the temperature of his fermentation setup. He had all of the data available to him, but it wasn’t automated. Everything from ice melting, environmental temperature changes, and other equipment in use emitting heat (i.e. controllers and pumps) can unintentionally heat the beer too much.

Oktoberfest Fun Fact #5: Locals call the festival Wies'n (short for Theresienwiese) which translates to "Theresa's fields". This refers to Princess Therese of Saxony-Hildurghausen who married the prince in 1810.

Stress-free beer monitoring

Automate everything! Luke now uses a myriad of technologies to collect and analyze metrics about his beer-making activities. He uses Telegraf to collect temperature metrics and sends them to InfluxDB. Once the metrics are in the purpose-built time series database, he easily creates useful dashboards and alerts.

Luke's IoT monitoring solution

Luke has a cooler that automatically turns on to reduce the temperature. After the dramatic drop, the temperature begins to slowly climb again. He added code to reject values above 28ºC — this resulted in a lot of erroneous notifications. His dashboards even show a time where he must have accidentally kicked the webcam while passing — his graph displayed a lot of rapid up and down. Luke created a rule where he gets an alert in Slack whenever the temperature rises above 21ºC. This indicates that he needs to check the ice level. He also received deadman alerts when there was no data coming into InfluxDB.

Examples of Luke's dashboards and notifications

Oktoberfest Fun Fact #6: In 1896, Albert Einstein was working as an electrician who helped set up one of the beer tents.

Oktoberfest at InfluxData

There are countless types of beers out there, and Influxers enjoy their hops! When polled, InfluxData employees shared some of their favorite beers.

To learn more about how Luke uses InfluxDB to monitor his homebrewing setup, check out the recording and presentation here. Whether you’re making your own beer at home or enjoying a local microbrew at home, we hope everyone has fun celebrating Oktoberfest!

P.S. If you’re using InfluxDB to monitor your own brewing project or if you’re using it in other ways, share your InfluxDB story here.

See You in October - InfluxDays North America 2021!

Caitlin Croft (InfluxData) — Thu, 09 Sep 2021 02:00:04 -0700

We’re super excited to announce our speaker lineup for InfluxDays North America 2021 Virtual Experience. InfluxDays is an event focused on the impact of time series data, where you’ll gain first-hand knowledge — from InfluxDB community members and InfluxData’s team — about time series applications and use cases. The conference is being held on October 26-27, 2021.

We will have speakers from InfluxData including members of our engineering teams, product management and leadership. There will also be sessions from community members and customers who will share their knowledge and expertise from hands-on experience with the tools.

Registration is open — it's free!

Trainings

Leading up to InfluxDays, we have two separate trainings available for our community! On October 11-12 2021, our popular Flux Training returns. The day before InfluxDays, on October 25, we will be providing our Telegraf Training. Both trainings will be held virtually. If you register for either one of these trainings (or both), you will be automatically registered for the conference. Spaces are filling up fast — so register now!

Day 1 speakers

InfluxData founder and CTO Paul Dix will kick off the conference! After Paul discusses the future of InfluxDB, VP of Products Tim Hall will follow up with a roadmap update deep-dive.

Bhagvan Kommadi is the Director of Engineering at Value Momentum. He will demonstrate how they created a DevOps delivery platform that is powered by AI which has been instrumental for doctors and other healthcare professionals during the COVID-19 pandemic.

The team from Socialgist, including Vijeta Hingorani, Milos Pavkovic and Mladen Maric, will show how they use Kafka, Slack, Grafana, Elasticsearch andInfluxDB to turn email reports into useful dashboards. These dashboards are self-service and used by their sales team. They are also able to run their anomaly detection model and store the data in InfluxDB, the purpose-built time series database.

Vijeta Hingorani	Milos Pavkovic	Mladen Maric
Socialgist team

Russ Savage will provide an update on the InfluxDB platform. The Telegraf team will highlight product updates and share tips on data collection. In case you don’t get enough from the Flux Training, Scott Anderson will show new features in Flux - including how to create and send notifications using raw Flux.

Day 2 speakers

Ryan Betts, our VP of Engineering, will kick off the second day of InfluxDays by highlighting advances made to the platform by the developers behind InfluxDB.

Michael Hall is our Community Manager who will give shoutouts to all of the amazing worldwide developers who contribute to InfluxDB via our communities, Slack workspace, GitHub repos, social media, etc. We couldn’t do it without our community of fierce supporters, and we always want to show our appreciation to everyone out there. ??

Timothy Spann from StreamNative will share how to use FLiP with InfluxDB for EdgeAI IoT at scale. Learn how to use various technologies, including Apache Pulsar, NiFi and InfluxDB, to develop a low-code IoT app. InfluxDB is used for data transformations, parsing, enrichment and querying.

Day 2’s speakers also include Jacob Marble and Brian Gilmore from InfluxData. Jacob is an engineer on the InfluxDB IOx team. In his session, he will show how to use OpenTelemtry to collect metrics and then use InfluxDB for storage. Brian is the Director of IoT Product Management. Sensors are everywhere! Join his session to understand more about how InfluxDB is a perfect solution for sensor data — regardless if it’s a sensor at your home or thousands of sensors monitoring an industrial manufacturing plant!

Anais Dotis-Georgiou from InfluxData will help you sharpen your Flux skills by querying a NOAA dataset! She'll also answer some of the common Flux questions from community members.
Hari Prasad Sudharshan from Fujitsu Network Communications will teach you how to use InfluxDB to make your software development process data-driven. He uses InfluxDB, Flux and InfluxDB Templates to make developers' lives easier!
Ana-Maria Calin works at InfluxData; in her capacity as a Senior Software Engineer, she deploys and maintains all app components that make up InfluxDB Cloud and she works on our CI/CD pipeline! Check out this session to learn more about running InfluxDB on a self-hosted cloud platform using Kubernetes — Containerize!

On-demand sessions

We will have a few on-demand sessions that will be available on October 26, 2021. Tobias Braun from Herrenknecht AG will highlight how they use InfluxDB to collect and store IIoT metrics from construction projects. The team at Eldor including Volkan Balikci, Nilden Tutalar and Uygar Zubari will demonstrate how to collect machine data using the MQTT protocol, Telegraf, InfluxDB and Kapacitor to produce actionable insights and to create useful data pipelines!

Volkan Balikci	Nilden Tutalar	Uygar Zubari
Eldor Corporation team

Can't wait to see everyone October 26-27, 2021. Save your free spot today!