InfluxData Blog - Jacob Marble

OpenTelemetry Tutorial: Collect Traces, Logs & Metrics with InfluxDB 3.0, Jaeger & Grafana

Jay Clifford, Jacob Marble (InfluxData) — Fri, 05 May 2023 07:00:00 +0000

Here at InfluxData, we recently announced InfluxDB 3.0, which expands the number of use cases that are feasible with InfluxDB. One of the primary benefits of the new storage engine that powers InfluxDB 3.0 is its ability to store traces, metrics, events, and logs in a single database.

Each of these types of time series data has unique workloads, which leaves some unanswered questions. For example:

What schema should I follow?
How do I convert my traces to line protocol?
How does InfluxDB connect with the larger observability ecosystem?

Luckily this is where our work within OpenTelemetry comes into play. If you want to learn more about what OpenTelemetry is, we recommend this blog by Charles Mahler. However, the aim of this blog is to take you through a working example of OpenTelemetry and InfluxDB 3.0.

Running the demo

Let’s start with the fun part by running the demo and then discussing the theory behind it. Here is the demo we are going to deploy:

This demo uses Hot R.O.D. to simulate traces, logs, and metrics. We then use OpenTelemetry Collector to collect that data and write it to InfluxDB 3.0. Finally, we use Grafana and Jaeger UI to visualize this data in a highly summarized view.

To run the demo you can use this KillerCoda demo environment.

Simply create an account and follow the tutorial to run through the demo without worrying about a local installation. For local installs, we have all the info you need in a GitHub repo, so check out the repository readme for up-to-date installation instructions.

Walkthrough

Let’s run through the demo with Killercoda so you can see what to expect:

You will see the demo provisioning script ticking away on the first load. It shouldn’t take too long. You will know it finished loading once InfluxDB OTEL Demo appears.
If you don’t already have one, follow the steps to create a free InfluxDB 3.0 Cloud account for the demo. Make sure to follow the instructions for creating a bucket in InfluxDB called otel and generating a read-and-write token for the new otel bucket.
Next, we provide credentials for InfluxDB 3.0 so the demo can write and query from our otel bucket. To do this, select the Editor tab within Killercoda and navigate to influxdb-observability/.env. This is where we update our connection credentials. Note: Make sure to update INFLUXDB_ADDR to point to your InfluxDB Cloud region. Also, remove any protocols from the address (e.g., HTTPS://).
Once we finish updating the environment, we can start the demo. Simply click this command to spin up the demo: docker-compose --file demo/docker-compose.yml --project-directory
Now we get to the fun part! Let's generate some traces. First of all, let's open the HotROD application. Within a local installation, this runs via localhost:8080. To access this address in Killercoda, follow the hyperlinks provided in the screenshot below.
From there we can start generating traces by clicking on the different buttons. Each one simulates ordering a car for the indicated task, which triggers the background service calls and creates the trace.
If you head over to your InfluxDB 3.0 Cloud instance, you can explore the otel bucket schema to see how we translate the OpenTelemtry data structure into InfluxDB’s line protocol, which consists of measurements, tags, and fields. (Note: We will cover line protocol more in-depth in the next blog.)
If we head back to our Killercoda instance and open up Grafana, we can explore our OpenTelemetry dashboard and configuration.

Grafana explained

This section breaks down some of the key features of the Grafana dashboard. Let’s start with the data sources.

As you can see, we connected to two data sources – Flight SQL and Jaeger. Both sources pull data directly from InfluxDB 3.0, but we will discuss how they work and their differences in more detail in the next blog. For now, you just need to know the following:

Flight SQL – This is the direct SQL query interface for InfluxDB 3.0. It is great for general-purpose time series-based queries and metrics summaries.
Jaeger – This functions as the bridging interface for metrics, logs, and traces between InfluxDB 3.0 and Grafana visualizations.

Let’s look at our dashboard again and map our data sources to the different panels.

We use Flight SQL to generate our general-purpose navigation and summary overviews:

Services queries InfluxDB for all unique services found within the otel bucket. We use the service names as data links for the rest of our visualization. For example, clicking redis filters my summary results and trace list to only include the service redis.
Duration returns the duration in nanoseconds of each spanID over the selected time period. As part of our SQL query, we convert this value to seconds to improve readability.
Error Rate calculates the service error rate, as a percentage, based on the number of errors flagged in the otel.status column.

We use Jaeger to drill into our traces with the following visualizations:

Service Latency Histogram creates a histogram based on the latency of traces within the service. This panel groups trace spans based on their detected latency range.
Traces provides a table of traces associated with the selected service. The table includes trace name, start time, and duration. Users can select TraceID’s to generate the next two visualizations: Relationships and Trace.
Relationships: After selecting a TraceID in the Traces panel, users can navigate span relationships within the trace using the span tree. We’ll have more on this in the next blog.
Trace: This panel allows users to navigate raw trace data for the selected TraceID and its associated log data.

Conclusion

We hope this tutorial provides you with a solid foundation to learn more about OpenTelemetry and how InfluxDB 3.0 will play a pivotal role in future observability solutions. Stick around for the next blog where we will delve into the theory of OpenTelemtry, breaking down each component in the demo architecture and discussing data schemas. Until then play with the demo, fork the repo, and see if you can apply these components to your own use case. If you have any questions please do not hesitate to reach out to us on Slack.

Tracing with InfluxDB IOx

Jason Myers, Jacob Marble (InfluxData) — Wed, 30 Nov 2022 07:00:00 +0000

Tracing has always been a key use case for time series data. But admittedly, it’s also one that past versions of InfluxDB could not handle as well as we wanted. One of the roadblocks was the cardinality issue. Tracing data is, almost by definition, high cardinality data and prior to InfluxDB IOx, high cardinality data could affect query performance.

InfluxDB IOx removes limits on cardinality, opening up the InfluxDB platform to handle a wider range of use cases, like traces and logs, in a performant manner. So, let’s answer some basic questions about tracing and address how InfluxDB IOx makes them possible.

What is tracing?

When you have a distributed system, generally, you want to know how all the different components are functioning in relation to each other. Some processes and services may be dependent on other ones, so errors, delays, and bottlenecks can impact overall system performance.

To understand how all these different pieces work in concert with each other we use a form of observability called tracing. A trace provides a view of a request, task, operation, job, or other useful unit of work, as it works its way through a distributed system. Any number of subtasks, (algorithms, network calls, database transactions, cache queries, etc) coordinate to satisfy the request. Each of these subtasks is a span.

So, a trace is a collection of spans that provide timing information on the finer details of a request.

How does tracing work?

Spans have zero to many sub-spans, called child spans, or children. A child span can have its own children, and so forth.

A trace begins with a root span, and has no parent. Because the root span is the recursive parent of all other spans in the trace, the duration of the root span represents the total time of the trace.

As indicated in the diagram, there are a lot of potential spans in a single trace.

To construct a trace from all these spans, and to ensure that everything fits together properly, every span needs identifying information. To that end each span has a trace ID, a span ID, and, when applicable, a parent span ID.

These building blocks create a hierarchical tree of subtasks that provide the key structure for building traces.

What does this have to do with cardinality?

Before IOx, InfluxDB’s time series merge tree (TSM) stored data in columns of series. A series is a unique combination of measurement, tags, and one field. In this model, cardinality is the total number of series, i.e., the number of columns stored on disk. So, if you have two series, your cardinality is two. High cardinality usually becomes troublesome because it correlates with slower query performance.

To avoid runaway cardinality in pre-IOx InfluxDB, the quantity of series must be bounded. The challenge with tracing is that each trace and span generate a unique ID (for tracing, most database schemata consider these IDs to be tags), creating unbounded tag values. Combine this with the volume and velocity of tracing data and you get runaway cardinality.

IOx: Solving for cardinality

Previous versions of InfluxDB stored each series as a column, which could result in a lot of columns. With the introduction of InfluxDB IOx, the measurement timestamp, tags, and fields all get saved to a respective column, so a measurement with two tag keys and three fields is represented by six columns. This design drastically reduces the number of columns the database has to deal with.

IOx stores the columns of a table in a Parquet file. When more data arrives, IOx writes those columns of the table to a new Parquet file. Parquet files compress columns of data very well, and IOx stores those files in object storage (S3), which is extremely scalable. The InfluxDB Cloud query tier automatically scales with your query workload.

The Parquet file format also provides fantastic query performance. For example, when IOx writes columns of data to Parquet, it includes hints in the Parquet metadata to describe column contents. This way, the query engine can skip over entire Parquet files, and/or unhelpful portions of Parquet files, at query time.

These updates to the InfluxDB storage engine, persistence format, and storage and query tiers combine to provide infinite cardinality and unlock use cases like tracing.

To get started with InfluxDB IOx and see what unlimited cardinality can unlock for you, sign up for the InfluxDB IOx beta program.

Why I Joined InfluxData - Jacob Marble

Jacob Marble (InfluxData) — Wed, 28 Mar 2018 06:00:27 -0700

My name is Jacob Marble, and I’m a new Influxer.

In my short career as a software engineer, I’ve worked for both BigCo and Startup, and learned volumes from each. BigCo likes systems, processes, standards, tests, specialists, chain-of-command. Startup likes nimble, scrappy, curious, pragmatic, generalist, impact. My biggest lesson so far?

I want to work with people who are (1) nice and (2) smart.

People

In my early programming days, I was impressed by the stereotypical programmer-in-a-closet. He (stereotype!) is brilliant, eats pizza, stays up late hacking, avoids people, and doesn’t care what other people think. This agrees strongly with my upbringing; my parents taught me to be independent, an attribute that I am proud of for myself and respect in others.

However, the harder I try to be an independent-whiz-kid, the more I realize that I need and enjoy interaction with people. In fact, I learn more and get more work done as a member of a team than I do alone! When I interviewed with InfluxData, I wanted to know about the people I would be working with, so I asked my interviewers questions like:

Who do you like to work with?
How often do you interact with people?
What did you do before coming to InfluxData?
What percentage of my prospective peers are remote like me?
What are the relationships like between SWE, SRE, and PM?

In the process of interviewing with InfluxData, I interacted with four software engineers. After each interview, I felt certain that this is someone I like, and that I can learn a lot from. Influxers are nice, outgoing, and smart!

Products

I used Telegraf, InfluxDB and Chronograf before joining InfluxData. My last employer uses InfluxDB and Grafana for service metrics (and they are excited for TSI in InfluxDB 1.5) and I use InfluxDB as the central persistence engine in a side project. I’ve read a lot of the online documentation and found answers to my questions there. All this to say, I’m confident that the products are solid, and support is top-notch.

As society continues to move into the age of Big Data, the relevance of time series will only continue to grow, and before I even considered InfluxData as an employer, I knew that InfluxData is my preferred solution for time series tools.

Tech

The people and product meet the standard, but will my inner nerd find satisfaction?

My background so far is strong with tools like Java, Golang, data pipeline frameworks, and building APIs. I look forward to working with the InfluxDB storage team to leverage, grow, and apply these skills to allow our users to reduce Time to Awesome.

InfluxData uses Golang a lot. If you haven’t tried Golang yet, then you need to take a peek. For me, it’s everything good about Python (expressive syntax; small surface area; garbage collected), plus some massive advantages (no GIL; compiled).

Some of the upcoming changes in 2.0 are really cool for anyone who has worked with data pipelines. IFQL is part of 2.0, and the language itself is interesting, but the implementation is where it gets really exciting.

Most of InfluxData’s product code is open source, and most development work is open source. It isn’t always right for a business to operate like this, and it’s awesome that the model here is to give the community something useful (the TICK Stack) and let them decide when it’s time to upgrade from OSS user to SaaS customer.

Summary

I’m excited to be here at InfluxData, primarily due to the people, products and technology, in that order.

Looking for a new challenge? Drop us a line!