Webhooks and DuckDB Integration
Powerful performance with an easy integration, powered by Telegraf, the open source data connector built by InfluxData.
5B+
Telegraf downloads
#1
Time series database
Source: DB Engines
1B+
Downloads of InfluxDB
2,800+
Contributors
Table of Contents
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Input and output integration overview
The Webhooks plugin allows Telegraf to receive and process HTTP requests from various external services via webhooks. This plugin enables users to collect real-time metrics and events and integrate them into their monitoring solutions.
This plugin enables Telegraf to write structured metrics into DuckDB using SQLite-compatible SQL connections, supporting lightweight local analytics and offline metric analysis.
Integration details
Webhooks
This Telegraf plugin is designed to act as a webhook listener by starting an HTTP server that registers multiple webhook endpoints. It provides a way to collect events from various services by capturing HTTP requests sent to defined paths. Each service can be configured with its specific authentication details and request handling options. The plugin stands out by allowing integration with any Telegraf output plugin, making it versatile for event-driven architectures. By enabling efficient reception of events, it opens possibilities for real-time monitoring and response systems, essential for modern applications that need instantaneous event handling and processing.
DuckDB
Use the Telegraf SQL plugin to write metrics into a local DuckDB database. DuckDB is an in-process OLAP database designed for efficient analytical queries on columnar data. Although it does not provide a traditional client-server interface, DuckDB can be accessed via SQLite-compatible drivers in embedded mode. This allows Telegraf to store time series metrics in DuckDB using SQL, enabling powerful analytics workflows using familiar SQL syntax, Jupyter notebooks, or integration with data science tools like Python and R. DuckDB’s columnar storage and vectorized execution make it ideal for compact and high-performance metric archives.
Configuration
Webhooks
[[inputs.webhooks]]
## Address and port to host Webhook listener on
service_address = ":1619"
## Maximum duration before timing out read of the request
# read_timeout = "10s"
## Maximum duration before timing out write of the response
# write_timeout = "10s"
[inputs.webhooks.filestack]
path = "/filestack"
## HTTP basic auth
#username = ""
#password = ""
[inputs.webhooks.github]
path = "/github"
# secret = ""
## HTTP basic auth
#username = ""
#password = ""
[inputs.webhooks.mandrill]
path = "/mandrill"
## HTTP basic auth
#username = ""
#password = ""
[inputs.webhooks.rollbar]
path = "/rollbar"
## HTTP basic auth
#username = ""
#password = ""
[inputs.webhooks.papertrail]
path = "/papertrail"
## HTTP basic auth
#username = ""
#password = ""
[inputs.webhooks.particle]
path = "/particle"
## HTTP basic auth
#username = ""
#password = ""
[inputs.webhooks.artifactory]
path = "/artifactory"
DuckDB
[[outputs.sql]]
## Use the SQLite driver to connect to DuckDB via Go's database/sql
driver = "sqlite3"
## DSN should point to the DuckDB database file
dsn = "file:/var/lib/telegraf/metrics.duckdb"
## SQL INSERT statement with placeholders for metrics
table_template = "INSERT INTO metrics (timestamp, name, value, tags) VALUES (?, ?, ?, ?)"
## Optional: manage connection pooling
# max_open_connections = 1
# max_idle_connections = 1
# conn_max_lifetime = "0s"
## DuckDB does not require TLS or authentication by default
Input and output integration examples
Webhooks
-
Real-time Notifications from Github: Integrate the Webhooks Input Plugin with Github to receive real-time notifications for events such as pull requests, commits, and issues. This allows development teams to instantly monitor crucial changes and updates in their repositories, improving collaboration and response times.
-
Automated Alerting with Rollbar: Use this plugin to listen for errors reported from Rollbar, enabling teams to react swiftly to bugs and issues in production. By forwarding these alerts into a centralized monitoring system, teams can prioritize their responses based on severity and prevent escalated downtime.
-
Performance Monitoring from Filestack: Capture events from Filestack to track file uploads, transformations, and errors. This setup helps businesses understand user interactions with file management processes, optimize workflow, and ensure high availability of file services.
-
Centralized Logging with Papertrail: Tie in all logs sent to Papertrail through webhooks, allowing you to consolidate your logging strategy. With real-time log forwarding, teams can analyze trends and anomalies efficiently, ensuring they maintain visibility over critical operations.
DuckDB
-
Embedded Metric Warehousing for Notebooks: Write metrics to a local DuckDB file from Telegraf and analyze them in Jupyter notebooks using Python or R. This workflow supports reproducible analytics, ideal for data science experiments or offline troubleshooting.
-
Batch Time-Series Processing on the Edge: Use Telegraf with DuckDB on edge devices to log metrics locally in SQL format. The compact storage and fast analytical capabilities of DuckDB make it ideal for batch processing and low-bandwidth environments.
-
Exploratory Querying of Historical Metrics: Accumulate system metrics over time in DuckDB and perform exploratory data analysis (EDA) using SQL joins, window functions, and aggregates. This enables insights that go beyond what typical time-series dashboards provide.
-
Self-Contained Metric Snapshots: Use DuckDB as a portable metrics archive by shipping
.duckdb
files between systems. Telegraf can collect and store data in this format, and analysts can later load and query it using the DuckDB CLI or integrations with tools like Tableau and Apache Arrow.
Feedback
Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Related Integrations
Related Integrations
HTTP and InfluxDB Integration
The HTTP plugin collects metrics from one or more HTTP(S) endpoints. It supports various authentication methods and configuration options for data formats.
View IntegrationKafka and InfluxDB Integration
This plugin reads messages from Kafka and allows the creation of metrics based on those messages. It supports various configurations including different Kafka settings and message processing options.
View IntegrationKinesis and InfluxDB Integration
The Kinesis plugin allows for reading metrics from AWS Kinesis streams. It supports multiple input data formats and offers checkpointing features with DynamoDB for reliable message processing.
View Integration