Basic Transformation
The Basic Transformation plugin cleans and standardizes raw time series data on ingest, so teams can normalize names, values, and formats without extra pipelines. It’s ideal for messy telemetry from sensors, apps, and infrastructure, making data easier to query, alert on, and act on. The result is analysis ready data faster, with less manual work and less operational overhead.
Configuration
Plugin parameters may be specified as key-value pairs in the --trigger-arguments flag (CLI) or in the trigger_arguments field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s config_file_path parameter.
If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use.
Plugin metadata
This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the InfluxDB 3 Explorer UI to display and configure the plugin.
Required parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
measurement |
string | required | Source measurement containing data to transform |
target_measurement |
string | required | Destination measurement for transformed data |
target_database |
string | current database | Database for storing transformed data |
dry_run |
string | “false” | When “true”, logs transformations without writing |
Transformation parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
names_transformations |
string | none | Field/tag name transformation rules. Format: ‘field1:”transform1 transform2”.field2:”transform3”’ |
values_transformations |
string | none | Field value transformation rules. Format: ‘field1:”transform1”.field2:”transform2”’ |
custom_replacements |
string | none | Custom string replacements. Format: ‘rule_name:”find=replace”’ |
custom_regex |
string | none | Regex patterns for field matching. Format: ‘pattern_name:”temp%”’ |
Data selection parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
window |
string | required (scheduled only) | Historical data window. Format: |
included_fields |
string | all fields and tags | Dot-separated list of fields and tags to include (e.g., “temp.humidity.location”) |
excluded_fields |
string | none | Dot-separated list of fields and tags to exclude |
filters |
string | none | Query filters. Format: ‘field:”operator value”’ |
TOML configuration
| Parameter | Type | Default | Description |
|---|---|---|---|
config_file_path |
string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) |
To use a TOML configuration file, set the PLUGIN_DIR environment variable and specify the config_file_path in the trigger arguments. This is in addition to the --plugin-dir flag when starting InfluxDB 3.
Example TOML configurations
- basic_transformation_config_scheduler.toml - for scheduled triggers
- basic_transformation_config_data_writes.toml - for data write triggers
For more information on using TOML configuration files, see the Using TOML Configuration Files section in the influxdb3_plugins/README.md.
Examples
Example 1: Temperature unit conversion
Convert temperature readings from Celsius to Fahrenheit while standardizing field names:
# Create the trigger
influxdb3 create trigger \
--database weather \
--path "gh:influxdata/basic_transformation/basic_transformation.py" \
--trigger-spec "every:30m" \
--trigger-arguments 'measurement=raw_temps,window=1h,target_measurement=temps_fahrenheit,names_transformations=Temperature:"snake",values_transformations=temperature:"convert_degC_to_degF"' \
temp_converter
# Write test data
influxdb3 write \
--database weather \
"raw_temps,location=office Temperature=22.5"
# Query transformed data (after trigger runs)
influxdb3 query \
--database weather \
"SELECT * FROM temps_fahrenheit"
Expected output
| location | temperature | time |
|---|---|---|
| office | 72.5 | 2024-01-01T00:00:00Z |
Transformation details:
- Before:
Temperature=22.5(Celsius) - After:
temperature=72.5(Fahrenheit, field name converted to snake_case)
Example 2: Field name standardization
Clean and standardize field names from various sensors:
# Create trigger with multiple transformations
influxdb3 create trigger \
--database sensors \
--path "gh:influxdata/basic_transformation/basic_transformation.py" \
--trigger-spec "all_tables" \
--trigger-arguments 'measurement=raw_sensors,target_measurement=clean_sensors,names_transformations=.*:"remove_special_chars snake collapse_underscore trim_underscore"' \
field_cleaner
# Write data with inconsistent field names
influxdb3 write \
--database sensors \
"raw_sensors,device=sensor1 \"Room Temperature\"=20.1,\"__Humidity_%\"=45.2"
# Query cleaned data
influxdb3 query \
--database sensors \
"SELECT * FROM clean_sensors"
Expected output
| device | room_temperature | humidity | time |
|---|---|---|---|
| sensor1 | 20.1 | 45.2 | 2024-01-01T00:00:00Z |
Transformation details:
- Before:
"Room Temperature"=20.1,"__Humidity_%"=45.2 - After:
room_temperature=20.1,humidity=45.2(field names standardized)
Example 3: Custom string replacements
Replace specific strings in field values:
# Create trigger with custom replacements
influxdb3 create trigger \
--database inventory \
--path "gh:influxdata/basic_transformation/basic_transformation.py" \
--trigger-spec "every:1d" \
--trigger-arguments 'measurement=products,window=7d,target_measurement=products_updated,values_transformations=status:"status_replace",custom_replacements=status_replace:"In Stock=available.Out of Stock=unavailable"' \
status_updater
Ready to get started?
Download InfluxDB 3 and have running in minutes.