Cisco Model-Driven Telemetry and Parquet Integration

Input and output integration overview

The Cisco Model-Driven Telemetry (MDT) plugin facilitates the collection of telemetry data from Cisco networking platforms, utilizing gRPC and TCP transport mechanisms. This plugin is essential for users looking to implement advanced telemetry solutions for better insights and operational efficiency.

This plugin writes metrics to parquet files, utilizing a schema based on the metrics grouped by name. It supports file rotation and buffered writing for optimal performance.

Integration details

Cisco Model-Driven Telemetry

Cisco model-driven telemetry (MDT) is designed to provide a robust means of consuming telemetry data from various Cisco platforms, including IOS XR, IOS XE, and NX-OS. This plugin focuses on the efficient transport of telemetry data using either TCP or gRPC protocols, offering flexibility based on the network environment and requirements. The gRPC transport is particularly advantageous as it supports TLS for enhanced security through encryption and authentication. The plugin is compatible with a range of software versions on Cisco devices, enabling organizations to leverage telemetry capabilities across their network operations. It is especially useful for network monitoring and analytics, as it enables real-time data collection directly from Cisco devices, enhancing visibility into network performance, resource utilization, and operational metrics.

Parquet

The Parquet output plugin for Telegraf writes metrics to parquet files, which are columnar storage formats optimized for analytics. By default, this plugin groups metrics by their name, writing them to a single file. If a metric’s schema does not align with existing schemas, those metrics are dropped. The plugin generates an Apache Arrow schema based on all grouped metrics, ensuring that the schema reflects the union of all fields and tags. It operates in a buffered manner, meaning it temporarily holds metrics in memory before writing them to disk for efficiency. Parquet files require proper closure to ensure readability, and this is crucial when using the plugin, as improper closure can lead to unreadable files. Additionally, the plugin supports file rotation after specific time intervals, preventing overwrites of existing files and schema conflicts when a file with the same name already exists.

Configuration

Cisco Model-Driven Telemetry

[[inputs.cisco_telemetry_mdt]]
 ## Telemetry transport can be "tcp" or "grpc".  TLS is only supported when
 ## using the grpc transport.
 transport = "grpc"

 ## Address and port to host telemetry listener
 service_address = ":57000"

 ## Grpc Maximum Message Size, default is 4MB, increase the size. This is
 ## stored as a uint32, and limited to 4294967295.
 max_msg_size = 4000000

 ## Enable TLS; grpc transport only.
 # tls_cert = "/etc/telegraf/cert.pem"
 # tls_key = "/etc/telegraf/key.pem"

 ## Enable TLS client authentication and define allowed CA certificates; grpc
 ##  transport only.
 # tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]

 ## Define (for certain nested telemetry measurements with embedded tags) which fields are tags
 # embedded_tags = ["Cisco-IOS-XR-qos-ma-oper:qos/interface-table/interface/input/service-policy-names/service-policy-instance/statistics/class-stats/class-name"]

 ## Include the delete field in every telemetry message.
 # include_delete_field = false

 ## Specify custom name for incoming MDT source field.
 # source_field_name = "mdt_source"

 ## Define aliases to map telemetry encoding paths to simple measurement names
 [inputs.cisco_telemetry_mdt.aliases]
   ifstats = "ietf-interfaces:interfaces-state/interface/statistics"
 ## Define Property Xformation, please refer README and https://pubhub.devnetcloud.com/media/dme-docs-9-3-3/docs/appendix/ for Model details.
 [inputs.cisco_telemetry_mdt.dmes]
#    Global Property Xformation.
#    prop1 = "uint64 to int"
#    prop2 = "uint64 to string"
#    prop3 = "string to uint64"
#    prop4 = "string to int64"
#    prop5 = "string to float64"
#    auto-prop-xfrom = "auto-float-xfrom" #Xform any property which is string, and has float number to type float64
#    Per Path property xformation, Name is telemetry configuration under sensor-group, path configuration "WORD         Distinguished Name"
#    Per Path configuration is better as it avoid property collision issue of types.
#    dnpath = '{"Name": "show ip route summary","prop": [{"Key": "routes","Value": "string"}, {"Key": "best-paths","Value": "string"}]}'
#    dnpath2 = '{"Name": "show processes cpu","prop": [{"Key": "kernel_percent","Value": "float"}, {"Key": "idle_percent","Value": "float"}, {"Key": "process","Value": "string"}, {"Key": "user_percent","Value": "float"}, {"Key": "onesec","Value": "float"}]}'
#    dnpath3 = '{"Name": "show processes memory physical","prop": [{"Key": "processname","Value": "string"}]}'

 ## Additional GRPC connection settings.
 [inputs.cisco_telemetry_mdt.grpc_enforcement_policy]
  ## GRPC permit keepalives without calls, set to true if your clients are
  ## sending pings without calls in-flight. This can sometimes happen on IOS-XE
  ## devices where the GRPC connection is left open but subscriptions have been
  ## removed, and adding subsequent subscriptions does not keep a stable session.
  # permit_keepalive_without_calls = false

  ## GRPC minimum timeout between successive pings, decreasing this value may
  ## help if this plugin is closing connections with ENHANCE_YOUR_CALM (too_many_pings).
  # keepalive_minimum_time = "5m"

Parquet

[[outputs.parquet]]
  ## Directory to write parquet files in. If a file already exists the output
  ## will attempt to continue using the existing file.
  # directory = "."
  
  ## Files are rotated after the time interval specified. When set to 0 no time
  ## based rotation is performed.
  # rotation_interval = "0h"
  
  ## Timestamp field name
  ## Field name to use to store the timestamp. If set to an empty string, then
  ## the timestamp is omitted.
  # timestamp_field_name = "timestamp"

Input and output integration examples

Cisco Model-Driven Telemetry

1. Real-Time Network Monitoring: Utilize the Cisco MDT plugin to collect network performance metrics from Cisco routers and switches. By feeding telemetry data into a visualization tool, network operators can observe traffic trends, bandwidth usage, and error rates in real-time. This proactive monitoring allows teams to swiftly address issues before they affect network performance, resulting in a more reliable service.

2. Automated Anomaly Detection: Integrate Cisco MDT with machine learning algorithms to create an automated anomaly detection system. By continuously analyzing telemetry data, the system can identify deviations from typical operational patterns, providing alerts for unusual conditions that may signify network problems or security threats, which can aid in maintaining operational integrity.

3. Dynamic Configuration Management: Leveraging the telemetry data collected from Cisco devices, organizations can implement dynamic configuration management solutions that automatically adjust network settings based on current performance indicators. For instance, if the telemetry indicates high utilization on certain links, the system could dynamically route traffic to underutilized paths, optimizing resource usage.

4. Enhanced Reporting and Analytics: Use the Cisco MDT plugin to feed detailed telemetry data into analytics platforms, enabling comprehensive reporting on network health and performance. Historical and real-time analysis can guide decision-making and strategic planning, helping organizations to allocate resources more effectively and understand their network’s operational landscape better.

Parquet

1. Data Lake Ingestion: Utilize the Parquet plugin to store metrics from various sources into a data lake. By writing metrics in parquet format, you establish a standardized and efficient way to manage time-series data, enabling faster querying capabilities and seamless integration with analytics tools like Apache Spark or AWS Athena. This setup can significantly improve data retrieval times and analysis workflows.

2. Long-term Storage of Metrics: Implement the Parquet plugin in a monitoring setup where metrics are collected over time from multiple applications. This allows for long-term storage of performance data in a compact format, making it cost-effective to store vast amounts of historical data while preserving the ability for quick retrieval and analysis later on. By archiving metrics in parquet files, organizations can maintain compliance and create detailed reports from historical performance trends.

3. Analytics and Reporting: After writing metrics to parquet files, leverage tools like Apache Arrow or PyArrow to perform complex analytical queries directly on the files without needing to load all the data into memory. This can enhance reporting capabilities, allowing teams to generate insights and visualization from large datasets efficiently, thereby improving decision-making processes based on accurate, up-to-date performance metrics.

4. Integrating with Data Warehouses: Use the Parquet plugin as part of a data integration pipeline that feeds into a modern data warehouse. By converting metrics to parquet format, the data can be easily ingested by systems like Snowflake or Google BigQuery, enabling powerful analytics and business intelligence capabilities that drive actionable insights from the collected metrics.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.