Home
...
Data Streaming
Details and Features of Data Streaming

Details and Features of Data Streaming

Understand delivery formats, differences between observability and analytics, trace dimensions, and examples of exported files.

Data Delivery

Data Streaming uses different technologies to export data, and the delivery reliability can vary depending on the format and destination chosen.

Export via OpenTelemetry (OTLP) When exporting data in OTLP format to destinations such as Dynatrace, Datadog, or AWS S3, inherent completeness limitations from the OpenTelemetry model may occur. This format is better suited for observability scenarios (such as performance monitoring and event tracing), where analyses based on samples are usually sufficient to identify failures and anomalies. It is not suitable for business data that requires integral precision.
Export in JSON format Exporting data in JSON format generally offers a more reliable delivery rate. This option is recommended for situations where data integrity has a higher priority, but even so, we do not recommend it for use in critical decision-making systems that depend on 100% of the data for correct operation.

NOTE
The JSON file sent is compressed in gz format.

TIP
See also: Data Streaming usage limits.

Observability vs. Data Analysis

Understand the difference between the two concepts that, although related, have distinct goals:

Observability: tracks system behavior and health in real time. It uses telemetry data, such as metrics, logs, and traces, and performs analyses through intelligent sampling, which can reduce the data volume when everything is operating normally. The pipeline is optimized for speed and efficiency, prioritizing rapid problem detection.
Data Analysis (Analytics): focuses on understanding the business and user behavior. It uses historical data for detailed analyses, report generation, and strategic decision-making. The pipeline prioritizes comprehensiveness.

IMPORTANT

Data Streaming and Sensedia Analytics process data through different pipelines, which can impact the volume and granularity of the available information.

Temporary interruptions in networks, messaging systems, or service providers can create gaps in observability, reinforcing the difference between observational and analytical data.

Trace Data and Dimensions

The traces sent in OTLP format are generated on the Sensedia gateway itself. This means they are separated from any distributed trace that your client application may have generated. Currently, we only send the trace dimension.

Examples

See examples of what the file generated and sent to different destinations looks like.

OTLP Example

See an example of a file in OTLP format, sent to AWS S3:
JSON Example

See an example of a file exported in JSON format for S3, Azure Blob, or GCS:

NOTE
The example above has been reduced for easier reading. In real files, the fields trace, requestHeaders, and responseHeaders may contain more extensive information. If you prefer, you can download a complete example file in JSON format here.
- Folder Structure
  
  The directory structure varies by storage provider. In the case of AWS S3 (JSON), Azure, and Google Cloud, the organization follows the pattern:
  
  Example:
  
  {internal_identifier} = Sensedia control point.

Responsibility matrix

Data Streaming delivers your data to the destination you configure. The destination environment, whether an APM, observability, or storage platform, is provisioned, sustained, and controlled by you. The table below summarizes what is Sensedia's responsibility and what is yours.

Responsibility	Sensedia	Customer
Delivery stream configuration
Provide the Data Streaming screen and infrastructure, with the supported destinations	✓
Open the ticket with the destination connection information		✓
Create and activate the delivery stream from the connection information provided in the ticket, for any destination	✓
Edit and delete the delivery stream on the self-service destinations: AWS S3 (OTLP), Datadog Agent (OTLP), Datadog, Dynatrace, and Grafana (OTLP)		✓
Edit and delete the delivery stream on the remaining destinations: AWS S3 (JSON), Azure, and Google Cloud	✓ *
Keep the credentials valid, renewing them when they expire or are rotated, and notify Sensedia through a ticket when the destination does not offer self-service editing		✓
Destination environment and data usage
Provision and sustain the account, the license, and the resources required by the destination		✓
Ensure the destination is reachable and compatible with the expected protocol, allowing Sensedia's access when there is a network restriction		✓
Grant Data Streaming the permissions required by the destination to write and ingest the data		✓
Sizing and sustaining the ingestion and storage consumption on the destination		✓
Usage of the data after delivery, including visualizations, alerts and queries, retention, backup, security, access control, and compliance in your environment		✓
Adapt the destination queries and visualizations to the exported format and fields		✓
Data sending
Collect the data on the Sensedia gateway, convert it to the destination format (OTLP or JSON), and deliver it to the configured destination, without applying filters or field selection	✓
Send the data generated from the activation of the delivery stream onward. There is no retroactive sending of data prior to the activation or of the period in which the stream is inactive	✓
Define the content recorded in the trace by the interceptors configured on the platform and decide whether to apply encryption to that content		✓
Run the retry attempts provided by the feature on momentary failures	✓ **
Handle rejections and unavailability on the destination		✓
Monitor internally whether the data sent is being accepted or rejected by the destination	✓
Track the arrival of the data in your environment, since Sensedia's monitoring is internal		✓
Support and evolution
Support for the Data Streaming feature and for the delivery of the data to the destination	✓
Support for the destination APM, observability, or storage platform		✓ ***
Maintain the documentation of the supported destinations and of the delivery formats	✓
Make new destinations available	✓

* Through a support ticket.

** Delivery is best effort. After the retry attempts, undelivered data is discarded and there is no reprocessing.

*** With the platform vendor.

IMPORTANT
Unavailability, rejections, or losses that occur in the destination environment are outside Sensedia's scope of responsibility.

How happy are you with this page?

Log Searching

Configuring your Delivery Stream