Query best practices — Sensedia Analytics API

To ensure fast and stable performance for all users, we strongly recommend following these best practices when querying the Sensedia Analytics API.

These recommendations are especially important in high-volume and multi-tenant environments, as they directly contribute to cluster stability, lower latency, and a better overall platform experience.

1. Always filter by time (date range)

Search on temporal data is optimized when indexes follow the time-series pattern.

• Recommended practice: always include a time range filter in your queries.
• Avoid: queries that attempt to retrieve data from the entire historical dataset without time restrictions.
• Optimization: limit the query to the smallest time window necessary. For example, querying 24 hours of data is significantly faster than querying 30 days.

2. Limit response size (pagination)

Restricting the number of documents returned in a single request is critical for both network performance and query efficiency.

• Use pagination: use parameters such as size and from (or page and limit) to control the number of results returned and the query offset.

Pagination example using the size and from parameters (recommended for small volumes):

• Smart limits: keep the size at a value whose response fits within the 6 MB payload limit. Returning all fields of the record, that limit is reached at around a few thousand records; selecting only the fields you need allows a larger page. See usage limits.

• Results larger than 10,000 records: the sum of from and size cannot exceed 10,000. To page through a larger result, see how to paginate the results of the traces/query endpoint.

3. Select only the required fields

By default, OpenSearch returns the full JSON document (_source). If your application requires only a subset of fields, explicitly specify which fields are needed.

• Use _source_includes or field parameters: request only the fields that will be used, for example: fields=timestamp,status_code,latency.

Benefit: this significantly reduces bandwidth consumption and processing time in the OpenSearch cluster.

4. Optimize aggregations

Aggregations are used to compute metrics such as counts, averages, and sums, and they often represent the most expensive part of a query.

• Filter before aggregating: always apply strict filters — especially the time filter — before executing aggregations. The fewer documents processed, the faster the response.

• Avoid high-cardinality aggregations: avoid grouping by fields with an extremely high number of unique values, such as transaction IDs. Prefer fields with controlled cardinality, such as status_code or api_name.

• Bucket limitation: do not request an excessive number of buckets in a single aggregation. If you need more than 100 buckets, consider performing multiple queries or using sampling techniques.

Example: if an aggregation is executed with a granularity of 1 second over a 24-hour period, the number of generated buckets will be 86,400. This volume exceeds the recommended limits and may lead to query rejection.

5. Prefer terms over wildcards

How you structure your search queries directly impacts performance.

• Correct query type selection: use term exclusively for exact matches on keyword fields and match for full-text searches on analyzed text fields, avoiding unnecessary text processing. Avoid wildcard (*) and regular expression queries whenever possible, as they require extensive index scanning and consume significantly more CPU and I/O.

• If you must use wildcard: avoid starting the expression with *. For example, prefer api_name* instead of *api_name.

6. Avoid unnecessary sorting

Sorting results is an expensive operation. Avoid sorting that is not necessary, and when it is necessary, choose fields that make it cheap.

• In exploratory searches: if the default sort by timestamp in descending order is sufficient, do not specify additional sort fields.

• When sorting is required: prefer numerical or date fields, and text fields only in the keyword version. Avoid sorting by unstructured text fields.

• In paginated queries: explicit sorting is no longer optional. Paging through results with search_after requires a stable sort, and that cost is justified because it is what keeps records from being duplicated or skipped between pages. See how to paginate the results of the traces/query endpoint.

Example: optimizing the view of recent traces

• Situation: you need to analyze the most recent traces captured by the system.
• Recommended action:
  1. Apply the time filter: select the desired period (e.g., last 15 minutes).
  2. Avoid manual sorting: do not click on column headers to reorder the data.
  3. Trust the default: results are already delivered sorted by timestamp (descending) when there is no text search.
• Benefits:
  • Speed: significantly faster queries.
  • Efficiency: lower CPU and memory consumption on the cluster.
  • Stability: consistent responses without overloading the search nodes.

By following these guidelines, your clients will maximize query performance and help maintain the optimal operation of the Sensedia Analytics environment.