Use the limit clause in your search query to reduce the data resultset in IBM Cloud Logs

Use the limit clause in your search query to reduce the data resultset and improve performance optimization. For example, source logs | ... | limit 500.

The limit command restricts the number of logs that are returned by a query to the specified count.

It is typically used after sorting or aggregating operations to retrieve only the most relevant or top results from a larger dataset.

This command is useful for performance optimization or for displaying only a subset of high-value results.

The limit command does not guarantee order unless it is used after an explicit orderby. If order matters, always pair it with orderby to ensure predictable results.

Reduce the data resultset by adding filters to a Dataprime query in IBM Cloud Logs in IBM Cloud Logs

Add explicit filter clauses or grep conditions early in your DataPrime Query Assistance or search bar to reduce the working data set before running transformations.

For example, the following sample shows a filter condition before an aggregation expression:

source logs
| filter $d.action == 'create'
| groupby $d.path aggregate count()

Reduce the data resultset by using dynamic range variables in widget Dataprime queries in IBM Cloud Logs

This recommendation only applies to Dataprime queries that you configure in the widget of a custom dashboard.

If you are running a Dataprime query in a widget in a custom dashboard, you can use dynamic range variables ($p) to reduce the the working data set before running transformations.

For more information, see Using query parameters in your DataPrime queries in IBM Cloud Logs.

Reduce the data resultset by using Dataprime commands in a query in IBM Cloud Logs

Reduce the data resultset by using Dataprime commands that apply filters or keep only relevant fields.

• filter command: Keeps rows where a condition is true.

  For example, you can select the logs where the status_code value is greater or equal to 500.

  filter status_code >= 500
  

• block command: Removes rows that match a condition.

  For example, you can ignore logs that include a field method and the value of the field is set to mm1.

  block method == 'mm1'
  

• choose command: Keeps only the specified fields and ignores other fields in the log record.

  For example, you can choose 2 fields from a log records.

  choose path, status_code
  

• distinct command: Returns one row per unique value.

  For example, you can identify all the user IDs in the your logs.

  distinct user_id
  

• find command: Free-text search within a field or across all data.

  Use the find command for quickly searching raw log fields across log messages or string fields where full parsing is unnecessary in DataPrime queries.

Narrow the timeframe of a query in IBM Cloud Logs

Shorten the default time range in the Explore Logs page or inside the query code to minimize the volume of data that is scanned.

Querying large time ranges (weeks or months) increases the scan volume and slows down results.

To narrow the timeframe of a query in IBM Cloud Logs:

• Use the time picker to select only what is necessary.

• Use relative time queries.

  Relative time queries are handled via DataPrime time expressions. For example, use now() combined with standard time arithmetic and intervals (such as 1h, 15m, or 30s) to filter events dynamically or filter logs that are newer than one hour ($m.timestamp > now() - 1h). For more information, see DataPrime date / time functions.

  You can also use dynamic time range variables ($p) to automatically respect active dashboard time pickers. For example, you can reference $p.timeRange.startTime or $p.timeRange.endTime in your queries. For more information, see Using query parameters in your DataPrime queries in IBM Cloud Logs.

  You can shift the evaluation window backward inside specific Dataprime commands to analyze historical baselines or compare current operational data against past performance. The timeshifted keyword modifies the timerange within the scope of the query. For example, when a custom dashboard has a global timerange of Today but your query only makes sense when looking at yesterday's data, you can use the keyword timeshifted followed by a time interval (source logs timeshifted 1h).

Change the querying mode to query data in the COS bucket to increase the amount of logs that are returned in IBM Cloud Logs

Change the querying mode to query data in the COS bucket to increase the amount of logs that are returned.

When you run a query in Priority insights that returns the message Scan limit exceeded, you have hit the cap for a query.

• You can switch to All Logs mode in the Logs page. This mode queries your full archive where scan limits are much higher.
• You can switch to *Analyze and Alert in a custom widget configuration panel. This mode queries data in the Priority insights and Analyze and alert data pipelines where scan limits are much higher.
• You can also simplify the query by removing unnecessary transformations, filters, or nested expressions that force full scans.