Querying data

Query data from the UI

In the Explore Logs page, you can:

• Filter log data
• Search log data using Lucene
• Search log data using DataPrime

Filtering can be used in conjunction with searching using Lucene or DataPrime.

After you define a query, you can save it for later reuse by creating a view. For more information, see Creating custom views.

Considerations when querying Priority insights data

There are considerations when querying log data in the IBM Cloud Logs Priority insights pipeline:

• Filtering can be used in conjunction with searching using Lucene or DataPrime.

• Logs in the Priority insights pipeline are indexed. If your instance reaches its maximum amount of indexed fields, additional fields are unavailable to query. For more information on indexing and data mapping, see Understanding indexing and field mapping.

• You can get a mapping exception when data that is ingested through the Priority insights data pipeline detects the same field sent by different log records with different types. Mapping exceptions make fields unavailable to query. For more information, see Mapping exceptions.

• Logs ingested through the Analyze and alert and Analyze and alert data pipelines can only be directly queried from the archive..

• You can query logs that are ingested and processed through the Priority insights data pipeline by using a Lucene query or a DataPrime query.

  For example, when you define a Lucene query, you can run queries such as free-text searches, regular RegEX expressions, or using field searches.

  If you are not seeing expected data:

  • Consider broadening your query or removing filters.

  • Note that some of your logs might not yet be indexed and will not be found by the filter or query.

Considerations when querying data from a bucket

You can query data from the Explorer logs page or by running an archive query.

There are considerations when querying log data from the data bucket:

• Data stored in the data bucket include data ingested through the Priority insights, Analyze and alert, and Store and search data pipelines.
• You maintain the data in the bucket. You can keep the data for as long as you need and query it via the Logs page, selecting the All logs option.
• Filtering can be used in conjunction with searching using Lucene or DataPrime.
• You can query data with unlimited time frames. There are no restrictions on how far back in time your data can go. You maintain the data and you ahve access to the data for as long as you keep it.
• You can query logs regardless of log priority and daily quota. Only blocked logs are not sent to the archive.
• Archive Query lets you to directly query your logs from your archive using any text or a wide range of syntax queries. You can query logs regardless of log priority, daily quota, or the time frame of your data. For more information, see Querying archived data.

Limitations querying data through the Explorer

Limits exist when querying data in IBM® Cloud Logs.

Log query limits for Direct HTTP API Archive Query

Results returned

The maximum number of rows that are returned from a query depends if you are querying from Priority insights or data that is stored in IBM Cloud Object Storage.

• The maximum number of results that are returned from Priority insights is 12 K.
• The maximum number of results that are returned from IBM Cloud Object Storageis 50 K.

Bytes scanned

A maximum of 100 MB is scanned for Priority insights data. No limit exists when data stored in IBM Cloud Object Storage is scanned.

Rate limiting

A maximum of 10 queries per minute can be submitted.

When the rate limit is exceeded, an HTTP 429 is returned.

Archive query limitations

The following are the limitations placed on queries.

You also need to be aware of the following when querying archived data:

• You can use the same query syntax (Lucene or DataPrime) on the Archive Queries page.

• You might see slight delays when querying archived data when compared with other Explore queries.

• Once a limit is reached, a warning message is displayed. Refine your query results to avoid reaching a limit.

Refining archive query results

You can refine your query results using the following methods:

• Apply more selective filters to your queries (for example, application or subsystem).

• If using the DataPrime extract operator and subsequently filtering its results, create a parsing rule and filter on the parsed field instead.

• Avoid regular expressions or wildcards in filters.

• In DataPrime, switch from using the contains operator on strings to the free text search operator (~).