Query aggregations

Add a filter

Use the filter() in the aggregation clause to filter results. For example, you can specify the same filter that was submitted separately in the previous example directly in the aggregation clause.

{
  "aggregation": "filter(enriched_text.entities:(text::Gilroy,type::Location)).term(enriched_text.entities.type)"
}

In this case, the filter().term() aggregation finds the same result as the earlier example with the separate filter and aggregation clauses. However, results are ranked differently when the filter clause is used. You can leverage this difference by using the filter() clause within the aggregation clause to filter results from a sequence of expressions, as shown in the next example.

Start with nested objects

In the previous examples, the "matching_counts" value represents the number of documents that match the filter and aggregation. You might want to count how many nested objects are present in the query response. The nested() aggregation allows you to change the set of documents that is used as input to other aggregation terms.

For example, in the following query the nested() segment selects all enriched_text.entities nested objects as the input used by the filter() and term() segments.

{
  "aggregation": "nested(enriched_text.entities).filter(enriched_text.entities.type::Organization).term(enriched_text.entities.text,count:3)"
}

The query results in an aggregations object that looks as follows:

{
  "aggregations": [
    {
      "type": "nested",
      "path": "enriched_text.entities",
      "matching_results": 1993,
      "aggregations": [
        {
          "type": "filter",
          "match": "enriched_text.entities.type::Organization",
          "matching_results": 645,
          "aggregations": [
            {
              "type": "term",
              "field": "enriched_text.entities.text",
              "count": 3,
              "results": [
                {
                  "key": "IBM",
                  "matching_results": 36
                },
                {
                  "key": "Docker",
                  "matching_results": 12
                },
                {
                  "key": "OpenShift",
                  "matching_results": 12
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

The nested() segment of the query found 1993 enriched_text.entities nested objects. The filter was applied to those objects and found 645 enriched_text.entities of type Organization.

Terminal operations

For most aggregation types, when you construct a query with multiple aggregation operations, the first operation is applied to the documents. Then, the output of that operation is used as the input for the next operation. However, a subset of the aggregation types are terminal operations. The output of a terminal operation is not used as input for the next aggregation. Instead, the output is returned in a discrete group.

For an example of a request that combines aggregation types and includes an aggregation that performs a terminal operation, see the second example for the average aggregation type.

Syntax

average(field)

Example

When the average aggregation type is applied to a set of documents in which the price field contains the values that are shown in Table 1, the result is 325.

average(price)=325

This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.

{
    "query":"brake",
    "aggregation": "term(field:STATE,count:3).average(field:VEH_SPEED).term(field:CITY,count:2)"
}

For each state returned by the first term aggregation operation, the response shows the average vehicle speed specified in the incident reports. Notice that the second term aggregation uses the output from the first term aggregation, not the average aggregation, as its input.

{
  "matching_results": 9064,
  "retrieval_details": {
    "document_retrieval_strategy": "untrained"
  },
  "aggregations": [
    {
      "type": "term",
      "field": "STATE",
      "count": 3,
      "results": [
        {
          "key": "CA",
          "matching_results": 1210,
          "aggregations": [
            {
              "type": "average",
              "field": "VEH_SPEED",
              "value": 26.239653512993264
            },
            {
              "type": "term",
              "field": "CITY",
              "count": 2,
              "results": [
                {
                  "key": "LOS ANGELES",
                  "matching_results": 77
                },
                {
                  "key": "SAN DIEGO",
                  "matching_results": 66
                }
              ]
            }
          ]
        }

Syntax

filter(field)

Example

The following example filters the matching document set to include only documents that mention IBM.

filter(enriched_text.entities.text:IBM)

When combined with other aggregations, filters the matching documents set to include only those documents that meet the condition you specify.

{
    "query":"brake",
    "aggregation": "filter(VEH_SPEED>50).term(field:STATE,count:3).term(field:CITY,count:2)"
}

The query response shows cities where incidents happen that involve the brakes and the vehicle speed is over 50.

{
  "matching_results": 9064,
  "retrieval_details": {
    "document_retrieval_strategy": "untrained"
  },
  "aggregations": [
    {
      "type": "filter",
      "match": "VEH_SPEED>50",
      "matching_results": 1075,
      "aggregations": [
        {
          "type": "term",
          "field": "STATE",
          "count": 3,
          "results": [
            {
              "key": "CA",
              "matching_results": 176,
              "aggregations": [
                {
                  "type": "term",
                  "field": "CITY",
                  "count": 2,
                  "results": [
                    {
                      "key": "FONTANA",
                      "matching_results": 6
                    },
                    {
                      "key": "ALTA LOMA",
                      "matching_results": 5
                    }
                  ]
                }
              ]
            }

Syntax

group_by(condition:[(condition 1),(condition 2)...])

Each condition must be specified as a valid Discovery Query Language expression surrounded by parentheses. For example, (age<20) or (flavor:chocolate). The maximum number of conditions that you can define is 50.

You can optionally include the relevancy parameter and set it to true to return the relevancy value of the set of documents that meet the specified condition. When true, the results are sorted by relevance. When false, the results are sorted by the highest number of matching_results.

Example

The following request looks for documents that mention the term engine, and groups them by car manufacturing year. The documents are sorted into 3 groups, one group of traffic incident reports involving cars that were manufactured before 2000, one group for cars manufactured in 2000, and one group for cars manufactured after 2000.

{
    "query":"engine",
    "aggregation": "group_by(condition:[(YEARTXT<2000),(YEARTXT=2000),(YEARTXT>2000)],relevancy:true)"
}

The results might look like this:

{
  "type": "group_by",
  "results": [
    {
    "key": "YEARTXT<2000",
      "matching_results": 2034,
      "relevancy": 1.0,
      "total_matching_documents": 2034,
      "estimated_matching_results": 2034
    },
    {
      "key": "YEARTXT=2000",
      "matching_results": 1738,
      "relevancy": 1.0,
      "total_matching_documents": 1738,
      "estimated_matching_results": 1738
    },
    {
      "key": "YEARTXT>2000",
      "matching_results": 32708,
      "relevancy": 1.0,
      "total_matching_documents": 32708,
      "estimated_matching_results": 32708
    }
  ]
}

Syntax

histogram({field},{interval})

Uses field values from a single numeric field to describe the category. The field that is used to create the histogram must have a number data type, such as integer, float, double, or date.

Nonnumber types such as string are not supported. For example, "price": 1.30 is a number value that works, and "price": "1.30" is a string, so it doesn’t work.

Use the interval argument to define the size of the sections for the results to be split into. Interval values must be whole, nonnegative numbers. Choose a value that makes sense for segmenting the typical values from the field.

Histograms can process decimal values that are specified in a field, but the interval must be a whole number.

You can optionally include a custom name by including a name parameter.

Example

For example, if your data set includes the price of several items, like: “price”: 1.30, “price”: 1.99, and “price”: 2.99, you might use intervals of 1, so that you see everything that is grouped in the range 1 - 2, and 2 and 3. You do not want to use an interval of 100 because then all of the data ends up in the same segment.

histogram(product_price,interval:1)

Syntax

max(field)

Example

When the max aggregation type is applied to a set of documents in which the price field contains the values that are shown in Table 2, the result is 450.

max(price)=450

This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.

Syntax

min(field)

Example

When the min aggregation type is applied to a set of documents in which the price field contains the values that are shown in Table 3, the result is 200.

min(price)=200

This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.

Syntax

pair(first:{aggregation},second:{aggregation})

The first and second {aggregation} values must be one of the following aggregation types:

• term
• group_by
• histogram
• timeslice

The relevancy parameter from the term or group_by aggregation is ignored. The pair aggregation type calculates relevancy values by using combinations of document sets from the results of the two aggregations.

Only one pair aggregation can be used per query request, and it cannot be combined with any other aggregations.

Example

For example, you might specify term(model_name) as the first aggregation and term(component_name) as the second. Each of the aggregations returns the following values as keys of aggregated document sets:

• term(model_name): Accord, CR-V
• term(component_name): engine, brake, radiator

The calculated relevancy values of combinations of each of the document sets might look like this:

• Accord x engine
• Accord x brake
• Accord x radiator
• CR-V x engine
• CR-V x brake
• CR-V x radiator

The response defines a two-dimensional array of aggregation results, which can be represented in a table.

Each array of columns and rows of the table is sorted in the same order of the results of the first and second aggregations. For example, if you specify the term aggregation as the first argument, the resulting column arrays are sorted by frequency of terms. If you use the timeslice aggregation as the second argument, the row arrays are sorted by date or time.

Syntax

sum(field)

Example

When the sum aggregation type is applied to a set of documents in which the price field contains the values that are shown in Table 6, the result is 975.

sum(price)=975

This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.

Syntax

term(field:{field_name})

You can optionally specify the following parameters:

• count: Specifies the maximum number of terms to return.

• name: You can optionally include a custom name. Not returned if relevancy information is included in the request.

• relevancy: Boolean value that indicates whether to include relevancy information in the result. You can use relevancy to get a score that indicates the level of relevancy between the term and keywords in the query. This parameter is false by default. If set to true, the following fields are returned also:

  • total_matching_documents: Number of documents in the collection where the term is mentioned in the specified field.
  • estimated_matching_results: Number of documents that are estimated to have the term in the specified field in the set of documents that are returned by the query.

Example

The following example returns the text from the recognized entities in the document, and specifies to return a maximum of 10 terms.

For example:

term(enriched_text.entities.text,count:10)

When relevancy is set to true, a relevancy score is shown in the results. Relevancy measures the level of uniqueness of the frequency count compared to other documents that match your query. If the relevancy shows 2.0, it means that the number of times that the two data points intersect is 2 times larger than expected.

For more examples, see Grouping documents and Combining aggregation types.

Syntax

The syntax is timeslice({field},{interval},{time_zone}).

• The field that you specify must have a date data type. For more information about date field, see How dates are handled.
• Valid interval values are 1second or {n}seconds, 1minute or {n}minutes, 1hour or {n}hours, 1day or {n}days, 1week or {n}weeks, 1month or {n}months, and 1year or {n}years where {n} is a number.
• You can optionally include a custom name by including a name parameter.

Example

The following example shows the number of matches for each day value.

timeslice(field:DATEA,interval:1day)

The results look as follows.

"aggregations": [
  {
    "type": "timeslice",
    "field": "DATEA",
    "interval": "1d",
    "results": [
        {
    "key": 1262304000000,
    "key_as_string": "2010-01-01T00:00:00.000Z",
    "matching_results": 5
        },
        {
    "key": 1262390400000,
    "key_as_string": "2010-01-02T00:00:00.000Z",
    "matching_results": 18
        },
        {
    "key": 1262476800000,
    "key_as_string": "2010-01-03T00:00:00.000Z",
    "matching_results": 38
        },
        {
    "key": 1262563200000,
    "key_as_string": "2010-01-04T00:00:00.000Z",
    "matching_results": 66
        }

Syntax

{aggregation}.top_hits({n})

Example

The following example returns the top hit for the term halt per city.

{
  "query":"halt",
  "aggregation": "term(CITY).top_hits(1)"
}

The response contains the top query results for the term halt grouped by cities mentioned in documents where the term is most mentioned. Ten results are returned by default. For each of the 10 cities, the document with the top score is returned as the hit object. The content for each hit in the hits array matches the content in each result in the results array. Only the order of the results is different.

"aggregations": [
  {
    "type": "term",
    "field": "CITY",
    "results": [
      {
        "key": "LOS ALTOS",
        "matching_results": 3,
        "aggregations": [
          {
            "type": "top_hits",
            "size": 1,
            "hits": {
              "matching_results": 3,
              "hits": [
                {
                  "document_id": "2bed19a9069442fd82542827ebe260d5_7015",
                  ...
                }
              ]
            }
          }
        ]
      },
      {
        "key": "ANDOVER",
        "matching_results": 2,
        "aggregations": [
          {
            "type": "top_hits",
            "size": 1,
            "hits": {
              "matching_results": 2,
              "hits": [
                {
                  "document_id": "2bed19a9069442fd82542827ebe260d5_18329",
                  ...
                }
              ]
            }
          }
        ]
      },
      ...
      {
        "key":"ACTON",
        "maatching_results": 1,
        "aggregations": []
      }
      ...

This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.

Sytnax

trend(facet:{aggregation},time_segments:{aggregation})

The first (facet) aggregation must be one of the following types of aggregations:

• term
• group_by

The relevancy parameter from the term or group_by aggregation is ignored.

The second (time_segments) aggregation must be an aggregation of type timeslice.

You can alternatively include the following parameters:

• show_estimated_matching_results:true: Indicates whether to include the estimated_matching_results information in the result. This field contains the number of documents that are estimated to have the term in the specified field or meet the conditions in the specified aggregation for the specified time interval in the set of documents that are returned by the query.
• show_total_matching_documents:true: Indicates whether to include the total_matching_documents information in the result. This field contains the number of documents in the collection where the term is mentioned in the specified field or the condition is met.

Only one trend aggregation can be used per query request, and it cannot be combined with any other aggregations.

Example

The following example calculates the trend indicator or trend index by using combinations of results from the following aggregations:

• term(flavor): vanilla, chocolate, mint
• timeslice(date, 1month): Jan 2020, Feb 2020, Mar 2020, Apr 2020, May 2020, Jun 2020

trend( facet: aggregation(<parameter>...), time_segments: timeslice(<parameter>...)), 
show_estimated_matching_results: <true_or_false>, show_total_matching_documents: <true_or_false> )

The resulting matrix can be represented in a table.

In the following sample response, the key information is the trend_indicator value. The trend indicator measures the increase ratio of the frequency of a given facet value for a given time interval compared to the expected average frequency. The excepted average frequency is calculated based on the changes in the past time interval frequencies of the given facet value, using a weighted arithmetic mean.

If the standarized residual value is less than -2, the observed frequency is less than the expected frequency. If it is greater than 2, the observed frequency is greater than the expected frequency. If the standardized residual is greater or less than the expected frequency by 3 or more, then something unusual is happening and suggests that there might be an anomaly that is worth investigating.

For example, the expected number of feedback submissions for the vanilla flavor in May is calculated from the number of feedback submissions that were received previously (from Jan to Apr). The result is 5.341. The actual number of feedback submissions in May is 10. The results indicate that the vanilla flavor got about twice the number of feedback submissions as expected. The standardized residual value is 2.016, which is greater than expected, but not unusually so.

{
"aggregations": [
  {
    "type": "trend",
    "facet": "term(flavor),",
    "time_segments": "timeslice(date, 1month)",
    "show_estimated_matching_results": true,
    "show_total_matching_documents": true,
    "results": [
      {
        "aggregations": [
          {
            "type": "term",
            "field": "flavor",
            "results": [
              {
                "key": "vanilla",
                "matching_results": 36,
                "aggregations": [
                  {
                    "type": "timeslice",
                    "field": "date",
                    "results": [
                      {
                        "key": 1577836800000,
                        "key_as_string": "2020-01-01T00:00:00.000Z",
                        "matching_results": 4,
                        "trend_indicator": 0.0,
                        "total_matching_documents": 7,
                        "estimated_matching_results": 0.0
                      },
                      {
                        "key": 1588291200000,
                        "key_as_string": "2020-05-01T00:00:00.000Z",
                        "matching_results": 10,
                        "trend_indicator": 2.016106745,
                        "total_matching_documents": 12,
                        "estimated_matching_results": 5.340760209
                      },
                      {
                        "key": 1590969600000,
                        "key_as_string": "2020-06-01T00:00:00.000Z",
                        "matching_results": 5,
                        "trend_indicator": -0.763212711,
                        "total_matching_documents": 11,
                        "estimated_matching_results": 7.022515985
                      }
                    ]
                  }
                ]
              },
              {
                "key": "chocolate",
                "matching_results": 10,
                "aggregations": [...]
              },
              {
                "key": "mint",
                "matching_results": 25,
                "aggregations": [...]
...
}  

Syntax

topic(facet:{aggregation},time_segments:{aggregation})

The first (facet) aggregation must be one of the following types of aggregations:

• term
• group_by

The relevancy parameter from the term or group_by aggregation is ignored.

The second (time_segments) aggregation must be an aggregation of type timeslice.

You can alternatively include the following parameters:

• show_estimated_matching_results:true: Indicates whether to include the estimated_matching_results information in the result. This field contains the number of documents that are estimated to have the term in the specified field or meet the conditions in the specified aggregation for the specified time interval in the set of documents that are returned by the query.
• show_total_matching_documents:true: Indicates whether to include the total_matching_documents information in the result. This field contains the number of documents in the collection where the term is mentioned in the specified field or the condition is met.

Only one topic aggregation can be used per query request, and it cannot be combined with any other aggregations.

Example

{
    "query: like",
    "aggregation": "topic( facet: term(flavor), time_segments: timeslice(date, 1month), show_estimated_matching_results: true, show_total_matching_documents: true )"
}

With the same data set and aggregation as is used in the term aggregation example, the results might look as follows.

Notice that the topic_indicator values are different from the trend_indicator values that are returned by the trend aggregation. While both are calculated from the actual and expected frequencies, they differ because their expected frequencies are computed differently. In the trend aggregation, the expected frequency of the feedback submissions for vanilla-flavored ice cream in May is computed from the number of feedback submissions that were received for vanilla previously (from Jan to Apr) and the total number of feedback submissions received for all of the flavors in May. However, in the topic aggregation, the expected frequency of feedback submissions for vanilla-flavored ice cream in May is calculated from the number of feedback submissions that were received for vanilla and the total number of feedback submissions received for all of the flavors in May. In this example, the expected frequency result is 12.169, the actual frequency is 10, and the topic_indicator is -0.621777032.

{
"aggregations": [
  {
    "type": "topic",
    "facet": "term(flavor)",
    "time_segments": "timeslice(date, 1month)",
    "show_estimated_matching_results": true,
    "show_total_matching_documents": true,
    "results": [
      {
        "aggregations": [
          {
            "type": "term",
            "field": "flavor",
            "results": [
              {
                "key": "vanilla",
                "matching_results": 36,
                "aggregations": [
                  {
                    "type": "timeslice",
                    "field": "date",
                    "results": [
                      {
                        "key": 1577836800000,
                        "key_as_string": "2020-01-01T00:00:00.000Z",
                        "matching_results": 4,
                        "topic_indicator": -0.027972712,
                        "total_matching_documents": 7,
                        "estimated_matching_results": 4.056338028
                      },
                      {
                        "key": 1588291200000,
                        "key_as_string": "2020-05-01T00:00:00.000Z",
                        "matching_results": 10,
                        "topic_indicator": -0.621777032,
                        "total_matching_documents": 12,
                        "estimated_matching_results": 12.16901408
                      },
                      {
                        "key": 1590969600000,
                        "key_as_string": "2020-06-01T00:00:00.000Z",
                        "matching_results": 5,
                        "topic_indicator": -0.787665504,
                        "total_matching_documents": 11,
                        "estimated_matching_results": 7.098591549
                      }
                    ]
                  }
                ]
              },
              {
                "key": "chocolate",
                ...
              },
              {
                "key": "mint",
                ...
              }
}

Syntax

unique_count(field)

Example

The following aggregation requests the number of unique enrichment types that are recognized in the query.

unique_count(enriched_text.keyword.type)

The result indicates that there are 17 matching results. In those 17 documents, 14 entity types are mentioned.

{
  "matching_results": 17,
  "retrieval_details": {
    "document_retrieval_strategy": "untrained"
  },
  "aggregations": [
    {
      "type": "unique_count",
      "field": "enriched_text.entities.type",
      "value": 14.0
    }
  ],
  "results": []
}

This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.

In the following example, the aggregation parameter requests for the results to show the first 45 most-frequently mentioned entities. Per entity, it indicates how many documents mention the term and how many times in total that the term occurs.

term(enriched_text.entities.text,count:45).unique_count(enriched_text.entities.type)

The results include several aggregations such as the following group for the term PostgreSQL. The aggregation indicates that the term appears in 4 documents and is mentioned 12 times.

{
  "key": "PostgreSQL",
  "matching_results": 4,
  "aggregations": [
    {
      "type": "unique_count",
      "field": "enriched_text.entities.type",
      "value": 12.0
    }
  ]
}