45_filtering.asciidoc

Filtering Queries and Aggregations

A natural extension to aggregation scoping is filtering. Because the aggregation operates in the context of the query scope, any filter applied to the query will also apply to the aggregation.

Filtered Query

If we want to find all cars over $10,000 and also calculate the average price for those cars, we can simply use a filtered query:

GET /cars/transactions/_search?search_type=count
{
    "query" : {
        "filtered": {
            "range": {
                "price": {
                    "gte": 10000
                }
            }
        }
    },
    "aggs" : {
        "single_avg_price": {
            "avg" : { "field" : "price" }
        }
    }
}

Fundamentally, using a filtered query is no different from using a match query like we discussed in the last section. The query (which happens to include a filter) returns a certain subset of documents, and the aggregation operates on those documents.

Filter bucket

But what if you would like to filter just the aggregation results? Imagine we are building the search page for our car dealership. We want to display search results according to what the user searches for. But we also want to enrich the page by including the average price of cars (matching the search) which were sold in the last month.

We can’t use simple scoping here, since there are two different criteria. The search results must match "ford", but the aggregation results must match "ford" AND "sold > now - 1M".

To solve this problem, we can use a special bucket called filter. You specify a filter, and when documents match the filter’s criteria, they are added to the bucket.

Here is the resulting query:

GET /cars/transactions/_search?search_type=count
{
   "query":{
      "match": {
         "make": "ford"
      }
   },
   "aggs":{
      "recent_sales": {
         "filter": { (1)
            "range": {
               "sold": {
                  "from": "now-1M"
               }
            }
         },
         "aggs": {
            "average_price":{
               "avg": {
                  "field": "price" (2)
               }
            }
         }
      }
   }
}

1. Using the filter bucket to apply a filter in addition to the query scope

2. This avg metric will therefore only average docs which are both "ford" and sold in the last month

Since the filter bucket operates like any other bucket, you are free to nest other buckets and metrics inside. All nested components will "inherit" the filter. This allows you to filter selective portions of the aggregation as required.

Post Filter

So far, we have a way to filter the both search results and aggregations (a filtered query), as well as filtering individual portions of the aggregation (filter bucket).

You may be thinking to yourself "hmm…​is there a way to filter just the search results but not the aggregation?". The answer is to use a post_filter.

This is a top-level search request element which accepts a filter. The filter is applied after the query has executed (hence the "post" moniker…​it runs post query execution). Because it operates after the query has executed, it does not affect the query scope…​and thus does not affect the aggregations either.

We can use this behavior to apply additional filters to our search criteria that don’t affect things like categorical facets in your UI. Let’s design another search page for our car dealer. This page will allow the user to search for a car and filter by color. Color choices are populated via an aggregation.

GET /cars/transactions/_search?search_type=count
{
    "query": {
        "match": {
            "make": "ford"
        }
    },
    "post_filter": {    (1)
        "term" : {
            "color" : "green"
        }
    },
    "aggs" : {
        "all_colors": {
            "terms" : { "field" : "color" }
        }
    }
}

1. The post_filter element is a "top-level" element and filters just the search hits

The query portion is finding all "ford" cars. We are then building a list of colors with a terms aggregation. Because aggregations operate in the query scope, the list of colors will correspond with the colors that Ford cars are painted.

Finally, the post_filter will filter the search results to show only green "ford" cars. This happens after the query is executed, so the aggregations are unaffected.

This is often important for coherent UIs. Imagine a user clicks a category in your UI (e.g. "green"). The expectation is that the search results are filtered, but not the UI options. If you applied a filtered query, the UI would instantly transform to show only "green" as an option…​not what the user wants!

Warning

Performance consideration Only use a post_filter if you need to differentially filter search results and aggregations. Sometimes people will use post_filter for regular searches. Don’t do this! The nature of the post_filter means it runs after the query, so any performance benefit of filtering (caches, etc) is lost completely. The post_filter should only be used in combination with aggregations, and only when you need differential filtering.

Recap

Choosing the appropriate type of filtering — search hits, aggregations or both — often boils down to how you want your user interface to behave. Choose the appropriate filter (or combinations) depending on how you want to display results to your user.

• filtered query: affects both search results and aggregations

• filter bucket: affects just aggregations

• post_filter: affects just search results