15_Configure_Analyzer.asciidoc

Configuring analyzers

The third important index setting is the analysis section, which is used to configure existing analyzers or to create new custom analyzers specific to your index.

In [analysis-intro] we introduced some of the built in analyzers, which are used to convert full text strings into an inverted index, suitable for searching.

The standard analyzer, which is the default analyzer used for full text fields, is a good choice for most Western languages. It consists of:

• the standard tokenizer, which splits the input text on word boundaries

• the standard token filter, which is intended to tidy up the tokens emitted by the tokenizer (but currently is a no-op)

• the lowercase token filter which converts all tokens into lowercase

• the stop token filter which removes stopwords — common words that have little impact on search relevance, such as a, the, and, is etc.

By default the stopwords filter is disabled. You can enable it by creating a custom analyzer based on the standard analyzer and setting the stopwords parameter. Either provide a list of stopwords or tell it to use a predefined stopwords list from a particular language.

In the following example, we create a new analyzer called the es_std analyzer, which uses the predefined list of Spanish stopwords:

PUT /spanish_docs
{
    "settings": {
        "analysis": {
            "analyzer": {
                "es_std": {
                    "type":      "standard",
                    "stopwords": "_spanish_"
                }
            }
        }
    }
}

The es_std analyzer is not global — it only exists in the spanish_docs index where we have defined it. To test it with the analyze API, we must specify the index name:

GET /spanish_docs/_analyze?analyzer=es_std
El veloz zorro marrón

The abbreviated results show that the Spanish stopword "El" has been removed correctly:

{
  "tokens" : [
    { "token" :    "veloz",   "position" : 2 },
    { "token" :    "zorro",   "position" : 3 },
    { "token" :    "marrón",  "position" : 4 }
  ]
}