WARNING: The 2.x versions of Elasticsearch have passed their EOL dates. If you are running a 2.x version, we strongly advise you to upgrade.
This documentation is no longer maintained and may be removed. For the latest information, see the current Elasticsearch documentation.
Pros and Cons of Stopwordsedit
We have more disk space, more RAM, and better compression algorithms than existed back in the day. Excluding the preceding 33 common words from the index will save only about 4MB per million documents. Using stopwords for the sake of reducing index size is no longer a valid reason. (However, there is one caveat to this statement, which we discuss in Stopwords and Phrase Queries.)
On top of that, by removing words from the index, we are reducing our ability to perform certain types of searches. Filtering out the words listed previously prevents us from doing the following:
- Distinguishing happy from not happy.
- Searching for the band The The.
- Finding Shakespeare’s quotation “To be, or not to be”
-
Using the country code for Norway:
no
The primary advantage of removing stopwords is performance. Imagine that we
search an index with one million documents for the word fox. Perhaps fox
appears in only 20 of them, which means that Elasticsearch has to calculate the
relevance _score for 20 documents in order to return the top 10. Now, we
change that to a search for the OR fox. The word the probably occurs in
almost all the documents, which means that Elasticsearch has to calculate
the _score for all one million documents. This second query simply cannot
perform as well as the first.
Fortunately, there are techniques that we can use to keep common words searchable, while still maintaining good performance. First, we’ll start with how to use stopwords.