Analyzers and rankers

Everything in the search is a bean, asked in @Order: SearchDocumentSource beans give documents, SearchAnalyzer beans cut the text of a language into terms, and SearchRanker beans change the score of each hit. An app adds or replaces any of them without touching the service.

Why

Languages differ: English has word endings and stop words, Japanese has no spaces, German has compounds. One analysis for all would miss matches in some and invent them in others. And what ranks first is often a business rule (the reference before the guides, a sold-out product last), not a matter of words.

Analyzers

A SearchAnalyzer is a tokenizer, then filters in order. The documents and the queries go through the same analyzer, so they meet.

Analyzer Tokenizer and filters Used for
standard StandardTokenizer, WordPartsFilter, LowerCaseFilter, AsciiFoldingFilter every language but English, unless named
english standard, then StopWordFilter(StopWordFilter.english) and EnglishStemFilter en
Part What it does
StandardTokenizer Words of letters and digits in any script; Chinese, Japanese and Korean, which write no spaces, as every two characters next to each other (依存性 gives 依存, 存性), so a query cut the same way finds them without a dictionary
WordPartsFilter Also the parts of a word in camel case or with digits: ApplicationContext is found by context too; the whole word stays
LowerCaseFilter Bean finds bean
AsciiFoldingFilter Latin letters without their accents: café finds cafe
StopWordFilter Leaves out words that are in every document (the, a)
EnglishStemFilter English endings removed: beans and bean, binding and bind. Light on purpose: a word under four letters stays, and an ending goes only when a vowel is left before it (string stays)

An analyzer of your own

Declare it as a bean and name it per language. A bean with a built-in name (standard, english) replaces it.

// Lower case, no accents, and the app's own stop words.
@Configuration()
class SearchConfig {
  @Bean()
  SearchAnalyzer french() => const SearchAnalyzer(
    'french',
    filters: [
      WordPartsFilter(),
      LowerCaseFilter(),
      AsciiFoldingFilter(),
      StopWordFilter({'le', 'la', 'les', 'de', 'des', 'un', 'une', 'et'}),
    ],
  );
}
spring:
  search:
    analyzers:
      fr: french
      ja: standard

A filter of your own implements SearchTokenFilter (Iterable<SearchToken> filter(Iterable<SearchToken> tokens)), a tokenizer SearchTokenizer. A SearchToken keeps its term (what is compared), the word before a stemmer shortened it (for a word still being typed), and its place in the text (start, end, for highlights).

The language of a search picks its analyzer: the whole tag (pt-BR) when named, else the part before - or _ (pt), else english for en and standard for the rest.

Rankers

A SearchRanker changes the score of a hit after the index ranked it. The rankers run in @Order, each on the score the one before gave, with the query searched (SearchQuery: its clauses, each a word or a phrase as written); the hits are sorted again after them.

// The reference pages before the guides.
@Component()
class ReferenceFirst implements SearchRanker {
  @override
  double rank(SearchHit hit, double score, SearchQuery query) =>
      (hit.document.attributes['path'] ?? '').contains('/reference/')
          ? score * 1.5
          : score;
}

The property

Key Type Default Meaning
spring.search.analyzers.<language> analyzer name english for en, standard otherwise The analyzer of a language

See also