Analyzers and rankers
Everything in the search is a bean, asked in @Order: SearchDocumentSource beans give documents, SearchAnalyzer beans cut the text of a language into terms, and SearchRanker beans change the score of each hit. An app adds or replaces any of them without touching the service.
Why
Languages differ: English has word endings and stop words, Japanese has no spaces, German has compounds. One analysis for all would miss matches in some and invent them in others. And what ranks first is often a business rule (the reference before the guides, a sold-out product last), not a matter of words.
Analyzers
A SearchAnalyzer is a tokenizer, then filters in order. The documents and the queries go through the same analyzer, so they meet.
| Analyzer | Tokenizer and filters | Used for |
|---|---|---|
standard |
StandardTokenizer, WordPartsFilter, LowerCaseFilter, AsciiFoldingFilter |
every language but English, unless named |
english |
standard, then StopWordFilter(StopWordFilter.english) and EnglishStemFilter |
en |
| Part | What it does |
|---|---|
StandardTokenizer |
Words of letters and digits in any script; Chinese, Japanese and Korean, which write no spaces, as every two characters next to each other (依存性 gives 依存, 存性), so a query cut the same way finds them without a dictionary |
WordPartsFilter |
Also the parts of a word in camel case or with digits: ApplicationContext is found by context too; the whole word stays |
LowerCaseFilter |
Bean finds bean |
AsciiFoldingFilter |
Latin letters without their accents: café finds cafe |
StopWordFilter |
Leaves out words that are in every document (the, a) |
EnglishStemFilter |
English endings removed: beans and bean, binding and bind. Light on purpose: a word under four letters stays, and an ending goes only when a vowel is left before it (string stays) |
An analyzer of your own
Declare it as a bean and name it per language. A bean with a built-in name (standard, english) replaces it.
// Lower case, no accents, and the app's own stop words.
@Configuration()
class SearchConfig {
@Bean()
SearchAnalyzer french() => const SearchAnalyzer(
'french',
filters: [
WordPartsFilter(),
LowerCaseFilter(),
AsciiFoldingFilter(),
StopWordFilter({'le', 'la', 'les', 'de', 'des', 'un', 'une', 'et'}),
],
);
}
spring:
search:
analyzers:
fr: french
ja: standard
A filter of your own implements SearchTokenFilter (Iterable<SearchToken> filter(Iterable<SearchToken> tokens)), a tokenizer SearchTokenizer. A SearchToken keeps its term (what is compared), the word before a stemmer shortened it (for a word still being typed), and its place in the text (start, end, for highlights).
The language of a search picks its analyzer: the whole tag (pt-BR) when named, else the part before - or _ (pt), else english for en and standard for the rest.
Rankers
A SearchRanker changes the score of a hit after the index ranked it. The rankers run in @Order, each on the score the one before gave, with the query searched (SearchQuery: its clauses, each a word or a phrase as written); the hits are sorted again after them.
// The reference pages before the guides.
@Component()
class ReferenceFirst implements SearchRanker {
@override
double rank(SearchHit hit, double score, SearchQuery query) =>
(hit.document.attributes['path'] ?? '').contains('/reference/')
? score * 1.5
: score;
}
The property
| Key | Type | Default | Meaning |
|---|---|---|---|
spring.search.analyzers.<language> |
analyzer name | english for en, standard otherwise |
The analyzer of a language |