
# Analyzers and rankers

Everything in the search is a bean, asked in `@Order`: `SearchDocumentSource` beans give documents, `SearchAnalyzer` beans cut the text of a language into terms, and `SearchRanker` beans change the score of each hit. An app adds or replaces any of them without touching the service.

## Why

Languages differ: English has word endings and stop words, Japanese has no spaces, German has compounds. One analysis for all would miss matches in some and invent them in others. And what ranks first is often a business rule (the reference before the guides, a sold-out product last), not a matter of words.

## Analyzers

A `SearchAnalyzer` is a tokenizer, then filters in order. The documents and the queries go through the same analyzer, so they meet.

| Analyzer | Tokenizer and filters | Used for |
|---|---|---|
| `standard` | `StandardTokenizer`, `WordPartsFilter`, `LowerCaseFilter`, `AsciiFoldingFilter` | every language but English, unless named |
| `english` | `standard`, then `StopWordFilter(StopWordFilter.english)` and `EnglishStemFilter` | `en` |

| Part | What it does |
|---|---|
| `StandardTokenizer` | Words of letters and digits in any script; Chinese, Japanese and Korean, which write no spaces, as every two characters next to each other (`依存性` gives `依存`, `存性`), so a query cut the same way finds them without a dictionary |
| `WordPartsFilter` | Also the parts of a word in camel case or with digits: `ApplicationContext` is found by `context` too; the whole word stays |
| `LowerCaseFilter` | `Bean` finds `bean` |
| `AsciiFoldingFilter` | Latin letters without their accents: `café` finds `cafe` |
| `StopWordFilter` | Leaves out words that are in every document (`the`, `a`) |
| `EnglishStemFilter` | English endings removed: `beans` and `bean`, `binding` and `bind`. Light on purpose: a word under four letters stays, and an ending goes only when a vowel is left before it (`string` stays) |

### An analyzer of your own

Declare it as a bean and name it per language. A bean with a built-in name (`standard`, `english`) replaces it.

```dart
// Lower case, no accents, and the app's own stop words.
@Configuration()
class SearchConfig {
  @Bean()
  SearchAnalyzer french() => const SearchAnalyzer(
    'french',
    filters: [
      WordPartsFilter(),
      LowerCaseFilter(),
      AsciiFoldingFilter(),
      StopWordFilter({'le', 'la', 'les', 'de', 'des', 'un', 'une', 'et'}),
    ],
  );
}
```

```yaml
spring:
  search:
    analyzers:
      fr: french
      ja: standard
```

A filter of your own implements `SearchTokenFilter` (`Iterable<SearchToken> filter(Iterable<SearchToken> tokens)`), a tokenizer `SearchTokenizer`. A `SearchToken` keeps its `term` (what is compared), the word before a stemmer shortened it (for a word still being typed), and its place in the text (`start`, `end`, for highlights).

The language of a search picks its analyzer: the whole tag (`pt-BR`) when named, else the part before `-` or `_` (`pt`), else `english` for `en` and `standard` for the rest.

## Rankers

A `SearchRanker` changes the score of a hit after the index ranked it. The rankers run in `@Order`, each on the score the one before gave, with the query searched (`SearchQuery`: its `clauses`, each a word or a phrase as written); the hits are sorted again after them.

```dart
// The reference pages before the guides.
@Component()
class ReferenceFirst implements SearchRanker {
  @override
  double rank(SearchHit hit, double score, SearchQuery query) =>
      (hit.document.attributes['path'] ?? '').contains('/reference/')
          ? score * 1.5
          : score;
}
```

## The property

| Key | Type | Default | Meaning |
|---|---|---|---|
| `spring.search.analyzers.<language>` | analyzer name | `english` for `en`, `standard` otherwise | The analyzer of a language |

## See also

- [Documents](documents.md)
- [Searching](searching.md)
- [All properties](properties.md)
