
# Scaling the isolates

The number of isolates that serve the port can change while the server runs. The application changes it through the `IsolateScaler` bean; with `server.isolates.scaling: auto` a policy changes it too, between `min` and `max`.

This page builds on [A pool of isolates](isolates.md). Everything said there about state per isolate holds for an isolate added later.

## Why

Load is not the same all day. A shop needs eight isolates for the hour of a sale and two for the night. Each isolate keeps its own beans in memory, so isolates nobody needs are memory nobody uses.

Adding an isolate takes as long as the application takes to start, and removing one must not cut the requests it is answering. The pool does both: a new isolate serves only when it is ready, and one that leaves finishes its requests first.

## Example

```yaml
# lib/resources/application.yml
server:
  isolates:
    count: 2
    min: 2
    max: 8
```

```dart
import 'package:spring_boot_starter_web/spring_boot_starter_web.dart';

@RestController()
class Operations {
  Operations(this.isolates);
  final IsolateScaler isolates;

  // Before the sale starts. Protect this path: see below.
  @PostMapping('/internal/scale')
  Future<int> scale(@RequestParam() int to) => isolates.scaleTo(to);
}
```

```console
$ curl -X POST 'localhost:8080/internal/scale?to=8'
8
```

```text
INFO  Scaling 2 → 8 isolates (manual, asked in isolate-1)
INFO  Isolate isolate-2 ready in 0.398 seconds
...
```

The request may be taken by any isolate. The pool lives in the first one, and a step asked elsewhere is done there; the log says where it was asked.

## IsolateScaler

The bean `isolateScaler` is there when the server has a pool: `server.isolates.count` above one (or `auto`), `server.isolates.scaling: auto`, or `server.isolates.max` above one. Then it is in every isolate of the pool: the pool itself in the first, and in the others a bean that hands the step to it.

| Member | Meaning |
|---|---|
| `count` | The isolates serving now, the first one included. One that is being drained no longer counts |
| `min` | The fewest `scaleTo` goes down to (`server.isolates.min`, at least one) |
| `max` | The most it goes up to (`server.isolates.max`) |
| `Future<int> scaleTo(int count, {String reason})` | Brings the pool to `count` isolates and completes with how many serve |
| `Future<int> scaleBy(int delta, {String reason})` | Adds `delta` isolates, or removes them when it is negative |

Rules:

- **The number is kept between `min` and `max`.** `scaleTo(0)` goes to `min`, `scaleTo(100)` to `max`; neither is an error. Read the answer for what was done.
- **The first isolate is never removed.**
- **`reason` is for the log**: `Scaling 2 → 4 isolates (manual)` without one. A step asked in another isolate than the first adds where: `(manual, asked in isolate-2)`.
- **Steps run one after the other.** A `scaleTo` called while another is under way waits for it.
- **The future completes when the step is done**: the new isolates are ready, and the removed ones have ended (their requests finished, or `server.isolates.drain-timeout` passed).
- **An isolate that fails to start is left out.** The answer is then less than what was asked for, and the failure is in the log.
- Before the server listens and after the context has closed, `scaleTo` changes nothing and answers the current count.
- To scale past one isolate the application must be [startable again](isolates.md#the-application-must-be-startable-again); otherwise `scaleTo` fails.

### In every isolate

A request reaches any isolate, so the scaler works in every one of them. Asked in another isolate than the first, `scaleTo` is sent to the pool in the first isolate (through [`PrimaryIsolate`](../../spring-core-dart/isolates/primary-isolate.md), handler `isolates.scale`) and completes with its answer.

- In the other isolates `count`, `min` and `max` are what the first isolate last told: it tells every isolate after each step, and when an isolate joins or ends.
- **Protect an endpoint that scales.** Whoever may call it decides how much memory and how many cores the server takes, and can drain its isolates. Keep it off the public paths and behind a rule of the security starter.
- **Do not await a step down in a request handler.** A step down removes the newest isolates, and the request may have been taken by one of them. That isolate finishes its requests before it ends, the step completes when it has ended, and the request waits for the step: neither moves until a timeout. Start the step and answer at once:

```dart
@PostMapping('/internal/scale')
ResponseEntity<void> scale(@RequestParam() int to) {
  unawaited(isolates.scaleTo(to)); // dart:async
  return ResponseEntity.accepted().build();
}
```

### No pool, no scaler

With `count: 1`, `scaling: manual` and no `max` above one, the port is not shared and nothing can be added later: there is no `isolateScaler` bean, and a bean that requires one stops the start. An application that also runs with one isolate (in development, in tests) takes it as optional:

```dart
Operations(this.isolates);
final IsolateScaler? isolates; // null without a pool
```

To start with one isolate and scale by hand, set `max`:

```yaml
server:
  isolates:
    count: 1
    max: 4
```

## Adding and removing

**Adding** starts new isolates, all of a step at once. Each runs the whole application and joins the port when its own beans are up, so requests are never sent to one that is not ready. A free number is taken: after `isolate-2` has left, the next isolate added is `isolate-2` again.

**Removing** takes the newest isolates, those with the highest numbers. Each one is drained:

1. It stops taking connections. The others take the new ones.
2. It finishes the requests it is answering, as a stopping application does (see [Shutdown](shutdown.md)).
3. Its event streams and WebSocket sessions are closed. Clients connect again and reach another isolate.
4. Its `@PreDestroy` methods run, and it ends: `Isolate isolate-3 stopped`.

An isolate that has not ended after `server.isolates.drain-timeout` (30 seconds) is stopped anyway, with a warning:

```text
Isolate isolate-3 did not stop within 30 seconds (server.isolates.drain-timeout): stopped anyway
```

What the isolate kept in memory goes with it: its in-memory sessions, its cached values.

## Automatic scaling

```yaml
server:
  isolates:
    count: 2
    scaling: auto
    min: 2
    max: 8
```

```text
INFO  Scaling the isolates automatically between 2 and 8 (LagScalingPolicy)
...
INFO  Scaling 2 → 3 isolates (auto: lag 14.2 ms, 180 in flight)
```

Every two seconds the pool asks its isolates how they are doing and shows the answer to the scaling policy. The policy says how many isolates it wants, or nothing. The pool keeps the answer between `min` and `max`, and after any scaling step, by the policy or by hand, it waits `server.isolates.cooldown` before it asks the policy again.

- `count` is where the pool starts. The policy may go below it, down to `min`: set `min` to what must always serve.
- `max` not set is the number of processor cores, or `count` when that is more.
- `IsolateScaler` works as before; the application and the policy share the pool.
- A look that fails is logged (`The scaling of the isolates failed`) and the next one is tried.

### The default policy: by lag

`LagScalingPolicy` decides by how late the isolates are. An isolate does one thing at a time. While it keeps up, its timers fire on time; when requests arrive faster than it answers, they queue, and its event loop runs late. That lateness is the [lag](../../spring-core-dart/isolates/load.md#how-lag-and-busy-are-measured), and the policy reads its average over the isolates.

| Step | When | Result |
|---|---|---|
| Up | The average lag stays at `scale-up-lag` (5 ms) or more for `scale-up-after` (5 seconds). One look under it starts the wait again | Half as many isolates again: 1 → 2 → 3 → 5 → 8 |
| Down | The average lag stays under a quarter of `scale-up-lag` for `scale-down-after` (2 minutes), the pool is above `min`, and the requests would fit in one isolate less | One isolate less |
| Neither | The lag is between a quarter of `scale-up-lag` and `scale-up-lag` | The pool stays as it is |

"Would fit" is decided from what the policy has seen:

- At each step up it remembers the requests per second each isolate took at that moment, as what one isolate can take (the average of that and what it remembered before).
- Once it knows that, the requests fit when the requests per second now are under 60 % of what one isolate less could take.
- Before the first step up it does not know, and the requests fit when fewer are in flight than there are isolates.

### What the policy cannot do

- **It scales up late.** Lag shows overload when requests already wait, not how close an isolate is to it: a compiled program cannot ask how much processor time its isolates use. The first seconds of a sudden peak are slow: five seconds of lag, then the start of the new isolates.
- **So keep `min` high enough for the load you expect**, and scale by hand before a peak you know of.
- **It reads the average.** One isolate that runs late among free ones may not reach the threshold. `maxLag`, the longest lag any isolate measured, is there for a policy of your own.
- **It scales for work, not for waiting.** A server whose requests wait for a database has little lag, and gets no isolate from this policy.

## A policy of your own

An `IsolateScalingPolicy` bean replaces `LagScalingPolicy`:

```dart
@Component()
class InFlightPolicy implements IsolateScalingPolicy {
  @override
  int? decide(IsolateScalingState state, Duration now) {
    final each = state.activeRequests / state.count;
    if (each > 200) return state.count + 1;
    if (each < 20 && state.count > state.min) return state.count - 1;
    return null; // leave the pool as it is
  }
}
```

`decide` is called every two seconds outside the cooldown, in the first isolate, once a lag has been measured. `now` is a time that only goes forward, for "how long has this lasted"; it is not the time of day.

| Field of `IsolateScalingState` | Meaning |
|---|---|
| `count` | The isolates serving |
| `min`, `max` | The fewest and the most allowed |
| `lag` | How late the isolates' event loops run, on average over them |
| `maxLag` | The longest lag any isolate measured |
| `activeRequests` | The requests in flight, all isolates together |
| `requestsPerSecond` | The requests taken per second since the last look, all isolates together; `0` at the first look |

- The answer is kept between `min` and `max` by the pool, and the cooldown still applies.
- `scale-up-lag`, `scale-up-after` and `scale-down-after` are the default policy's; a policy of your own reads its own settings.
- The step is logged with the lag and the requests in flight, whatever the policy decided by.
- The bean exists in every isolate; only the first isolate's is asked.

## The properties

| Key | Type | Default | Meaning |
|---|---|---|---|
| `server.isolates.scaling` | `manual`, `auto` | `manual` | Who changes the number of isolates while the server runs: only the application, through `IsolateScaler`, or the scaling policy too |
| `server.isolates.min` | `int` | `1` | The fewest isolates scaling goes down to |
| `server.isolates.max` | `int` | not set: the processor cores, or `count` when that is more | The most isolates scaling goes up to |
| `server.isolates.scale-up-lag` | `Duration` | `5ms` | How late the isolates' event loops run on average when the default policy calls them overloaded: requests wait about that long before they are even looked at |
| `server.isolates.scale-up-after` | `Duration` | `5s` | How long the isolates must be late before the default policy adds some |
| `server.isolates.scale-down-after` | `Duration` | `2m` | How long the isolates must be free before the default policy removes one |
| `server.isolates.cooldown` | `Duration` | `15s` | The pause after a scaling step before the policy is asked again |
| `server.isolates.drain-timeout` | `Duration` | `30s` | How long an isolate that leaves may finish its requests before it is stopped anyway |

`server.isolates.count` is on [A pool of isolates](isolates.md#the-properties).

## In Spring (Java)

Spring has no counterpart. Nearest is changing a thread pool's size over JMX, which adds threads that share the beans; here a step adds or removes a whole copy of the application.

## See also

- [A pool of isolates](isolates.md)
- [Overload](overload.md)
- [Requests per second](qps.md)
- [The load of the isolates](../../spring-core-dart/isolates/load.md)
- [Asking the first isolate](../../spring-core-dart/isolates/primary-isolate.md)
- [The `isolates` endpoint](../../spring-boot-actuator/isolates.md)
- [Shutdown](shutdown.md)
- [@Scheduled](../../spring-boot-starter-scheduling/scheduled.md)
- [Load testing](../../../docs/load-testing.md)
- [Durations](../../spring-core-dart/conversion/durations.md)
