Scaling the isolates

The number of isolates that serve the port can change while the server runs. The application changes it through the IsolateScaler bean; with server.isolates.scaling: auto a policy changes it too, between min and max.

This page builds on A pool of isolates. Everything said there about state per isolate holds for an isolate added later.

Why

Load is not the same all day. A shop needs eight isolates for the hour of a sale and two for the night. Each isolate keeps its own beans in memory, so isolates nobody needs are memory nobody uses.

Adding an isolate takes as long as the application takes to start, and removing one must not cut the requests it is answering. The pool does both: a new isolate serves only when it is ready, and one that leaves finishes its requests first.

Example

# lib/resources/application.yml
server:
  isolates:
    count: 2
    min: 2
    max: 8
import 'package:spring_boot_starter_web/spring_boot_starter_web.dart';

@RestController()
class Operations {
  Operations(this.isolates);
  final IsolateScaler isolates;

  // Before the sale starts. Protect this path: see below.
  @PostMapping('/internal/scale')
  Future<int> scale(@RequestParam() int to) => isolates.scaleTo(to);
}
$ curl -X POST 'localhost:8080/internal/scale?to=8'
8
INFO  Scaling 2 → 8 isolates (manual, asked in isolate-1)
INFO  Isolate isolate-2 ready in 0.398 seconds
...

The request may be taken by any isolate. The pool lives in the first one, and a step asked elsewhere is done there; the log says where it was asked.

IsolateScaler

The bean isolateScaler is there when the server has a pool: server.isolates.count above one (or auto), server.isolates.scaling: auto, or server.isolates.max above one. Then it is in every isolate of the pool: the pool itself in the first, and in the others a bean that hands the step to it.

Member Meaning
count The isolates serving now, the first one included. One that is being drained no longer counts
min The fewest scaleTo goes down to (server.isolates.min, at least one)
max The most it goes up to (server.isolates.max)
Future<int> scaleTo(int count, {String reason}) Brings the pool to count isolates and completes with how many serve
Future<int> scaleBy(int delta, {String reason}) Adds delta isolates, or removes them when it is negative

Rules:

In every isolate

A request reaches any isolate, so the scaler works in every one of them. Asked in another isolate than the first, scaleTo is sent to the pool in the first isolate (through PrimaryIsolate, handler isolates.scale) and completes with its answer.

@PostMapping('/internal/scale')
ResponseEntity<void> scale(@RequestParam() int to) {
  unawaited(isolates.scaleTo(to)); // dart:async
  return ResponseEntity.accepted().build();
}

No pool, no scaler

With count: 1, scaling: manual and no max above one, the port is not shared and nothing can be added later: there is no isolateScaler bean, and a bean that requires one stops the start. An application that also runs with one isolate (in development, in tests) takes it as optional:

Operations(this.isolates);
final IsolateScaler? isolates; // null without a pool

To start with one isolate and scale by hand, set max:

server:
  isolates:
    count: 1
    max: 4

Adding and removing

Adding starts new isolates, all of a step at once. Each runs the whole application and joins the port when its own beans are up, so requests are never sent to one that is not ready. A free number is taken: after isolate-2 has left, the next isolate added is isolate-2 again.

Removing takes the newest isolates, those with the highest numbers. Each one is drained:

  1. It stops taking connections. The others take the new ones.
  2. It finishes the requests it is answering, as a stopping application does (see Shutdown).
  3. Its event streams and WebSocket sessions are closed. Clients connect again and reach another isolate.
  4. Its @PreDestroy methods run, and it ends: Isolate isolate-3 stopped.

An isolate that has not ended after server.isolates.drain-timeout (30 seconds) is stopped anyway, with a warning:

Isolate isolate-3 did not stop within 30 seconds (server.isolates.drain-timeout): stopped anyway

What the isolate kept in memory goes with it: its in-memory sessions, its cached values.

Automatic scaling

server:
  isolates:
    count: 2
    scaling: auto
    min: 2
    max: 8
INFO  Scaling the isolates automatically between 2 and 8 (LagScalingPolicy)
...
INFO  Scaling 2 → 3 isolates (auto: lag 14.2 ms, 180 in flight)

Every two seconds the pool asks its isolates how they are doing and shows the answer to the scaling policy. The policy says how many isolates it wants, or nothing. The pool keeps the answer between min and max, and after any scaling step, by the policy or by hand, it waits server.isolates.cooldown before it asks the policy again.

The default policy: by lag

LagScalingPolicy decides by how late the isolates are. An isolate does one thing at a time. While it keeps up, its timers fire on time; when requests arrive faster than it answers, they queue, and its event loop runs late. That lateness is the lag, and the policy reads its average over the isolates.

Step When Result
Up The average lag stays at scale-up-lag (5 ms) or more for scale-up-after (5 seconds). One look under it starts the wait again Half as many isolates again: 1 → 2 → 3 → 5 → 8
Down The average lag stays under a quarter of scale-up-lag for scale-down-after (2 minutes), the pool is above min, and the requests would fit in one isolate less One isolate less
Neither The lag is between a quarter of scale-up-lag and scale-up-lag The pool stays as it is

"Would fit" is decided from what the policy has seen:

What the policy cannot do

A policy of your own

An IsolateScalingPolicy bean replaces LagScalingPolicy:

@Component()
class InFlightPolicy implements IsolateScalingPolicy {
  @override
  int? decide(IsolateScalingState state, Duration now) {
    final each = state.activeRequests / state.count;
    if (each > 200) return state.count + 1;
    if (each < 20 && state.count > state.min) return state.count - 1;
    return null; // leave the pool as it is
  }
}

decide is called every two seconds outside the cooldown, in the first isolate, once a lag has been measured. now is a time that only goes forward, for "how long has this lasted"; it is not the time of day.

Field of IsolateScalingState Meaning
count The isolates serving
min, max The fewest and the most allowed
lag How late the isolates' event loops run, on average over them
maxLag The longest lag any isolate measured
activeRequests The requests in flight, all isolates together
requestsPerSecond The requests taken per second since the last look, all isolates together; 0 at the first look

The properties

Key Type Default Meaning
server.isolates.scaling manual, auto manual Who changes the number of isolates while the server runs: only the application, through IsolateScaler, or the scaling policy too
server.isolates.min int 1 The fewest isolates scaling goes down to
server.isolates.max int not set: the processor cores, or count when that is more The most isolates scaling goes up to
server.isolates.scale-up-lag Duration 5ms How late the isolates' event loops run on average when the default policy calls them overloaded: requests wait about that long before they are even looked at
server.isolates.scale-up-after Duration 5s How long the isolates must be late before the default policy adds some
server.isolates.scale-down-after Duration 2m How long the isolates must be free before the default policy removes one
server.isolates.cooldown Duration 15s The pause after a scaling step before the policy is asked again
server.isolates.drain-timeout Duration 30s How long an isolate that leaves may finish its requests before it is stopped anyway

server.isolates.count is on A pool of isolates.

In Spring (Java)

Spring has no counterpart. Nearest is changing a thread pool's size over JMX, which adds threads that share the beans; here a step adds or removes a whole copy of the application.

See also