Scaling the isolates
The number of isolates that serve the port can change while the server runs. The application changes it through the IsolateScaler bean; with server.isolates.scaling: auto a policy changes it too, between min and max.
This page builds on A pool of isolates. Everything said there about state per isolate holds for an isolate added later.
Why
Load is not the same all day. A shop needs eight isolates for the hour of a sale and two for the night. Each isolate keeps its own beans in memory, so isolates nobody needs are memory nobody uses.
Adding an isolate takes as long as the application takes to start, and removing one must not cut the requests it is answering. The pool does both: a new isolate serves only when it is ready, and one that leaves finishes its requests first.
Example
# lib/resources/application.yml
server:
isolates:
count: 2
min: 2
max: 8
import 'package:spring_boot_starter_web/spring_boot_starter_web.dart';
@RestController()
class Operations {
Operations(this.isolates);
final IsolateScaler isolates;
// Before the sale starts. Protect this path: see below.
@PostMapping('/internal/scale')
Future<int> scale(@RequestParam() int to) => isolates.scaleTo(to);
}
$ curl -X POST 'localhost:8080/internal/scale?to=8'
8
INFO Scaling 2 → 8 isolates (manual, asked in isolate-1)
INFO Isolate isolate-2 ready in 0.398 seconds
...
The request may be taken by any isolate. The pool lives in the first one, and a step asked elsewhere is done there; the log says where it was asked.
IsolateScaler
The bean isolateScaler is there when the server has a pool: server.isolates.count above one (or auto), server.isolates.scaling: auto, or server.isolates.max above one. Then it is in every isolate of the pool: the pool itself in the first, and in the others a bean that hands the step to it.
| Member | Meaning |
|---|---|
count |
The isolates serving now, the first one included. One that is being drained no longer counts |
min |
The fewest scaleTo goes down to (server.isolates.min, at least one) |
max |
The most it goes up to (server.isolates.max) |
Future<int> scaleTo(int count, {String reason}) |
Brings the pool to count isolates and completes with how many serve |
Future<int> scaleBy(int delta, {String reason}) |
Adds delta isolates, or removes them when it is negative |
Rules:
- The number is kept between
minandmax.scaleTo(0)goes tomin,scaleTo(100)tomax; neither is an error. Read the answer for what was done. - The first isolate is never removed.
reasonis for the log:Scaling 2 → 4 isolates (manual)without one. A step asked in another isolate than the first adds where:(manual, asked in isolate-2).- Steps run one after the other. A
scaleTocalled while another is under way waits for it. - The future completes when the step is done: the new isolates are ready, and the removed ones have ended (their requests finished, or
server.isolates.drain-timeoutpassed). - An isolate that fails to start is left out. The answer is then less than what was asked for, and the failure is in the log.
- Before the server listens and after the context has closed,
scaleTochanges nothing and answers the current count. - To scale past one isolate the application must be startable again; otherwise
scaleTofails.
In every isolate
A request reaches any isolate, so the scaler works in every one of them. Asked in another isolate than the first, scaleTo is sent to the pool in the first isolate (through PrimaryIsolate, handler isolates.scale) and completes with its answer.
- In the other isolates
count,minandmaxare what the first isolate last told: it tells every isolate after each step, and when an isolate joins or ends. - Protect an endpoint that scales. Whoever may call it decides how much memory and how many cores the server takes, and can drain its isolates. Keep it off the public paths and behind a rule of the security starter.
- Do not await a step down in a request handler. A step down removes the newest isolates, and the request may have been taken by one of them. That isolate finishes its requests before it ends, the step completes when it has ended, and the request waits for the step: neither moves until a timeout. Start the step and answer at once:
@PostMapping('/internal/scale')
ResponseEntity<void> scale(@RequestParam() int to) {
unawaited(isolates.scaleTo(to)); // dart:async
return ResponseEntity.accepted().build();
}
No pool, no scaler
With count: 1, scaling: manual and no max above one, the port is not shared and nothing can be added later: there is no isolateScaler bean, and a bean that requires one stops the start. An application that also runs with one isolate (in development, in tests) takes it as optional:
Operations(this.isolates);
final IsolateScaler? isolates; // null without a pool
To start with one isolate and scale by hand, set max:
server:
isolates:
count: 1
max: 4
Adding and removing
Adding starts new isolates, all of a step at once. Each runs the whole application and joins the port when its own beans are up, so requests are never sent to one that is not ready. A free number is taken: after isolate-2 has left, the next isolate added is isolate-2 again.
Removing takes the newest isolates, those with the highest numbers. Each one is drained:
- It stops taking connections. The others take the new ones.
- It finishes the requests it is answering, as a stopping application does (see Shutdown).
- Its event streams and WebSocket sessions are closed. Clients connect again and reach another isolate.
- Its
@PreDestroymethods run, and it ends:Isolate isolate-3 stopped.
An isolate that has not ended after server.isolates.drain-timeout (30 seconds) is stopped anyway, with a warning:
Isolate isolate-3 did not stop within 30 seconds (server.isolates.drain-timeout): stopped anyway
What the isolate kept in memory goes with it: its in-memory sessions, its cached values.
Automatic scaling
server:
isolates:
count: 2
scaling: auto
min: 2
max: 8
INFO Scaling the isolates automatically between 2 and 8 (LagScalingPolicy)
...
INFO Scaling 2 → 3 isolates (auto: lag 14.2 ms, 180 in flight)
Every two seconds the pool asks its isolates how they are doing and shows the answer to the scaling policy. The policy says how many isolates it wants, or nothing. The pool keeps the answer between min and max, and after any scaling step, by the policy or by hand, it waits server.isolates.cooldown before it asks the policy again.
countis where the pool starts. The policy may go below it, down tomin: setminto what must always serve.maxnot set is the number of processor cores, orcountwhen that is more.IsolateScalerworks as before; the application and the policy share the pool.- A look that fails is logged (
The scaling of the isolates failed) and the next one is tried.
The default policy: by lag
LagScalingPolicy decides by how late the isolates are. An isolate does one thing at a time. While it keeps up, its timers fire on time; when requests arrive faster than it answers, they queue, and its event loop runs late. That lateness is the lag, and the policy reads its average over the isolates.
| Step | When | Result |
|---|---|---|
| Up | The average lag stays at scale-up-lag (5 ms) or more for scale-up-after (5 seconds). One look under it starts the wait again |
Half as many isolates again: 1 → 2 → 3 → 5 → 8 |
| Down | The average lag stays under a quarter of scale-up-lag for scale-down-after (2 minutes), the pool is above min, and the requests would fit in one isolate less |
One isolate less |
| Neither | The lag is between a quarter of scale-up-lag and scale-up-lag |
The pool stays as it is |
"Would fit" is decided from what the policy has seen:
- At each step up it remembers the requests per second each isolate took at that moment, as what one isolate can take (the average of that and what it remembered before).
- Once it knows that, the requests fit when the requests per second now are under 60 % of what one isolate less could take.
- Before the first step up it does not know, and the requests fit when fewer are in flight than there are isolates.
What the policy cannot do
- It scales up late. Lag shows overload when requests already wait, not how close an isolate is to it: a compiled program cannot ask how much processor time its isolates use. The first seconds of a sudden peak are slow: five seconds of lag, then the start of the new isolates.
- So keep
minhigh enough for the load you expect, and scale by hand before a peak you know of. - It reads the average. One isolate that runs late among free ones may not reach the threshold.
maxLag, the longest lag any isolate measured, is there for a policy of your own. - It scales for work, not for waiting. A server whose requests wait for a database has little lag, and gets no isolate from this policy.
A policy of your own
An IsolateScalingPolicy bean replaces LagScalingPolicy:
@Component()
class InFlightPolicy implements IsolateScalingPolicy {
@override
int? decide(IsolateScalingState state, Duration now) {
final each = state.activeRequests / state.count;
if (each > 200) return state.count + 1;
if (each < 20 && state.count > state.min) return state.count - 1;
return null; // leave the pool as it is
}
}
decide is called every two seconds outside the cooldown, in the first isolate, once a lag has been measured. now is a time that only goes forward, for "how long has this lasted"; it is not the time of day.
Field of IsolateScalingState |
Meaning |
|---|---|
count |
The isolates serving |
min, max |
The fewest and the most allowed |
lag |
How late the isolates' event loops run, on average over them |
maxLag |
The longest lag any isolate measured |
activeRequests |
The requests in flight, all isolates together |
requestsPerSecond |
The requests taken per second since the last look, all isolates together; 0 at the first look |
- The answer is kept between
minandmaxby the pool, and the cooldown still applies. scale-up-lag,scale-up-afterandscale-down-afterare the default policy's; a policy of your own reads its own settings.- The step is logged with the lag and the requests in flight, whatever the policy decided by.
- The bean exists in every isolate; only the first isolate's is asked.
The properties
| Key | Type | Default | Meaning |
|---|---|---|---|
server.isolates.scaling |
manual, auto |
manual |
Who changes the number of isolates while the server runs: only the application, through IsolateScaler, or the scaling policy too |
server.isolates.min |
int |
1 |
The fewest isolates scaling goes down to |
server.isolates.max |
int |
not set: the processor cores, or count when that is more |
The most isolates scaling goes up to |
server.isolates.scale-up-lag |
Duration |
5ms |
How late the isolates' event loops run on average when the default policy calls them overloaded: requests wait about that long before they are even looked at |
server.isolates.scale-up-after |
Duration |
5s |
How long the isolates must be late before the default policy adds some |
server.isolates.scale-down-after |
Duration |
2m |
How long the isolates must be free before the default policy removes one |
server.isolates.cooldown |
Duration |
15s |
The pause after a scaling step before the policy is asked again |
server.isolates.drain-timeout |
Duration |
30s |
How long an isolate that leaves may finish its requests before it is stopped anyway |
server.isolates.count is on A pool of isolates.
In Spring (Java)
Spring has no counterpart. Nearest is changing a thread pool's size over JMX, which adds threads that share the beans; here a step adds or removes a whole copy of the application.