A pool of isolates
By default the server answers from one isolate, the application's own, and so uses one processor core. server.isolates.count adds more: each isolate runs the whole application, and all of them serve the same port.
It changes more than the speed. Read What changes with more than one isolate before you turn it on.
Why
An isolate does one thing at a time and uses one processor core. A server whose handlers compute (large JSON, hashing, pictures) reaches the limit of that core while the others are idle. More isolates use them.
Tomcat does this with threads that share the beans. Dart's isolates share nothing, so here each isolate has the application for itself. That keeps code free of locks, and it means that whatever a bean keeps in memory is kept once per isolate.
It does not help a server that mostly waits: for a database, for another service. One isolate can wait for many answers at once; what it cannot do is compute two things at once. Measure first: see Load testing.
Example
# lib/resources/application.yml
server:
isolates:
count: 4 # or auto: one per processor core
// bin/main.dart
import 'package:spring_boot_starter_web/spring_boot_starter_web.dart';
import 'spring_gen/main.spring.dart';
@SpringBootApplication()
void main(List<String> args) => runApplication(args);
Nothing else changes in the code. The first isolate logs:
INFO Web server started on port 8080 (http)
INFO Scaling 1 → 4 isolates (server.isolates.count=4)
WARN Each isolate keeps its own sessions in memory, and a request may reach any isolate: what one keeps the others do not see. Keep it where all of them do (a database, a shared store), or accept one copy per isolate.
INFO Isolate isolate-1 ready in 0.412 seconds
INFO Isolate isolate-2 ready in 0.415 seconds
INFO Isolate isolate-3 ready in 0.431 seconds
INFO Started neon-books in 0.52 seconds
A controller can tell which isolate answers:
@RestController()
class WhoController {
WhoController(this.isolate);
final IsolateInfo isolate;
@GetMapping('/who')
String who() => isolate.name; // main, isolate-1, isolate-2 or isolate-3
}
What changes with more than one isolate
| What | With more than one isolate | What to do |
|---|---|---|
| The beans | Every isolate runs the whole application: each bean exists once per isolate, and what the beans keep in memory is kept that many times | Count what the application keeps in memory times the isolates |
| A request | May reach any isolate. A connection goes to whichever isolate takes it first, and the requests of a kept-alive connection stay with that isolate. Two requests of one browser can be answered by two isolates | Do not expect the next request to find what this one left in memory |
| In-memory sessions | Each isolate has its own. A session made by one is unknown to the next: the user is logged out at random. server.session.max-sessions counts per isolate |
Declare an HttpSessionRepository that keeps them where all isolates see them. See Sessions |
Caches (@Cacheable) |
Each isolate has its own cached values: a value is computed once per isolate, and @CacheEvict evicts in the isolate that runs it only |
Accept it where a stale value for a while is fine, or use a cache outside the isolates. See The in-memory caches |
| Counters, rate limiters, any field of a bean | Count what their own isolate saw. A limit of 100 a minute kept in memory lets 100 through per isolate | Keep them in a shared store, or divide the limit by the isolates |
WebSocketSessionRegistry.broadcast |
Reaches the clients connected to the isolate that sends, not the others' | Send through something every isolate listens to (a message broker, a database notification) |
| Server-sent events | An SseEmitter lives in the isolate that took the request. A bean that sends to "all" its emitters sends to those of its isolate |
The same |
@Scheduled methods |
Run in the first isolate only, so a job runs once for the application | spring.task.scheduling.isolates: all for what each isolate must do for itself. See @Scheduled |
ApplicationRunner and CommandLineRunner beans, @PostConstruct, @EventListener for the context's events |
Run once in every isolate: a migration, a warm-up, an e-mail "the server started" happens four times | @ConditionalOnPrimaryIsolate() on the bean, or if (isolate.primary) |
The banner and Started … in |
Logged once, by the first isolate; Started … in when the others are ready |
Nothing |
Signals (Ctrl+C, SIGTERM) |
With spring.main.register-shutdown-hook: true the first isolate takes them and stops the others with its context. The others register no hook |
Nothing |
| State all isolates must see | Is the application's job: a database, a shared store. The framework does not synchronise beans. For what must exist once inside the application it has one building block: a bean in the first isolate that the others ask (PrimaryIsolate) |
Design for it before you raise count |
| A database connection pool | Is a bean: one pool per isolate. spring.datasource.pool.max-size: 10 with four isolates is up to 40 connections at the database |
Divide the pool's size, or raise the database's limit |
server.max-connections, server.max-concurrent-requests |
Are per isolate: the server as a whole takes that many times the isolates. See Overload | Set them per isolate |
The isolateTaskExecutor |
Each isolate has its own pool of task isolates (isolate-2-task-1) |
Count core-size times the isolates |
| The actuator's endpoints | isolates and qps ask the first isolate, so every isolate answers for the whole application. The others, health among them, answer for the isolate that takes the request. See The isolates endpoint |
Nothing for those two; do not read one isolate's health as the application's |
| An isolate that ends unexpectedly | Is started again. The requests it was answering get no answer: their connections stay open until the VM collects the isolate, or the client gives up | Give clients a timeout; look for ended unexpectedly in the logs |
| A Flutter app | Has no pool: the setting is ignored with a warning | Nothing; see In a Flutter app |
| Hot reload | When a reload restarts the context, the pool stops with it and the new context starts the isolates again | Nothing |
The pool names what it can see. A bean that implements IsolateLocalState is listed in the one warning the pool logs when it first adds an isolate. The in-memory session repository (sessions), the cache starter's managers, the default in-memory one and SimpleCacheManager (cached values), and the WebSocketSessionRegistry (WebSocket sessions) do. The pool finds them among the beans that exist when it grows, whatever type they were declared with. A bean of your own can be listed too:
@Service()
class LoginAttempts implements IsolateLocalState {
final _failed = <String, int>{};
@override
String get isolateLocalState => 'failed logins';
}
WARN Each isolate keeps its own sessions, failed logins in memory, and a request may reach any isolate: what one keeps the others do not see. Keep it where all of them do (a database, a shared store), or accept one copy per isolate.
The warning is not an error. A cache can often live with one copy per isolate; sessions usually cannot.
Once for the application
@ConditionalOnPrimaryIsolate() registers a bean in the first isolate only:
@Component()
@ConditionalOnPrimaryIsolate()
class Migrations implements ApplicationRunner {
Migrations(this.db);
final Database db;
// Once for the application, not once per isolate.
@override
Future<void> run(ApplicationArguments args) => db.migrate();
}
IsolateInfo (the bean isolateInfo) decides inside a bean that every isolate needs:
@PostConstruct()
void warmUp() {
cache.load(); // every isolate needs its cache
if (isolate.primary) migrate(); // the database is migrated once
}
- A bean under the condition does not exist in the other isolates. A bean that takes it in its constructor fails there, and that isolate does not start (
Isolate isolate-1 failed to start): put the condition on both, or take it as a nullable type. - A controller under the condition is mapped in the first isolate only. A request for its path that another isolate takes finds no handler.
- Without a pool the condition always matches.
When the other isolates need what that one bean knows, they ask it: a bean of the first isolate that implements PrimaryIsolateHandler answers primaryIsolate.ask(...) from every isolate. See Asking the first isolate.
How the pool starts
- The first isolate starts as always. Its server binds the port, shared, so the same program can bind it again.
- The pool starts the others, all at once, when that server listens. Each is the application started again in a new isolate, with the same arguments, properties and profiles, and with these over its own configuration: the port of the first (also when that asked for any free port with
server.port=0), no banner, no startup lines, no shutdown hook. - An isolate joins the port when its own beans are up. Requests are never sent to one that is not ready. It then tells the first isolate, which logs
Isolate isolate-1 ready in … seconds. - The application waits for them.
Started … in, the runners of the first isolate andApplicationReadyEventcome after every isolate is ready or has failed. An isolate that is not ready after two minutes is given up.
An isolate that fails to start is logged (Isolate isolate-2 failed to start, with the error) and left out: the server runs with the others. It is not tried again.
The isolates are named isolate-1, isolate-2, …; the first stays main. IsolateInfo.index is the number, 0 for main.
The application must be startable again
The pool starts the application in another isolate through the ApplicationLaunch bean, which exists when the application was run with what returns it. The generated runApplication(args) and runApplicationAsync(args) do that. An application started by hand must pass it:
SpringApplication.run(args, hotReload: () => springApplication);
Without it the pool cannot add an isolate, logs The pool of isolates did not start, and the first isolate serves alone:
Isolates cannot be added: the application was not started in a way that can be repeated in another isolate. Start it with the generated runApplication(args), or pass what returns it: SpringApplication.run(args, hotReload: () => springApplication).
What returns the application crosses to another isolate, so it must not use anything from around it. The generated () => springApplication does not.
The logs
The pool logs on spring.web.isolates. Every start, every scaling step and every failure is a line:
| Level | Line | When |
|---|---|---|
info |
Scaling 1 → 4 isolates (server.isolates.count=4) |
At the start, and at every scaling step, with its reason |
warn |
Each isolate keeps its own … in memory, … |
Once, when the pool first adds an isolate and some bean is an IsolateLocalState |
info |
Isolate isolate-1 ready in 0.412 seconds |
Its server listens |
error |
Isolate isolate-1 failed to start |
Its start failed, or took longer than two minutes |
error |
The pool of isolates did not start |
The isolates of count could not be added |
error |
Uncaught error in isolate-1: … |
An error nothing caught, with its stack trace. The isolate goes on |
error |
Isolate isolate-1 ended unexpectedly: starting it again |
It ended without being asked to |
error |
Isolate isolate-1 ended unexpectedly, the fourth in a minute: not started again. 3 isolates serve. |
See below |
info |
Draining isolate isolate-3 |
It leaves the pool: at a scaling step down, or when the context closes |
warn |
Isolate isolate-3 did not stop within 30 seconds (server.isolates.drain-timeout): stopped anyway |
Its requests did not finish in time |
info |
Isolate isolate-3 stopped |
It has ended, as asked |
info |
Stopping 3 isolates |
The context closes |
The other isolates log like the first. Their lines carry their name where main has main: %thread in a pattern of the logging starter, [isolate-2] at the start of the default line. See Isolates.
Each isolate runs its own logging. Whether a log file (logging.file.name) and its rollover work when several isolates write the same file has not been verified.
An isolate that ends
An isolate can end without being asked, for example by Isolate.exit() in the application's code.
- The pool logs it and starts an isolate of the same name again.
- Its requests in flight get no answer. Their connections stay open until the VM collects the isolate, or the client gives up. The other isolates go on serving meanwhile.
- What it kept in memory is gone: its sessions, its cache.
- The fourth isolate to end within a minute is not started again, so a broken application does not restart in a loop. The pool then serves with fewer isolates until it is scaled or the application restarts.
An error a handler throws does not end an isolate: it is answered as a 500, as with one isolate.
Stopping
The pool stops with the context, before the first isolate's server: Stopping 3 isolates, then each one is drained. A draining isolate stops taking connections, finishes its requests as a stopping application does (see Shutdown), runs its @PreDestroy methods and ends. One that has not ended after server.isolates.drain-timeout is stopped anyway.
In a Flutter app
The pool is for server programs. A Flutter app that runs the server keeps its one isolate: an isolate of the pool would run the whole application, and the widgets, the assets and the plugins of a Flutter app exist in its first isolate only. With server.isolates.* set, the app logs this warning and starts as without it:
server.isolates is ignored in a Flutter app: an isolate of the pool runs the whole application, and the widgets, the assets and the plugins of a Flutter app exist in its first isolate only.
For heavy work in a Flutter app use the task executor.
The properties
| Key | Type | Default | Meaning |
|---|---|---|---|
server.isolates.count |
a number, or auto |
1 |
How many isolates serve at the start. auto: one per processor core. 1: the application's own isolate only. Anything else that is not a number counts as 1 |
server.isolates.drain-timeout |
Duration |
30s |
How long an isolate that leaves the pool may finish its requests before it is stopped anyway |
server.isolates.scaling, min, max and the keys of the scaling policy are on Scaling the isolates.
With count: 1, scaling: manual and no max above one, there is no pool at all: no serverIsolatePool bean, and the port is not shared.
In Spring (Java)
Spring has no counterpart. Tomcat's worker threads (server.tomcat.threads.*) share one heap and the beans: a session, a cache and a counter are the same for every thread. The nearest in Java is several instances of the application behind a load balancer: the questions of the table above are the ones asked there.