A pool of isolates

By default the server answers from one isolate, the application's own, and so uses one processor core. server.isolates.count adds more: each isolate runs the whole application, and all of them serve the same port.

It changes more than the speed. Read What changes with more than one isolate before you turn it on.

Why

An isolate does one thing at a time and uses one processor core. A server whose handlers compute (large JSON, hashing, pictures) reaches the limit of that core while the others are idle. More isolates use them.

Tomcat does this with threads that share the beans. Dart's isolates share nothing, so here each isolate has the application for itself. That keeps code free of locks, and it means that whatever a bean keeps in memory is kept once per isolate.

It does not help a server that mostly waits: for a database, for another service. One isolate can wait for many answers at once; what it cannot do is compute two things at once. Measure first: see Load testing.

Example

# lib/resources/application.yml
server:
  isolates:
    count: 4        # or auto: one per processor core
// bin/main.dart
import 'package:spring_boot_starter_web/spring_boot_starter_web.dart';

import 'spring_gen/main.spring.dart';

@SpringBootApplication()
void main(List<String> args) => runApplication(args);

Nothing else changes in the code. The first isolate logs:

INFO  Web server started on port 8080 (http)
INFO  Scaling 1 → 4 isolates (server.isolates.count=4)
WARN  Each isolate keeps its own sessions in memory, and a request may reach any isolate: what one keeps the others do not see. Keep it where all of them do (a database, a shared store), or accept one copy per isolate.
INFO  Isolate isolate-1 ready in 0.412 seconds
INFO  Isolate isolate-2 ready in 0.415 seconds
INFO  Isolate isolate-3 ready in 0.431 seconds
INFO  Started neon-books in 0.52 seconds

A controller can tell which isolate answers:

@RestController()
class WhoController {
  WhoController(this.isolate);
  final IsolateInfo isolate;

  @GetMapping('/who')
  String who() => isolate.name; // main, isolate-1, isolate-2 or isolate-3
}

What changes with more than one isolate

What With more than one isolate What to do
The beans Every isolate runs the whole application: each bean exists once per isolate, and what the beans keep in memory is kept that many times Count what the application keeps in memory times the isolates
A request May reach any isolate. A connection goes to whichever isolate takes it first, and the requests of a kept-alive connection stay with that isolate. Two requests of one browser can be answered by two isolates Do not expect the next request to find what this one left in memory
In-memory sessions Each isolate has its own. A session made by one is unknown to the next: the user is logged out at random. server.session.max-sessions counts per isolate Declare an HttpSessionRepository that keeps them where all isolates see them. See Sessions
Caches (@Cacheable) Each isolate has its own cached values: a value is computed once per isolate, and @CacheEvict evicts in the isolate that runs it only Accept it where a stale value for a while is fine, or use a cache outside the isolates. See The in-memory caches
Counters, rate limiters, any field of a bean Count what their own isolate saw. A limit of 100 a minute kept in memory lets 100 through per isolate Keep them in a shared store, or divide the limit by the isolates
WebSocketSessionRegistry.broadcast Reaches the clients connected to the isolate that sends, not the others' Send through something every isolate listens to (a message broker, a database notification)
Server-sent events An SseEmitter lives in the isolate that took the request. A bean that sends to "all" its emitters sends to those of its isolate The same
@Scheduled methods Run in the first isolate only, so a job runs once for the application spring.task.scheduling.isolates: all for what each isolate must do for itself. See @Scheduled
ApplicationRunner and CommandLineRunner beans, @PostConstruct, @EventListener for the context's events Run once in every isolate: a migration, a warm-up, an e-mail "the server started" happens four times @ConditionalOnPrimaryIsolate() on the bean, or if (isolate.primary)
The banner and Started … in Logged once, by the first isolate; Started … in when the others are ready Nothing
Signals (Ctrl+C, SIGTERM) With spring.main.register-shutdown-hook: true the first isolate takes them and stops the others with its context. The others register no hook Nothing
State all isolates must see Is the application's job: a database, a shared store. The framework does not synchronise beans. For what must exist once inside the application it has one building block: a bean in the first isolate that the others ask (PrimaryIsolate) Design for it before you raise count
A database connection pool Is a bean: one pool per isolate. spring.datasource.pool.max-size: 10 with four isolates is up to 40 connections at the database Divide the pool's size, or raise the database's limit
server.max-connections, server.max-concurrent-requests Are per isolate: the server as a whole takes that many times the isolates. See Overload Set them per isolate
The isolateTaskExecutor Each isolate has its own pool of task isolates (isolate-2-task-1) Count core-size times the isolates
The actuator's endpoints isolates and qps ask the first isolate, so every isolate answers for the whole application. The others, health among them, answer for the isolate that takes the request. See The isolates endpoint Nothing for those two; do not read one isolate's health as the application's
An isolate that ends unexpectedly Is started again. The requests it was answering get no answer: their connections stay open until the VM collects the isolate, or the client gives up Give clients a timeout; look for ended unexpectedly in the logs
A Flutter app Has no pool: the setting is ignored with a warning Nothing; see In a Flutter app
Hot reload When a reload restarts the context, the pool stops with it and the new context starts the isolates again Nothing

The pool names what it can see. A bean that implements IsolateLocalState is listed in the one warning the pool logs when it first adds an isolate. The in-memory session repository (sessions), the cache starter's managers, the default in-memory one and SimpleCacheManager (cached values), and the WebSocketSessionRegistry (WebSocket sessions) do. The pool finds them among the beans that exist when it grows, whatever type they were declared with. A bean of your own can be listed too:

@Service()
class LoginAttempts implements IsolateLocalState {
  final _failed = <String, int>{};

  @override
  String get isolateLocalState => 'failed logins';
}
WARN  Each isolate keeps its own sessions, failed logins in memory, and a request may reach any isolate: what one keeps the others do not see. Keep it where all of them do (a database, a shared store), or accept one copy per isolate.

The warning is not an error. A cache can often live with one copy per isolate; sessions usually cannot.

Once for the application

@ConditionalOnPrimaryIsolate() registers a bean in the first isolate only:

@Component()
@ConditionalOnPrimaryIsolate()
class Migrations implements ApplicationRunner {
  Migrations(this.db);
  final Database db;

  // Once for the application, not once per isolate.
  @override
  Future<void> run(ApplicationArguments args) => db.migrate();
}

IsolateInfo (the bean isolateInfo) decides inside a bean that every isolate needs:

@PostConstruct()
void warmUp() {
  cache.load();                       // every isolate needs its cache
  if (isolate.primary) migrate();     // the database is migrated once
}

When the other isolates need what that one bean knows, they ask it: a bean of the first isolate that implements PrimaryIsolateHandler answers primaryIsolate.ask(...) from every isolate. See Asking the first isolate.

How the pool starts

  1. The first isolate starts as always. Its server binds the port, shared, so the same program can bind it again.
  2. The pool starts the others, all at once, when that server listens. Each is the application started again in a new isolate, with the same arguments, properties and profiles, and with these over its own configuration: the port of the first (also when that asked for any free port with server.port=0), no banner, no startup lines, no shutdown hook.
  3. An isolate joins the port when its own beans are up. Requests are never sent to one that is not ready. It then tells the first isolate, which logs Isolate isolate-1 ready in … seconds.
  4. The application waits for them. Started … in, the runners of the first isolate and ApplicationReadyEvent come after every isolate is ready or has failed. An isolate that is not ready after two minutes is given up.

An isolate that fails to start is logged (Isolate isolate-2 failed to start, with the error) and left out: the server runs with the others. It is not tried again.

The isolates are named isolate-1, isolate-2, …; the first stays main. IsolateInfo.index is the number, 0 for main.

The application must be startable again

The pool starts the application in another isolate through the ApplicationLaunch bean, which exists when the application was run with what returns it. The generated runApplication(args) and runApplicationAsync(args) do that. An application started by hand must pass it:

SpringApplication.run(args, hotReload: () => springApplication);

Without it the pool cannot add an isolate, logs The pool of isolates did not start, and the first isolate serves alone:

Isolates cannot be added: the application was not started in a way that can be repeated in another isolate. Start it with the generated runApplication(args), or pass what returns it: SpringApplication.run(args, hotReload: () => springApplication).

What returns the application crosses to another isolate, so it must not use anything from around it. The generated () => springApplication does not.

The logs

The pool logs on spring.web.isolates. Every start, every scaling step and every failure is a line:

Level Line When
info Scaling 1 → 4 isolates (server.isolates.count=4) At the start, and at every scaling step, with its reason
warn Each isolate keeps its own … in memory, … Once, when the pool first adds an isolate and some bean is an IsolateLocalState
info Isolate isolate-1 ready in 0.412 seconds Its server listens
error Isolate isolate-1 failed to start Its start failed, or took longer than two minutes
error The pool of isolates did not start The isolates of count could not be added
error Uncaught error in isolate-1: … An error nothing caught, with its stack trace. The isolate goes on
error Isolate isolate-1 ended unexpectedly: starting it again It ended without being asked to
error Isolate isolate-1 ended unexpectedly, the fourth in a minute: not started again. 3 isolates serve. See below
info Draining isolate isolate-3 It leaves the pool: at a scaling step down, or when the context closes
warn Isolate isolate-3 did not stop within 30 seconds (server.isolates.drain-timeout): stopped anyway Its requests did not finish in time
info Isolate isolate-3 stopped It has ended, as asked
info Stopping 3 isolates The context closes

The other isolates log like the first. Their lines carry their name where main has main: %thread in a pattern of the logging starter, [isolate-2] at the start of the default line. See Isolates.

Each isolate runs its own logging. Whether a log file (logging.file.name) and its rollover work when several isolates write the same file has not been verified.

An isolate that ends

An isolate can end without being asked, for example by Isolate.exit() in the application's code.

An error a handler throws does not end an isolate: it is answered as a 500, as with one isolate.

Stopping

The pool stops with the context, before the first isolate's server: Stopping 3 isolates, then each one is drained. A draining isolate stops taking connections, finishes its requests as a stopping application does (see Shutdown), runs its @PreDestroy methods and ends. One that has not ended after server.isolates.drain-timeout is stopped anyway.

In a Flutter app

The pool is for server programs. A Flutter app that runs the server keeps its one isolate: an isolate of the pool would run the whole application, and the widgets, the assets and the plugins of a Flutter app exist in its first isolate only. With server.isolates.* set, the app logs this warning and starts as without it:

server.isolates is ignored in a Flutter app: an isolate of the pool runs the whole application, and the widgets, the assets and the plugins of a Flutter app exist in its first isolate only.

For heavy work in a Flutter app use the task executor.

The properties

Key Type Default Meaning
server.isolates.count a number, or auto 1 How many isolates serve at the start. auto: one per processor core. 1: the application's own isolate only. Anything else that is not a number counts as 1
server.isolates.drain-timeout Duration 30s How long an isolate that leaves the pool may finish its requests before it is stopped anyway

server.isolates.scaling, min, max and the keys of the scaling policy are on Scaling the isolates.

With count: 1, scaling: manual and no max above one, there is no pool at all: no serverIsolatePool bean, and the port is not shared.

In Spring (Java)

Spring has no counterpart. Tomcat's worker threads (server.tomcat.threads.*) share one heap and the beans: a session, a cache and a counter are the same for every thread. The nearest in Java is several instances of the application behind a load balancer: the questions of the table above are the ones asked there.

See also