
# The load of the isolates

`IsolateLoads` answers one question: which isolates does the application have, and how busy is each? It gives one row per isolate, or per thread of a Flutter engine, with how late its event loop runs, the share of the time it works, and what it works on. With Spring Boot it is the bean `isolateLoads`.

Two things read it: the [task manager of a Flutter app](../../spring-boot-starter-flutter/isolates.md) and the actuator's [`isolates` endpoint](../../spring-boot-actuator/isolates.md). Your own code can too. The same measurements decide the [scaling of the web server's isolates](../../spring-boot-starter-web/server/scaling.md).

## Why

"The server is slow" has two very different causes. Either an isolate computes and cannot take the next request, or it waits for a database and is free. The first is fixed by another isolate, the second is not. The rows tell which: a busy isolate runs late, a waiting one does not.

A compiled Dart program cannot list its isolates, and cannot ask how much processor time one uses. So the framework measures what can be measured from inside, and this page says where that falls short.

## Example

```dart
import 'dart:isolate';

import 'package:spring_boot_dart/spring_boot_dart.dart';

@Service()
class Diagnostics {
  Diagnostics(this.loads);
  final IsolateLoads loads;

  // One line per row: "main (main): busy 0.02, lag 0:00:00.000180".
  List<String> rows() => [
    for (final load in loads.all())
      '${load.name} (${load.kind}): busy ${load.busy}, lag ${load.lag}',
  ];

  // An isolate the application starts itself is a row while it runs.
  Future<Report> build() => loads.track('report', Isolate.run(buildReport));
}
```

The first `all()` starts the measuring and has no lag yet. Read it again a moment later.

## A row

A row is an `IsolateLoad`. Every value but `name` and `kind` is `null` when its source cannot measure it.

| Field | In JSON | Meaning |
|---|---|---|
| `name` | `name` | The name the logs show: `main`, `isolate-2`, `task-1`, `raster` |
| `kind` | `kind` | What it is; see below |
| `busy` | `busy` | The share of the time it was working, `0` to `1` |
| `lag` | `lagMicros` | How late the event loop ran a timer, on average: near zero when the isolate is free, the length of what blocks it when it is not |
| `maxLag` | `maxLagMicros` | The longest lag measured |
| `active` | `active` | What it is working on now: requests in flight, tasks running |
| `memory` | `memory` | Memory in bytes. Isolates share the memory of their process, so only the `main` row has it: the resident size of the process |
| `details` | `details` | What only one source measures, by name: `requests`, `queued`, `tasks`. A duration is in microseconds, under a name that ends in `Micros` |

`toJson()` and `IsolateLoad.fromJson` use the JSON names. That is how a row crosses from one isolate to another, and what the actuator answers. A value that is `null` is left out.

| `kind` | The row is |
|---|---|
| `main` | The application's first isolate |
| `server` | An isolate of the web server's pool |
| `task` | An isolate of the task executor |
| `engine` | A thread of the Flutter engine |
| `tracked` | Work shown with `track` |
| another word | A source of your own |

## The sources

`IsolateLoads` measures nothing itself. It asks every `IsolateLoadSource` bean, in order:

| Bean | Package | Rows |
|---|---|---|
| `isolateLoadSampler` | `spring_boot_dart` | The isolate it runs in: `lag`, `maxLag`, `busy`, and `memory` in the first isolate |
| `isolateTaskExecutor` | `spring_boot_dart` | One per isolate of the [task executor](task-executor.md) that exists now |
| `webServerLoadSource` | `spring_boot_starter_web` | Adds to the isolate's own row: `active` (requests in flight) and the details `requests` (all it has taken) and `refused` (answered 503 because the server was full) |
| `serverIsolatePool` | `spring_boot_starter_web` | The rows of the other isolates of the [server's pool](../../spring-boot-starter-web/server/isolates.md), as each last sent them |
| `flutterEngineLoads` | `spring_boot_starter_flutter` | The engine's threads (`raster`, `platform`, `io`) and the frames' build times in `main`, with the Flutter task manager in the build |

A source of your own is a bean that implements the interface, for isolates the application starts and manages itself:

```dart
@Component()
class EncoderLoads implements IsolateLoadSource {
  EncoderLoads(this.encoders);
  final EncoderPool encoders;

  @override
  List<IsolateLoad> loads() => [
    for (final encoder in encoders.all)
      IsolateLoad(
        name: encoder.name,
        kind: 'encoder',
        active: encoder.running,
        details: {'frames': encoder.frames},
      ),
  ];
}
```

`loads()` is called at every read, perhaps once a second. Answer from what you already know; do not wait for anything.

## Rows of one name are one row

A source may add to a row another source gives, by giving a load of the same name. The web server does: the sampler gives `main` its lag, the server gives `main` its requests in flight, and the reader sees one row with both.

The merge takes, for each value, the first source that has it. So a later source cannot replace the `busy` of an earlier one, only fill what is missing. `name` and `kind` are the first source's. The details of both are kept; for a detail of the same name the later source wins.

## track

No source sees an isolate started with `Isolate.run` or `compute`. `track` shows it:

```dart
final report = await loads.track('report', Isolate.run(buildReport));
```

While the future runs there is a row named `report`, of kind `tracked`, with `busy: 1` and `active: 1`. Two tracked futures of one name are one row with `active: 2`. The row goes when the last of them completes, also with an error. `track` returns the future it was given.

It shows that the work runs, not how hard: nothing is measured inside that isolate. Work that fits the [task executor](task-executor.md) gets real rows without `track`.

## How lag and busy are measured

`IsolateLoadSampler` measures the isolate it runs in with one timer.

- A timer is due every `interval` (100 ms). How late it fires is the lag. A free isolate fires it on time; one that computes for 80 ms fires it 80 ms late.
- `load()` looks at the last `window` (5 seconds): `lag` is the average of the lags in it, `maxLag` the longest.
- `busy` is estimated from the lag: the sum of the lags over the time measured (the intervals plus the lags).
- **It measures only while someone reads.** The first `load()` starts the timer and has no `lag` and no `busy` yet. Without a read for `idleAfter` (30 seconds) the timer stops by itself. An application nobody watches pays nothing.

What that cannot tell:

- **`busy` is not processor time.** It is how much the event loop was held up. A compiled program cannot ask how much processor time an isolate uses.
- **Short work between two timers is not seen.** A timer is late only when something runs at the moment it is due. An isolate that computes for 30 ms and then rests until after the next timer shows no lag for it. Many short pieces of work show as less `busy` than they are; one long piece shows in full.
- **Waiting is not busy.** An isolate that awaits a database has a free event loop: no lag, `busy` near zero, whatever its `active` says. A high `active` with a low `busy` points at what the isolate waits for, not at the isolate.
- **The first seconds are missing.** Nothing was measured before the first read.
- **Memory is the process's.** It cannot be split by isolate; only `main` has it. In a browser it is the JavaScript heap in use, and only where the browser tells it.
- **Only known isolates are rows.** An isolate without a source and without `track` does not exist for the task manager, though it uses a core.

A task executor's row has another `busy`: the share of the time during which the isolate had a task, over the last second or more. A task that awaits counts for all its time. It is worked out at most once a second, so several readers see the same value.

## With a pool of server isolates

The rows of the other isolates are measured there and sent to the first isolate, which keeps the last answer of each.

- Reading `isolateLoads` in the first isolate asks the others again, at most once a second. The answers are there for the next read, so a row of another isolate is about one read old.
- `await isolateReports.refresh()` asks now and completes when they have answered, or after a second without those that did not. Read the rows after it when they must be current.
- **Only the first isolate's `isolateLoads` has every row.** In `isolate-2` the bean gives its own row and its task isolates, nothing else. Code that may run in any isolate asks the first one: `await primaryIsolate.ask(IsolateLoadsHandler.handlerId)` answers every row as JSON, fresh, whichever isolate asks (see [Asking the first isolate](primary-isolate.md)). The actuator's endpoint does that.

## Reports from every isolate

Numbers a bean keeps are kept once per isolate. An `IsolateReporter` bean makes them available for the whole application: each isolate is asked in its own isolate, and the answers are kept in the first isolate's `IsolateReports`, the bean `isolateReports`.

```dart
@Component()
class BasketCount implements IsolateReporter {
  BasketCount(this.baskets);
  final Baskets baskets;

  @override
  String get id => 'baskets';

  @override
  Object? report() => baskets.open; // of this isolate
}

// In the first isolate:
await reports.refresh();
final open = reports.of('baskets', own: 'main');
// {main: 12, isolate-1: 9, isolate-2: 14}
```

| Member of `IsolateReports` | Meaning |
|---|---|
| `own()` | What this isolate's reporters tell now, by id |
| `of(id, own: name)` | The reports named `id` by isolate: this isolate's first, under the name given, then the others in the order of their numbers (`isolate-2` before `isolate-10`) |
| `refresh()` | Asks the other isolates now. Completes at once without a pool |
| `put`, `remove` | Called by the pool |

- `report()` must answer JSON values only: numbers, text, lists and maps of them. Anything else cannot cross.
- A report is as old as the last refresh. Reading the task manager refreshes; otherwise call `refresh()`.
- Without a pool it has the reports of the one isolate, asked when read.
- Read it in the first isolate. Another isolate has only its own; to answer from every isolate, put the reading into a [`PrimaryIsolateHandler`](primary-isolate.md), as the web starter's `RequestRates` does.

The web starter's [requests per second](../../spring-boot-starter-web/server/qps.md) are built this way: each isolate counts, the first adds up.

## In Spring (Java)

Spring has no counterpart: a JVM's threads share their beans, so a number kept in a bean is the application's. Nearest is the actuator's `threaddump` endpoint. Here each isolate measures itself and tells the first one.

## See also

- [Isolates](index.md)
- [The task executor](task-executor.md)
- [Asking the first isolate](primary-isolate.md)
- [The `isolates` endpoint](../../spring-boot-actuator/isolates.md)
- [The task manager of a Flutter app](../../spring-boot-starter-flutter/isolates.md)
- [Scaling the isolates](../../spring-boot-starter-web/server/scaling.md)
- [Observations and the start](../observations/index.md)
