
# Overload

A server can be given two limits: how many requests it answers at a time, and how many connections it keeps open. Past either, a request is answered `503 Service Unavailable` with `Retry-After` at once. Neither limit is set by default.

## Why

A server that takes every request does not answer every request. Past what it can do, requests wait behind the others: each answers later than the one before, clients give up and try again, and every waiting request takes memory. In the end the server answers nobody in time.

A full server that says so at once stays fast for the requests it has taken. The client knows after a few milliseconds, not after its timeout, and a load balancer or a client that honours `Retry-After` sends the request again a moment later, or elsewhere.

These two limits are one part of protecting a server. The others exist already:

| Protects against | Property | Default | Past it |
|---|---|---|---|
| More requests at once than the server can answer | `server.max-concurrent-requests` | no limit | `503` with `Retry-After`; the connection stays open for the next try |
| More open connections than the server should hold | `server.max-connections` | no limit | `503` with `Retry-After`, the connection closed |
| A body that is too large | `server.max-http-request-body-size` | `10MB` | `413` |
| A client that sends nothing, idle connections | `server.connection-timeout` | `60s` | `408`, the connection closes; an idle connection closes |
| A handler that takes too long | `server.request-timeout` | no limit | `503`; the request's `RequestDeadline` has passed |

See [Timeouts](timeouts.md) for the last two and [The embedded server](index.md#servermax-http-request-body-size) for the body limit.

## Example

```yaml
# lib/resources/application.yml
server:
  max-concurrent-requests: 500
  max-connections: 5000
  retry-after: 2s
```

```properties
server.max-concurrent-requests=500
server.max-connections=5000
server.retry-after=2s
```

The request that would be the 501st at a time gets:

```text
HTTP/1.1 503 Service Unavailable
Retry-After: 2
Content-Type: application/problem+json

{"type":"about:blank","title":"Service Unavailable","status":503,"detail":"The server answers its most requests at a time (500)"}
```

## `server.max-concurrent-requests`

- **Type:** `int`. **Default:** not set: no limit. `0` or a negative value: no limit either.
- **What it does:** the most requests the server answers at a time. A request that arrives while that many are being answered is refused at once: `503`, `Retry-After`, and the detail `The server answers its most requests at a time (500)`. Its connection stays open for the next try.
- **Event streams are not counted.** A response that is an event stream (`text/event-stream`) or NDJSON stays open for minutes and does no work most of that time. Such requests take no place under the limit.
- **Room comes back by itself.** When one of the requests is answered, the next one is taken.
- **Why set it:** when you know from a [load test](../../../docs/load-testing.md) how many requests at a time the server answers within the time you promise. Set it there. Without the limit, the latency of every request grows with the queue.

## `server.max-connections`

- **Type:** `int`. **Default:** not set: no limit. `0` or a negative value: no limit either.
- **What it does:** the most connections the server keeps open. While it has more, every request is refused, on whichever connection it arrives, until the number is back under the limit: `503`, `Retry-After`, the detail `The server has its most connections open (5000)`, `Connection: close`, and the connection is closed after the answer.
- **A connection that sends nothing takes its place too.** Idle keep-alive connections count; `server.connection-timeout` closes them after a while.
- **A connection is refused when it sends a request**, not when it is opened: the server needs the request to answer it. A connection past the limit that sends nothing stays open until `server.connection-timeout`.
- **The count is taken at most every 100 ms**, because counting goes through every connection. A burst of new connections within that time passes the limit by a little.
- **Why set it:** to bound what clients can hold open, whatever they send or do not send.

## The answer

Both limits answer alike, but for the connection:

- `503 Service Unavailable` as a `ProblemDetail`, like every error of the starter.
- **`Retry-After`** in whole seconds, from `server.retry-after` (`1s` by default), at least `1`: `500ms` is sent as `1`.
- **The connection.** Past `server.max-concurrent-requests` it stays open: the server has too much to do, not too many connections, and the client tries again on the same one. Past `server.max-connections` the answer has `Connection: close` and the connection is closed: one connection too many goes.
- **Before everything else.** The server answers before the request reaches the filters, the security filter chain or a handler. So the refusal costs almost nothing, and nothing of the application sees it: no filter, no `@ExceptionHandler`, no access rule.
- **Counted.** The row of the isolate in the [task manager](../../spring-core-dart/isolates/load.md) has `refused`, the number of requests refused since the server started, and they are part of its `requests`. The [requests per second](qps.md) of the debug feature do not include them: its filter never sees them.

## With a pool of isolates

Both limits are per isolate. With `server.isolates.count: 4` and `server.max-concurrent-requests: 500`, the server as a whole answers up to 2000 at a time, and one isolate can be full while another has room: a connection goes to whichever isolate takes it. See [A pool of isolates](isolates.md).

## In Spring (Java)

Spring Boot's nearest keys are Tomcat's: `server.tomcat.max-connections` and `server.tomcat.threads.max`. There a request past the threads waits for one. Here nothing waits: past a limit the client is told at once, with a time to come back. `server.max-connections`, `server.max-concurrent-requests` and `server.retry-after` are this starter's own keys.

## See also

- [Timeouts](timeouts.md)
- [Rate limiting](../rate-limiting.md): how much of the server one client gets
- [The embedded server](index.md)
- [A pool of isolates](isolates.md)
- [Requests per second](qps.md)
- [ProblemDetail](../errors/problem-detail.md)
- [Load testing](../../../docs/load-testing.md)
- [All properties](../properties.md)
