Overload
A server can be given two limits: how many requests it answers at a time, and how many connections it keeps open. Past either, a request is answered 503 Service Unavailable with Retry-After at once. Neither limit is set by default.
Why
A server that takes every request does not answer every request. Past what it can do, requests wait behind the others: each answers later than the one before, clients give up and try again, and every waiting request takes memory. In the end the server answers nobody in time.
A full server that says so at once stays fast for the requests it has taken. The client knows after a few milliseconds, not after its timeout, and a load balancer or a client that honours Retry-After sends the request again a moment later, or elsewhere.
These two limits are one part of protecting a server. The others exist already:
| Protects against | Property | Default | Past it |
|---|---|---|---|
| More requests at once than the server can answer | server.max-concurrent-requests |
no limit | 503 with Retry-After; the connection stays open for the next try |
| More open connections than the server should hold | server.max-connections |
no limit | 503 with Retry-After, the connection closed |
| A body that is too large | server.max-http-request-body-size |
10MB |
413 |
| A client that sends nothing, idle connections | server.connection-timeout |
60s |
408, the connection closes; an idle connection closes |
| A handler that takes too long | server.request-timeout |
no limit | 503; the request's RequestDeadline has passed |
See Timeouts for the last two and The embedded server for the body limit.
Example
# lib/resources/application.yml
server:
max-concurrent-requests: 500
max-connections: 5000
retry-after: 2s
server.max-concurrent-requests=500
server.max-connections=5000
server.retry-after=2s
The request that would be the 501st at a time gets:
HTTP/1.1 503 Service Unavailable
Retry-After: 2
Content-Type: application/problem+json
{"type":"about:blank","title":"Service Unavailable","status":503,"detail":"The server answers its most requests at a time (500)"}
server.max-concurrent-requests
- Type:
int. Default: not set: no limit.0or a negative value: no limit either. - What it does: the most requests the server answers at a time. A request that arrives while that many are being answered is refused at once:
503,Retry-After, and the detailThe server answers its most requests at a time (500). Its connection stays open for the next try. - Event streams are not counted. A response that is an event stream (
text/event-stream) or NDJSON stays open for minutes and does no work most of that time. Such requests take no place under the limit. - Room comes back by itself. When one of the requests is answered, the next one is taken.
- Why set it: when you know from a load test how many requests at a time the server answers within the time you promise. Set it there. Without the limit, the latency of every request grows with the queue.
server.max-connections
- Type:
int. Default: not set: no limit.0or a negative value: no limit either. - What it does: the most connections the server keeps open. While it has more, every request is refused, on whichever connection it arrives, until the number is back under the limit:
503,Retry-After, the detailThe server has its most connections open (5000),Connection: close, and the connection is closed after the answer. - A connection that sends nothing takes its place too. Idle keep-alive connections count;
server.connection-timeoutcloses them after a while. - A connection is refused when it sends a request, not when it is opened: the server needs the request to answer it. A connection past the limit that sends nothing stays open until
server.connection-timeout. - The count is taken at most every 100 ms, because counting goes through every connection. A burst of new connections within that time passes the limit by a little.
- Why set it: to bound what clients can hold open, whatever they send or do not send.
The answer
Both limits answer alike, but for the connection:
503 Service Unavailableas aProblemDetail, like every error of the starter.Retry-Afterin whole seconds, fromserver.retry-after(1sby default), at least1:500msis sent as1.- The connection. Past
server.max-concurrent-requestsit stays open: the server has too much to do, not too many connections, and the client tries again on the same one. Pastserver.max-connectionsthe answer hasConnection: closeand the connection is closed: one connection too many goes. - Before everything else. The server answers before the request reaches the filters, the security filter chain or a handler. So the refusal costs almost nothing, and nothing of the application sees it: no filter, no
@ExceptionHandler, no access rule. - Counted. The row of the isolate in the task manager has
refused, the number of requests refused since the server started, and they are part of itsrequests. The requests per second of the debug feature do not include them: its filter never sees them.
With a pool of isolates
Both limits are per isolate. With server.isolates.count: 4 and server.max-concurrent-requests: 500, the server as a whole answers up to 2000 at a time, and one isolate can be full while another has room: a connection goes to whichever isolate takes it. See A pool of isolates.
In Spring (Java)
Spring Boot's nearest keys are Tomcat's: server.tomcat.max-connections and server.tomcat.threads.max. There a request past the threads waits for one. Here nothing waits: past a limit the client is told at once, with a time to come back. server.max-connections, server.max-concurrent-requests and server.retry-after are this starter's own keys.
See also
- Timeouts
- Rate limiting: how much of the server one client gets
- The embedded server
- A pool of isolates
- Requests per second
- ProblemDetail
- Load testing
- All properties