Overload

A server can be given two limits: how many requests it answers at a time, and how many connections it keeps open. Past either, a request is answered 503 Service Unavailable with Retry-After at once. Neither limit is set by default.

Why

A server that takes every request does not answer every request. Past what it can do, requests wait behind the others: each answers later than the one before, clients give up and try again, and every waiting request takes memory. In the end the server answers nobody in time.

A full server that says so at once stays fast for the requests it has taken. The client knows after a few milliseconds, not after its timeout, and a load balancer or a client that honours Retry-After sends the request again a moment later, or elsewhere.

These two limits are one part of protecting a server. The others exist already:

Protects against Property Default Past it
More requests at once than the server can answer server.max-concurrent-requests no limit 503 with Retry-After; the connection stays open for the next try
More open connections than the server should hold server.max-connections no limit 503 with Retry-After, the connection closed
A body that is too large server.max-http-request-body-size 10MB 413
A client that sends nothing, idle connections server.connection-timeout 60s 408, the connection closes; an idle connection closes
A handler that takes too long server.request-timeout no limit 503; the request's RequestDeadline has passed

See Timeouts for the last two and The embedded server for the body limit.

Example

# lib/resources/application.yml
server:
  max-concurrent-requests: 500
  max-connections: 5000
  retry-after: 2s
server.max-concurrent-requests=500
server.max-connections=5000
server.retry-after=2s

The request that would be the 501st at a time gets:

HTTP/1.1 503 Service Unavailable
Retry-After: 2
Content-Type: application/problem+json

{"type":"about:blank","title":"Service Unavailable","status":503,"detail":"The server answers its most requests at a time (500)"}

server.max-concurrent-requests

server.max-connections

The answer

Both limits answer alike, but for the connection:

With a pool of isolates

Both limits are per isolate. With server.isolates.count: 4 and server.max-concurrent-requests: 500, the server as a whole answers up to 2000 at a time, and one isolate can be full while another has room: a connection goes to whichever isolate takes it. See A pool of isolates.

In Spring (Java)

Spring Boot's nearest keys are Tomcat's: server.tomcat.max-connections and server.tomcat.threads.max. There a request past the threads waits for one. Here nothing waits: past a limit the client is told at once, with a time to come back. server.max-connections, server.max-concurrent-requests and server.retry-after are this starter's own keys.

See also