Per-Call Worker Cost in Synchronous Services

Yes, for most synchronous services, a per-call worker (thread or process) is worth it for simplicity and isolation, but it is not always the most efficient choice. This article explains what per-call workers are, how they compare to alternatives like event loops, and how to decide based on your workload.

Formula

The number of workers needed to handle a given load without queueing is:

Required workers = (Concurrent calls × Average call duration in seconds) / 1 second

Where:

• Concurrent calls – number of requests arriving per second (throughput).
• Average call duration – time each request occupies a worker, in milliseconds (convert to seconds by dividing by 1000).

This is a direct application of Little’s Law: the number of in-flight requests equals throughput times latency. If you have fewer workers than this, requests will queue or be rejected.

Worked Example

Using the default values: 100 concurrent calls, average duration 50 ms.

1. Convert duration to seconds: 50 ms ÷ 1000 = 0.05 s
2. Multiply by throughput: 100 × 0.05 = 5
3. So you need at least 5 workers to handle 100 requests per second with 50 ms each.

The calculator shows 5 required workers. With 10 workers, utilization is 5/10 = 50%.

What Is a Per-Call Worker?

A per-call worker is a thread (or process) assigned to handle one request from start to finish. The service accepts a connection, spawns or borrows a worker, and that worker reads the request, does the work, writes the response, and returns to a pool. This is the classic thread-per-request model used by many web servers and database clients.

The alternative is an event-driven model, where a single thread handles many requests by multiplexing I/O (like Node.js or nginx). In that model, no thread is tied to a specific call; instead, callbacks or coroutines are scheduled when I/O completes.

When Per-Call Workers Shine

Per-call workers are worth it when:

• Work is CPU-bound or blocking – if your code does heavy computation, file I/O, or locks, an event loop would block all other requests. A per-call worker isolates that stall.
• Code simplicity matters – writing synchronous code is easier to reason about, debug, and test. No callbacks, no state machines, no async pitfalls.
• Latency is low and predictable – if calls finish quickly (milliseconds), thread overhead is negligible.
• You need isolation – a crash in one request won’t take down the whole service (if using processes).

For example, a REST API that queries a local database and returns JSON typically fits this model well. Each request is short, and the database driver is blocking.

When It’s Not Worth It

Per-call workers become a liability when:

• Concurrency is extremely high – thousands of simultaneous connections (like chat servers or streaming) would require thousands of threads, exhausting memory. Each thread reserves stack space (typically 1-8 MB), so 10,000 threads could use 10-80 GB just for stacks.
• Work is I/O-bound with long waits – if a request spends 90% of its time waiting for a downstream service, a worker is idle but still occupies memory. An event loop could handle thousands of such waits with a few threads.
• Frequent spawning – creating a new thread per call adds overhead. Use a thread pool to reuse workers.

For instance, a service that proxies to a slow external API with 2-second latency and 10,000 requests per second would need 20,000 workers (by the formula) – impractical. An event-driven proxy could handle that with a handful of threads.

How to Choose: A Practical Decision Tree

Ask these questions in order:

1. Is your workload primarily CPU-bound? → Per-call workers are fine.
2. Is your workload I/O-bound with short waits (< 100 ms)? → Per-call workers are fine.
3. Is your workload I/O-bound with long waits (> 100 ms) and high concurrency (> 1000)? → Consider event-driven or async.
4. Do you need to handle 10,000+ simultaneous connections? → Event-driven is usually necessary.
5. Is your team more productive with synchronous code? → Per-call workers may be worth the extra hardware.

Also consider the language/platform. Java’s virtual threads (Project Loom) blur the line: they give you per-call simplicity with event-loop scalability. Go’s goroutines are similar. If you use such a runtime, the per-call model is almost always worth it because the overhead is tiny.

Common Problems and Misconceptions

• “Threads are expensive” – true for OS threads, but less so with lightweight threads. Measure before optimizing.
• “Event loops are always faster” – not for CPU-bound work. They only help when waiting on I/O.
• “I need a worker per connection” – no, you need a worker per active request. A keep-alive connection with no request in flight doesn’t need a worker.
• “More workers = better” – beyond the number of CPU cores, extra threads only help if they are blocked on I/O. Too many threads cause context-switching overhead.
• “Thread pool size = max concurrency” – if the pool is exhausted, requests queue or fail. Use the formula to size it.

Costs and Trade-offs

The main costs of per-call workers are memory and context switching. Each OS thread reserves stack memory (often 1-8 MB, though only part is used). A pool of 100 threads uses ~100-800 MB of virtual memory. Context switching happens when the OS switches between threads; if you have more runnable threads than cores, this adds overhead.

Event loops avoid these costs but add complexity: you must write non-blocking code, handle partial I/O, and manage state across callbacks. Debugging is harder because stack traces don’t show the full request flow.

Hybrid approaches exist: use an event loop for I/O and offload CPU-heavy tasks to a small thread pool. This is common in Node.js (worker_threads) and Python (asyncio with run_in_executor).

Real-World Examples

• Web server (e.g., Apache) – uses a thread/process per connection. Works well for moderate traffic. Under massive load, it consumes memory.
• Nginx – event-driven, handles 10,000+ connections with few workers. Better for static files and reverse proxying.
• Database client – a connection pool is effectively a per-call worker pool. Each query occupies a connection, which is a worker.
• Microservice – if your service calls 3 other services, each request occupies a worker for the entire chain. If the downstream is slow, you need many workers.

Frequently Asked Questions

What is the difference between a thread and a process?

A thread is a lightweight execution unit within a process; threads share memory. A process has its own memory space. For per-call workers, threads are more common because they are cheaper to create and communicate. Processes offer better isolation (a crash doesn’t kill others) but use more memory.

How do I size a thread pool?

Use the formula: required workers = throughput × average latency. For example, 500 requests/sec × 0.2 sec = 100 workers. Add some headroom (e.g., 20%) for spikes. If the work is CPU-bound, cap the pool at the number of CPU cores to avoid thrashing.

Is a per-call worker the same as thread-per-request?

Yes, they are the same model. Each incoming request gets a dedicated thread (or borrowed from a pool) for its entire duration. The terms are often used interchangeably.

Can I use virtual threads with the per-call model?

Yes. Virtual threads (Java) or goroutines (Go) allow you to write synchronous code while the runtime multiplexes them onto OS threads. This gives you the simplicity of per-call workers without the memory overhead, making the model viable for high concurrency.

What happens if I have fewer workers than needed?

Requests will queue up. If the queue is unbounded, memory grows until the service crashes. If bounded, requests are rejected with errors (e.g., 503). The formula tells you the minimum to avoid queueing.

Does the calculator account for queueing?

No, it computes the steady-state number of workers needed to keep up with arrival rate. It assumes no queueing and no variability. In real systems, you should add a safety margin (e.g., 20-30%) to handle bursts.

Related

• Protecting Internet Privacy: A Practical Guide
• Quantum Computing Simulators & Cloud Access Guide
• How to Create Custom Meta Boxes with CMB2 in WordPress

in t | Average call duration | ms | 50 | 1 | 60000
in c | Concurrent calls | calls | 100 | 1 | 100000
in w | Worker threads | threads | 10 | 1 | 1000
out r | Required workers | threads | c * t / 1000 | 0
out u | Worker utilization | % | (c * t / 1000) / w * 100 | 1