Understanding Nested Parallelism,FastAPI, OpenMP Thread Contention, and the Right Way to Deploy CPU-Bound Machine Learning Models

Imagine This…

You have trained a Random Forest classifier using Scikit-Learn.

Everything works perfectly during development.

You deploy it behind FastAPI, add a bit of asynchronous programming using run_in_executor(), and your API happily serves requests.

Then someone runs a load test.

At around 50 concurrent requests, your API becomes completely unresponsive.

CPU usage jumps to 100%.

Latency shoots through the roof.

Eventually even /health stops responding.

Your first thought?

“AsyncIO is broken.”

Spoiler: it isn’t.

The real culprit is something much more subtle—a phenomenon called Nested Parallelism.

In this article, we’ll understand why this happens, why many production ML APIs unknowingly suffer from it, and how companies deploy Scikit-Learn models without bringing down their ASGI servers.

The Architecture Looks Perfect… Right?

Our inference pipeline looks something like this.

At first glance, everything appears correct.

• FastAPI remains asynchronous.
• CPU-bound work is moved off the event loop.
• ThreadPoolExecutor handles inference.

So why does everything freeze?

Step 1: AsyncIO Isn’t Running Your Model

Many developers think:

“FastAPI is async, therefore my ML inference is async.”

Not exactly.

Scikit-Learn is completely synchronous.

When you write:

prediction = await loop.run_in_executor(
    executor,
    predict_sync,
    data
)

AsyncIO simply says:

“I’ll let another thread execute this.”

The model itself never becomes asynchronous.

Instead:

So far, everything is fine.

Step 2: Random Forest Has Its Own Parallelism

Most people train their Random Forest like this:

RandomForestClassifier(
    n_estimators=200,
    n_jobs=-1
)

That innocent-looking n_jobs=-1 means:

Use every available CPU core.

Internally, Scikit-Learn delegates work to Joblib, which in turn relies on OpenMP.

Instead of one thread predicting trees…

Predict()
     │
     ├── Tree 1
     ├── Tree 2
     ├── Tree 3
     ├── Tree 4

…it becomes…

Predict()
     ├── CPU Thread 1
     ├── CPU Thread 2
     ├── CPU Thread 3
     ├── CPU Thread 4
     ├── ...
     └── CPU Thread N

One prediction can now consume your entire CPU.

Step 3: Now Add Concurrent Requests

Suppose your server has:

• 8 CPU cores
• ThreadPoolExecutor with 4 workers
• Random Forest using n_jobs=-1

Four requests arrive simultaneously.

Instead of this…

Request 1 → Worker 1
Request 2 → Worker 2
Request 3 → Worker 3
Request 4 → Worker 4

Each worker says…

“I’d also like all 8 CPU cores.”

Now the operating system sees:

Worker 1
 ├──8 OpenMP Threads
Worker 2
 ├──8 OpenMP Threads
Worker 3
 ├──8 OpenMP Threads
Worker 4
 ├──8 OpenMP Threads

Total runnable threads:

4 × 8 = 32

On an 8-core machine.

The Thread Explosion

The situation becomes even worse under load.

Suppose:

• 50 concurrent requests
• Executor size = 4

Only four requests execute immediately.

But each worker creates another pool of OpenMP threads.

The scheduler now has to coordinate:

This is called Nested Parallelism.

Instead of spending CPU time predicting trees…

the operating system spends most of its time doing this:

The result is:

• huge context switching
• cache invalidation
• scheduler thrashing

Eventually even Uvicorn struggles to get CPU time.

The API appears frozen.

Why AsyncIO Gets Blamed

People often say:

“FastAPI froze.”

But the event loop isn’t blocked by Python code.

It’s simply starving.

The operating system keeps scheduling dozens of OpenMP worker threads instead.

Your event loop is waiting…

"I'm ready to run..."
Scheduler:
"Sorry.
32 OpenMP threads are ahead of you."

Why loky Doesn’t Solve It

One popular suggestion is:

with joblib.parallel_backend("loky"):
    prediction = model.predict(X)

Technically…

Yes.

This isolates prediction into processes.

But every prediction now requires:

For batch inference this is excellent.

For real-time APIs predicting a single row?

The IPC overhead often dominates the actual inference.

It’s common to add 100–200 ms of latency just from process communication.

Why ProcessPoolExecutor Isn’t a Silver Bullet

Another recommendation is:

“Just use a ProcessPool.”

That has trade-offs too.

Each process owns its own Python interpreter.

Depending on how the model is loaded and how workers are started, memory usage can increase substantially because the model exists in multiple processes.

Large tree ensembles can quickly consume gigabytes of RAM.

The Production Pattern

Interestingly…

Most production Scikit-Learn deployments do not let every prediction use every CPU core.

Instead they choose request-level parallelism.

Each worker:

• loads the model once
• predicts using one thread
• serves requests independently

Concurrency comes from multiple workers, not multiple OpenMP thread pools.

This scales far better.

Another Option: Limit OpenMP Threads

If you still want some internal parallelism, don’t use -1.

Instead:

from threadpoolctl import threadpool_limits
def predict_sync(data):
    with threadpool_limits(limits=2):
        return model.predict(data)

Or configure environment variables:

OMP_NUM_THREADS=2
MKL_NUM_THREADS=2
OPENBLAS_NUM_THREADS=2

A useful rule of thumb:

Executor Threads × OpenMP Threads
        ≤
Available CPU Cores

Example:

Avoid oversubscription.

An Interesting Surprise

Many developers assume:

n_jobs=-1

must always be faster.

It often isn’t.

For single-row inference, thread creation and synchronization can cost more than the prediction itself.

Benchmarks frequently show:

• nearly identical latency
• dramatically better throughput

when using:

n_jobs=1

This surprises many engineers.

Architecture Comparison

❌ Nested Parallelism

                                    Client
                                    
                                      ↓
                                    
                                    FastAPI
                                    
                                      ↓
                                    
                                    ThreadPoolExecutor
                                    
                                        ↓
                                    
                                    RandomForest
                                    
                                        ↓
                                    
                                    OpenMP
                                    
                                        ↓
                                    
                                    CPU Contention

✅ Production Deployment

Scaling happens by adding workers rather than creating more threads inside each prediction.

Key Takeaways

• AsyncIO isn’t the problem—nested parallelism is.
• run_in_executor() combined with n_jobs=-1 creates competing thread pools.
• OpenMP oversubscription causes scheduler thrashing and event loop starvation.
• loky introduces IPC overhead, making it unsuitable for low-latency, single-row inference.
• ProcessPoolExecutor can increase memory usage because each process has its own model instance.
• Production systems typically use multiple ASGI worker processes with n_jobs=1 (or a small bounded thread count) and scale horizontally.

Final Thoughts

One of the biggest misconceptions in machine learning deployment is believing that more threads automatically mean better performance.

In reality, modern serving systems perform best when only one layer of the stack owns parallelism.

Either let your web server handle concurrency or let your ML library parallelize computation—but rarely both at full capacity.

If you found this article helpful, you may also enjoy:

• FastAPI Documentation — https://fastapi.tiangolo.com/
• my blog: https://geekycodes.in/
• Scikit-Learn Parallelism Guide — https://scikit-learn.org/stable/computing/parallelism.html
• Python AsyncIO Documentation — https://docs.python.org/3/library/asyncio.html
• Joblib Documentation — https://joblib.readthedocs.io/
• Uvicorn Documentation — https://www.uvicorn.org/