Production agent latency is rarely just an LLM problem. Here’s how to find the real bottleneck.

Your AI agent takes 12 seconds to answer a user.

You check the LLM metrics.

The model itself takes only 2 seconds.

So where did the other 10 seconds go?

This is one of the most important questions to understand when moving AI agents from prototypes into production.

A typical agent might look like this:

User Request
     ↓
    LLM
   2.1s
     ↓
  Tool A
   1.8s
     ↓
  Tool B
   2.4s
     ↓
  Tool C
   1.7s
     ↓
    LLM
   2.0s
     ↓
  Response

Total latency:

2.1 + 1.8 + 2.4 + 1.7 + 2.0 = 10.2 seconds

And that’s before accounting for additional orchestration, network overhead, serialization, retries, queueing, and other infrastructure costs.

If you’re optimizing this system by simply replacing the LLM with a faster model, you’re optimizing the wrong component.

The real question is:

What is the critical path through the agent?

1. Start With the Execution Graph

The first thing I would do in production is not change the model.

I’d inspect the execution graph.

An agent is not just:

Prompt → LLM → Answer

A production agent often looks more like:

                  ┌→ Search
                  │
User → LLM → Router├→ Database
                  │
                  └→ API
                       ↓
                    LLM
                       ↓
                    Answer

Every node can introduce latency.

And more importantly, the way those nodes are connected determines the total latency.

Suppose your agent needs three independent pieces of information:

• customer profile
• current account balance
• recent transactions

You might accidentally implement:

LLM
 ↓
Customer API
 ↓
Balance API
 ↓
Transaction API
 ↓
LLM

If each API takes roughly 2 seconds, you’ve created a 6-second critical path.

But what if those calls don’t depend on each other?

Then you can execute:

              ┌→ Customer API ─┐
             │                │
LLM → Router ├→ Balance API ──┼→ LLM
             │                │
             └→ Transaction ──┘

Now the tool execution time approaches:

max(T_A,T_B,T_C)

rather than:

T_A+T_B+T_C

If:

Tool A = 1.8s
Tool B = 2.4s
Tool C = 1.7s

Sequential execution costs approximately:

1.8 + 2.4 + 1.7 = 5.9s

Parallel execution can reduce the tool portion to approximately:

max(1.8, 2.4, 1.7) = 2.4s

That’s a massive reduction.

2. Think in Terms of the Critical Path

This is one of the most useful concepts for agent optimization.

The critical path is the sequence of operations that determines the minimum time required to produce the final answer.

Imagine:

LLM:        2.0s
Tool A:     1.5s
Tool B:     2.5s
Tool C:     1.8s
Final LLM:  2.0s

If A, B and C are sequential:

2.0 + 1.5 + 2.5 + 1.8 + 2.0
= 9.8s

But if A, B and C can execute concurrently:

2.0 + max(1.5, 2.5, 1.8) + 2.0
= 6.5s

You haven’t changed the LLM.

You haven’t changed the APIs.

You haven’t optimized the database.

You simply changed the execution strategy.

And you saved more than 3 seconds.

This is why agent optimization should begin with execution topology, not model benchmarking.

3. But Be Careful: Not Everything Can Run in Parallel

Parallelism isn’t automatically correct.

Consider:

LLM
 ↓
Search
 ↓
Read Search Result
 ↓
Database Query

The database query might depend on the search result.

For example:

Search → returns customer_id
             ↓
       query customer_id

You cannot execute both simultaneously because the second operation depends on the output of the first.

This gives us a useful rule:

Parallelize independent work. Preserve dependencies.

A good agent graph makes these dependencies explicit.

For example:

             ┌→ Tool A ─┐
             │          │
Planner ─────┼→ Tool B ─┼→ Synthesizer
             │          │
             └→ Tool C ─┘

versus:

Planner → Tool A → Tool B → Tool C → Synthesizer

The second graph may be correct, but if those tools are independent, you’re paying a latency penalty for no reason.

4. Tool Calls Are Often the Hidden Bottleneck

One of the biggest mistakes in agent design is assuming:

“The LLM is slow.”

In many production systems, external tools are slower.

For example:

LLM                    1.8s
Vector search          0.2s
Internal database      0.4s
External REST API      2.7s
CRM API                1.9s
Another LLM            2.2s

The LLM isn’t necessarily the biggest problem.

The external API might be.

This becomes especially painful when agents make multiple tool calls.

Consider an agent that decides:

"Let me check the CRM."
→ CRM API
"Now let me check the database."
→ Database API
"Now let me verify this through another service."
→ External API
"Now I'll ask the LLM to summarize everything."

Each individual call may look reasonable.

Together, they create a slow system.

5. Reduce Unnecessary Tool Calls

The fastest tool call is the one you don’t make.

This sounds obvious, but agentic systems can make unnecessary calls because the LLM has too much freedom.

Suppose the agent receives:

“What is the status of order 12345?”

The agent might do:

Search documentation
        ↓
Search customer profile
        ↓
Query order database
        ↓
Call shipping API
        ↓
LLM

But perhaps only this was necessary:

Order database
      ↓
    LLM

The other calls add latency without improving the answer.

This is why tool selection should be treated as an optimization problem.

Ask:

• Does this tool actually contribute to the answer?
• Is the information already available?
• Can one tool replace three?
• Can the LLM answer without a tool?
• Can a deterministic router select the correct tool directly?

Reducing tool count can sometimes produce a bigger latency improvement than changing the model.

6. Repeated LLM Calls Are Another Major Problem

Agent architectures frequently contain multiple model calls:

LLM → Planner
      ↓
    Tool
      ↓
    LLM → Reasoning
      ↓
    Tool
      ↓
    LLM → Validation
      ↓
    LLM → Final answer

Each model invocation introduces:

• network latency
• queueing
• token processing
• input serialization
• output generation
• potentially another round of context construction

Even if each call takes only 1–2 seconds, the total can become significant.

For example:

Planner LLM       1.5s
Reasoning LLM     1.8s
Validation LLM    1.4s
Final LLM         1.7s
Total             6.4s

Ask whether every call is actually necessary.

Sometimes:

Planner → Tool → Final LLM

is sufficient.

The goal isn’t to minimize the number of LLM calls at all costs.

The goal is to eliminate unnecessary sequential model calls.

7. Context Size Directly Affects Latency

Another common bottleneck is context.

Suppose your final LLM receives:

System prompt
+
Conversation history
+
20 retrieved documents
+
Tool outputs
+
Previous reasoning
+
Metadata

The model now has to process a large input before generating the answer.

Large context can increase:

• time to first token
• total inference time
• token cost
• memory usage
• downstream processing

This creates an important optimization opportunity:

Don’t give the model everything you have. Give it what it needs.

Instead of:

20 documents
→ 15,000 tokens
→ LLM

you might use:

100 candidates
     ↓
Retriever
     ↓
Top 10
     ↓
Reranker
     ↓
Top 3
     ↓
LLM

The final model receives a much smaller and more relevant context.

This can improve both latency and answer quality.

8. Retrieval Can Also Become a Latency Trap

RAG-based agents introduce another layer of complexity.

A retrieval pipeline might look like:

Query
 ↓
Embedding model
 ↓
Vector database
 ↓
BM25
 ↓
Merge results
 ↓
Cross-encoder reranker
 ↓
Context construction
 ↓
LLM

Every component adds latency.

For example:

Embedding       120ms
Vector search    80ms
BM25             60ms
Reranker        400ms
LLM            2000ms

The retrieval pipeline adds another 660ms before the final generation even begins.

That may be perfectly acceptable if it dramatically improves answer quality.

But you should measure it.

A sophisticated retrieval architecture isn’t automatically a better production architecture.

Sometimes a simpler pipeline is enough.

9. External APIs Can Destroy Your Latency Budget

Agent systems often depend on services outside your infrastructure.

For example:

Agent
 ↓
Payment API
 ↓
CRM API
 ↓
Shipping API
 ↓
Analytics API

You don’t control all of these systems.

They may have:

• variable response times
• rate limits
• network latency
• cold starts
• overloaded servers
• retries
• intermittent failures

A tool that normally takes 300ms might occasionally take 3 seconds.

That’s where p95 and p99 latency become important.

Average latency isn’t enough.

Suppose:

Average API latency = 400ms
p95                 = 1.2s
p99                 = 3.8s

Your average dashboard may look healthy.

But users experiencing the p99 request are waiting almost 10× longer.

Production optimization should therefore look at:

p50 → p95 → p99

not just the mean.

10. Retries Can Secretly Multiply Latency

Retries are essential for reliability.

But they can also create surprising latency.

Imagine:

Tool call
  ↓
Timeout after 2s
  ↓
Retry
  ↓
Success after 1.5s

The user experiences:

2 + 1.5 = 3.5 seconds

The tool’s successful response was only 1.5 seconds.

The additional 2 seconds came from failure handling.

Now imagine multiple tools each retrying independently.

Your agent can quickly accumulate several seconds of hidden latency.

This is why traces should include:

• attempt number
• timeout duration
• retry reason
• backoff duration
• final status

Without this information, the latency profile can be misleading.

11. Caching Can Remove Entire Classes of Work

If the same expensive operation happens repeatedly, caching can be extremely effective.

Suppose your agent repeatedly asks:

“What is our refund policy?”

The retrieval pipeline might execute every time:

Embedding
 ↓
Vector search
 ↓
Reranking
 ↓
LLM

If the answer or relevant retrieval result is stable, caching can eliminate some of that work.

Possible cache layers include:

Response cache

Cache the final answer for identical or equivalent requests.

Retrieval cache

Cache search results for repeated queries.

Embedding cache

Avoid recomputing embeddings for identical text.

Tool-result cache

Cache expensive external API responses where the data’s freshness requirements allow it.

Caching doesn’t work everywhere.

For dynamic data such as account balances or real-time inventory, stale results may be unacceptable.

But for stable information, caching can provide enormous latency improvements.

12. Model Selection Matters — But Later

Now we get to the optimization everyone wants to talk about:

“Should we use a faster LLM?”

Yes, potentially.

But not first.

Suppose your current system is:

LLM  = 2 seconds
Tools = 8 seconds

You replace the LLM with a model that takes 1 second.

You saved:

1 second

Your system went from:

10s → 9s

But if you parallelize three independent tools and reduce their contribution from 5 seconds to 2 seconds, you might save 3 seconds without touching the model.

This doesn’t mean model selection is unimportant.

It means you should optimize based on the largest contributor to the critical path.

13. Smaller Models Can Be Extremely Useful for Routing

Not every agent decision requires a powerful model.

For example:

User Query
    ↓
Small/Fast Model
    ↓
Tool selection
    ↓
Expensive LLM
    ↓
Final response

A lightweight model can handle:

• intent classification
• routing
• tool selection
• simple extraction
• structured decisions

while a more capable model handles complex reasoning.

This creates a useful architecture:

Simple tasks → cheap/fast model
Complex tasks → powerful model

Instead of sending every decision through your most expensive and slowest model.

14. Streaming Improves Perceived Latency

There is another important distinction:

Actual latency vs. perceived latency

Suppose the final response takes 5 seconds.

If the user sees nothing for 5 seconds:

Request
  ↓
  ↓
  ↓
  ↓
Answer

it feels extremely slow.

But if the model starts streaming after 1 second:

Request
  ↓
First token
  ↓
"Based..."
  ↓
"on..."
  ↓
"your..."
  ↓
"request..."

the user receives feedback almost immediately.

Streaming doesn’t necessarily reduce total computation time.

But it can dramatically improve the user experience.

Therefore, measure both:

Time to first token (TTFT)

and

Time to final token (TTLT)

depending on your application.

15. Don’t Optimize Without Tracing

This is perhaps the most important production lesson.

You cannot optimize what you cannot observe.

Your agent should generate a trace resembling:

Request                    0ms
│
├── Router LLM             0–600ms
│
├── Tool A                 600–1,400ms
│
├── Tool B                 600–1,900ms
│
├── Tool C                 600–1,200ms
│
└── Final LLM              1,900–3,900ms

Now the bottleneck becomes obvious.

Without tracing, you might look only at:

LLM latency = 2 seconds

and incorrectly conclude:

“The LLM is too slow.”

With tracing, you discover:

Tool B = 1.3 seconds
Final LLM = 2 seconds
Router = 600ms

and perhaps several operations were unnecessarily sequential.

Observability turns optimization from guessing into engineering.

16. What Should You Measure?

For every agent execution, capture at least:

Model metrics

• model name
• input tokens
• output tokens
• time to first token
• total generation latency
• number of model calls

Tool metrics

• tool name
• execution time
• success/failure
• retry count
• timeout count

Retrieval metrics

• embedding latency
• vector search latency
• reranking latency
• number of documents retrieved
• final context size

Agent metrics

• total execution time
• number of steps
• number of tool calls
• number of LLM calls
• parallel vs. sequential execution
• termination reason

These metrics let you answer the most important question:

Where is the time actually going?

17. A Practical Optimization Hierarchy

If I were handed a production agent taking 10–12 seconds, I’d investigate in roughly this order.

Step 1: Build the execution trace

Break the request into every individual operation.

Total = 12s
LLM 1       1.8s
Tool A      1.5s
Tool B      2.2s
Tool C      1.7s
Retry       1.0s
LLM 2       2.0s
Other       1.8s

Don’t optimize yet.

Measure first.

Step 2: Identify the critical path

Ask:

Which operations actually determine the final response time?

Some operations may already be running concurrently.

Others may be irrelevant to the critical path.

Step 3: Parallelize independent operations

Convert:

A → B → C

into:

A ┐
B ├→ D
C ┘

where dependencies allow it.

Step 4: Remove unnecessary work

Look for:

• redundant tools
• duplicate retrieval
• repeated LLM calls
• unnecessary validation
• excessive context
• duplicate API requests

Step 5: Optimize the slowest tools

Investigate:

• database queries
• API calls
• network overhead
• serialization
• connection pooling
• cold starts
• external service latency

Step 6: Add caching where appropriate

Cache stable and frequently repeated operations.

Step 7: Reduce context

Retrieve fewer documents.

Rerank more effectively.

Remove redundant conversation history.

Compress tool outputs.

Step 8: Optimize model selection

Only after understanding the architecture should you decide:

• smaller model
• faster model
• fewer model calls
• lower token count
• streaming
• speculative or parallel generation where appropriate

18. Turning a 10-Second Agent Into a 3-Second Agent

Let’s take the original example.

Before

LLM 1        2.1s
Tool A       1.8s
Tool B       2.4s
Tool C       1.7s
LLM 2        2.0s
Total       10.0s

Suppose A, B and C are independent.

Parallelize them:

LLM 1                  2.1s
    ↓
A ┐
B ├── parallel         2.4s
C ┘
    ↓
LLM 2                  2.0s
Total ≈ 6.5s

We’re already down from 10 seconds to roughly 6.5 seconds.

Now suppose Tool C is unnecessary.

LLM 1                  2.1s
    ↓
A ┐
B ├── parallel         2.4s
    ↓
LLM 2                  2.0s
Total ≈ 6.5s

The removal doesn’t change the critical path because B was already slower than C.

This is an important optimization lesson:

Removing work doesn’t always reduce latency. Removing work from the critical path does.

Now suppose the final LLM can use a smaller model and go from 2 seconds to 1 second:

2.1 + 2.4 + 1.0
= 5.5s

Still not 3 seconds.

So you continue profiling.

Maybe the first LLM is only being used for routing.

Replace it with a lightweight classifier:

Router          300ms
Tools          2.4s
Final LLM      1.0s
Total          3.7s

Then optimize the slowest tool:

Router          300ms
Tools          1.8s
Final LLM       900ms
Total          ~3.0s

Notice what happened.

We didn’t find one magical optimization.

We reduced latency across the entire execution graph.

19. The Interview Answer

This is exactly how I’d answer the question in a system design or AI engineering interview:

“I’d first profile the complete agent execution graph rather than assuming the LLM is the bottleneck. I’d measure latency for every LLM call, tool call, retrieval step, retry, and external API. Then I’d identify the critical path. If independent tools are currently executed sequentially, I’d parallelize them. I’d remove unnecessary tool and model calls, reduce excessive context, investigate slow external APIs and retries, add caching where appropriate, and only then consider switching to a faster or smaller model. I’d also monitor p95 and p99 latency, not just average latency. The goal is to reduce the critical path rather than simply making the LLM faster.”

That’s a much stronger answer than:

“Use a faster LLM.”

Because it demonstrates that you understand systems, not just models.

20. The Bigger Lesson

AI agents are distributed systems disguised as AI applications.

They involve:

LLMs
+
APIs
+
Databases
+
Retrieval systems
+
Vector stores
+
Queues
+
Network calls
+
Retries
+
Orchestration
+
Caching

The LLM is only one component.

When an agent takes 12 seconds to respond, asking:

“How fast is my LLM?”

is only part of the question.

The more important question is:

“What does my agent do between the user’s request and the final response?”

Trace that execution.

Find the critical path.

Parallelize what you can.

Remove what you don’t need.

Cache what you repeatedly compute.

Reduce unnecessary context.

Optimize slow dependencies.

Then choose the right model.

Final Takeaway

If your AI agent takes 12 seconds and your LLM takes 2 seconds, don’t immediately blame the LLM.

There are probably 10 seconds of system behavior waiting to be understood.

The fastest agent isn’t necessarily the one with the fastest model.

It’s the one that does the least necessary work, in the shortest possible critical path.

And that’s the mindset that separates a prototype agent from a production-grade one.

Author

Ved Prakash

Senior  Data Scientist | AI Engineer | Generative AI

Writing practical tutorials on RAG, LLMs, AI Agents, LangGraph, MCP,  Machine Learning, and production AI systems.