RAG, LLMs, Agentic AI, LangGraph, and Production GenAI

Part 1: AI Engineer Interview Questions 
Part 2: AI Engineer Interview Questions
Part 3: AI Engineer Interview Questions
Part 4: AI Engineer Interview Questions 
Part 5: AI Engineer Interview Questions

AI Engineer interviews are no longer limited to questions like:

“What is an embedding?”

or:

“What is RAG?”

Interviewers increasingly want to know whether you can design, debug, deploy, and operate AI systems in production.

They may ask you to explain:

• Why you selected RAG instead of fine-tuning
• How you handle hallucinations
• How you design a multi-agent architecture
• Why you chose LangGraph
• How you evaluate an LLM application
• How you protect sensitive data
• What happens when retrieval fails
• How you control LLM costs
• How your AI system behaves in production

This is Part 1 of an interview-focused series covering the most important questions for AI Engineer and GenAI Engineer roles.

1. Explain Your GenAI Project

Interview Question

Can you explain your most recent GenAI project?

Answer

One of my key projects was an AI-Powered Clinical Care Navigation Platform for a healthcare client.

The business problem was that healthcare organizations have large volumes of information spread across clinical guidelines, insurance policies, SOPs, compliance documents, and operational knowledge bases. Employees often had to manually search across multiple systems, which was time-consuming and inefficient.

To solve this, we built an Agentic AI platform using LangGraph, where different specialized agents handled different stages of the workflow.

The architecture included:

• Planner Agent for understanding intent and decomposing complex tasks
• Retrieval Agent for RAG-based enterprise knowledge retrieval
• Reasoning Agent for synthesizing information from multiple sources
• Governance Agent for hallucination, PHI, and policy compliance checks

The RAG pipeline used embeddings, vector search, hybrid retrieval, metadata filtering, and role-based access controls.

For orchestration, we used LangGraph because we needed stateful workflows, conditional routing, retries, and human approval checkpoints.

Since this was a healthcare use case, governance was particularly important. We implemented PHI masking, response validation, source attribution, confidence thresholds, and human-in-the-loop escalation.

The platform was deployed using cloud infrastructure with containerization, CI/CD, monitoring, model and prompt versioning, and observability.

The overall objective was to make enterprise healthcare knowledge easier to access while maintaining strong governance, security, and reliability.

2. Why Did You Use Agentic AI Instead of Simple RAG?

Interview Question

Why couldn’t you just use a traditional RAG chatbot? Why did you need an Agentic AI architecture?

Answer

The main reason was workflow complexity.

A traditional RAG pipeline is usually:

User Query
    ↓
Retrieve Documents
    ↓
Generate Answer

That works well for straightforward questions.

But our use cases involved complex tasks requiring:

• Query decomposition
• Multiple retrieval steps
• Multi-hop reasoning
• Policy validation
• Tool usage
• Conditional routing
• Governance checks

For example, if a user asks:

“Summarize the latest diabetes treatment guidelines and identify insurance policy changes affecting approval.”

This isn’t a single retrieval task.

The system needs to:

User Query
     ↓
Planner
     ↓
 ┌───────────────┐
 │               │
 ▼               ▼
Clinical      Insurance
Guidelines    Policies
 │               │
 └───────┬───────┘
         ▼
    Compare Changes
         ↓
      Reasoning
         ↓
     Governance
         ↓
      Response

An Agentic architecture allowed us to represent this workflow explicitly.

The Planner decomposed the task, the Retrieval Agent gathered relevant information, the Reasoning Agent synthesized the results, and the Governance Agent validated the final response.

So the key distinction is:

RAG retrieves knowledge. Agentic AI orchestrates complex workflows around knowledge retrieval and reasoning.

3. Why Did You Choose LangGraph?

Interview Question

Why did you choose LangGraph instead of implementing everything using LangChain?

Answer

We chose LangGraph because our workflow required stateful, multi-step, conditional execution.

A simple LangChain pipeline is suitable for linear workflows:

Input
 ↓
Retriever
 ↓
LLM
 ↓
Output

But our architecture had multiple agents with dependencies between them.

For example:

User Query
    ↓
Planner
    ↓
Intent / Task Decomposition
    ↓
Retrieval
    ↓
Reasoning
    ↓
Governance
    ↓
Final Response

The workflow could also branch.

For example, if the Governance Agent detected low confidence or unsupported claims, we could route the request back for additional retrieval or escalate it to a human.

LangGraph gave us:

• Stateful execution
• Directed workflows
• Conditional routing
• Agent coordination
• Retry mechanisms
• Human-in-the-loop checkpoints
• Persistence of intermediate state

So my decision was based on the workflow requirement.

We didn’t choose LangGraph simply because it was a popular framework. We chose it because the problem required explicit state management and controlled orchestration of multiple agents.

4. What Does Your Planner Agent Do?

Interview Question

Explain the role of the Planner Agent.

Answer

The Planner Agent acts as the orchestrator of the workflow.

Its primary responsibility is to understand the user’s request and determine what steps are required to solve it.

For example:

            User Query
                ↓
            Planner
                ↓
            Task Decomposition
                ↓
 ┌─────────────┬─────────────┐
 ▼             ▼             ▼
Retrieve       Compare       Validate
Guidelines     Policies       Compliance

The Planner determines:

1. What is the user’s intent?
2. Is the task simple or multi-step?
3. Which agents should be invoked?
4. In what order should they execute?
5. What information should be passed between agents?

The resulting plan is stored in the LangGraph workflow state.

That state is then used by downstream agents to execute the appropriate workflow.

So, instead of every agent independently deciding what to do, the Planner provides structured orchestration and routing.

5. How Does Your Retrieval Agent Work?

Interview Question

Walk me through your RAG pipeline.

Answer

Our RAG pipeline followed a standard ingestion and retrieval architecture, but with additional enterprise and healthcare-specific controls.

The high-level flow was:

Enterprise Documents
        ↓
Document Ingestion
        ↓
OCR / Text Extraction
        ↓
Cleaning
        ↓
Metadata Extraction
        ↓
PHI-Sensitive Filtering
        ↓
Chunking
        ↓
Embedding Generation
        ↓
Vector Index

At query time:

User Query
    ↓
Query Processing
    ↓
Embedding Generation
    ↓
Hybrid Retrieval
    ↓
Metadata Filtering
    ↓
Top-K Documents
    ↓
Context Ranking
    ↓
LLM
    ↓
Governance
    ↓
Final Answer

We used semantic vector retrieval along with keyword-based retrieval to improve recall, particularly for domain-specific healthcare terminology.

We also applied metadata and role-based filtering so that the system retrieved information relevant to the user’s context and access permissions.

The goal was not just to retrieve semantically similar documents, but to retrieve relevant, authorized, and trustworthy context.

6. Why Use Hybrid Search Instead of Pure Vector Search?

Interview Question

Why did you use hybrid retrieval? Why not just use embeddings?

Answer

Pure vector search is excellent for understanding semantic similarity, but it can struggle with exact terminology.

Healthcare and enterprise documents often contain:

• Policy IDs
• Procedure codes
• Acronyms
• Drug names
• Legal terms
• Exact phrases

For example, if a user searches for:

“Policy ABC-123”

A keyword-based search may be more effective than semantic similarity.

That’s why we combined:

                 User Query
                     ↓
            ┌────────┴────────┐
            ▼                 ▼
       BM25 Search       Vector Search
            │                 │
            ▼                 ▼
      Keyword Match     Semantic Match
            │                 │
            └────────┬────────┘
                     ▼
                Fusion / Rank
                     ↓
                Top-K Context

The advantage is that hybrid search combines:

• Lexical matching for exact terms
• Semantic matching for conceptual similarity

This is particularly useful in enterprise and healthcare environments where both natural-language questions and exact domain terminology are common.

7. How Do You Decide Chunk Size?

Interview Question

How do you decide the chunk size and overlap in a RAG system?

Answer

I don’t choose chunk size purely based on a fixed number of tokens.

I first consider the structure of the documents and the retrieval use case.

For example:

• Legal or insurance documents → clause-aware chunking
• Technical documentation → heading-aware chunking
• Research papers → section-aware chunking
• FAQs → question-answer pair chunking

The goal is to preserve semantic completeness.

If chunks are too large:

• Retrieval becomes less precise
• Irrelevant context increases
• Prompt size increases
• Token cost increases

If chunks are too small:

• Important context can be split
• Meaning can be lost
• Retrieval may return incomplete information

So I generally use:

Document
    ↓
Identify Semantic Boundaries
    ↓
Create Meaningful Chunks
    ↓
Add Metadata
    ↓
Generate Embeddings

I then tune chunk size and overlap using retrieval evaluation metrics such as:

• Recall@K
• Precision@K
• MRR
• NDCG

The final choice should be based on retrieval quality and downstream answer quality, not just an arbitrary token count.

8. Your RAG Retrieves Relevant Documents, but the LLM Hallucinates. What Do You Do?

Interview Question

The retriever is returning relevant chunks, but the LLM is still hallucinating. How would you debug it?

Answer

I would debug the pipeline layer by layer rather than immediately changing the LLM.

I would check:

Query
 ↓
Retrieval
 ↓
Context
 ↓
Prompt
 ↓
LLM
 ↓
Response
 ↓
Governance

First, I would verify that the retrieved chunks actually contain enough information to answer the question.

Then I would check whether the context is being passed correctly to the LLM.

Next, I would inspect the prompt to ensure the model is explicitly instructed to answer only from the provided context and abstain when information is unavailable.

I would also check whether irrelevant or contradictory chunks are being included.

Then I would evaluate the generated answer for:

• Faithfulness
• Groundedness
• Citation correctness
• Unsupported claims

In our governance layer, we can use an LLM-as-a-judge approach along with deterministic checks to evaluate whether claims are supported by retrieved sources.

If the answer fails the required confidence or groundedness threshold, the system can:

Low Confidence
      ↓
Additional Retrieval
      ↓
Regenerate
      ↓
Validate Again
      ↓
Still Failing?
      ↓
Fallback / Human Escalation

So my approach is to treat hallucination as a pipeline problem, not simply an LLM problem.

9. How Do You Validate Groundedness?

Interview Question

You mentioned a Governance Agent. How exactly do you validate that an answer is grounded?

Answer

We validate groundedness by checking whether the claims made in the generated response are supported by the retrieved context.

For example, suppose the retrieved document says:

“Patients should consult a physician.”

But the LLM generates:

“The patient has diabetes and should take medication X.”

The second statement is not supported by the source.

The Governance Agent should identify that as unsupported content.

The validation pipeline can be:

Retrieved Context
       +
Generated Answer
       ↓
Claim Extraction
       ↓
Evidence Matching
       ↓
Groundedness Evaluation
       ↓
Confidence Score

We can use an LLM-as-a-judge approach for semantic evaluation and deterministic checks where possible.

We also evaluate the system offline using curated test datasets and metrics such as:

• Faithfulness
• Groundedness
• Retrieval quality
• Hallucination rate
• Citation correctness

If the response falls below the required threshold, we can regenerate the answer, retrieve additional context, provide a safe fallback, or escalate to a human.

The important point is:

We don’t assume that retrieving relevant documents automatically guarantees a grounded answer. We explicitly validate the generated claims against the evidence.

10. How Do You Handle PHI and Sensitive Data?

Interview Question

Since your project was in healthcare, how did you handle PHI and sensitive information?

Answer

Because the system operated in a healthcare environment, security and governance were part of the architecture rather than an afterthought.

We implemented multiple layers of controls.

At the data layer:

• PHI-sensitive filtering
• Data masking
• Encryption at rest
• Encryption in transit
• Role-based access control

At the application layer:

• Secure API access
• Role-aware retrieval
• Metadata filtering
• Input validation

At the LLM layer:

• Prompt controls
• Output validation
• PHI leakage checks
• Governance checks
• Source attribution

At the operational layer:

• Audit logging
• Monitoring
• Access control
• Human escalation

The overall architecture was:

User
 ↓
Authentication / Authorization
 ↓
Input Validation
 ↓
PHI Controls
 ↓
Retrieval with Access Filtering
 ↓
LLM
 ↓
Governance
 ↓
Output Validation
 ↓
User

The key principle is defense in depth.

We don’t rely on a single guardrail to protect sensitive information.

11. What Happens When the Governance Agent Finds a Hallucination?

Interview Question

Suppose the Governance Agent detects that the response is not grounded. What happens next?

Answer

We don’t simply return the response to the user.

The system can follow a controlled recovery workflow.

For example:

Generated Response
       ↓
Governance Check
       ↓
Grounded?
  ┌────┴────┐
 Yes        No
  │          │
  ▼          ▼
Return    Re-Retrieve
Answer       │
             ▼
          Regenerate
             │
             ▼
          Validate
             │
       ┌─────┴─────┐
       │           │
    Pass         Fail
       │           │
       ▼           ▼
    Return      Fallback /
                 Escalate

The exact action depends on the severity.

For a minor retrieval issue, we can perform another retrieval attempt.

For a low-confidence response, we can provide a safe fallback.

For high-risk healthcare scenarios, we can escalate to a human.

This is particularly important because in high-risk domains, the system should be designed to fail safely, not simply maximize answer generation.

12. How Do You Evaluate an LLM Application?

Interview Question

How do you know your GenAI application is actually working well?

Answer

I evaluate it at multiple levels.

1. Retrieval Evaluation

I measure whether the correct information is being retrieved.

Metrics include:

• Recall@K
• Precision@K
• MRR
• NDCG

2. Generation Evaluation

I evaluate:

• Faithfulness
• Groundedness
• Answer relevance
• Citation correctness

3. Safety Evaluation

I measure:

• Hallucination rate
• PHI leakage
• Toxicity
• Policy violations

4. System Evaluation

I monitor:

• Latency
• Token consumption
• Cost
• Error rates
• Agent failures

5. Business Evaluation

Ultimately, the system should improve business outcomes.

For example:

• Reduced manual search time
• Improved knowledge retrieval
• Faster resolution
• Higher user satisfaction
• Reduced operational effort

So I don’t evaluate an LLM application using only an LLM benchmark.

I evaluate:

Retrieval
   +
Generation
   +
Safety
   +
System Performance
   +
Business Impact

13. What Is the Role of the Governance Agent?

Interview Question

What exactly does the Governance Agent do?

Answer

The Governance Agent acts as a quality and safety control layer before the final response reaches the user.

Its responsibilities include:

• Groundedness validation
• Hallucination detection
• PHI leakage checks
• Policy compliance
• Response validation
• Source attribution
• Confidence evaluation

The workflow is:

User Query
    ↓
Planner
    ↓
Retrieval
    ↓
Reasoning
    ↓
Generated Response
    ↓
Governance Agent
    ↓
 ┌──────────────┐
 │              │
Pass           Fail
 │              │
 ▼              ▼
User       Retry / Fallback /
           Human Escalation

This is especially important in healthcare because the system should not blindly generate an answer simply because the LLM is confident.

The Governance Agent adds a separate validation layer between generation and delivery.

14. Why Not Fine-Tune the LLM Instead of Using RAG?

Interview Question

Why did you use RAG instead of fine-tuning the LLM on the healthcare documents?

Answer

The primary reason was the nature of the knowledge.

Healthcare policies, guidelines, and enterprise documents can change frequently.

If we fine-tuned the model every time the knowledge changed, we would have:

• Higher training costs
• Longer update cycles
• More complex versioning
• Potential knowledge staleness

With RAG, the knowledge remains external to the model.

New Document
    ↓
Ingest
    ↓
Chunk
    ↓
Embed
    ↓
Update Vector Index
    ↓
Available for Retrieval

This allows us to update knowledge without retraining the LLM.

Fine-tuning would be more appropriate when we want to change:

• Model behavior
• Output format
• Domain-specific style
• Task-specific capabilities

So my decision framework is:

Need new knowledge?
        ↓
       RAG
Need different behavior?
        ↓
   Fine-tuning
Need better instructions?
        ↓
Prompt Engineering

In practice, these techniques can also be combined.

15. How Do You Reduce LLM Costs in Production?

Interview Question

Suppose your LLM costs suddenly increase significantly. What would you do?

Answer

I would first identify where the cost increase is coming from.

I would monitor:

• Token usage
• Request volume
• Input token growth
• Output token growth
• Model usage by endpoint
• Agent execution frequency
• Retry rates

Then I would optimize across several layers.

1. Prompt Optimization

Reduce unnecessary context and instructions.

2. Retrieval Optimization

Retrieve fewer but more relevant chunks.

3. Model Routing

Use smaller models for simple tasks and larger models only when necessary.

Simple Query → Small Model
Complex Query → Large Model

4. Caching

Cache repeated queries and deterministic intermediate results.

5. Reduce Unnecessary Agent Calls

Ensure agents are invoked only when required.

6. Control Retries

Excessive retries can multiply LLM costs.

7. Monitor Token Usage

Set budgets and alerts for abnormal usage.

The architecture becomes:

Request
  ↓
Complexity Detection
  ↓
Model Router
  ├── Simple → Small Model
  └── Complex → Large Model
             ↓
         RAG / Agents
             ↓
          Response

The goal is not simply to reduce the number of LLM calls.

The goal is to optimize:

Cost per successful task.

16. How Do You Handle Conflicting Outputs From Multiple Agents?

Interview Question

What happens if two agents return conflicting information?

Answer

I would not allow the final response to be generated by blindly concatenating agent outputs.

Instead, I would introduce a validation or consensus mechanism.

For example:

Agent A ─────┐
             │
Agent B ─────┼──→ Validation / Consensus
             │
Agent C ─────┘
                     ↓
                Final Decision

The Reasoning Agent can compare the outputs and identify contradictions.

Then we can:

1. Check the original sources.
2. Assign confidence scores.
3. Prefer information from higher-authority sources.
4. Ask for additional retrieval if necessary.
5. Escalate when the conflict cannot be resolved safely.

For healthcare use cases, unresolved contradictions should not be hidden.

The system should explicitly communicate uncertainty or escalate rather than generate a confident but potentially incorrect answer.

17. How Do You Monitor a Multi-Agent AI System?

Interview Question

What metrics would you monitor in production?

Answer

I would monitor the system at three levels.

Application Metrics

• Request volume
• Error rate
• Response latency
• User feedback

LLM Metrics

• Token usage
• Cost
• Model latency
• Generation quality
• Hallucination rate
• Groundedness

Agent Metrics

• Agent execution time
• Agent failure rate
• Number of agent calls
• Retry frequency
• Workflow completion rate
• Routing accuracy

For example:

User Request
     ↓
Planner
     │
     ├── Latency
     ├── Success Rate
     └── Routing Accuracy
     ↓
Retriever
     │
     ├── Recall@K
     ├── Retrieval Latency
     └── Relevance
     ↓
Reasoning
     │
     ├── Token Usage
     └── Generation Quality
     ↓
Governance
     │
     ├── Groundedness
     ├── Hallucination
     └── Policy Violations
     ↓
Final Response

This helps us identify exactly where a production issue is occurring rather than treating the entire AI system as a black box.

18. What Is the Most Important Lesson From Your Project?

Interview Question

What was the biggest technical lesson you learned from building this system?

Answer

The biggest lesson was that building a production GenAI application is not just about choosing a powerful LLM.

The model is only one component.

A reliable AI system requires:

Good Data
   +
Reliable Retrieval
   +
Effective Orchestration
   +
Strong Evaluation
   +
Governance
   +
Security
   +
Observability
   +
Cost Control

For example, even if the LLM is highly capable, the system can still fail if:

• Retrieval returns the wrong documents.
• The context is incomplete.
• The model hallucinates.
• The agent workflow loops.
• Sensitive data is exposed.
• Costs become unpredictable.
• There is no monitoring or rollback mechanism.

So my biggest takeaway is:

Production GenAI is a systems engineering problem, not just an LLM problem.

19. How Should You Answer AI Engineer Questions in Interviews?

Based on my experience, the strongest answers follow this structure:

1. Business Problem
        ↓
2. Why This Approach?
        ↓
3. High-Level Architecture
        ↓
4. Your Specific Contribution
        ↓
5. Technical Implementation
        ↓
6. Challenges & Trade-offs
        ↓
7. Business Impact

For example, don’t start with:

“We used LangGraph, BM25, embeddings, Azure OpenAI, and vector databases.”

Start with:

“The business problem was that healthcare teams had to search across fragmented enterprise knowledge sources. We needed a system capable of handling multi-step workflows rather than simple question answering. That’s why we designed a multi-agent architecture with separate planning, retrieval, reasoning, and governance components.”

Then explain the technologies.

This approach demonstrates that you understand why the architecture exists, not just which tools were used.

Final Takeaway

AI Engineer interviews increasingly test your ability to think beyond individual models and frameworks.

You should be comfortable discussing:

LLMs
 ↓
Prompt Engineering
 ↓
RAG
 ↓
Hybrid Retrieval
 ↓
Vector Databases
 ↓
Agentic AI
 ↓
LangGraph
 ↓
Evaluation
 ↓
Governance
 ↓
Security
 ↓
MLOps / LLMOps
 ↓
Production Monitoring

The strongest candidate is not necessarily the person who can name the most AI frameworks.

It’s the person who can explain:

What problem are we solving?

Why did we choose this architecture?

What happens when something fails?

How do we measure whether it works?

How do we make it secure, scalable, reliable, and cost-effective?

That’s the difference between knowing GenAI concepts and demonstrating production-level AI engineering experience.

Part 2 can go deeper into advanced RAG, embeddings, chunking, reranking, hybrid search, vector databases, RAG evaluation, hallucination debugging, and RAG system design interview questions.

If you’re on medium follow me here