Forget memorizing another list of LangChain APIs. If you understand these five concepts deeply, you can handle the questions that separate AI application builders from AI engineers.

Introduction

I’ve noticed a pattern in AI Engineer interviews.

The candidate’s resume looks impressive:

• RAG
• LangChain
• LangGraph
• Vector databases
• AI Agents
• MCP
• Fine-tuning
• LLMs

Then the interviewer asks:

“Why does your RAG system still hallucinate even though retrieval is working?”

Or:

“Why does KV caching reduce inference latency?”

Or:

“When would you use RAG instead of fine-tuning?”

Suddenly, knowing the framework isn’t enough.

That’s because good AI Engineer interviews increasingly test something deeper:

Do you understand what is happening underneath the abstractions?

You don’t need to memorize every new AI framework.

But before walking into an AI Engineer or GenAI Engineer interview, I’d make sure you can confidently explain these five concepts:

1. Embeddings & Retrieval
2. RAG & Hallucination
3. Attention & KV Cache
4. Fine-tuning & LoRA
5. Evaluation & Observability

These five concepts form a surprisingly large portion of modern production AI systems.

Table of Contents

1. What You’ll Learn
2. Concept #1 — Embeddings & Retrieval
3. Concept #2 — RAG & Hallucinations
4. Concept #3 — Attention & KV Cache
5. Concept #4 — Fine-tuning & LoRA
6. Concept #5 — Evaluation & Observability
7. How These Concepts Connect
8. A Practical Interview Framework
9. Common Interview Mistakes
10. Production Perspective
11. Interview Questions
12. Key Takeaways
13. Related Tutorials
14. Next Tutorial
15. Author

What You’ll Learn

By the end of this article, you should be able to explain:

• Why embeddings are useful
• Why vector similarity isn’t the same as relevance
• Why RAG doesn’t automatically prevent hallucinations
• Why attention replaced recurrent architectures
• Why KV cache improves autoregressive inference
• When to fine-tune an LLM
• What LoRA actually changes
• Why evaluation is more complicated than accuracy
• How these concepts fit into a production AI architecture

1. Embeddings & Retrieval

Let’s start with one of the most fundamental components of modern AI applications.

What is an embedding?

An embedding converts an object such as text into a numerical vector.

For example:

"How do I reset my password?"

might become something conceptually like:

[0.12, -0.41, 0.87, 0.23, ...]

The actual vector may contain hundreds or thousands of dimensions.

The important idea isn’t the individual numbers.

It’s the geometric relationship between vectors.

Conceptually:

                  Vector Space

        password reset
              ●
             / \
            /   \
           /     \
          ●       ●
     forgot     account
     password   recovery

                          ●
                       pizza recipe

Texts with similar meanings should ideally be located closer together.

Why do we need embeddings?

Imagine a user asks:

“How can I change my login password?”

But the document says:

“Users can update their authentication credentials from the account security page.”

A keyword search may struggle because the exact words don’t match.

Semantic retrieval can recognize that:

change password

and

update authentication credentials

are related concepts.

Vector Search

Once documents have been embedded, we can search for vectors close to the query vector.

A common similarity measure is cosine similarity:$$\text{cosine similarity}(A,B) = \frac{A \cdot B} {\|A\|\|B\|}$$

The closer the value is to 1, the more similar the vectors are in direction.

A simplified pipeline:

Documents
    ↓
Chunking
    ↓
Embedding Model
    ↓
Vectors
    ↓
Vector Database

At query time:

User Question
      ↓
Embedding Model
      ↓
Query Vector
      ↓
Vector Search
      ↓
Top-K Chunks

The Interview Question

“Is the highest vector similarity document always the most relevant document?”

No.

This is an important distinction.

Embedding similarity measures semantic similarity according to the embedding model.

It doesn’t necessarily mean:

“This is exactly the information required to answer the question.”

That’s why production RAG systems often combine:

• dense retrieval
• BM25
• hybrid search
• reranking
• metadata filtering

The embedding model is part of the retrieval system—not the retrieval system itself.

2. RAG & Hallucinations

Now we get to one of the most frequently discussed concepts in AI Engineering.

What is RAG?

Retrieval-Augmented Generation combines retrieval with generation.

Instead of asking an LLM:

Question
   ↓
LLM
   ↓
Answer

we introduce external context:

                  User Question
                        │
                        ▼
                    Retriever
                        │
                        ▼
                 Relevant Documents
                        │
                        ▼
              Question + Context
                        │
                        ▼
                       LLM
                        │
                        ▼
                      Answer

The idea is simple:

Give the model relevant information before asking it to generate an answer.

But RAG Doesn’t Eliminate Hallucinations

This is where many candidates give an incomplete interview answer.

A common assumption is:

“If retrieval is correct, hallucinations disappear.”

Not necessarily.

Suppose retrieval returns:

Document A ✓
Document B ✓
Document C ✓

The LLM can still produce an incorrect answer.

Why?

Because there are multiple failure points:

Query
 ↓
Retrieval
 ↓
Context
 ↓
Prompt
 ↓
LLM
 ↓
Answer

A failure at any stage can affect the final result.

A Better Debugging Framework

If a RAG application hallucinates, investigate in this order:

1. Was the correct information retrieved?

Measure:

• Recall@K
• Precision@K
• MRR
• nDCG

2. Is the context relevant?

Retrieving a document that merely contains related words isn’t enough.

3. Is the context complete?

You may retrieve the right section but miss the paragraph containing an important exception.

4. Did the prompt correctly instruct the model?

For example:

Answer only using the supplied context.
If the answer cannot be found in the context,
say that the information is unavailable.

5. Is the generated answer grounded?

The final answer should be supported by the retrieved evidence.

This gives you a much better interview answer than simply saying:

“I’d improve the prompt.”

3. Attention & KV Cache

If you mention LLMs in your resume, you should understand attention.

You don’t necessarily need to derive every Transformer equation during an interview.

But you should understand the intuition.

The Problem With Sequential Models

RNNs process sequences sequentially.

For:

The cat sat on the mat because it was tired.

the model processes tokens step by step.

This makes it difficult to efficiently capture long-range relationships and limits parallelization during training.

Transformers introduced a different idea:

Let tokens directly interact with other tokens through attention.

Self-Attention

For every token, we generate:

Query
Key
Value

The core attention operation is:$$Attention(Q,K,V) = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V$$

The intuition:

• Query: What information am I looking for?
• Key: What information do I contain?
• Value: What information should I provide?

For example:

"The animal didn't cross the road because it was tired."

When processing:

"it"

attention allows the model to determine which other tokens are relevant to understanding “it.”

Then Comes KV Cache

This is one of my favorite AI Engineer interview topics.

During autoregressive generation, the model generates:

Token 1
Token 2
Token 3
Token 4
...

Without caching, previously computed attention information would need to be recomputed repeatedly.

KV caching stores previously calculated:

Key
Value

representations.

Conceptually:

Prompt
  ↓
Transformer
  ↓
K,V
  ↓
KV Cache
  │
  ├── Token 1
  ├── Token 2
  ├── Token 3
  └── Token 4

When the next token is generated, the model can reuse the cached K/V information.

Why Does This Matter?

Because production LLM inference isn’t just about model accuracy.

You care about:

• latency
• throughput
• GPU memory
• cost
• concurrency

KV cache can substantially reduce redundant computation during autoregressive decoding, but it isn’t free.

The cache grows with sequence length and consumes GPU memory.

So you get an important engineering trade-off:

Longer Context
      ↓
More KV Cache
      ↓
Higher Memory Consumption

This is why someone designing an LLM inference system needs to understand more than just Transformer architecture.

4. Fine-Tuning & LoRA

Another common interview question:

“When would you fine-tune an LLM instead of using RAG?”

A useful way to think about it is:

Use prompting when:

The model already knows how to perform the task.

You simply need to provide:

• instructions
• examples
• constraints
• output format

Use RAG when:

The model needs access to external or changing knowledge.

For example:

Company policies
Product documentation
Legal documents
Internal knowledge
Current information

Consider fine-tuning when:

You want to change the model’s behavior or specialize it for a task.

For example:

Classification style
Domain-specific language patterns
Output behavior
Structured response patterns
Specialized task performance

A simplified decision tree:

                 Problem
                    │
        ┌───────────┼───────────┐
        ↓           ↓           ↓
   Instructions  Knowledge    Behavior
        │           │           │
        ↓           ↓           ↓
    Prompting       RAG       Fine-tuning

These aren’t mutually exclusive.

Production systems can use all three.

What Is LoRA?

Full fine-tuning updates the model’s parameters.

If your model has billions of parameters, that’s expensive.

LoRA—Low-Rank Adaptation—takes a different approach.

Instead of directly updating the original weight matrix:$$W$$

LoRA introduces a low-rank update:$$W’ = W + BA$$

where:

B = smaller matrix
A = smaller matrix

The original model weights can remain frozen while the smaller adapter matrices are trained.

Conceptually:

             Base Model
           ┌─────────────┐
Input ────►│ Frozen      │────► Output
           │ Weights     │
           └──────┬──────┘
                  │
                  ▼
            LoRA Adapter
             A → B

Why Is LoRA Cheaper?

Because you’re training far fewer parameters.

That means:

• lower GPU memory requirements
• fewer trainable parameters
• smaller checkpoints
• cheaper experimentation

But there’s a trade-off.

You aren’t simply getting “free fine-tuning.”

The adapter has limited capacity compared with updating all model parameters.

Catastrophic Forgetting

Another question interviewers may ask:

“You fine-tuned an LLM, but it became worse at general tasks. Why?”

One possible explanation is catastrophic forgetting.

The model learns the new training distribution so aggressively that performance on previously learned capabilities can degrade.

Conceptually:

Before fine-tuning
General Knowledge █████████
Task Knowledge    █████
After aggressive fine-tuning
General Knowledge █████
Task Knowledge    ███████████

Mitigation strategies can include:

• better dataset composition
• smaller learning rates
• fewer training epochs
• parameter-efficient fine-tuning
• mixing general-purpose data with domain data where appropriate
• evaluation against general-task benchmarks

The key interview point:

Fine-tuning isn’t automatically an improvement. You have to measure what you gained and what you lost.

5. Evaluation & Observability

This is perhaps the most underestimated skill.

Many candidates say:

“We evaluated our RAG system.”

Then the interviewer asks:

“How?”

And the answer becomes:

“We manually checked some responses.”

That isn’t enough for a production system.

AI Systems Need Multiple Layers of Evaluation

Consider a RAG application.

You can evaluate:

                 RAG Evaluation
                       │
        ┌──────────────┼──────────────┐
        ↓              ↓              ↓
    Retrieval      Generation     Production
        │              │              │
     Recall@K      Faithfulness    Latency
     Precision@K   Correctness     Cost
     MRR           Relevance       Errors
     nDCG          Completeness    User feedback

Retrieval Evaluation

Suppose the correct document is:

D42

and your system returns:

D1, D8, D42, D72, D91

Then:

Recall@5 = 1

because the relevant document appeared in the top five.

But you may also want to understand ranking quality.

That’s where metrics such as:

• MRR
• nDCG
• Precision@K

become useful.

Generation Evaluation

Now suppose retrieval is perfect.

The LLM receives:

Correct Context

but generates:

Incorrect Answer

That’s a generation problem.

You can evaluate dimensions such as:

Faithfulness

Is the answer supported by the provided context?

Correctness

Does the answer actually answer the question correctly?

Relevance

Does it answer what the user asked?

Completeness

Did it omit important information?

LLM-as-a-judge can help automate some of these evaluations, but it should itself be validated rather than blindly trusted.

Production Evaluation

Offline evaluation isn’t enough.

Imagine:

Offline Score = 95%

but production shows:

Latency = 4 seconds
Cost = $0.15/request
Error Rate = 8%
User Satisfaction = Low

Your model may look excellent in a benchmark and still be a bad production system.

That’s why you need observability.

Track things like:

Request
  ↓
Prompt
  ↓
Retrieval
  ↓
Retrieved Documents
  ↓
LLM
  ↓
Response
  ↓
Latency
  ↓
Cost
  ↓
User Feedback

Now when something breaks, you can determine where it broke.

How These Five Concepts Connect

This is the most important part.

These aren’t five isolated interview topics.

They form a production AI system.

Consider a RAG application:

                         User
                           │
                           ▼
                       Query
                           │
                           ▼
                    Query Embedding
                           │
                           ▼
                 Hybrid / Vector Search
                           │
                           ▼
                       Reranking
                           │
                           ▼
                        Context
                           │
                           ▼
                     Prompt + LLM
                           │
                           ▼
                     Token Generation
                           │
                     ┌─────┴─────┐
                     │           │
                 Attention    KV Cache
                     │           │
                     └─────┬─────┘
                           ▼
                       Response
                           │
                           ▼
                      Evaluation
                           │
                           ▼
                     Observability

And if the model needs specialized behavior:

Base LLM
   +
LoRA Adapter
   ↓
Specialized Model

Everything connects.

A Practical Interview Framework

When an interviewer gives you a production AI problem, don’t immediately jump to a technology.

Instead, ask yourself five questions.

1. What information does the system need?

This leads you toward:

Embeddings / Retrieval / RAG

2. What behavior does the model need to learn?

This leads you toward:

Prompting / Fine-tuning / LoRA

3. What happens during inference?

This leads you toward:

Attention / KV Cache / Quantization / Batching

4. How do I know it works?

This leads you toward:

Evaluation

5. How do I know it keeps working?

This leads you toward:

Observability / Monitoring / Regression testing

That thought process is far more valuable than memorizing another framework API.

Common Interview Mistakes

Mistake #1: Starting With Tools

Bad:

“I’d use LangChain.”

Better:

“First I’d identify whether the problem is retrieval, generation, or orchestration. Then I’d choose the appropriate components.”

Mistake #2: Treating RAG as a Hallucination Cure

RAG can provide evidence.

It doesn’t guarantee that the model will use that evidence correctly.

Mistake #3: Saying “Accuracy”

Ask:

Accuracy of what?

Retrieval?

Generation?

Task completion?

Classification?

Tool selection?

Production behavior?

Define the evaluation target first.

Mistake #4: Knowing LoRA but Not Why It Exists

Don’t just memorize:

“LoRA is parameter-efficient fine-tuning.”

Understand the problem:

Full Fine-tuning
       ↓
Huge number of trainable parameters
       ↓
High memory + compute cost
LoRA
       ↓
Small trainable low-rank matrices
       ↓
Lower training cost

Mistake #5: Ignoring Production Constraints

An AI system isn’t finished when the answer looks good.

You also need to consider:

• latency
• cost
• scalability
• reliability
• security
• monitoring
• evaluation

Production Perspective

If I had to compress modern AI Engineering into one diagram, it would look something like this:

                    ┌──────────────────┐
                    │     User Query   │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Query Processing │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │    Retrieval     │
                    │ Embeddings/BM25  │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │    Reranking     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │      LLM         │
                    │ Attention + KV   │
                    │      Cache       │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │     Response     │
                    └────────┬─────────┘
                             │
              ┌──────────────┴──────────────┐
              ▼                             ▼
       ┌──────────────┐              ┌──────────────┐
       │  Evaluation  │              │ Observability│
       └──────────────┘              └──────────────┘

This is what separates building a demo from engineering an AI system.

Interview Questions You Should Be Able to Answer

If you understand the five concepts above, try answering these without looking anything up.

Embeddings & Retrieval

1. Why can semantic search retrieve irrelevant documents?
2. When would BM25 outperform vector search?
3. Why use hybrid retrieval?
4. What does Recall@K measure?
5. Why use a reranker after vector retrieval?

RAG

6. Why can a RAG system hallucinate even when retrieval is correct?
7. How would you debug a RAG pipeline?
8. How would you choose chunk size?
9. When would you use parent-child retrieval?
10. How would you evaluate context relevance?

LLM Inference

11. Why does self-attention outperform RNNs for many sequence modeling workloads?
12. What is KV caching?
13. Why does KV cache increase memory consumption?
14. What is the difference between prefill and decode?
15. How would you reduce LLM inference latency?

Fine-tuning

16. When would you choose fine-tuning over RAG?
17. What problem does LoRA solve?
18. What is catastrophic forgetting?
19. How would you debug a fine-tuned model that performs worse than the base model?
20. How would you reproduce a fine-tuning run six months later?

Evaluation

21. Why isn’t one accuracy number enough for an AI application?
22. What is LLM-as-a-judge?
23. How would you validate an LLM judge?
24. What production metrics would you monitor?
25. How would you determine whether a new model actually improved the system?

If you can answer these questions with reasoning rather than definitions, you’re in a much stronger position.

Key Takeaways

You don’t need to know every AI framework before an interview.

You need to understand the fundamentals underneath them.

The five concepts I’d prioritize are:

1. Embeddings & Retrieval

Understand how information becomes searchable and why semantic similarity isn’t the same as relevance.

2. RAG & Hallucinations

Understand the complete retrieval → context → generation pipeline and where it can fail.

3. Attention & KV Cache

Understand how Transformers process context and how inference is optimized.

4. Fine-tuning & LoRA

Understand when changing model behavior makes sense and the trade-offs involved.

5. Evaluation & Observability

Understand how to prove that an AI system actually works—and continues to work in production.

The bigger lesson is this:

Don’t prepare for AI Engineer interviews by memorizing more frameworks. Prepare by understanding why the systems work.

Frameworks will change.

The fundamentals won’t.

Related Tutorials

RAG Fundamentals

• What Is RAG?
• How RAG Works
• Why LLMs Hallucinate
• Dense vs Sparse Retrieval
• BM25 Explained
• Hybrid Search: BM25 + Vector Search(You are here)
• Cross-Encoder Reranking
• Parent-Child Retrieval
• Multi-Query Retrieval

Next Tutorial

← Previous

Multi-Query Retrieval: How to Improve RAG Search Recall

You are here:

AI Engineer Interview Preparation → Core AI Engineering Concepts

Next →

25 AI Engineer Interview Questions That Test Whether You Actually Understand LLMs

Author

Ved Prakash

Data Scientist | AI Engineer | Generative AI

Writing about Generative AI, RAG, LLMs, Machine Learning, Data Engineering, and production AI systems.