After interviewing 50+ AI Engineer candidates, I noticed a pattern: impressive GenAI projects can get you through the first 10 minutes—but strong fundamentals are what separate candidates who build AI systems from those who truly understand them.

I’ve interviewed 50+ AI Engineer candidates over the past few months.

Almost every resume looked impressive.

RAG.

AI Agents.

MCP.

Multi-agent workflows.

LangGraph.

Production chatbots.

The projects sounded great.

The technology stacks looked modern.

The GitHub repositories were full of impressive keywords.

But after about 10 minutes, I usually changed the conversation.

I stopped asking:

“Have you built a RAG application?”

And started asking:

“Why does self-attention outperform RNNs?”

Or:

“Why does KV caching reduce LLM inference latency?”

Or:

“Why can two embedding models retrieve completely different documents for the same query?”

Or:

“When would you choose RAG over fine-tuning?”

And one of my favorites:

“Why can an LLM hallucinate even when your RAG pipeline retrieves the correct documents?”

That’s where things got interesting.

Many candidates knew how to build AI applications.

Far fewer could explain why those applications worked.

And that distinction matters.

Because frameworks change.

Libraries change.

Model APIs change.

Today’s popular architecture may be replaced by something completely different two years from now.

But the fundamentals?

They stay with you.

If you’re preparing for an AI Engineer, GenAI Engineer, or LLM Engineer interview, this is the roadmap I would recommend.

The AI Engineer Learning Roadmap

I would divide the journey into five stages:

                    AI Engineer
                         │
                         ▼
              1. ML Fundamentals
                         │
                         ▼
              2. Deep Learning
                         │
                         ▼
                 3. LLM Fundamentals
                         │
                         ▼
                4. AI Engineering
                         │
                         ▼
             5. Production Engineering

The mistake I see most often?

Candidates jump directly from:

"I know Python"
       ↓
"Let's build a RAG chatbot"

The strongest candidates build the foundation first.

Let’s break it down.

Step 1: Build Strong Machine Learning Fundamentals

Before learning LLMs, you should understand the fundamentals of machine learning.

Not necessarily at the level of becoming a research scientist.

But you should be comfortable answering questions such as:

• What is overfitting?
• What is bias-variance tradeoff?
• How does gradient descent work?
• What is regularization?
• Why do we need train/validation/test splits?
• What is data leakage?
• How do you handle class imbalance?
• What is cross-validation?
• How do you select evaluation metrics?

You should also understand the mathematical foundations.

Linear Algebra

Focus on:

• Vectors
• Matrices
• Matrix multiplication
• Dot products
• Eigenvalues and eigenvectors
• Vector spaces
• Cosine similarity

Why does this matter?

Because modern AI systems are built around vectors.

Embeddings are vectors.

Attention uses matrix operations.

Neural networks perform transformations using matrices.

Vector databases perform similarity search.

If you understand the underlying mathematics, concepts like embeddings and attention become much easier to reason about.

Probability and Statistics

You should understand:

• Probability distributions
• Conditional probability
• Bayes’ theorem
• Expectation
• Variance
• Mean
• Standard deviation
• Sampling
• Hypothesis testing

This becomes especially useful when thinking about:

• Model uncertainty
• Evaluation
• A/B testing
• Sampling strategies
• Probabilistic outputs

Machine Learning

You should be comfortable with:

• Linear regression
• Logistic regression
• Decision trees
• Random forests
• Gradient boosting
• XGBoost
• Clustering
• Feature engineering
• Model evaluation

You don’t necessarily need to use every algorithm in an AI Engineer role.

But you should understand why models behave the way they do.

Step 2: Master Deep Learning

Once your ML fundamentals are strong, move to deep learning.

Start with:

Neural Networks
      ↓
Backpropagation
      ↓
CNNs
      ↓
RNNs
      ↓
LSTMs / GRUs
      ↓
Attention
      ↓
Transformers

This progression is important.

Don’t jump directly into:

“Transformers are all you need.”

Understand what came before them.

Neural Networks

Know:

• Forward propagation
• Backpropagation
• Activation functions
• Loss functions
• Gradient descent
• Optimizers

You should be able to explain what happens when a neural network makes a prediction and how its weights are updated.

CNNs

Understand:

• Convolution
• Filters
• Pooling
• Feature maps
• Receptive fields

CNNs help you understand how neural networks can learn hierarchical representations.

RNNs

RNNs introduce an important concept:

sequence modeling.

You should understand:

• Hidden states
• Sequential processing
• Vanishing gradients
• Exploding gradients
• LSTMs
• GRUs

This leads naturally to the question:

If sequences are processed one step at a time, how can we efficiently model long-range dependencies?

And that’s where attention becomes extremely important.

Step 3: Understand LLMs Deeply

Now we reach the part most candidates want to start with.

But by this point, the concepts become much easier.

You should understand the full LLM pipeline:

Text
 ↓
Tokenization
 ↓
Token IDs
 ↓
Embeddings
 ↓
Positional Information
 ↓
Transformer Layers
 ↓
Attention
 ↓
Feed-Forward Networks
 ↓
Logits
 ↓
Sampling
 ↓
Generated Tokens

You should be able to explain every stage.

Tokenization

Understand:

• What tokens are
• BPE
• WordPiece
• SentencePiece
• Why one word can become multiple tokens
• Why tokenization affects cost

A question I often ask is:

“Why doesn’t the number of words equal the number of tokens?”

If you can’t answer this, you probably need to spend more time understanding tokenization.

Embeddings

Understand:

• What embeddings represent
• Dense vectors
• Semantic similarity
• Cosine similarity
• Embedding dimensions
• Embedding model objectives

A particularly important interview question is:

Why can two embedding models retrieve different documents for the same query?

Because embeddings aren’t universal representations.

Different models are trained with different objectives, datasets, architectures, and optimization strategies.

The embedding space reflects those training choices.

This becomes extremely important when designing RAG systems.

Positional Encoding

Transformers process tokens in parallel.

But language is sequential.

The model therefore needs information about token positions.

Understand:

• Why positional information is needed
• Absolute positional embeddings
• Relative position representations
• Rotary Position Embeddings (RoPE)

You should be able to explain why:

"Dog bites man"

is different from:

"Man bites dog"

even though both contain the same tokens.

Attention and Transformers

This is one of the most important areas for AI Engineer interviews.

You should understand:

Query
Key
Value
   ↓
Attention Scores
   ↓
Softmax
   ↓
Weighted Values

Know the intuition behind:

Attention(Q, K, V)
=
softmax(QKᵀ / √dₖ)V

You don’t need to memorize the equation without understanding it.

You should be able to explain what Q, K, and V represent.

And you should understand:

Why did attention-based architectures become so successful compared with recurrent architectures?

Think about:

• Parallel computation
• Long-range dependencies
• Better scalability
• Efficient hardware utilization

These are the fundamentals behind today’s LLMs.

LLM Inference

A surprisingly common interview topic is inference.

You should understand:

• Prefill
• Decode
• KV cache
• Sampling
• Temperature
• Top-K
• Top-P
• Quantization
• Speculative decoding
• Batching

One question I frequently find useful is:

Why does KV caching reduce inference latency?

If the model had to recompute all previous keys and values every time it generated a new token, generation would become extremely inefficient.

KV caching allows the model to reuse previously computed attention information.

This is the kind of question that separates someone who has used an API from someone who understands LLM inference.

Fine-Tuning vs. RAG vs. Prompt Engineering

This is another essential area.

If an interviewer asks:

“When would you use RAG instead of fine-tuning?”

You should have a clear framework.

Prompt Engineering

Use when:

• The model already has the required knowledge
• You need to improve instructions
• You need to control response style
• You want the cheapest and fastest iteration

RAG

Use when:

• Knowledge changes frequently
• You need external or private information
• You need source attribution
• You want to update knowledge without retraining the model

Fine-Tuning

Use when:

• You need to change model behavior
• You need consistent output style
• You need domain-specific patterns
• You need specialized task performance

The key is understanding that these technologies solve different problems.

Step 4: Learn AI Engineering

Now you are ready for the technologies that appear on most modern AI Engineer resumes.

This is where RAG, agents, LangGraph, MCP, and vector databases come in.

But now you’re learning them from a foundation.

Not just copying tutorials.

RAG

Understand the complete pipeline:

Documents
    ↓
Parsing
    ↓
Chunking
    ↓
Embedding
    ↓
Vector Database
    ↓
Retrieval
    ↓
Reranking
    ↓
Context Construction
    ↓
LLM
    ↓
Grounded Answer

You should understand every stage.

Interviewers may ask:

“Your RAG system retrieves the correct documents, but the LLM still hallucinates. Why?”

Possible causes include:

• Retrieved context is irrelevant despite appearing semantically similar
• Too much context creates distraction
• The model ignores retrieved information
• Prompt design is weak
• Context contains conflicting information
• Retrieval ranking is poor
• The model lacks sufficient grounding constraints

RAG isn’t simply:

Vector DB + LLM

It’s a complete information retrieval and generation system.

Chunking

Understand:

• Fixed-size chunking
• Recursive chunking
• Semantic chunking
• Heading-aware chunking
• Clause-aware chunking

Ask yourself:

What happens when a chunk is too large?

You may retrieve unnecessary information and increase context size.

What happens when it’s too small?

You may lose the context needed to understand the information.

The right answer depends on the data.

A legal contract may require clause-aware chunking.

A technical document may benefit from heading-aware chunking.

A simple FAQ may work with smaller semantic units.

There is no universal chunk size.

Hybrid Search

Don’t assume vector search is always superior.

Sometimes keywords matter.

For example:

"ICD-10 code E11.9"

or:

"Policy ID ABC-12345"

Exact identifiers can be better handled by lexical retrieval.

That’s why production RAG systems often combine:

BM25
+
Dense Vector Search
+
Reranking

The strongest AI Engineers understand both:

Information Retrieval

and

Generative AI.

AI Agents

Understand:

• Tool calling
• Planning
• Memory
• State management
• Reflection
• Human-in-the-loop
• Agent orchestration

But don’t just learn:

“How to create an agent.”

Understand:

When should you use an agent at all?

If a deterministic workflow can solve the problem, an autonomous agent may add unnecessary complexity.

That’s an important production engineering decision.

MCP

Understand the basic concepts behind:

• Tools
• Resources
• Prompts
• MCP clients
• MCP servers

But also understand the security implications.

For example:

What happens if a malicious instruction enters the context and causes an agent to invoke a powerful tool?

AI Engineering isn’t only about making agents more capable.

It’s about making them safe.

Evaluation

This is where many AI Engineer candidates are surprisingly weak.

If I ask:

“How do you know your RAG system is improving?”

“Users liked it” isn’t enough.

You should think about:

Retrieval Metrics
      ↓
Generation Metrics
      ↓
End-to-End Evaluation
      ↓
Human Evaluation
      ↓
Production Monitoring

Depending on the application, you might measure:

• Retrieval precision
• Recall
• Context relevance
• Faithfulness
• Answer correctness
• Citation accuracy
• Latency
• Cost
• User satisfaction

Build golden datasets.

Create regression tests.

Track changes across model and prompt versions.

An AI system needs continuous evaluation.

Guardrails and Observability

Production AI systems need controls.

Think about:

• Input validation
• Output validation
• PII detection
• Prompt injection detection
• Toxicity filtering
• Tool permission controls
• Human approval
• Rate limits

Then add observability.

Monitor:

Latency
Token Usage
Cost
Errors
Hallucination Rate
Retrieval Quality
Tool Failures
User Feedback

If you can’t observe your AI system, you can’t reliably improve it.

Step 5: Think Like a Production Engineer

This is where I see the strongest candidates stand out.

They don’t just ask:

“Does the model work?”

They ask:

“Can this system work reliably at scale?”

Imagine your LLM application suddenly has:

10× the traffic.

What happens?

Or your inference bill increases from:

$2,000/day

to:

$35,000/day

What do you do?

Or latency increases from:

300 ms

to:

5 seconds

How do you debug it?

Production AI Engineering requires thinking about:

• Scalability
• Reliability
• Latency
• Cost
• Security
• Observability
• Reproducibility

The Interview Questions I Would Prepare For

If I were preparing for an AI Engineer interview today, I’d make sure I could answer questions like:

Fundamentals

Why does self-attention outperform RNNs for many sequence modeling tasks?

Why do Transformers need positional information?

What happens during backpropagation?

LLMs

Why does KV caching improve autoregressive generation?

What is the difference between prefill and decode?

What happens when you increase the context window?

How does quantization reduce memory requirements?

RAG

Why is your RAG system hallucinating despite retrieving relevant documents?

How would you choose chunk size?

When would BM25 outperform vector search?

How would you improve retrieval quality?

Fine-Tuning

When would you fine-tune instead of using RAG?

What is LoRA?

What is catastrophic forgetting?

How would you evaluate a fine-tuned model?

Agents

When should you use an agent instead of a deterministic workflow?

How do you prevent recursive agent loops?

How do you secure tool calling?

Production

How would you reduce LLM inference cost?

How would you reduce latency?

How would you monitor hallucinations?

How would you reproduce an AI model issue from six months ago?

These questions aren’t testing whether you memorized a framework’s API.

They’re testing whether you understand the system.

The Biggest Mistake I See

The most common learning path looks like this:

Python
   ↓
LangChain
   ↓
RAG
   ↓
LangGraph
   ↓
Agents
   ↓
MCP

The problem?

There is no foundation.

A stronger path looks like:

ML Fundamentals
        ↓
Deep Learning
        ↓
RNNs + Attention
        ↓
Transformers
        ↓
LLMs
        ↓
Inference
        ↓
RAG
        ↓
Agents
        ↓
Production AI

The second path takes longer.

But it creates engineers who can adapt.

When a new framework appears, they don’t panic.

They learn it.

Because they understand the underlying concepts.

The Real Difference Between Candidates

After interviewing dozens of candidates, one pattern has become clear to me.

There are candidates who can say:

“I built a RAG chatbot using LangChain and Pinecone.”

And there are candidates who can explain:

“We chose hybrid retrieval because our documents contained both natural-language descriptions and exact policy identifiers. We used dense embeddings for semantic retrieval, BM25 for lexical matching, and reranking to improve the final context. We evaluated retrieval separately from generation because our initial hallucinations were actually caused by context selection rather than the LLM itself.”

The second candidate stands out.

Not because they know more frameworks.

Because they understand why the architecture exists.

My Recommended AI Engineer Roadmap

If I had to summarize everything into one roadmap, it would be:

                ┌──────────────────────┐
                │  ML Fundamentals     │
                │  Math + Statistics   │
                └──────────┬───────────┘
                           ↓
                ┌──────────────────────┐
                │  Deep Learning       │
                │  NN → CNN → RNN      │
                └──────────┬───────────┘
                           ↓
                ┌──────────────────────┐
                │  Transformers        │
                │  Attention + LLMs    │
                └──────────┬───────────┘
                           ↓
                ┌──────────────────────┐
                │  LLM Engineering     │
                │  RAG + Fine-tuning   │
                └──────────┬───────────┘
                           ↓
                ┌──────────────────────┐
                │  AI Engineering      │
                │ Agents + MCP + Eval  │
                └──────────┬───────────┘
                           ↓
                ┌──────────────────────┐
                │ Production Systems   │
                │ Cost + Scale + SRE   │
                └──────────────────────┘

Final Takeaway

The AI industry moves incredibly fast.

New frameworks appear every few months.

New model architectures emerge.

New agent protocols become popular.

Today’s “must-have” tool may be irrelevant a few years from now.

But the fundamentals remain.

Linear algebra.

Probability.

Machine learning.

Deep learning.

Attention.

Transformers.

LLM inference.

Information retrieval.

Distributed systems.

These are the concepts that allow you to understand the next generation of AI systems—not just the current one.

So if you’re preparing for an AI Engineer or GenAI interview, don’t just learn how to build a RAG pipeline.

Learn why retrieval works.

Don’t just learn how to call an LLM API.

Learn what happens during inference.

Don’t just learn how to create an agent.

Learn when an agent is actually necessary.

Don’t just learn how to deploy an AI application.

Learn how to make it reliable, observable, secure, and cost-efficient.

Most candidates jump straight to Step 4.

The strongest candidates build from Step 1.

And in my experience, that’s the difference interviewers notice.

Follow me on medium

Read all Data Engineering Tutorials here