A practical AI/ML and GenAI interview preparation guide covering RAG, embeddings, Knowledge Graphs, LLM evaluation, system design, optimization, monitoring, and DSA

AI/ML interviews are changing.

Knowing definitions like “What is an embedding?” or “What is overfitting?” is no longer enough for senior AI/ML and GenAI roles.

A recent Associate Architect (ML) interview experience at Quantiphi is a good example.

The interview started with fundamentals but quickly moved into architecture and production-oriented questions:

Semantic vs. keyword search
Embedding models and chunking
Regularization mathematics
RAG and hybrid retrieval
Knowledge Graph construction
LLM-as-a-Judge
Legal-document processing
Microservice optimization
ML/GenAI observability
NL-to-SQL at scale
Deploying a 70B LLM
And even DSA

What makes these questions interesting is that most of them don’t have a single “correct” answer.

The interviewer wants to understand how you reason about trade-offs.

This article breaks down the interview into its major themes and provides an interview-ready framework for answering each question.

Note: The questions below are based on the interview experience provided. The explanations are preparation guidance rather than claims about Quantiphi’s official interview process.

What This Interview Actually Tested
Round 1 — ML + GenAI Fundamentals
Project-Based Questions
Semantic Search vs Keyword Search
Embeddings and Chunk Size
Dense vs Sparse Vectors
Overfitting, Underfitting and Regularization
DSA Question — Fibonacci
Round 2 & 3 — RAG and System Design
Evaluating LLM-as-a-Judge
Designing an Ingestion Pipeline
Knowledge Graphs
Knowledge Graph Construction
Cost of Knowledge Graphs
Legal Document RAG
Microservice Optimization
Improving Semantic and Hybrid Retrieval
Offline vs Online Monitoring
NL-to-SQL at Scale
Cold Start With a 70B LLM
How to Approach These Questions
Common Mistakes
Interview Preparation Checklist
Key Takeaways
Related Tutorials
Next Tutorial

1. What This Interview Actually Tested

At first glance, the interview looks like a collection of unrelated questions.

But there’s a common pattern.

The interviewer is testing five major capabilities:

                AI/ML Engineer
                     │
       ┌─────────────┼─────────────┐
       ↓             ↓             ↓
   Fundamentals   Architecture   Production
       │             │             │
       └─────────────┼─────────────┘
                     ↓
                 Reasoning
                     ↓
              Trade-off Analysis

You need to demonstrate that you can move between different abstraction levels.

For example:

			
Embedding
   ↓
Retriever
   ↓
RAG pipeline
   ↓
Microservices
   ↓
Production system
   ↓
Monitoring

		

That’s very different from simply knowing what an embedding is.

Round 1 — ML + GenAI Fundamentals

The first round covered:

Project experience
Search
Embeddings
Chunking
ML fundamentals
Regularization
DSA

Let’s go through the major questions.

2. “Tell Me About Your Project”

This sounds easy.

It isn’t.

For senior roles, interviewers often use your project as a gateway to deeper technical questions.

A weak answer sounds like:

“We built a RAG chatbot using LangChain and Pinecone.”

A stronger answer explains:

			
Business Problem
      ↓
Data
      ↓
Ingestion
      ↓
Chunking
      ↓
Embedding
      ↓
Retrieval
      ↓
Reranking
      ↓
LLM
      ↓
Evaluation
      ↓
Production Monitoring

		

Then be prepared for:

Why did you choose this chunking strategy?

Why this embedding model?

Why vector search?

How did you evaluate retrieval?

What happened when retrieval failed?

How did you control hallucinations?

For senior interviews, your project architecture is effectively an open-book exam.

3. Semantic Search vs Keyword Search

This was one of the questions mentioned in the interview.

Keyword Search

Keyword search looks for lexical overlap.

For example:

			
Query:
"How to fix CUDA out of memory?"

A keyword search system looks for terms such as:

			
CUDA
out
of
memory

BM25 is a common approach.

Semantic Search

Semantic search uses embeddings.

The query becomes a vector:

			
Query
 ↓
Embedding Model
 ↓
[0.12, -0.42, 0.81, ...]

		

Documents are also converted into vectors.

Then we retrieve semantically similar documents.

This allows:

"How do I reduce GPU memory usage?"

to potentially retrieve:

"Techniques for preventing CUDA memory exhaustion"

even though the wording differs.

So Which One Should You Use?

Don’t answer:

“Vector search is better.”

Instead:

“They solve different problems. Keyword search is strong for exact identifiers, technical terms, error messages, and legal clauses, while semantic search is strong when the query and document use different wording. In production RAG, I’d benchmark both and consider hybrid retrieval when both signals are important.”

That’s a much stronger answer.

4. How Do Embedding Models Affect Chunk Size?

This is a subtle question.

Many candidates treat chunk size as completely independent from the embedding model.

It isn’t.

Different embedding models have different:

context limits
embedding dimensions
training objectives
semantic representations
domain strengths

Suppose your documents contain highly technical content.

A generic embedding model might produce weaker representations than a model optimized for technical or retrieval-oriented data.

Similarly, if your chunks are extremely large, a single vector has to represent many concepts.

Consider:

			
Large Chunk
Insurance
 ├── Eligibility
 ├── Premiums
 ├── Claims
 ├── Exclusions
 └── Renewals

		

One embedding now represents multiple topics.

The retriever may struggle to determine which part is relevant.

With smaller, semantically coherent chunks:

			
Chunk 1 → Eligibility
Chunk 2 → Premium
Chunk 3 → Claims
Chunk 4 → Exclusions

retrieval becomes more targeted.

5. How Would You Select an Embedding Model?

Don’t select an embedding model solely because it has a high benchmark score.

I’d evaluate:

1. Retrieval quality

Measure:

Recall@K
Precision@K
MRR
nDCG

2. Domain relevance

Does it work with:

			
Legal documents?
Healthcare?
Financial documents?
Technical documentation?

3. Context length

Can it handle the chunk sizes required by your application?

4. Latency

Embedding generation may become a bottleneck during ingestion.

5. Cost

Especially important when embedding millions of documents.

6. Infrastructure

Consider:

API-based model
self-hosted model
GPU requirements
deployment complexity

The final decision should come from evaluation on your own dataset.

6. Which Vector Is Longer — Dense or Sparse?

This is a classic conceptual question.

Suppose the vocabulary contains:

V = 1,000,000 words

A sparse representation could conceptually have:

1,000,000 dimensions

but most values are zero.

For example:

[0, 0, 0, 0, 0.7, 0, 0, ...]

A dense embedding might have:

768 dimensions

with most values non-zero:

[0.21, -0.34, 0.17, 0.52, ...]

Therefore:

Sparse vectors are typically much higher-dimensional, while dense vectors are lower-dimensional and information-dense.

The important distinction is not merely length.

It’s also how information is represented.

7. Overfitting vs Underfitting

The interview also tested classical ML fundamentals.

Underfitting

The model is too simple to capture the underlying pattern.

			
High training error
High validation error

Overfitting

The model memorizes training data rather than learning generalizable patterns.

			
Low training error
High validation error

Conceptually:

			
Underfitting → Good training? No
              Poor generalization
Good fit     → Good training
              Good generalization
Overfitting  → Excellent training
              Poor generalization

		

8. The Mathematics Behind Regularization

This is where senior candidates should go beyond definitions.

Without regularization: $J(\theta)=Loss(\theta)$

With L2 regularization: $J(\theta)=Loss(\theta)+\lambda\sum_i\theta_i^2$

The additional term penalizes large weights.

For L1: $J(\theta)=Loss(\theta)+\lambda\sum_i|\theta_i|$

The difference matters.

L1

Encourages sparsity.

L2

Penalizes large weights and generally encourages smaller, distributed weights.

The important interview point is:

Regularization introduces a trade-off between fitting the training data and constraining model complexity.

9. DSA: Fibonacci Question

The interview reportedly included a DSA question:

Given a number, append all its Fibonacci numbers to a list, along with relevant test cases.

The key isn’t just writing the code.

Explain your assumptions first.

For example:

			
def fibonacci_upto(n):
    result = []
    a, b = 0, 1
    while a <= n:
        result.append(a)
        a, b = b, a + b
    return result

		

Test cases:

			
assert fibonacci_upto(0) == [0]
assert fibonacci_upto(1) == [0, 1]
assert fibonacci_upto(10) == [
    0, 1, 1, 2, 3, 5, 8
]
assert fibonacci_upto(20) == [
    0, 1, 1, 2, 3, 5, 8, 13
]

		

Also discuss complexity.

The number of generated Fibonacci values is approximately logarithmic in n, so the algorithm is effectively: $O(k)$

where k is the number of Fibonacci numbers generated.

Round 2 & 3 — RAG + Knowledge Graphs + System Design

This is where the interview becomes much more architecture-heavy.

10. How Do You Evaluate an LLM-as-a-Judge?

This is a particularly good interview question.

Suppose an LLM evaluates another LLM.

Who evaluates the evaluator?

That’s the problem.

You need a judge evaluation dataset.

For example:

			
Question
Expected Answer
Candidate Answer
Human Score
Judge Score

		

Then compare:

			
Human Evaluation
       ↕
LLM Judge

Measure:

agreement
correlation
consistency
false positives
false negatives

You can also test the judge for:

position bias
verbosity bias
model preference
prompt sensitivity
inconsistent scoring

A good answer should mention that LLM-as-a-Judge itself requires validation.

11. How Would You Improve the Judge?

Possible techniques include:

Better rubric

Instead of:

“Is the answer good?”

define:

			
Correctness: 0–5
Relevance: 0–5
Groundedness: 0–5
Completeness: 0–5

Structured output

Force the judge to produce:

			
{
  "correctness": 4,
  "relevance": 5,
  "groundedness": 3,
  "reason": "..."
}

		

Reference answers

Give the judge high-quality reference answers.

Calibration

Compare judge scores against human evaluations.

Multiple judges

For high-stakes evaluation, multiple independent evaluations can reduce dependence on one judge.

12. What Challenges Can Occur in an Ingestion Pipeline?

A production RAG system can fail before retrieval even begins.

Consider:

			
Documents
    ↓
Extraction
    ↓
Cleaning
    ↓
Chunking
    ↓
Metadata
    ↓
Embedding
    ↓
Vector DB

		

Potential failures include:

corrupted PDFs
scanned documents
OCR errors
tables
duplicated documents
malformed text
missing metadata
incorrect chunk boundaries
embedding failures
API rate limits
inconsistent document versions

A robust pipeline should have:

			
Validation
    ↓
Processing
    ↓
Quality Checks
    ↓
Indexing

		

and failures should be observable and retryable.

13. How Have You Used Knowledge Graphs?

A Knowledge Graph represents entities and relationships.

For example:

        Patient
           │
      diagnosed_with
           ↓
       Diabetes
           │
       treated_by
           ↓
       Metformin

Instead of storing only text, we represent relationships explicitly.

This can be particularly useful when the question requires multi-hop reasoning.

For example:

“Which medications are associated with diseases treated by a particular specialist?”

A graph can traverse:

			
Specialist
   ↓
Treats
   ↓
Disease
   ↓
Medication

		

14. How Does Knowledge Graph Construction Work?

A simplified pipeline:

			
Documents
    ↓
Document Processing
    ↓
Entity Extraction
    ↓
Relation Extraction
    ↓
Entity Resolution
    ↓
Graph Construction
    ↓
Graph Database

		

Suppose a document says:

“Metformin is commonly used to treat type 2 diabetes.”

We might extract:

			
Entity 1:
Metformin
Relation:
treats
Entity 2:
Type 2 Diabetes

		

and construct:

(Metformin) ──treats──> (Type 2 Diabetes)

The difficult part is that extraction isn’t always perfect.

You also need to deal with:

duplicate entities
ambiguous entities
conflicting relationships
ontology design
schema evolution
provenance

15. What Are the Costs of Knowledge Graph Construction?

Knowledge Graphs can become expensive because construction isn’t simply:

PDF → Graph

You may need:

			
Extraction
+
Entity Resolution
+
Relation Extraction
+
Validation
+
Storage
+
Maintenance

		

Large document collections can create significant processing costs.

There is also a maintenance problem.

If the underlying documents change:

			
Source Document
      ↓
Graph

the graph needs to remain synchronized.

Therefore, before building a Knowledge Graph, ask:

Does the business problem actually require graph relationships?

Don’t introduce a graph merely because it’s technically interesting.

16. How Would You Handle Legal Documents End-to-End?

This is a very practical RAG architecture question.

Legal documents are difficult because they contain:

sections
subsections
clauses
tables
footnotes
references
appendices
definitions

A naive:

PDF → fixed-size chunks

pipeline can destroy important structure.

A better architecture is:

			
Legal PDF
   ↓
Layout-aware extraction
   ↓
Section detection
   ↓
Clause identification
   ↓
Table extraction
   ↓
Semantic chunking
   ↓
Metadata enrichment
   ↓
Embedding
   ↓
Hybrid Retrieval
   ↓
Reranking
   ↓
LLM

		

Metadata might include:

			
document_id
document_type
section
subsection
clause
page_number
effective_date
version

		

This metadata becomes extremely valuable during retrieval.

17. Apart From Horizontal Scaling, How Do You Optimize Microservices?

A weak answer:

“Add more instances.”

That’s horizontal scaling.

A stronger architecture discussion includes:

Caching

Cache:

embeddings
retrieval results
repeated LLM requests

Asynchronous processing

Move expensive operations out of synchronous request paths.

Connection pooling

Reuse database and HTTP connections.

Batch processing

Batch embedding or inference requests.

Model optimization

Consider:

quantization
distillation
smaller models
optimized runtimes

Parallel execution

Run independent operations concurrently.

For example:

             Query
          /    |    \
         ↓     ↓     ↓
      Search  Auth  Metadata
         \     |     /
          \    |    /
             Merge

Payload optimization

Avoid sending unnecessary data between services.

Observability

Measure:

			
Latency
CPU
Memory
GPU
Errors
Throughput
Queue depth

		

18. Challenges With Semantic and Hybrid Retrieval

Semantic retrieval can fail because:

embeddings don’t capture domain terminology
chunks are poorly constructed
query intent is ambiguous
relevant documents have weak semantic similarity

Hybrid retrieval introduces additional complexity.

Now you have:

			
BM25
+
Vector Search

and need to combine the results.

Possible strategies:

Weighted score fusion

$S = \alpha S_{vector}+(1-\alpha)S_{BM25}$

Reciprocal Rank Fusion

Combine rankings rather than raw scores.

Reranking

Retrieve candidates first:

			
BM25 → Top 20
Vector → Top 20

then:

			
Merge
 ↓
Reranker
 ↓
Top 5

		

Again, don’t assume hybrid is automatically better.

Evaluate it.

19. How Would You Monitor an ML/GenAI System?

This is one of the most important production questions.

You need both offline and online evaluation.

Offline Evaluation

Before deployment:

			
Golden Dataset
      ↓
Retrieval Evaluation
      ↓
LLM Evaluation
      ↓
Regression Tests

		

Metrics can include:

			
Recall@K
Precision@K
MRR
nDCG
Faithfulness
Answer correctness
Context relevance

		

Online Monitoring

After deployment:

			
User Requests
      ↓
Production System
      ↓
Telemetry

		

Monitor:

Reliability

error rate
timeout rate
failed requests

Performance

p50 latency
p95 latency
p99 latency

Cost

tokens/request
cost/request
daily spend

Quality

user feedback
task completion
hallucination signals
escalation rate

Infrastructure

CPU
GPU
memory
queue depth

A production GenAI system needs all of these.

20. How Would You Build NL-to-SQL?

Imagine a database containing:

1,000 tables

A naive approach would send the entire schema to the LLM.

That’s expensive and can overwhelm the context window.

Instead:

			
User Question
      ↓
Intent Understanding
      ↓
Schema Retrieval
      ↓
Relevant Tables
      ↓
Relevant Columns
      ↓
SQL Generation
      ↓
SQL Validation
      ↓
Execution
      ↓
Result Validation

		

For example:

“What were our highest-value customers last quarter?”

The system first identifies relevant tables:

			
customers
orders
payments

instead of passing all 1,000 tables.

21. How Would You Scale NL-to-SQL?

A scalable architecture might use metadata retrieval.

Create embeddings or searchable metadata for:

			
Table descriptions
Column descriptions
Relationships
Business definitions
Sample queries

		

Then:

			
Question
   ↓
Schema Retriever
   ↓
Relevant Tables
   ↓
Relevant Columns
   ↓
LLM
   ↓
SQL

		

Then validate the generated SQL.

Important safeguards include:

read-only database access
SQL parser validation
table allowlists
query timeout
row limits
cost controls

Never blindly execute arbitrary LLM-generated SQL against a production database.

22. Cold Start With a 70B LLM

This is an infrastructure-heavy question.

A 70B model is expensive to load and serve.

Cold start can involve:

			
Container startup
      ↓
GPU allocation
      ↓
Model download
      ↓
Model loading
      ↓
Weight initialization
      ↓
KV cache allocation
      ↓
Ready

		

If this happens on demand, latency can become unacceptable.

Potential solutions include:

Warm instances

Keep model replicas running.

Model weight caching

Avoid downloading weights repeatedly.

Persistent infrastructure

Use long-lived GPU workers.

Quantization

Reduce memory requirements.

Smaller fallback model

Use:

			
Small Model
   ↓
Fast response

for simpler requests and route complex requests to the 70B model.

Autoscaling with minimum capacity

Keep a baseline number of warm replicas.

The right solution depends on:

traffic pattern
latency SLA
GPU availability
cost constraints

23. The Pattern Behind These Questions

Notice something.

The interview repeatedly moves from:

"What is X?"

toward:

"How would you build X?"

and finally:

"What happens when X fails at scale?"

That’s the progression you should prepare for.

For example:

Embeddings

Basic:

What is an embedding?

Intermediate:

How do you choose an embedding model?

Advanced:

How do embedding models affect chunking?

Production:

Your retrieval quality dropped after changing the embedding model. How would you diagnose it?

24. Common Mistakes in AI Engineer Interviews

1. Talking only about frameworks

Saying:

“I used LangChain.”

doesn’t demonstrate much.

Explain:

Why LangChain was used and what architectural problem it solved.

2. Giving only definitions

Don’t stop at:

“RAG retrieves documents and gives them to an LLM.”

Explain:

			
Ingestion
→ Chunking
→ Embedding
→ Retrieval
→ Reranking
→ Prompt construction
→ Generation
→ Evaluation

		

3. Ignoring trade-offs

Senior candidates should talk about:

			
Accuracy
Latency
Cost
Complexity
Scalability
Reliability

		

4. Forgetting failure modes

For every system, ask:

What happens when this component fails?

For example:

			
Embedding API fails
       ↓
Retry?
Vector DB unavailable
       ↓
Fallback?
LLM timeout
       ↓
Retry / fallback model?
Bad retrieval
       ↓
Regenerate query?

		

25. Interview Preparation Checklist

Before your next AI Engineer interview, make sure you can explain:

ML

Bias vs variance
Overfitting
Underfitting
Regularization
Gradient descent
Model evaluation

Embeddings

Dense vs sparse
Embedding dimensions
Similarity metrics
Embedding model selection
Domain-specific embeddings

RAG

Chunking
Embeddings
Vector search
BM25
Hybrid search
Reranking
Query rewriting
RAG evaluation

Knowledge Graphs

Entities
Relationships
Entity resolution
Graph construction
Graph RAG
Graph maintenance

LLM Evaluation

Golden datasets
LLM-as-a-Judge
Human evaluation
Regression testing
Faithfulness
Context relevance

Production

Caching
Batching
Quantization
Model routing
Autoscaling
Observability
Cost optimization

System Design

Be able to design:

			
RAG System
NL-to-SQL System
LLM Serving Platform
Agentic AI System
Knowledge Graph Pipeline

		

from scratch.

Key Takeaways

The biggest lesson from this interview isn’t a particular technology.

It’s the depth of reasoning expected from senior AI engineers and architects.

You may start with:

“What is semantic search?”

But the conversation can quickly become:

			
Semantic Search
      ↓
Embeddings
      ↓
Retrieval
      ↓
RAG
      ↓
Evaluation
      ↓
Microservices
      ↓
Scaling
      ↓
Monitoring
      ↓
Cost Optimization

		

That’s why memorizing GenAI terminology isn’t enough.

You need to understand how the pieces interact.

The strongest interview answers usually follow this pattern:

Define the problem → explain the architecture → discuss trade-offs → identify failure modes → explain how you would measure success.

That’s the mindset that separates someone who has experimented with AI tools from someone who can design and operate AI systems.

Next Tutorial

← Previous

AI Engineer Interview Questions — Part 6

You are here:

AI Engineer Interview Questions Series

A practical AI/ML and GenAI interview preparation guide covering RAG, embeddings, Knowledge Graphs, LLM evaluation, system design, optimization, monitoring, and DSA

Table of Contents

1. What This Interview Actually Tested

Round 1 — ML + GenAI Fundamentals

2. “Tell Me About Your Project”

3. Semantic Search vs Keyword Search

Keyword Search

Semantic Search

So Which One Should You Use?

4. How Do Embedding Models Affect Chunk Size?

5. How Would You Select an Embedding Model?

1. Retrieval quality

2. Domain relevance

3. Context length

4. Latency

5. Cost

6. Infrastructure

6. Which Vector Is Longer — Dense or Sparse?

7. Overfitting vs Underfitting

Underfitting

Overfitting

8. The Mathematics Behind Regularization

L1

L2

9. DSA: Fibonacci Question

Round 2 & 3 — RAG + Knowledge Graphs + System Design

10. How Do You Evaluate an LLM-as-a-Judge?

11. How Would You Improve the Judge?

Better rubric

Structured output

Reference answers

Calibration

Multiple judges

12. What Challenges Can Occur in an Ingestion Pipeline?

13. How Have You Used Knowledge Graphs?

14. How Does Knowledge Graph Construction Work?

15. What Are the Costs of Knowledge Graph Construction?

16. How Would You Handle Legal Documents End-to-End?

17. Apart From Horizontal Scaling, How Do You Optimize Microservices?

Caching

Asynchronous processing

Connection pooling

Batch processing

Model optimization

Parallel execution

Payload optimization

Observability

18. Challenges With Semantic and Hybrid Retrieval

Weighted score fusion

Reciprocal Rank Fusion

Reranking

19. How Would You Monitor an ML/GenAI System?

Offline Evaluation

Online Monitoring

Reliability

Performance

Cost

Quality

Infrastructure

20. How Would You Build NL-to-SQL?

21. How Would You Scale NL-to-SQL?

22. Cold Start With a 70B LLM

Warm instances

Model weight caching

Persistent infrastructure

Quantization

Smaller fallback model

Autoscaling with minimum capacity

23. The Pattern Behind These Questions

Embeddings

24. Common Mistakes in AI Engineer Interviews

1. Talking only about frameworks

2. Giving only definitions

3. Ignoring trade-offs

4. Forgetting failure modes

25. Interview Preparation Checklist

ML

Embeddings

RAG

Knowledge Graphs