Programming, AI, Machine Learning & Software Development Tutorials
Understanding transformations, actions, execution plans, and why Spark waits before processing your data. If you have worked with PySpark, you may have noticed something unusual. You can write several transformations: And Spark may appear to do nothing. No computation.No immediate scan of the entire dataset.No obvious execution. But the moment you run: or: Spark suddenly … Read more
A practical AI/ML and GenAI interview preparation guide covering RAG, embeddings, Knowledge Graphs, LLM evaluation, system design, optimization, monitoring, and DSA AI/ML interviews are changing. Knowing definitions like “What is an embedding?” or “What is overfitting?” is no longer enough for senior AI/ML and GenAI roles. A recent Associate Architect (ML) interview experience at Quantiphi … Read more
Production agent latency is rarely just an LLM problem. Here’s how to find the real bottleneck. Your AI agent takes 12 seconds to answer a user. You check the LLM metrics. The model itself takes only 2 seconds. So where did the other 10 seconds go? This is one of the most important questions to … Read more
Training a machine learning model is, at its core, a search problem. Your model starts with random weights and tries to find a configuration that minimizes the loss function. Gradient descent provides the direction, but the learning rate determines how far the optimizer moves in that direction. And that creates a fundamental problem. Take steps that … Read more
A practical guide to lexical + semantic retrieval, score fusion, reranking, Python implementation, and production RAG architecture Introduction A common mistake when building a RAG system is assuming that vector search is enough. You convert documents into embeddings, store them in a vector database, retrieve the top-k chunks, and send them to the LLM. It … Read more
A practical guide to BM25, the ranking formula, TF-IDF, Python implementation, and why BM25 is still essential for modern RAG systems Introduction When people build a modern RAG system, the conversation often starts with embeddings: “Which embedding model should we use?” But before vector databases and semantic search became popular, search engines had already solved … Read more
The future of Retrieval-Augmented Generation (RAG) is shifting from reliance on dedicated vector databases to utilizing existing databases like PostgreSQL with pgvector. Production needs are leading to integrated systems that manage both vector similarity and traditional filtering. This consolidation promotes efficiency, reducing operational complexities and costs.
The article explains the roles of the Driver and Executors in Apache Spark. The Driver coordinates the entire application, planning execution and scheduling tasks, while Executors carry out computations by processing data partitions. Understanding this distinction is crucial for effectively using Spark to handle large datasets across distributed systems.
Spark can process data across dozens or thousands of machines—but before any Executor can run your tasks, Spark needs a way to acquire and manage those resources. That is the job of a Cluster Manager. Introduction In the previous tutorials, we learned what Apache Spark is, why Spark can be faster than Hadoop MapReduce, and … Read more
The content discusses the evolution of AI engineer interview questions, shifting from basic LLM concepts to intricate topics involving AI agents. Key areas include the differences between agents and chatbots, the importance of state management, decision-making in agent design, tool selection, and handling infinite loops. Understanding these aspects is crucial for aspiring AI engineers.
This content serves as a comprehensive guide to retrieval strategies in Retrieval-Augmented Generation (RAG) systems. It contrasts sparse retrieval, which emphasizes keyword matching, and dense retrieval, focused on semantic understanding. Both strategies are essential for answering user queries effectively. The tutorial also highlights hybrid approaches for optimal performance.
Apache Spark is a distributed data processing engine that simplifies executing large-scale data pipelines through an organized structure. This structure includes a Driver for coordination, Cluster Manager for resource allocation, and Executors for data processing. Understanding the hierarchy of Application, Job, Stage, and Task is crucial for effective use of Spark.
Apache Spark’s dominance in big data processing stems from its innovative architecture, which optimizes distributed computation beyond simple speed. It utilizes in-memory processing, Directed Acyclic Graphs (DAGs), and multi-stage execution to efficiently manage complex workloads. Unlike Hadoop, Spark’s architecture supports improved query optimization, caching, and parallel processing, offering a more versatile computing model.
The most-cited AI deployment in enterprise history has a second chapter almost nobody reads carefully — and it isn’t the story you think it is. On 27 February 2024, Klarna published a press release that would end up in more consulting decks than possibly any other document of the AI era. The Swedish buy-now-pay-later company … Read more
From documents to embeddings, retrieval, reranking, prompt construction, and grounded answers — a practical walkthrough of how modern RAG pipelines actually work. Introduction In the previous article, we looked at Why LLMs Hallucinate is and why RAG has become one of the most important architectures for building LLM applications. But knowing what RAG is isn’t … Read more
Learn what Apache Spark is, why it has become the standard for big data processing, how it works at a high level, and where it fits into today’s data engineering ecosystem. Introduction If you’ve started exploring data engineering, you’ve probably noticed one technology appearing everywhere—Apache Spark. Whether you’re working with Databricks, Delta Lake, Azure Data … Read more
Learn why ChatGPT, Claude, Gemini, and Llama sometimes generate incorrect information, what causes hallucinations, and how production AI engineers minimize them using RAG, grounding, guardrails, and evaluation. Find all tutorials here Table of Contents Introduction If you’ve used ChatGPT long enough, you’ve probably experienced something like this. You ask: Who won the FIFA World Cup in … Read more
Learn how RAG works, why LLMs hallucinate, and build your first Retrieval-Augmented Generation pipeline in Python. Find all tutorials here Introduction Large Language Models (LLMs) have transformed how we build AI applications. Today, we can ask models to: Tools like ChatGPT, Claude, Gemini, and Llama make these tasks feel almost magical. But there’s one major … Read more
After the fundamentals, RAG, LLM inference, and GenAI architecture questions, here are 20 fresh AI Engineer interview questions designed to test whether you can reason about real-world AI systems — not just explain frameworks. Part 1: AI Engineer Interview Questions Part 2: AI Engineer Interview QuestionsPart 3: AI Engineer Interview QuestionsPart 4: AI Engineer Interview Questions Part 5: AI … Read more
After interviewing 50+ AI Engineer candidates, I noticed a pattern: impressive GenAI projects can get you through the first 10 minutes—but strong fundamentals are what separate candidates who build AI systems from those who truly understand them. I’ve interviewed 50+ AI Engineer candidates over the past few months. Almost every resume looked impressive. RAG. AI … Read more