Geeky Codes | AI Engineering, Data Engineering & Machine Learning Tutorials
Geeky Codes Learn how Spark reads data from files and storage systems, how schema inference works, and how to build reliable data ingestion pipelines with PySpark Almost every Spark application starts with one simple question: Where is the data, and how do I get it into Spark? In PySpark, the answer usually starts with: For example: or: … Read more
This text explains the differences between narrow and wide transformations in Apache Spark. Narrow transformations process independently without redistributing data, while wide transformations necessitate data movement and often introduce shuffle operations, affecting performance. Understanding these concepts aids in Spark’s optimization and execution strategies, influencing stages, tasks, and overall efficiency.
Apache Spark makes distributed data processing look deceptively simple. You write something like: It looks like a few lines of Python. But Spark has to answer several questions before this computation can actually run: This is where jobs, stages, and tasks become important. In the previous tutorial, we discussed how Spark builds a DAG (Directed … Read more
Understanding how Spark moves data across partitions — and why shuffle is one of the biggest causes of slow Spark jobs You write a simple Spark query: It looks harmless. But behind the scenes, Spark may need to move millions or billions of records across the cluster so that all records belonging to the same … Read more
Machine Learning is everywhere. When Netflix recommends a movie, your bank detects a suspicious transaction, Google ranks search results, or an e-commerce website recommends a product, Machine Learning may be working behind the scenes. But what exactly is Machine Learning? The simplest definition is: Machine Learning is a way of building systems that learn patterns … Read more
Forget memorizing another list of LangChain APIs. If you understand these five concepts deeply, you can handle the questions that separate AI application builders from AI engineers. Introduction I’ve noticed a pattern in AI Engineer interviews. The candidate’s resume looks impressive: Then the interviewer asks: “Why does your RAG system still hallucinate even though retrieval … Read more
Understanding Directed Acyclic Graphs, transformations, actions, dependencies, stages, shuffles, and how Spark turns your PySpark code into distributed execution. You write a few lines of PySpark: It looks like a simple sequence of operations. But Spark doesn’t simply execute these lines one after another. Instead, Spark builds a representation of the computation, analyzes the dependencies … Read more
Understanding transformations, actions, execution plans, and why Spark waits before processing your data. If you have worked with PySpark, you may have noticed something unusual. You can write several transformations: And Spark may appear to do nothing. No computation.No immediate scan of the entire dataset.No obvious execution. But the moment you run: or: Spark suddenly … Read more
A practical AI/ML and GenAI interview preparation guide covering RAG, embeddings, Knowledge Graphs, LLM evaluation, system design, optimization, monitoring, and DSA AI/ML interviews are changing. Knowing definitions like “What is an embedding?” or “What is overfitting?” is no longer enough for senior AI/ML and GenAI roles. A recent Associate Architect (ML) interview experience at Quantiphi … Read more
Production agent latency is rarely just an LLM problem. Here’s how to find the real bottleneck. Your AI agent takes 12 seconds to answer a user. You check the LLM metrics. The model itself takes only 2 seconds. So where did the other 10 seconds go? This is one of the most important questions to … Read more
Training a machine learning model is, at its core, a search problem. Your model starts with random weights and tries to find a configuration that minimizes the loss function. Gradient descent provides the direction, but the learning rate determines how far the optimizer moves in that direction. And that creates a fundamental problem. Take steps that … Read more
A practical guide to lexical + semantic retrieval, score fusion, reranking, Python implementation, and production RAG architecture Introduction A common mistake when building a RAG system is assuming that vector search is enough. You convert documents into embeddings, store them in a vector database, retrieve the top-k chunks, and send them to the LLM. It … Read more
A practical guide to BM25, the ranking formula, TF-IDF, Python implementation, and why BM25 is still essential for modern RAG systems Introduction When people build a modern RAG system, the conversation often starts with embeddings: “Which embedding model should we use?” But before vector databases and semantic search became popular, search engines had already solved … Read more
The future of Retrieval-Augmented Generation (RAG) is shifting from reliance on dedicated vector databases to utilizing existing databases like PostgreSQL with pgvector. Production needs are leading to integrated systems that manage both vector similarity and traditional filtering. This consolidation promotes efficiency, reducing operational complexities and costs.
The article explains the roles of the Driver and Executors in Apache Spark. The Driver coordinates the entire application, planning execution and scheduling tasks, while Executors carry out computations by processing data partitions. Understanding this distinction is crucial for effectively using Spark to handle large datasets across distributed systems.
Spark can process data across dozens or thousands of machines—but before any Executor can run your tasks, Spark needs a way to acquire and manage those resources. That is the job of a Cluster Manager. Introduction In the previous tutorials, we learned what Apache Spark is, why Spark can be faster than Hadoop MapReduce, and … Read more
The content discusses the evolution of AI engineer interview questions, shifting from basic LLM concepts to intricate topics involving AI agents. Key areas include the differences between agents and chatbots, the importance of state management, decision-making in agent design, tool selection, and handling infinite loops. Understanding these aspects is crucial for aspiring AI engineers.
This content serves as a comprehensive guide to retrieval strategies in Retrieval-Augmented Generation (RAG) systems. It contrasts sparse retrieval, which emphasizes keyword matching, and dense retrieval, focused on semantic understanding. Both strategies are essential for answering user queries effectively. The tutorial also highlights hybrid approaches for optimal performance.
Apache Spark is a distributed data processing engine that simplifies executing large-scale data pipelines through an organized structure. This structure includes a Driver for coordination, Cluster Manager for resource allocation, and Executors for data processing. Understanding the hierarchy of Application, Job, Stage, and Task is crucial for effective use of Spark.
Apache Spark’s dominance in big data processing stems from its innovative architecture, which optimizes distributed computation beyond simple speed. It utilizes in-memory processing, Directed Acyclic Graphs (DAGs), and multi-stage execution to efficiently manage complex workloads. Unlike Hadoop, Spark’s architecture supports improved query optimization, caching, and parallel processing, offering a more versatile computing model.