Geeky Codes | AI Engineering, Data Engineering & Machine Learning Tutorials
Part 5 moves beyond Spark API knowledge into the decisions that actually matter when a PySpark pipeline is slow, unstable, expensive, or producing incorrect results. If you’re preparing for PySpark interviews or working with Spark in production, follow me for practical data engineering and AI/ML content. 📩 Subscribe by email to get new tutorials and … Read more
You can write dozens of Spark transformations without Spark processing a single record. The moment you call an action, the lazy plan becomes an actual computation. If you have already learned about Spark transformations, you have seen code like: At this point, Spark has not necessarily read and processed all the data. Now add: Something … Read more
Learn how Spark DataFrames transform data lazily, how transformations build execution plans, and which operations can trigger expensive shuffles. Apache Spark Learning Path APACHE SPARK LEARNING PATH Where Are We Now? We’ve already learned how Spark reads data and how its execution engine works. Now we need to understand the most common thing we actually … Read more
Stop using ChatGPT only to get answers. Learn how to use it to plan, execute, verify, and improve complete tasks. AI assistants are becoming much more useful than traditional question-and-answer chatbots. Instead of asking: “Explain how to build a RAG application.” you can structure the task as: “Design the architecture, implement the application, test it, … Read more
This tutorial covers how Apache Spark writes data to files, tables, and databases, detailing the use of DataFrameWriter and various formats like Parquet, CSV, and JSON. It explores save modes, partitioning techniques, and common mistakes while writing data, ultimately aiming to enhance efficiency in data processing and querying.
A practical guide to uploading, downloading, filtering, and managing files in AWS S3 with Python Amazon S3 is one of the most commonly used AWS services for storing files and datasets. In the previous tutorial, we learned how to list and print the contents of an S3 bucket using Python and Boto3. But listing files … Read more
Understand the three core Spark data abstractions, how they differ, why DataFrames became the preferred choice for most workloads, and when RDDs or Datasets still make sense. If you’ve started learning Apache Spark, you’ve probably encountered three terms repeatedly: RDD. DataFrame. Dataset. At first, they can seem interchangeable. After all, all three allow you to … Read more
Understand what SparkSession is, why modern Spark applications need it, what it actually creates, and how it differs from SparkContext If you’ve written PySpark code, you’ve probably started with something like this: And then immediately started working with DataFrames: But what exactly is spark? Why do we need a SparkSession? What happens when getOrCreate() is … Read more
Learn how Spark reads data from files and storage systems, how schema inference works, and how to build reliable data ingestion pipelines with PySpark Almost every Spark application starts with one simple question: Where is the data, and how do I get it into Spark? In PySpark, the answer usually starts with: For example: or: … Read more
This text explains the differences between narrow and wide transformations in Apache Spark. Narrow transformations process independently without redistributing data, while wide transformations necessitate data movement and often introduce shuffle operations, affecting performance. Understanding these concepts aids in Spark’s optimization and execution strategies, influencing stages, tasks, and overall efficiency.
Apache Spark makes distributed data processing look deceptively simple. You write something like: It looks like a few lines of Python. But Spark has to answer several questions before this computation can actually run: This is where jobs, stages, and tasks become important. In the previous tutorial, we discussed how Spark builds a DAG (Directed … Read more
Understanding how Spark moves data across partitions — and why shuffle is one of the biggest causes of slow Spark jobs You write a simple Spark query: It looks harmless. But behind the scenes, Spark may need to move millions or billions of records across the cluster so that all records belonging to the same … Read more
Machine Learning is everywhere. When Netflix recommends a movie, your bank detects a suspicious transaction, Google ranks search results, or an e-commerce website recommends a product, Machine Learning may be working behind the scenes. But what exactly is Machine Learning? The simplest definition is: Machine Learning is a way of building systems that learn patterns … Read more
Forget memorizing another list of LangChain APIs. If you understand these five concepts deeply, you can handle the questions that separate AI application builders from AI engineers. Introduction I’ve noticed a pattern in AI Engineer interviews. The candidate’s resume looks impressive: Then the interviewer asks: “Why does your RAG system still hallucinate even though retrieval … Read more
Understanding Directed Acyclic Graphs, transformations, actions, dependencies, stages, shuffles, and how Spark turns your PySpark code into distributed execution. You write a few lines of PySpark: It looks like a simple sequence of operations. But Spark doesn’t simply execute these lines one after another. Instead, Spark builds a representation of the computation, analyzes the dependencies … Read more
Understanding transformations, actions, execution plans, and why Spark waits before processing your data. If you have worked with PySpark, you may have noticed something unusual. You can write several transformations: And Spark may appear to do nothing. No computation.No immediate scan of the entire dataset.No obvious execution. But the moment you run: or: Spark suddenly … Read more
A practical AI/ML and GenAI interview preparation guide covering RAG, embeddings, Knowledge Graphs, LLM evaluation, system design, optimization, monitoring, and DSA AI/ML interviews are changing. Knowing definitions like “What is an embedding?” or “What is overfitting?” is no longer enough for senior AI/ML and GenAI roles. A recent Associate Architect (ML) interview experience at Quantiphi … Read more
Production agent latency is rarely just an LLM problem. Here’s how to find the real bottleneck. Your AI agent takes 12 seconds to answer a user. You check the LLM metrics. The model itself takes only 2 seconds. So where did the other 10 seconds go? This is one of the most important questions to … Read more
Training a machine learning model is, at its core, a search problem. Your model starts with random weights and tries to find a configuration that minimizes the loss function. Gradient descent provides the direction, but the learning rate determines how far the optimizer moves in that direction. And that creates a fundamental problem. Take steps that … Read more
A practical guide to lexical + semantic retrieval, score fusion, reranking, Python implementation, and production RAG architecture Introduction A common mistake when building a RAG system is assuming that vector search is enough. You convert documents into embeddings, store them in a vector database, retrieve the top-k chunks, and send them to the LLM. It … Read more