Before self-attention changed AI forever, recurrent neural networks tried to solve sequence modeling. Here’s why they eventually hit a wall.
This is Part 1 of a 5-part series on Transformers.
📚 Blog Series
- Part 1: Why RNNs Could Never Scale to Modern AI?(This Article)
- Part 2: Backpropagation Through Time and the Vanishing Gradient Problem
- Part 3: How LSTMs Tried to Fix RNNs (and Why They Still Fell Short)
- Part 4: Attention Is All You Need — Understanding Self-Attention
- Part 5: Building the Transformer Encoder From Scratch
Every Revolution Starts With a Problem
In 2017, Google published a paper that fundamentally changed artificial intelligence.
Attention Is All You Need.
Today, almost every breakthrough model — ChatGPT, Claude, Gemini, Llama, DeepSeek, Stable Diffusion, GitHub Copilot — is built upon ideas introduced in that paper.
Whenever someone talks about Large Language Models, you’ll hear words like:
- Attention
- Self-Attention
- Query
- Key
- Value
- Multi-Head Attention
- Positional Encoding
But there’s an important question many engineers never ask:
Why was attention invented in the first place?
The answer isn’t simply “because it works better.”
It was invented because the neural networks that came before it were fundamentally difficult to train.
To appreciate why Transformers were revolutionary, we first need to understand the problem they solved.
That journey begins with Recurrent Neural Networks (RNNs).
Why Traditional Neural Networks Couldn’t Understand Language
Imagine building a neural network that predicts the next word in a sentence.
For example:
I love drinking hot ______
Most of us immediately think:
coffee
Why?
Because we remembered everything that came before.
Now consider a standard feed-forward neural network.
flowchart LR
A[Input] --> B[Hidden Layer]
B --> C[Hidden Layer]
C --> D[Output]
Each prediction is completely independent.
It has no memory.
If you feed one word at a time:
I
love
drinking
hot
the network treats every word as an isolated observation.
It has no mechanism to remember previous words.
This is acceptable for problems like:
- House Price Prediction
- Fraud Detection
- Customer Churn
- Image Classification
But language isn’t independent.
Every word depends on previous words.
Sequence Data Is Different
Unlike tabular datasets, sequence data has order.
Consider these two sentences.
Dog bites man.
Man bites dog.
Both contain exactly the same words.
Yet they mean completely different things.
The only difference?
The order.
This immediately tells us something important.
A neural network that ignores history cannot properly model language.
It needs memory.
Enter the Recurrent Neural Network
Instead of throwing away previous information after every prediction, researchers introduced a simple but elegant idea.
What if the neural network could pass information from one time step to the next?
That’s exactly what an RNN does.
flowchart LR
X1["x₁"] --> H1["Hidden State h₁"]H1 --> H2["Hidden State h₂"]X2["x₂"] --> H2H2 --> H3["Hidden State h₃"]X3["x₃"] --> H3H3 --> H4["..."]
Notice something interesting.
Unlike traditional neural networks, the hidden layer is no longer isolated.
It becomes a memory that travels through time.
Thinking of an RNN as a Python For Loop
Forget the equations for a moment.
The easiest way to understand an RNN is to imagine a Python loop.
hidden_state = initial_state
for word in sentence:
hidden_state = update(hidden_state, word)
That’s essentially an RNN.
Every iteration receives:
- the current input
- everything remembered so far
and produces a new memory.
Simple.
In fact, RNNs are one of the few deep learning architectures whose intuition is surprisingly close to their implementation.
Unrolling the RNN
Although an RNN is represented as a single recurrent block, during training it is “unrolled” across time.
Instead of one network, imagine many copies sharing the same weights.
flowchart LR
This visualization reveals something profound.
There aren’t actually multiple neural networks.
There is one neural network reused repeatedly across time.
Every time step shares exactly the same parameters.
This parameter sharing is one of the biggest advantages of RNNs.
No matter whether the sentence contains:
- 5 words
- 50 words
- 500 words
the model size remains constant.
Understanding the Hidden State
The hidden state is often described as the network’s memory.
But what exactly is stored there?
The answer is:
A compressed representation of everything the model has seen so far.
Suppose our sentence is:
The cat sat on the mat.
After processing:
The
the hidden state may encode:
Subject probably coming...
After processing:
The cat
the hidden state becomes
Subject = cat
After processing
The cat sat
it may additionally encode
Action = sat
Notice something important.
The hidden state doesn’t literally store words.
It stores features learned during training.
In modern terminology, you can think of it as a learned embedding representing the sequence seen so far.
Forward Propagation Inside an RNN
Now let’s look under the hood.
At every time step, the RNN combines:
- the current input vector
- the previous hidden state
to compute a new hidden state.
Mathematically:
where:
- (x_t) is the current input
- (h_{t-1}) is the previous hidden state
- (W_{xh}) learns how to process new information
- (W_{hh}) learns how to preserve past information
- (\tanh) keeps values bounded between -1 and 1
Finally,
produces the prediction.
Although these equations may look intimidating, the computation follows exactly the Python loop we saw earlier.
Each iteration performs only three steps:
- Read the current word.
- Update memory.
- Produce an output.
Repeat.
A Step-by-Step Example
Consider the sentence:
Machine Learning is amazing
Step 1
Input:
Machine
Hidden state:
Mostly empty
The network learns an initial representation.
Step 2
Input:
Learning
The hidden state now combines:
- Machine
- Learning
Step 3
Input:
is
Memory becomes richer.
The model now understands we’re likely describing something.
Step 4
Input:
amazing
The prediction now depends on every previous word.
At least…
that’s the theory.
Why RNNs Initially Felt Revolutionary
When RNNs were introduced, they solved problems that traditional neural networks simply couldn’t.
For the first time, one architecture could naturally handle:
- Machine Translation
- Speech Recognition
- Language Modeling
- Text Generation
- Time-Series Forecasting
- Video Analysis
Researchers finally had a neural network capable of remembering the past.
For nearly two decades, RNNs dominated sequence modeling.
It genuinely felt like the problem had been solved.
Until researchers tried training them on longer sequences.
That’s when the real challenge began.
What’s Coming Next
Everything we’ve discussed so far focuses on forward propagation — how information flows through an RNN.
But training a neural network isn’t just about moving information forward.
Eventually, the model must learn from its mistakes.
That means propagating gradients backward through every previous time step.
Unfortunately, that’s exactly where RNNs begin to break.
In the next article, we’ll explore Backpropagation Through Time (BPTT), uncover why gradients vanish (or explode), and see why RNNs become nearly impossible to train on long sequences.
Trust me — the mathematics behind this problem explains why Transformers were not just an improvement, but a necessity.