Regression is one of the most important areas of supervised machine learning. It is used when the goal is to predict a continuous numerical value from one or more input features.
From predicting house prices and customer demand to forecasting sales and estimating electricity consumption, regression algorithms are widely used in real-world machine learning applications.
This guide introduces the fundamentals of regression and provides a structured path to learning the most important regression algorithms.
In this section, you will learn:
- What Regression is in Machine Learning
- How regression models work
- Simple and Multiple Linear Regression
- Polynomial Regression
- Ridge Regression
- Lasso Regression
- Decision Tree Regression
- How to evaluate regression models
- How to choose the right regression algorithm
What Is Regression in Machine Learning?
Regression is a supervised learning technique used to predict a continuous numerical target variable based on one or more input features.
For example, suppose we want to predict the price of a house.
Our dataset could contain:
| Area | Bedrooms | Age | Price |
|---|---|---|---|
| 1,000 sq ft | 2 | 10 years | $180,000 |
| 1,500 sq ft | 3 | 7 years | $260,000 |
| 2,000 sq ft | 3 | 5 years | $350,000 |
| 2,500 sq ft | 4 | 3 years | $450,000 |
Here:
- Area, Bedrooms, and Age are features.
- Price is the target variable.
- Price is continuous, making this a regression problem.
The model learns a relationship between the input features and the target and then uses that relationship to predict the value for new observations.
For example, after training, the model might receive a house with:
Area = 1,800 sq ft, Bedrooms = 3, Age = 6 years
and predict:
Price = $315,000
The exact prediction depends on the data and the model being used.
Regression vs Classification
Regression and classification are both supervised learning problems, but they predict different types of outputs.
Regression
Predicts a continuous numerical value.
Examples:
- House price
- Revenue
- Temperature
- Sales
- Electricity demand
- Delivery time
Classification
Predicts a discrete category or class.
Examples:
- Spam / Not Spam
- Fraud / Not Fraud
- Churn / No Churn
- Positive / Negative
A simple way to remember the difference is:
Regression predicts “how much” or “how many”; classification predicts “which category.”
Learn more:
→ Regression vs Classification
How Does Regression Work?
A regression model attempts to learn a mathematical relationship between input variables and a target variable.
Suppose we have a single feature, such as house size.
We could represent the relationship as:
House Size → House Price
The model learns parameters that minimize the difference between the actual values and predicted values.
For a simple linear regression model, the relationship can be represented as a straight line.
The model tries to find the line that provides the best fit to the observed data.
The difference between an actual value and the model’s prediction is called a residual or prediction error.
A good regression model attempts to minimize these errors while still generalizing well to unseen data.
Types of Regression Algorithms
There is no single regression algorithm that works best for every dataset.
Different algorithms make different assumptions about the relationship between features and the target.
The main regression algorithms covered in this section are:
1. Linear Regression
Linear Regression is one of the simplest and most widely used regression algorithms.
It assumes that the target variable can be approximated using a linear combination of the input features.
For a single feature, the model attempts to find the best-fitting straight line.
When to use Linear Regression
Linear Regression can be useful when:
- The relationship between variables is approximately linear.
- Interpretability is important.
- You need a strong baseline model.
- The dataset is relatively simple.
Learn More
→ Linear Regression: A Complete Guide
2. Polynomial Regression
Not every relationship between variables is linear.
For example, the relationship between an input variable and the target may follow a curve.
Polynomial Regression extends linear regression by introducing polynomial features.
For example:
x → x² → x³
This allows the model to capture nonlinear relationships while still using a regression framework.
When to use Polynomial Regression
Polynomial Regression can be useful when:
- The relationship between variables is nonlinear.
- A curved relationship provides a better fit.
- You want a relatively simple way to model nonlinear patterns.
However, increasing the polynomial degree can make the model increasingly complex and can lead to overfitting.
Learn More
→ Polynomial Regression: How It Works
3. Ridge Regression
Ridge Regression is a regularized version of Linear Regression.
It introduces a penalty on large model coefficients to reduce model complexity.
This can help when the model suffers from:
- Multicollinearity
- Large coefficients
- Overfitting
Ridge Regression uses L2 regularization.
The regularization parameter controls how strongly large coefficients are penalized.
Why Ridge Regression?
Suppose several features contain highly correlated information.
Ordinary Linear Regression may assign unstable or excessively large coefficients.
Ridge Regression constrains those coefficients, often producing a more stable model.
Learn More
→ Ridge Regression: L2 Regularization Explained
4. Lasso Regression
Lasso Regression is another regularized version of Linear Regression.
Unlike Ridge Regression, Lasso uses L1 regularization.
One important property of Lasso is that it can shrink some model coefficients all the way to zero.
This means Lasso can perform a form of feature selection.
For example, suppose a dataset contains 100 features.
After training a Lasso model, some coefficients may become zero.
Those features effectively contribute nothing to the final prediction.
When to use Lasso
Lasso can be useful when:
- You have many features.
- You suspect some features are irrelevant.
- Feature selection is desirable.
- You want a more interpretable sparse model.
Learn More
→ Lasso Regression: L1 Regularization Explained
5. Decision Tree Regression
Decision Tree Regression takes a different approach from linear models.
Instead of fitting a straight line or polynomial curve, a Decision Tree recursively divides the feature space into regions and makes predictions within those regions.
For example, a tree might learn rules such as:
Is Area < 1,500 sq ft?
→ Yes → Continue to another split
→ No → Continue to another split
Eventually, the model reaches a leaf node and produces a numerical prediction.
Decision Trees can model nonlinear relationships and interactions between features without requiring them to be explicitly specified.
Advantages
- Can model nonlinear relationships.
- Easy to visualize.
- Does not require feature scaling.
- Can capture feature interactions.
Limitation
An unrestricted Decision Tree can easily become too complex and overfit the training data.
Learn More
→ Decision Tree Regression: How It Works
Regression Algorithms at a Glance
| Algorithm | Main Idea | Regularization | Handles Nonlinearity |
|---|---|---|---|
| Linear Regression | Fits a linear relationship | No | Limited |
| Polynomial Regression | Adds polynomial features | Optional | Yes |
| Ridge Regression | Linear Regression + L2 penalty | L2 | Limited |
| Lasso Regression | Linear Regression + L1 penalty | L1 | Limited |
| Decision Tree Regression | Learns decision rules | Tree constraints | Yes |
This table provides a high-level overview. The individual tutorials explain the mathematical intuition, implementation, advantages, limitations, and practical use cases of each algorithm.
How Do You Evaluate a Regression Model?
Training a regression model is only the first step.
We also need to determine how accurately it predicts unseen data.
Common regression evaluation metrics include:
Mean Absolute Error — MAE
MAE measures the average absolute difference between actual and predicted values.
It is easy to interpret because the error is expressed in the same units as the target.
For example, if a house-price model has an MAE of $20,000, its predictions are off by about $20,000 on average in absolute terms.
Mean Squared Error — MSE
MSE calculates the average squared prediction error.
Because errors are squared, large errors receive substantially more weight.
This makes MSE useful when large prediction errors are particularly undesirable.
Root Mean Squared Error — RMSE
RMSE is the square root of MSE.
It is expressed in the same units as the target variable while still penalizing larger errors more heavily.
R² Score
R², or the coefficient of determination, measures how much of the variation in the target variable is explained by the model relative to a baseline.
It is commonly used alongside error-based metrics such as MAE or RMSE.
Regression Model Selection
Choosing a regression algorithm depends on the characteristics of your dataset and the objective of your project.
A useful starting point is:
Start with Linear Regression
Use Linear Regression as a baseline when you expect approximately linear relationships.
↓
Try Polynomial Regression
If the relationship appears nonlinear and a polynomial representation makes sense.
↓
Use Ridge or Lasso
When regularization is needed or when features are correlated / numerous.
↓
Try Decision Tree Regression
When the relationship is nonlinear and rule-based splits can capture the underlying patterns.
For more complex datasets, you can later explore ensemble algorithms such as:
- Random Forest Regression
- Gradient Boosting
- XGBoost
- LightGBM
- CatBoost
These algorithms can be studied after understanding the fundamental regression models.
Common Problems in Regression
A regression model can perform poorly for several reasons.
Overfitting
The model learns the training data too closely and performs poorly on unseen data.
A complex polynomial model or deep decision tree can be particularly susceptible to overfitting.
Underfitting
The model is too simple to capture the underlying relationship.
For example, fitting a straight line to strongly nonlinear data may result in underfitting.
Multicollinearity
Two or more features may contain highly correlated information.
This can make coefficient estimates unstable, particularly for ordinary Linear Regression.
Outliers
Extreme observations can have a significant influence on regression models, especially models based on squared errors.
Poor Features
Even a sophisticated algorithm may produce poor predictions when the input features do not contain enough useful information about the target.
This is why feature engineering and data quality are often just as important as algorithm selection.
Regression Learning Path
If you are learning regression from scratch, follow these tutorials in order:
1. Linear Regression
Learn the basic idea behind fitting a model to continuous data.
↓
2. Polynomial Regression
Learn how regression can model nonlinear relationships.
↓
3. Ridge Regression
Understand L2 regularization and how it helps control model complexity.
↓
4. Lasso Regression
Learn L1 regularization and how it can perform feature selection.
↓
5. Decision Tree Regression
Learn how tree-based models can capture nonlinear relationships.
↓
6. Regression Model Evaluation
Learn MAE, MSE, RMSE, R² and other evaluation techniques.
This progression takes you from simple linear models → nonlinear models → regularization → tree-based models.
Frequently Asked Questions
What is regression in Machine Learning?
Regression is a supervised machine learning technique used to predict a continuous numerical value from one or more input features.
What are some examples of regression problems?
Common examples include predicting house prices, sales revenue, demand, temperature, electricity consumption, customer lifetime value, and delivery times.
What is the difference between regression and classification?
Regression predicts continuous numerical values, whereas classification predicts discrete categories or classes.
Which regression algorithm should I learn first?
Linear Regression is generally the best starting point because it introduces fundamental ideas such as features, coefficients, predictions, residuals, loss functions, and model fitting.
Is Linear Regression only used for simple problems?
No. Linear Regression can be a powerful baseline and remains useful in many real-world applications, particularly when interpretability and simplicity are important.
What is regularization in regression?
Regularization adds a penalty to the model’s objective function to discourage excessive model complexity. Ridge uses L2 regularization, while Lasso uses L1 regularization.
Which metrics are used for regression?
Common metrics include MAE, MSE, RMSE, and R². The appropriate metric depends on the business problem and how prediction errors should be treated.
Start Learning Regression
Regression provides the foundation for understanding how machine learning models predict continuous numerical outcomes.
If you are new to regression, start with:
→ Linear Regression
Then progress through Polynomial Regression, Ridge Regression, Lasso Regression, and Decision Tree Regression.
Once you understand these algorithms, you can move on to more powerful ensemble-based approaches such as Random Forest, Gradient Boosting, and XGBoost.
Machine Learning → Regression → Individual Regression Algorithms