Random Forest Regression in Python: A Complete Guide

Random Forest Regression is one of the most reliable ensemble methods for predicting continuous values. By combining predictions from hundreds of decision trees, it resists overfitting, handles non-linear relationships, and requires surprisingly little preprocessing. This guide walks through building, evaluating, and tuning a Random Forest Regressor in Python using scikit-learn.

A single decision tree can memorize training data and fail on anything new. Random Forest solves this by building many trees, each trained on a different random sample of the data, and averaging their predictions. The result is a model that generalizes well across a wide range of regression problems, from housing prices to energy consumption forecasting.

What Is Random Forest Regression?

Random Forest Regression is an ensemble learning technique that constructs multiple decision trees during training and outputs the average of their individual predictions. Unlike a single decision tree that can easily overfit by memorizing noise in the data, a random forest introduces two layers of randomness: each tree is trained on a bootstrapped (randomly sampled with replacement) subset of the training data, and at each split within a tree, only a random subset of features is considered.

This dual randomness forces the individual trees to be different from one another. When their predictions are averaged together, the errors from individual trees tend to cancel out, producing a more stable and accurate result.

How the Algorithm Works

The Random Forest Regression algorithm follows a three-stage process: bootstrap sampling, tree construction with feature randomness, and prediction aggregation.

Step 1: Bootstrap Sampling (Bagging)

For each tree in the forest, the algorithm creates a new training set by randomly sampling rows from the original data with replacement. This means some rows may appear multiple times in a single tree's training set, while others may be left out entirely. The rows left out are called "out-of-bag" (OOB) samples and can be used for internal validation without needing a separate test set.

Step 2: Tree Construction with Feature Randomness

Each decision tree is grown using its bootstrapped sample. At every split, instead of evaluating all available features to find the best split, the algorithm randomly selects a subset of features and chooses the best split from that subset. For regression, the default in scikit-learn is to consider all features (max_features=1.0), but reducing this value increases diversity among trees and can improve generalization.

Step 3: Aggregation

When making a prediction on new data, each tree in the forest produces its own prediction independently. The final output is the arithmetic mean of all individual tree predictions. This averaging process is what gives the random forest its stability and resistance to overfitting.

Building Your First Random Forest Regressor

The following example uses scikit-learn's built-in California Housing dataset, which contains features like median income, house age, and average number of rooms for block groups across California. The target variable is the median house value for each block group.

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

# Load the California Housing dataset
data = fetch_california_housing()
X, y = data.data, data.target
feature_names = data.feature_names

print(f"Dataset shape: {X.shape}")
print(f"Features: {feature_names}")
print(f"Target range: {y.min():.2f} - {y.max():.2f}")

The dataset contains 20,640 samples and 8 features. Before training, the data needs to be split into training and test sets so the model can be evaluated on data it has not seen during training.

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples: {X_test.shape[0]}")

Now the model can be created and trained. The RandomForestRegressor class accepts several parameters, but reasonable defaults work well for an initial model.

# Create and train the Random Forest Regressor
rf_model = RandomForestRegressor(
    n_estimators=100,     # number of trees in the forest
    max_depth=None,       # trees grow until leaves are pure
    min_samples_split=2,  # minimum samples to split a node
    min_samples_leaf=1,   # minimum samples in a leaf node
    random_state=42,      # reproducibility
    n_jobs=-1             # use all available CPU cores
)

rf_model.fit(X_train, y_train)
print("Model training complete.")

Evaluating Model Performance

After training, the model's quality needs to be measured on the held-out test set. The two standard metrics for regression models are Mean Squared Error (MSE) and R-squared (R2). MSE measures the average squared difference between predicted and actual values -- lower is better. R2 represents the proportion of variance in the target that the model explains -- closer to 1.0 is better.

# Generate predictions on the test set
y_pred = rf_model.predict(X_test)

# Calculate evaluation metrics
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_test, y_pred)

print(f"Mean Squared Error:  {mse:.4f}")
print(f"Root Mean Squared Error: {rmse:.4f}")
print(f"R-squared Score:     {r2:.4f}")

A well-tuned Random Forest on the California Housing dataset typically achieves an R2 score above 0.80, meaning the model explains over 80% of the variance in house values. The RMSE gives a more interpretable sense of error magnitude since it is in the same units as the target variable.

Out-of-Bag (OOB) Score

Random Forest offers a built-in validation mechanism through OOB scoring. Each tree is trained on a bootstrap sample, and the rows not included in that sample can be used to evaluate that specific tree. By aggregating these OOB predictions across all trees, the model produces a validation score without needing a separate validation set.

# Rebuild the model with OOB scoring enabled
rf_oob = RandomForestRegressor(
    n_estimators=100,
    oob_score=True,
    random_state=42,
    n_jobs=-1
)

rf_oob.fit(X_train, y_train)
print(f"OOB R-squared Score: {rf_oob.oob_score_:.4f}")

Feature Importance

One of the strengths of Random Forest is its ability to rank features by importance. Scikit-learn computes feature importance based on how much each feature reduces impurity (variance, in the case of regression) across all trees in the forest. Features that produce larger reductions in variance are considered more important.

import pandas as pd

# Extract and display feature importances
importances = rf_model.feature_importances_

# Create a sorted DataFrame
feat_imp_df = pd.DataFrame({
    "Feature": feature_names,
    "Importance": importances
}).sort_values("Importance", ascending=False)

print("Feature Importances:")
print("-" * 35)
for _, row in feat_imp_df.iterrows():
    bar = "#" * int(row["Importance"] * 50)
    print(f"{row['Feature']:>12}  {row['Importance']:.4f}  {bar}")

For the California Housing dataset, MedInc (median income) typically dominates as the most important feature, which aligns with the intuition that income is a strong predictor of housing prices in a given area.

Permutation Importance

The default impurity-based importance can be biased toward high-cardinality features. Permutation importance provides an alternative by measuring how much the model's performance drops when a single feature's values are randomly shuffled. This approach is model-agnostic and generally considered more reliable.

from sklearn.inspection import permutation_importance

# Calculate permutation importance on the test set
perm_result = permutation_importance(
    rf_model, X_test, y_test,
    n_repeats=10,
    random_state=42,
    n_jobs=-1
)

# Display results
perm_imp_df = pd.DataFrame({
    "Feature": feature_names,
    "Importance Mean": perm_result.importances_mean,
    "Importance Std": perm_result.importances_std
}).sort_values("Importance Mean", ascending=False)

print("\nPermutation Importances (test set):")
print("-" * 50)
for _, row in perm_imp_df.iterrows():
    print(f"{row['Feature']:>12}  "
          f"{row['Importance Mean']:.4f} "
          f"+/- {row['Importance Std']:.4f}")

Hyperparameter Tuning

While default parameters often produce a solid baseline, tuning hyperparameters can meaningfully improve performance. The key parameters to focus on are n_estimators, max_depth, min_samples_split, min_samples_leaf, and max_features.

Here is a summary of what each parameter controls:

• n_estimators -- The number of trees in the forest. More trees generally improve performance up to a point of diminishing returns. Values between 100 and 500 are common starting points.
• max_depth -- The maximum depth each tree can grow. Setting this limits complexity and helps prevent overfitting. None means trees grow until all leaves are pure.
• min_samples_split -- The minimum number of samples required to split an internal node. Higher values create simpler trees.
• min_samples_leaf -- The minimum number of samples required in a leaf node. Increasing this smooths the model's predictions.
• max_features -- The number of features considered at each split. Lower values increase tree diversity. Options include "sqrt", "log2", a float (fraction), or an integer.

Grid Search with Cross-Validation

Scikit-learn's GridSearchCV systematically tests combinations of hyperparameters using cross-validation to find the best configuration.

from sklearn.model_selection import GridSearchCV

# Define the parameter grid
param_grid = {
    "n_estimators": [100, 200, 300],
    "max_depth": [10, 20, None],
    "min_samples_split": [2, 5, 10],
    "min_samples_leaf": [1, 2, 4],
    "max_features": ["sqrt", "log2", None]
}

# Set up GridSearchCV
grid_search = GridSearchCV(
    estimator=RandomForestRegressor(random_state=42, n_jobs=-1),
    param_grid=param_grid,
    cv=3,                  # 3-fold cross-validation
    scoring="r2",
    verbose=1,
    n_jobs=-1
)

grid_search.fit(X_train, y_train)

print(f"\nBest Parameters: {grid_search.best_params_}")
print(f"Best CV R2 Score: {grid_search.best_score_:.4f}")

Randomized Search Alternative

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

# Define parameter distributions
param_distributions = {
    "n_estimators": randint(100, 500),
    "max_depth": [10, 20, 30, None],
    "min_samples_split": randint(2, 20),
    "min_samples_leaf": randint(1, 10),
    "max_features": uniform(0.1, 0.9)
}

# Run randomized search
random_search = RandomizedSearchCV(
    estimator=RandomForestRegressor(random_state=42, n_jobs=-1),
    param_distributions=param_distributions,
    n_iter=50,             # test 50 random combinations
    cv=3,
    scoring="r2",
    random_state=42,
    verbose=1,
    n_jobs=-1
)

random_search.fit(X_train, y_train)

print(f"\nBest Parameters: {random_search.best_params_}")
print(f"Best CV R2 Score: {random_search.best_score_:.4f}")

Evaluating the Tuned Model

# Use the best model from the search
best_model = random_search.best_estimator_

# Evaluate on the test set
y_pred_tuned = best_model.predict(X_test)
mse_tuned = mean_squared_error(y_test, y_pred_tuned)
r2_tuned = r2_score(y_test, y_pred_tuned)

print(f"\nTuned Model Performance:")
print(f"  RMSE: {np.sqrt(mse_tuned):.4f}")
print(f"  R2:   {r2_tuned:.4f}")

print(f"\nBaseline Model Performance:")
print(f"  RMSE: {rmse:.4f}")
print(f"  R2:   {r2:.4f}")

Common Pitfalls and Best Practices

Pitfall: Ignoring Feature Scaling

Random Forest does not require feature scaling. Unlike algorithms such as linear regression, SVMs, or neural networks, decision trees split data based on threshold comparisons, not on distances or magnitudes. Standardizing or normalizing features before training a Random Forest has no effect on the model's performance. However, if Random Forest is part of a pipeline that includes other models, scaling may still be necessary for those other components.

Pitfall: Too Many Trees Without Benefit

Adding more trees always reduces variance, but the gains diminish rapidly. Going from 10 trees to 100 produces a significant improvement. Going from 100 to 1,000 produces a smaller improvement. Going from 1,000 to 10,000 may produce almost no measurable improvement while multiplying training time and memory usage. A practical approach is to plot the OOB error or validation score as a function of n_estimators and look for where the curve flattens.

Pitfall: Overfitting with Unrestricted Depth

With max_depth=None, individual trees can grow very deep and memorize training data. While the averaging process of the random forest mitigates this, setting a reasonable max_depth (such as 15-30 for tabular datasets) or increasing min_samples_leaf can reduce memory usage and training time without sacrificing accuracy.

Best Practice: Use Cross-Validation

A single train/test split can give misleading results, especially on smaller datasets. Use cross_val_score with 5 or 10 folds to get a more stable estimate of model performance.

from sklearn.model_selection import cross_val_score

# 5-fold cross-validation
cv_scores = cross_val_score(
    RandomForestRegressor(n_estimators=200, random_state=42, n_jobs=-1),
    X, y,
    cv=5,
    scoring="r2"
)

print(f"Cross-Validation R2 Scores: {cv_scores}")
print(f"Mean R2: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")

Best Practice: Save Your Trained Model

Training a Random Forest can take time, especially with many trees and a large dataset. Save the trained model using joblib so it can be loaded later without retraining.

import joblib

# Save the model to disk
joblib.dump(best_model, "random_forest_model.joblib")

# Load the model later
loaded_model = joblib.load("random_forest_model.joblib")

# Verify it works
y_pred_loaded = loaded_model.predict(X_test[:5])
print(f"Sample predictions: {y_pred_loaded}")

Key Takeaways

1. Ensemble strength: Random Forest Regression builds many decision trees on random data subsets and averages their predictions, producing more accurate and stable results than any single tree.
2. Minimal preprocessing: The algorithm handles non-linear relationships natively and does not require feature scaling, making it a strong choice when you want reliable results without extensive data preparation.
3. Feature importance built in: Both impurity-based and permutation importance methods help identify which features drive predictions, supporting model interpretability and feature selection.
4. Tuning matters: While defaults work well, tuning n_estimators, max_depth, min_samples_leaf, and max_features through cross-validated search can meaningfully improve model quality.
5. Know the tradeoffs: Random Forest cannot extrapolate beyond the range of its training data, and very large forests consume significant memory. For tabular data, it remains one of the strongest baseline models available in scikit-learn.

Random Forest Regression strikes a practical balance between power and simplicity. It works well out of the box, rewards thoughtful tuning, and provides interpretable feature importance scores. Whether you are predicting housing prices, forecasting demand, or modeling any continuous variable, RandomForestRegressor is a reliable tool to have in your machine learning workflow.