A perfect score
is a warning

First, what a regression model actually does

Skip this if it is familiar. If it is not, everything below depends on it.

You have pairs of numbers. An input, call it \(x\), and an output, \(y\). A regression model is a rule for guessing \(y\) when you are handed a new \(x\) you have not seen before. Fitting the model means searching for the rule that comes closest to the pairs you already have.

The simplest rule is a straight line: pick a slope and a height, and the model is fixed. Linear regression does exactly that, choosing the line whose total squared distance from the points is smallest. It is the whole method.

Other models allow other shapes:

• Polynomial regression is still a straight-line fit, but on invented columns. Give it \(x\), \(x^2\) and \(x^3\) as separate inputs and the resulting “line” bends, because it is a line in a space you built.
• Random forest does not draw any curve. It repeatedly splits the input range into regions and predicts the average of each region, so its output looks like a staircase rather than a curve.
• SVR fits a smooth surface but only penalises predictions that miss by more than a set tolerance, so small errors cost it nothing.

Three shapes, and the one that humiliates a straight line

The study names three functions. They look similar on paper and behave nothing alike.

\[f_1(x) = x \sin(x) + 2x \qquad f_2(x) = 10 \sin(x) + x^2 \qquad f_3(x) = \operatorname{sign}(x)(x^2 + 300) + 20 \sin(x)\]

Fitted on raw \(x\) alone, with no engineered columns, they separate the models sharply:

Look at linear regression. On \(f_1\) it manages a respectable 0.892. On \(f_2\) it scores below zero, which is the worst result any model records on clean data anywhere in this project.

The reason is worth sitting with, because the usual explanation is wrong. It is not that \(f_2\) is “too non-linear”. \(f_1\) is non-linear too and the line did fine there. The difference is symmetry. \(f_1\) contains \(2x\), a steady climb that a line can follow. \(f_2\) is a parabola centred on zero: it falls, then rises by the same amount. The best straight line through it is flat, and a flat line is precisely the “always guess the average” baseline that R squared scores as zero.

And \(f_3\) shows the reverse. It contains a jump at the origin, a vertical cliff that no smooth curve can reproduce, yet random forest scores a perfect 1.000 on it. A staircase model finds a step trivial, because stepping is all it does. The shape that is hardest for every equation-shaped model is the easiest one here.

Why build data you already understand

A real dataset cannot tell you whether a model is right. It can only tell you whether the model agrees with the labels you happen to have, and if those labels are wrong or incomplete, agreement is worthless.

Synthetic data removes that problem by construction. Each function above is sampled at 100 points across \([-20, 20]\) and split 70/30 at a fixed seed. The point is not that these curves are interesting. The point is that when a model gets one wrong, you can prove it.

The rest of this page follows \(f_1\), because it is the shape the noise and feature-engineering experiments were run against.

Handing the model what you already know

The models never see the formula. They see engineered columns: \(x\), \(\sin x\), \(\cos x\), \(x^2\), \(x^3\). That choice came from looking at a plot and noticing the curve was both wavy and rising.

It changes everything, and the table shows how unevenly:

The five columns are built by one function, and it never sees \(f_1\):

def create_features(x):
    return np.column_stack([
        x,
        np.sin(x),
        np.cos(x),
        x**2,
        x**3
    ])

That restraint is the point. np.sin(x) is a guess drawn from looking at a plot, not a copy of the answer. A function that returned x * np.sin(x) + 2 * x would score 1.000 too and would have proved nothing.

Feature engineering did not make anything smarter. It rewrote the question. “Approximate this curve” became “find four coefficients”, and the second question has an exact answer. Random forest gained nothing because it was never asking the first question: it slices the input into regions and averages inside them, so a wavy curve costs it no more than a straight one.

Correlated inputs break the linear models

The second dataset is multivariate: 2,000 samples from make_regression, then deliberately spoiled by making features carry overlapping information.

The correlation is not simulated by hand. make_low_rank_matrix produces ten columns that genuinely contain fewer than ten independent directions:

X_corr = make_low_rank_matrix(
    n_samples=2000, n_features=10,
    effective_rank=5, tail_strength=0.5, random_state=SEED
)
weights = np.random.RandomState(SEED).randn(5)
noise   = np.random.RandomState(SEED).normal(0, 10, size=2000)
y_corr  = X_corr[:, :5] @ weights + noise

Only the first five columns drive the target. The other five are along for the ride, and they overlap with the first five because the matrix has rank 5.

There is a trap in this table. The mean squared error improved for every model while R² collapsed. Both are correct. The correlated target has far less variance, so smaller errors are easier to achieve and explaining the remaining variation is harder. A single metric would have told you the opposite story.

What a model does with features that mean nothing

The brief asks for something sharper than a score here: build a dataset where some inputs are known to be pure noise, fit a linear regressor, and read the coefficients it learned for them.

The useless columns are drawn from a normal distribution and glued on beside the real ones, so nothing distinguishes them except that they carry no signal:

def create_harder_features(x, n_non_informative=5):
    base_features = np.column_stack([
        x, x * np.sin(x), np.sin(x), np.cos(x), x**2, x**3
    ])
    non_informative = np.random.randn(len(x), n_non_informative)
    return np.hstack([base_features, non_informative])

X_enhanced = create_harder_features(X_base, n_non_informative=10)

With strong target noise and ten uninformative columns added, linear regression manages R² 0.082 with an MSE of 1845.6. Barely above useless. The coefficients explain why:

The expectation is that informative features get large weights and noise features get roughly zero. That is not what happened. Feature 6 carries no signal and was handed a coefficient of −32.5, comparable in size to the genuine one at the top.

This is worth internalising, because it is how feature importance is commonly misread. A large coefficient does not mean a feature matters. Least squares distributes weight to minimise error on the training set, and if a random column happens to correlate with the noise in that particular sample, it will be paid for it. Run it again with a different seed and the same column gets a different number.

Two ways to cut features down, both refusing to help

Once the multivariate data is deliberately correlated, an obvious response is to reduce the inputs. The notebook tries the two standard approaches and scores them against the untouched baseline.

SelectKBest with f_regression picks columns: it scores each feature against the target and keeps the best. PCA builds new ones: it rotates the data onto axes of maximum variance, which are combinations of the originals and are uncorrelated by construction.

Both reduce ten columns to five, and both are handed to the same evaluate_models helper so nothing else differs between the runs:

selector = SelectKBest(score_func=f_regression, k=5)
X_train_selected = selector.fit_transform(X_train_corr, y_train_corr)
X_test_selected  = selector.transform(X_test_corr)

pca = PCA(n_components=5)
X_train_pca = pca.fit_transform(X_train_scaled)
X_test_pca  = pca.transform(X_test_scaled)

Note that PCA is given scaled inputs and SelectKBest is not. PCA looks for directions of maximum variance, so a column measured in thousands would dominate the first component for no reason other than its units.

Neither rescues anything. The best result is polynomial regression moving from −0.049 to −0.008, which is still a model worse than the mean.

That is the useful lesson. Both techniques address redundancy among features, and redundancy was not the binding problem. The target simply did not have much explainable variance left once the correlation was introduced, and no amount of rearranging the inputs creates signal that is not there. Dimensionality reduction is a remedy for a specific illness, not a tonic.

A pipeline built for noise, rather than against it

After watching the polynomial disintegrate, the notebook assembles a second pipeline designed with noise assumed rather than treated as an accident: smoothing on the targets, scaling, and models that regularise by default.

Compare that against the first attempt, where polynomial regression fell from 1.000 to −0.917. These models start lower and end far higher. Giving up the perfect fit bought resilience worth more than the fit was.

Real data, where nothing can be checked

The last stage removes the last guarantee. The data is daily weather from sensor 22508 near Honolulu, recorded between 1940 and 1945. No formula produced it. There is no truth to recover, only yesterday and tomorrow.

A rolling window converts the series into a supervised problem: the previous \(W\) days become the inputs and the next day becomes the target, with the window sliding one day at a time.

def create_rolling_windows(data, window_size=7):
    X, y = [], []
    for i in range(len(data) - window_size):
        X.append(data[i:i+window_size])
        y.append(data[i+window_size])
    return np.array(X), np.array(y)

X_rolling, y_rolling = create_rolling_windows(df_sensor['MeanTemp'].values, 7)

Seven columns in, one out. A time series has become an ordinary supervised table, and every regression model above now applies to it unchanged.

Training uses 1940 to 1944, testing uses 1945, so the model always predicts forward and never peeks at a period it has seen.

Linear regression wins here, having lost on the synthetic curve. Tomorrow’s temperature mostly resembles today’s, and that relationship is close to a straight line. The flexible models have no hidden structure to find, so their flexibility only gives them more ways to be wrong. SVR at 0.398 is the clearest case.

Across five TimeSeriesSplit folds, Bayesian Ridge averages about 0.73, with folds ranging from 0.628 to 0.795. The spread is the more useful number: it says the answer depends on which stretch of the war you test on.

Tuning bought nothing, and that is worth saying

GridSearchCV made it fractionally worse. Against a fold standard deviation of 0.057, a change of 0.008 is noise. The honest reading is that the pipeline was already where it wanted to be and the search had nothing to find, which is a result rather than a failure to report.

Four things that held up across every round

Flexibility is only an asset when the truth is complicated. Random forest won on a wavy curve and lost on temperatures. SVR was competitive on synthetic data and worst on real data.

A perfect score is a warning. The only model to reach 1.000 was the only one to reach −24.

One metric can invert the story. The correlated round had error falling and R² collapsing at the same time, from identical predictions.

Regularisation never wins and never loses badly. Ridge held no best score in any clean comparison and avoided every disaster in the spoiled ones.

Run it and disagree with it

Every table above is printed by one notebook, executed top to bottom, and the excerpts on this page are lifted from it unedited. Nothing was fitted somewhere else and pasted in.

git clone https://github.com/O-2wice/regression-benchmark-and-forecasting
cd regression-benchmark-and-forecasting
pip install -r requirements.txt
jupyter lab notebooks/regression-benchmark-and-forecasting.ipynb

It runs on a laptop CPU in a few minutes. There is nothing to download by hand and no key to supply: the weather archive is fetched over urllib and unpacked with zipfile, so it behaves the same on Windows, macOS, Linux and Colab.

The most useful thing you can do with it is change the seed. Several results here, especially the coefficient handed to a pure-noise column, are stable in their lesson and unstable in their exact value. Watching that number move is the lesson.