The K Nearest Neighbors
The Prediction
◆ The Blueprint
What You're Looking At

KNN regression on 391 real cars. To predict the selected car's mpg, find the K closest cars in (horsepower, weight) and average their mpg.

The Prediction Rule
$$\hat{y}_0 = \frac{1}{K} \sum_{i \in N_K(x_0)} y_i$$
Distance, Two Ways

Raw units:

$$d = \sqrt{(\Delta \text{hp})^2 + (\Delta \text{lbs})^2}$$

Weight spans thousands of lbs and horsepower spans tens, so the weight gap dominates every distance and raw distance behaves like a weight-only model.

Normalized: rescale each variable to \(z = (x - \bar{x})/s\) first, so both count in standard deviations.

The Shaded Region

Everything inside the boundary lies closer to the selected car than the K-th neighbor. In raw units a large horsepower gap barely changes the distance, so the region is a flat slab. Normalized, it is round.

Try This

Select the 48-hp VW Dasher diesel. Raw distance matches it with 75–85 hp gasoline compacts (25.8 mpg predicted); normalized distance finds the other Volkswagens (37.1). Its actual economy: 43.4 mpg.

Grid Search: Held-Out RMSE Across K
At Your K
◆ The Blueprint
What You're Looking At

A grid search fits every candidate K and selects the one with the smallest held-out RMSE. Both curves are U-shaped.

Minimum vs Elbow

The star marks the smallest RMSE. The diamond marks the elbow, where the curve stops improving quickly: the point farthest from the straight line joining the curve's ends. Analysts who value simplicity stop at the elbow; the grid search selects the minimum.

The Two Failures

Small K overfits: the predictions track noise in individual cars (high variance). Large K includes neighbors far from the query, so the average drifts toward the overall mean (high bias). At K = 390 every prediction is the same number.

Why Leave-One-Out
$$\text{RMSE} = \sqrt{\tfrac{1}{391}\textstyle\sum_i (y_i - \hat{y}_{(-i)})^2}$$

\(\hat{y}_{(-i)}\) is car \(i\)'s prediction from the other 390 cars. If a car's own row helped predict it, the error would shrink toward zero and K = 1 would look nearly perfect.

5-Fold Cross-Validation

The L05 boardsheet runs the same search with 5-fold cross-validation inside a 300-car training set, saving a 91-car test set for the final grade. The logic is identical; only the split differs.