Chapter 1 — Machine Learning Fundamentals¶

Learning Objectives¶

By the end of this chapter you will be able to:

1. State the formal definition of machine learning and map a new problem onto the correct learning paradigm.
2. Derive the bias–variance decomposition and explain what each term implies for model selection.
3. Implement stratified k-fold cross-validation from scratch and verify it against scikit-learn.
4. Identify all four common sources of data leakage and redesign a pipeline to eliminate them.
5. Explain the No Free Lunch theorem and use it to justify empirical algorithm comparison.

1.1 What Is Machine Learning?¶

1.1.1 Mitchell's Formal Definition¶

Tom Mitchell's 1997 definition remains the clearest:

A computer program is said to learn from experience E with respect to some task T and performance measure P, if its performance at task T, as measured by P, improves with experience E.

In the language of functions, we seek a mapping:

where \(\mathcal{X}\) is the input space (features) and \(\mathcal{Y}\) is the output space (labels or values). We do not hand-code \(f\); instead we learn it from data:

1.1.2 The Learning Problem¶

The learner sees \(\mathcal{D}\), searches a hypothesis class \(\mathcal{H}\) for the best \(h \in \mathcal{H}\) according to some loss function \(\mathcal{L}\):

The fundamental challenge: we want \(h^*\) to minimise loss on unseen data (the true distribution \(p(x, y)\)), not just the training set \(\mathcal{D}\).

1.2 The ML Workflow¶

flowchart LR
    A["Raw Data"] --> B["Data Cleaning\n& EDA"]
    B --> C["Feature\nEngineering"]
    C --> D["Train/Val/Test\nSplit"]
    D --> E["Model\nSelection"]
    E --> F["Training\n(Optimisation)"]
    F --> G["Evaluation"]
    G -->|"Iterate"| E
    G -->|"Satisfied"| H["Deployment"]
    H --> I["Production\nMonitoring"]
    I -->|"Drift detected"| A

    style A fill:#0d6efd,color:#fff
    style H fill:#198754,color:#fff
    style I fill:#fd7e14,color:#fff

Each arrow represents a decision point where engineering judgement is required. The feedback loops are intentional — ML development is inherently iterative.

1.3 Three Types of Learning¶

1.3.1 Supervised, Unsupervised, and Reinforcement Learning¶

1.3.2 Semi-supervised and Self-supervised Learning¶

A fourth paradigm sits between supervised and unsupervised: semi-supervised learning uses a small labelled set and a large unlabelled set. Self-supervised learning (the basis of large language models) creates synthetic supervision from the data itself — e.g., predicting the next token.

1.4 The Bias–Variance Tradeoff¶

1.4.1 Formal Decomposition¶

Consider a regression setting with true function \(f(x)\) and a learner \(\hat{f}\) trained on dataset \(\mathcal{D}\). The expected mean squared error at a point \(x\) is:

Total Error = Bias² + Variance + Irreducible Error

1.4.2 What Each Term Means¶

1.4.3 The Tradeoff Visualised¶

xychart-beta
    title "Bias-Variance Tradeoff"
    x-axis ["Simple", "Medium", "Medium-High", "High", "Very High"] 
    y-axis "Error" 0 --> 10
    line [8, 5, 3, 4, 6]
    line [1, 2, 3, 5, 8]
    line [9, 7, 6, 9, 14]

1.4.4 Practical Implications¶

• A training error much lower than validation error signals high variance → regularise or get more data.
• Both training error and validation error are high signals high bias → use a more powerful model or engineer better features.
• The optimal complexity depends on dataset size: with 10× more data you can afford a more complex model.

1.5 Overfitting and Underfitting¶

1.5.1 Definitions¶

Overfitting occurs when a model memorises the training data, capturing noise rather than the underlying signal. It achieves low training error but high generalisation error.

Underfitting occurs when a model is too simple to capture the true pattern. Both training and generalisation errors are high.

1.5.2 Detection¶

1.5.3 Remedies¶

1.6 The No Free Lunch Theorem¶

1.6.1 Statement¶

Wolpert and Macready (1997) proved that, averaged over all possible data-generating distributions, every learning algorithm performs equally well. No algorithm is universally superior.

where \(L\) and \(L'\) are any two learning algorithms and \(f\) ranges over all possible target functions.

1.6.2 Practical Implications¶

1.7 Features: Types and Encoding¶

1.7.1 Feature Types¶

1.7.2 Encoding Strategies¶

One-Hot Encoding (nominal categories with small cardinality):

Ordinal Encoding (when order is meaningful):

Target Encoding (high-cardinality categorical):

Standardisation (continuous features for distance-based models):

Min-Max Normalisation (when bounded range is needed):

from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

def build_preprocessor(
    numerical_cols: list[str],
    categorical_cols: list[str],
) -> ColumnTransformer:
    """Build a column-aware preprocessing pipeline."""
    return ColumnTransformer(
        transformers=[
            ("num", StandardScaler(), numerical_cols),
            ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), categorical_cols),
        ],
        remainder="drop",
    )

1.8 Train / Validation / Test Split¶

1.8.1 Why Three Splits?¶

• Training set: the data the model sees and learns from.
• Validation set: used during development to tune hyperparameters and make model selection decisions.
• Test set: touched once, at the very end, to report the final unbiased performance estimate.

Using the test set more than once inflates reported performance because you are implicitly fitting to it. The test set must remain completely sealed until you have committed to a final model.

1.8.2 Choosing Ratios¶

1.8.3 Stratification¶

For classification, always stratify so that class proportions are preserved in every split:

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y,
    test_size=0.2,
    stratify=y,   # <-- preserves class ratio
    random_state=42,
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp,
    test_size=0.5,
    stratify=y_temp,
    random_state=42,
)

Without stratification on a 5 % minority class, your test set might have 0 % or 10 % minority examples by chance, making metrics unreliable.

1.9 Cross-Validation¶

1.9.1 Why Cross-Validation?¶

A single validation split is noisy — different random seeds can swing accuracy by several percentage points on small datasets. Cross-validation uses all data for both training and validation by rotating which fold is held out.

1.9.2 K-Fold Cross-Validation¶

The dataset is partitioned into \(k\) equal folds \(F_1, F_2, \ldots, F_k\). In round \(i\), fold \(F_i\) is the validation set and the remaining \(k-1\) folds form the training set. The CV score is:

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold, LeaveOneOut
from sklearn.base import BaseEstimator
from sklearn.metrics import accuracy_score


def manual_kfold_cv(
    model: BaseEstimator,
    X: np.ndarray,
    y: np.ndarray,
    n_splits: int = 5,
    random_state: int = 42,
) -> dict[str, float]:
    """Run stratified k-fold CV and return mean ± std accuracy."""
    skf = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=random_state)
    scores: list[float] = []

    for fold, (train_idx, val_idx) in enumerate(skf.split(X, y)):
        X_tr, X_val = X[train_idx], X[val_idx]
        y_tr, y_val = y[train_idx], y[val_idx]

        model.fit(X_tr, y_tr)
        y_pred = model.predict(X_val)
        fold_score = accuracy_score(y_val, y_pred)
        scores.append(fold_score)
        print(f"  Fold {fold + 1}/{n_splits}: accuracy = {fold_score:.4f}")

    scores_arr = np.array(scores)
    return {"mean": scores_arr.mean(), "std": scores_arr.std()}

1.9.3 Variants¶

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

rfc = RandomForestClassifier(n_estimators=100, random_state=42)

cv_scores = cross_val_score(
    rfc, X_train, y_train,
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring="f1_macro",
    n_jobs=-1,
)

print(f"F1 Macro: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

1.10 Data Leakage¶

1.10.1 Definition¶

Data leakage occurs when information from outside the training boundary flows into the model, causing artificially optimistic performance estimates that collapse at deployment.

1.10.2 The Four Common Sources¶

Source 1: Target Leakage¶

A feature is computed using the target variable itself, or contains information available only after the outcome is known.

Example: predicting loan default with a feature "collections_account_opened", which is created as a consequence of defaulting.

Fix: Audit every feature for temporal causality. Ask: could this feature value be known at the time a real prediction would be made?

Source 2: Preprocessing on the Full Dataset¶

Fitting a scaler, imputer, or encoder on the full dataset (including validation/test) leaks statistical information about held-out examples into the training pipeline.

# WRONG — scaler sees test data during fit
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(X)  # leaks test statistics into training

# CORRECT — fit only on training data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)   # use training statistics

Source 3: Temporal Leakage¶

In time-series problems, using future data to predict the past. A model trained on data from 2024 predicting 2023 outcomes, or features computed from a rolling window that inadvertently looks forward.

Fix: Use TimeSeriesSplit from scikit-learn, ensure all feature engineering uses strictly past data.

Source 4: Duplicate / Overlapping Rows¶

If the same (or very similar) rows appear in both training and test sets, the model effectively memorises test examples. Common with oversampling (SMOTE applied before splitting) or with dataset pipelines that don't deduplicate.

Fix: Deduplicate before splitting. Apply oversampling inside the training fold only.

1.10.3 Leakage Prevention Checklist¶

☑ Every preprocessing step is inside a Pipeline that is fit only on training data.
☑ All features are temporally valid at prediction time.
☑ Oversampling (SMOTE, etc.) is applied after splitting, inside the training fold.
☑ Target encoding uses fold-level statistics only.
☑ No duplicate rows across splits.
☑ No features derived from the target (or from post-outcome data).

1.11 Exercises¶

scaler.fit_transform(X)          # step 1
smote.fit_resample(X_scaled, y)  # step 2
X_train, X_test = train_test_split(X_resampled)  # step 3

Summary¶

Next: Chapter 2 — Supervised Learning