Machine learning 1, 2025-2026

General Information

Meeting times

1.5 hours per week

• Friday, 10h45 - 12h15

---

Syllabus (course description)

This course introduces students to the foundational ideas of machine learning, with a strong emphasis on methodology, experimental design, and statistical thinking. Rather than rushing through many algorithms, the course spends significant time on regression, using it as a laboratory for understanding:

• how models work,
• how they fail,
• how to diagnose problems
• how to improve them

Students learn to formulate ML problems, explore datasets, construct baselines, build models, evaluate them properly, and interpret results with a scientific mindset.

Hands‑on mini‑projects accompany each module, allowing students to practice embedding ML into correct workflows. Datasets are intentionally synthetic but carefully designed to illustrate classical modeling challenges such as nonlinearity, heteroscedasticity, correlated features, and overfitting.

---

Learning objectives

• Foundational Understanding: Explain what machine learning is, what problems it can solve, and the central role of generalization. Distinguish between features and targets, and articulate how data represents real‑world phenomena. Describe the machine learning experiment loop.

• Modeling & Evaluation Skills: Correctly perform train/validation/test splits and justify why they are necessary. Build and evaluate baseline models (mean/median for regression, majority class for classification). Train, interpret, and evaluate linear regression models, comparing them against baselines using correct error metrics. Diagnose model failures using residuals and identify signs of nonlinearity, heteroscedasticity, and multicollinearity. Improve regression models by designing and transforming features and identifying redundant or unstable predictors. Recognize and explain overfitting, underfitting, and the bias–variance tradeoff, and apply regularization techniques (Ridge, Lasso). Use cross‑validation and systematic hyperparameter tuning to select models without leaking test information.

• Broader ML Competence: Implement and evaluate key classical ML models: logistic regression, k‑nearest neighbors, decision trees, random forests, and PCA. Choose and justify appropriate metrics for regression and classification (MSE/MAE, precision/recall, F1, ROC‑AUC). Identify and avoid data leakage, and implement evaluation in a scientifically valid manner. Produce reproducible machine learning workflows using pipelines, controlled randomness, and documented experiment design.

• Communication & Applied Skills: Conduct small‑scale ML experiments following best practices. Interpret models using diagnostic plots and qualitative reasoning. Write clear, concise reports summarizing methodology, results, and limitations. Collaborate effectively in small groups to build and evaluate ML models.

---

Books and references

• James, G., Witten, D., Hastie, T., Tibshirani, R., & Taylor, J. (2023). Statistical learning. In An introduction to statistical learning: With applications in Python (pp. 15-67). Cham: Springer International Publishing.
• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning.
• Bishop, C. M., & Nasrabadi, N. M. (2006). Pattern recognition and machine learning (Vol. 4, No. 4, p. 738). New York: springer.
• Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.

There are also online ressources that can be very useful for students:

1. StatQuest with Josh Starmer

---

Evaluation

• Continuous assessment: 60%. This includes all assessments done in class, which include oral presentations throughout the semester, homeworks, mini-projects and in-class quizzes.

• Final project (report & presentation): 40%

---

Modules (week-by-week)

The course follows a 15‑week, hands‑on progression — each week lists core topics, demos, and expected mini‑project prompts.

Week 1 — Foundations: ML, Data, and Evaluation

• ML framing: supervised vs unsupervised
• Data representation: features vs target
• Experiment loop; what generalization means
• Train/validation/test splits; random seed control
• Baselines: mean/median, majority class
• Demo: Synthetic dataset → train_test_split → baseline MSE

Week 2 — Regression I: Linear Regression as Optimization & Residuals

• Linear regression as least‑squares optimization
• Loss functions: MSE vs MAE
• Normal equations (conceptual); gradient descent intuition
• Pred vs true plots; residuals as diagnostics
• Detecting heteroscedasticity; intro to nonlinearity
• Demo: Residual plots; funnel‑shape example
• Mini‑project: Load provided dataset; compute baseline test error. Fit linear model; interpret errors & residuals

Week 3 — Regression II: Feature Design & Multicollinearity

• Feature engineering basics: polynomial & domain transforms
• Correlation matrix and pairwise scatter
• Multicollinearity & coefficient instability
• Predictive vs interpretive tradeoffs
• Demo: Duplicate features → unstable coefficients

Week 4 — Regression III: Overfitting & Regularization

• Bias–variance intuition
• Polynomial degree sweep: train vs test curves
• Ridge (L2) for stability; Lasso (L1) for sparsity
• Impact on coefficients & prediction error
• Demo: Ridge rescues high‑degree polynomial; Lasso zeros out coords
• Mini‑project: Degree sweep + Ridge/Lasso tuning and plots of coefficients w.r.t regularization parameter

Week 5 — Model Evaluation: Cross‑Validation & Tuning

• k‑fold cross‑validation
• Hyperparameter grids & estimator variability
• Pipelines to prevent leakage
• Learning curves (conceptual)
• Demo: Pipeline: PolynomialFeatures → StandardScaler → Ridge

Week 6 — Regression IV: Robustness & Error Analysis

• Outliers & heavy‑tailed noise
• MSE vs MAE sensitivity
• Huber (conceptual), quantile regression (light)
• Error distribution interpretation
• Demo: Inject outliers → compare LinearRegression vs Huber
• Mini‑project: CV‑based model selection for polynomial regression. Robust metrics comparison + error distribution study

Week 7 — Transition Week: Regression Wrap‑Up & Intro to Classification

• Comparison of regression approaches
• When linear models fail vs succeed
• Conceptual bridge: predicting categories vs numbers
• Logistic regression
• Decision boundaries; thresholding

Week 8 — Classification: Metrics & Imbalance

• Confusion matrix, precision, recall, F1
• ROC‑AUC & PR curves; class imbalance pitfalls
• Mitigation: threshold adjustment, class weights
• Demo: 90/10 dummy baseline → accuracy trap
• Mini‑project: Train logistic model → accuracy + ROC‑AUC, PR curve analysis & threshold selection

Week 9 — Distance‑Based Models: kNN & Scaling

• kNN for regression/classification
• Scaling necessity (StandardScaler)
• Curse of dimensionality
• Choosing k via CV
• Demo: kNN fails without scaling; k‑sweep curve

Week 10 — Work on final project

• Students can work in groups on the final project, ask questions to the instructor.

Week 11 — Decision trees

• Decision trees: splits, impurity, structure
• Depth control; high variance
• Feature importance caveats
• Demo: Depth sweep; tree visualizations

Week 12 — Ensembles: Random Forests & Boosting Intuition

• Bagging & variance reduction
• Random Forests: trees + randomness
• OOB intuition; tuning n_estimators and max_depth
• Intro to boosting (concept only)
• Demo: RF vs single tree performance & stability

Week 13 — Interpretability

• Linear models: coefficients, scaling issues
• Decision trees: decision paths
• Random forests: permutation importance; PDP (intuitive)

Week 14 — Presentations

• Group presentations of project results

Week 15 — Presentations

• Group presentations of project results