Project III: Machine Learning Theory
Description
Machine learning is an important technology of our time and thus studying its underlying mathematical foundations is of interest to both mathematicians and computer scientists. This project will introduce students to the theoretical frameworks for analyzing when and why learning is possible. We will study empirical risk minimization as a principle for fitting models to data, and analyze its guarantees through PAC learning and uniform convergence. Key topics include generalization theory, the bias–complexity tradeoff, and measures of hypothesis class capacity such as VC dimension. The project emphasizes precise mathematical formulation, proof techniques, and conceptual understanding, and is suitable for students with an interest in rigorous mathematical theory for machine learning.
This project does not involve programming or applications of machine learning.
Group project
The group project will focus on the foundations of machine learning theory. The main topics are:
- The statistical learning framework and empirical risk minimization;
- The PAC learning framework;
- Learning via uniform convergence;
- The bias-complexity tradeoff;
- The VC dimension;
- Nonuniform learnability.
Mode of operation and evidence of learning
The group project will revolve around learning through reading with focus on the underlying theory, mathematical rigour, and the development of deep conceptual understanding. Students will demonstrate their understanding by solving relevant problems, exploring examples and theoretical applications of the material, and clearly communicating it in both written and oral formats.
Individual Project
The individual project will build on the knowledge we have gained in the group project and will explore additional advanced topics. A few examples of topics you would be able to investigate include (but are not limited to):
- Theory for linear predictors;
- Theory for boosting;
- Convex learning problems;
- Regularization and stability;
- Rademacher complexities.
Mode of operation and evidence of learning
The individual project will revolve around learning through reading with focus on the underlying theory, mathematical rigour, and the development of deep conceptual understanding. Students will demonstrate their understanding by solving relevant problems, exploring examples and theoretical applications of the material, and clearly communicating it in both written and oral formats.
Prerequisites
- Analysis I, Probability I, Statistics I
Reference
- Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. (Link to online version).