Information-theoretic characterization for learning algorithms

Summary

Information theory and statistical learning theory are closely related, as both fields are rooted in statistics. Numerous works have shown that information-theoretic analysis and techniques provide useful insight to machine learning algorithms. Examples include that information measures (eg, mutual information) provide tight bounds for generalisation errors of learning algorithms, and Fano inequalities can be used to give lower bounds on non-parametric regression problems. In this project, we will use both information-theoretic measures and techniques to study learning algorithms. Examples include important machine learning problem as transfer learning and learning with causality consideration.

Key Articles

A Bayesian approach to (online) transfer learning: Theory and algorithms”, 2023

An information-theoretic analysis for transfer learning: Error bounds and applications, 2022

On the tightness of information-theoretic bounds on generalization error of learning algorithms, 2023