Algebraic Theory of Information Quantities

Algebraic Theory of Information Quantities

Information theory is a discipline at the intersection of statistics, engineering and computer science. As the study of information quantities, such as compression or communication capacities, information content or measures of statistical dependency, it is one of the theoretical underpinnings of data science. Computational problems in information theory are highly non-linear. The goal of this project is to transfer state-of-the-art methods of computational non-linear algebra to information theory, to study the inherent algebraic complexity of information-theoretical problems and to provide tools for solving them in practice. The algebraic point of view has proven to be fruitful in seemingly unrelated areas, as witnessed by a surge of recent work in algebraic statistics, in particular on likelihood geometry. However, maximizing the likelihood function is the same as minimizing relative entropy — a specific information quantity. Hence, this project also aims at generalizing the techniques developed in likelihood geometry. One focus is on the practical computation of information quantities using numerical and differential algebraic geometry. Such quantities are defined via non-linear optimization problems and we aim to pinpoint the algebraic complexity of these problems in instances of general interest, such as common information measures. The final objective is finding fundamental laws and limits of data science imposed by non-linear inequalities constraining the entropic region. These inequalities provide, by duality, universal bounds for many of the optimization problems studied in information theory.