LANS Seminar

Seminar Title: Accurate estimation of information theoretic quantities with a correction-based approach and their application in Generative models

Speaker: Anirban Samaddar, Postdoctoral Researcher, Mathematics and Computer Science (MCS), Argonne National Laboratory

Date/Time: Thursday, March 20, 2025/ 2:30 PM – 3:30 PM (In-Person)
Location: See Meeting URL on the cels-seminars website which will require an Argonne login.

Description: Information theoretic quantities play a central role in machine learning. The recent surge in the complexity of data and models has increased the demand for accurate estimation of these quantities. However, as the dimension grows the estimation presents significant challenges, with existing methods struggling already in relatively low dimensions. In this talk, we introduce REMEDI for efficient and accurate estimation of differential entropy, a fundamental information theoretic quantity. The approach combines the minimization of the cross-entropy for simple, adaptive base models and the estimation of their deviation, in terms of the relative entropy, from the data density. Our approach demonstrates improvement across a broad spectrum of entropy estimation tasks. We illustrate how the framework can be naturally extended to information theoretic supervised learning models, with a specific focus on the Information Bottleneck approach. In addition, we explore a natural connection between REMEDI and generative modeling using rejection sampling and Langevin dynamics. We further demonstrate the utility of this approach in unsupervised learning and downstream time series forecasting tasks to predict disruptions in plasma inside Tokamak fusion reactors.

Bio: Anirban Samaddar is a postdoctoral researcher with Dr. Sandeep Madireddy in the Mathematics and Computer Science (MCS) division at Argonne National Laboratory. Anirban is working on Bayesian machine learning methods and their application to natural sciences. His areas of interest include – diffusion models, information-theoretic deep learning, and neural architecture search. He is involved in a wide range of projects on the application of machine learning in areas such as physics, biology, and material science. Anirban has a Ph.D. degree in Statistics from Michigan State University under the guidance of Dr. Gustavo de los Campos and Dr. Taps Maiti

Please note that the meeting URL for this event can be seen on the cels-seminars website which requires an Argonne login.