CyLab Seminar - Yaoquing Yao March 31, 2025 12:00pm — 1:00pm Location: In Person and Virtual - ET - Panther Hollow Room, Mehrabian Collaborative Innovation Center 4105 Speaker: YAOQUING YAO , Assistant Professor, Department of Computer Science, Dartmouth College https://sites.google.com/site/yangyaoqingcmu/ Weight Matrix Diagnostics for Improved Neural Network Training and Compression This talk will introduce several useful metrics derived from studying the heavy-tail phenomenon in neural network weight matrices. I will begin by motivating these metrics through random matrix theory and discussing their connection to recent studies on feature learning. I will then demonstrate how these metrics can be applied to various neural network applications, including layer-wise pruning of large language models, tuning training and fine-tuning learning rates, training scientific machine learning models with limited data, adjusting the architectural hyperparameters of LoRA networks, and model selection on Hugging Face Transformers without access to training or testing data. In a recent study, we show that these theory-driven metrics can be scaled to prune large language models with up to 65 billion parameters, outperforming some of the latest pruning methods. — My current research focuses on diagnosing and mitigating failures in machine learning models. For example, I analyze shape and geometric features in high-dimensional spaces, such as loss landscapes, weight matrix spectral densities, and decision boundaries, to provide actionable insights for addressing common failure modes in these models. I also apply these techniques to applications such as 3D point clouds and graphs. My research draws inspiration from statistical learning and information theory. Faculty Host: Pulkit Grover In Person and Zoom Participation. See announcement. Event Website: https://www.cylab.cmu.edu/events/2025/03/31-seminar-yang.html Add event to Google Add event to iCal
