Some Advances in Foundation Models for Physics
Please login to view abstract download link
A foundation model for physics would provide an easy tool for few and zero-shot predictions of numerous physical processes. Coupled into a scientific agentic system, these models could be fine-tuned to solve numerous inverse and system design problems in areas ranging from astrophysics to energy production. In this talk, we will present a vision for these models and our work developing them. We will touch upon two major areas of research. In the first, we focus on test-time adaptation for PDE foundation models. PDE foundation models have advanced computational efficiency and the potential to be adapted for numerous downstream physics tasks, but they can struggle with autoregressive rollout. Inspired by advances in test-time-compute for LLMs, we introduce a test-time-adaptation scheme for PDEs to achieve more accurate predictions. We accomplish this with a learned reward model that evaluates patio-temporal consistency. We demonstrate improved accuracy on the PDEGym benchmark relative to standard approaches. In the second, we introduce MOPRH, a shape-agnostic foundation model for PDEs that seamlessly handles data of varying dimensionality (1D-3D) at different resolutions. This will ultimately allow us to build a model from diverse set of training data. The architecture combines component-wise convolution, inter-field cross-attention, and axial attention. We train several variants and evaluate transfer to a range of downstream prediction tasks. Across extensive evaluations, MORPH matches or surpasses recent state-of-the-art models.
