Partial differential equations (PDEs) are a powerful and widely used tool for simulating complex physical systems. Many practical applications involve multi-phase, heterogeneous media—such as porous membranes, concrete, metal alloys, and fiber-reinforced composites [1]—where microscale structures strongly influence macroscale behavior. Solving the associated inverse problem, i.e., recovering an unknown microstructure from incomplete or noisy observations, is extremely challenging, since each forward PDE solve is already computationally expensive. Although Bayesian formulations of these inverse problems are principled, they are severely limited by the curse of dimensionality for realistic heterogeneous media. Recent deep learning approaches attempt to bypass these issues by learning direct inverse maps or by using generative models to infer conditional microstructure distributions. However, such models require large training datasets, lack physical consistency with the underlying PDE, struggle in extrapolative settings, and typically treat forward and inverse tasks with separate architectures. We propose GenPANIS, a unified generative model for jointly solving forward and inverse problems in two-phase heterogeneous media. Physical knowledge is embedded directly into the architecture via a vectorized, auto-differentiable coarse-grained solver [4]. Microstructures are efficiently compressed into a low-dimensional latent space, accelerating and regularizing the inverse solution. We demonstrate GenPANIS on the heat and Helmholtz equations in two-phase media. The model significantly outperforms state-of-the-art methods [2, 3], despite having orders of magnitude fewer parameters—on challenging out-of-distribution forward and inverse problems with sparse and noisy observations.