Scientific deep learning has enabled accurate surrogate modeling of complex, PDE-governed systems, yet the opacity of many deep learning approaches limits their use for scientific understanding. In particular, most neural operator and autoencoder-based reduced-order models achieve efficiency and accuracy at the cost of interpretability. Additionally, post-hoc explainers require further calculations and are known to sometimes produce inconsistent results. This work addresses these limitations by introducing two complementary explainable AI strategies for operator learning and reduced-order modeling First, we introduce the DIfferentiable Autoencoding Neural Operator (DIANO), a deterministic, mesh-invariant framework that performs nonlinear dimensional and geometric reduction while preserving interpretability. DIANO learns an interpretable coarse-grid latent space using neural operators and embeds fully differentiable PDE solvers directly within this latent space, enabling physics-consistent temporal evolution governed by differential equations rather than purely data-driven dynamics. This design allows physical priors to shape latent representations and facilitates direct visualization of latent dynamics in a grid, as opposed to the more prevalent autoencoder modes that cannot be visualized in space. We demonstrate DIANO's reconstruction accuracy, data compression capability, and interpretability on benchmark fluid flow problems. Second, we present a self-explainable operator learning framework that reformulates operator learning as a linear combination of integral equations. By decomposing operators into learnable localized kernel-based contributions, the model provides a by-design explainable AI (XAI), directly linking input spatial features to output patterns. Applied to function-to-scalar and function-to-function mappings in fluid dynamics, the framework reveals physically meaningful spatial dependencies and agrees with SHAP (an established post-hoc tool), while extending XAI methods to produce output-level linear decompositions. Together, these approaches advance interpretable operator learning by embedding differentiable physics directly into model structure or by leveraging the spatial decomposition of integral equations.