Classically, model order reduction for parameterized systems is based on a so-called offline phase, where reduced order models (ROMs) are constructed, followed by an online phase, where the ROMs can be cheaply evaluated in a multi-query context. In this contribution, we present a comprehensive multi-fidelity framework for accelerating solution processes in parameter optimization problems with parabolic PDE constraints or inverse problems, focusing on the efficient on-the-fly learning of ROMs. We first introduce a novel hierarchical reduced basis - machine learning - reduced order model (RB-ML-ROM) framework which is adaptively refined during parametric requests using rigorous a posteriori error estimates to certify accuracy and control adaptation. We then address computational bottlenecks in PDE constrained optimization problems by leveraging the adaptive RB-ML-ROM techniques to accelerate state and adjoint evaluations in derivative-based optimization methods like the reduced basis - trust region approach. Finally, we generalize to inverse problems with high or infinite dimensional parameter spaces and discuss adaptive ROM construction that combines parameter and state space reduction and is embedded within an error-aware trust-region iteratively regularized Gauss-Newton method (TR-IRGNM) guaranteeing reliability. We demonstrate effectiveness in time-dependent parameter identification problems in dynamic reaction-diffusion systems. Numerical results from parameter optimization, and inverse problems showcase the efficiency gains achieved through these adaptive, error-certified reduced order modeling techniques.