The integration of Scientific Machine Learning (SciML) into computational simulations has transformed approaches to complex scientific challenges, particularly in national security applications. Numerous efforts are underway to verify the credibility of computational simulations by assessing geometry fidelity, evaluating physics models, and conducting code and solution verification, validation, and uncertainty quantification (UQ). However, the integration of machine learning black-box models into scientific simulations elevates the importance of establishing credibility, as these models often lack explicit geometric representations and mathematical equations. In this context, this presentation introduces an adapted framework based on the SciML Predictive Capability Maturity Model (PCMM), which encompasses critical components such as verification, validation, and UQ. We highlight the importance of explainability to bolster the credibility and reliability of SciML applications. Rigorous verification processes are essential, utilizing sensitivity analysis to ensure solution fidelity, which are vital for maintaining trust in computational models. Validation strategies focus on cross-validation techniques to assess model performance and generalizability across diverse datasets. We also addressed the role of UQ in quantifying prediction uncertainties, facilitating informed decision-making in critical scenarios. To enhance interpretability, we explored methods like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) that clarify feature contributions and improve transparency. Reproducibility is a core aspect of the PCMM framework, acknowledging the stochastic nature of machine learning that can introduce variability. This talk will focus on the SciML PCMM framework to assess the credibility of the developed models. This framework can be applied across various exemplars; we will specifically focus on the data-driven turbulence modeling exemplar, demonstrating a systematic approach to ensuring effective, transparent, and actionable insights. Through case studies involving Reynolds-averaged Navier-Stokes simulations, we will illustrate the practical application of the PCMM framework in refining predictive capabilities and enhancing confidence in the deployment of machine learning techniques in critical applications. This work is supported by the DOE-NNSA ASC program.