The properties and performance of polycrystalline metal alloys are greatly influenced by voids given their significant effect on the initiation and evolution of damage. The integration of crystal plasticity theories into finite element methods (CPFEM) protocols are capable of providing fundamental understanding of the underlying behavior that connects the internal polycrystalline structure to its corresponding damage behavior and evolution. Nevertheless, despite their accuracy, CPFEM-based linkages are unable to efficiently explore the extensive crystallographic orientation space given their iterative nature coupled with the fact that they need significant computational resources. Moreover, CPFEM-based linkages are not capable of extracting knowledge from previously obtained results since each simulation yields large amount of data that complicates the development of accurate surrogate models. Therefore, there is a critical need for developing protocols that can distill knowledge from the vast amount of information generated and as such enable a computationally efficient linkages between the crystallographic orientation and the resultant damage behavior and its evolution when a void is present. In this work, we address this challenge by obtaining a robust latent representation of the complex distribution function of crystal orientations using Generalized Spherical Harmonics (GSH) and parametrizing the extreme values of the evolution of the damage field of a single crystal with an embedded spherical void. Subsequently, we leverage recurrent neural networks (RNNs) to connect the latent representation to the parameterized distribution evolution of extreme values to effectively predict the evolution of damage in a computationally efficient, yet accurate manner. The resultant model is highly accurate and capable of predicting damage performance and its evolution at a fraction of the cost. Furthermore, the insights and predictions obtained with the model showed great agreement with high-fidelity CPFEM simulation results. SNL is managed and operated by NTESS under DOE NNSA contract DE-NA0003525. SAND No. SAND2025-14520A