The increasing density of orbital debris poses a significant threat to the sustainability of our space environment. Collisions and breakups trigger an uncontrolled proliferation of fragments, heightening the risk of subsequent impacts; a cascade effect known as the Kessler syndrome [1]. To assess these risks, space agencies rely on breakup models, which in turn depend on empirical parameters. Physics-based models of dynamic fragmentation offer a path to significantly refine these models. Such simulations must capture multiple time scales---from crack initiation and propagation to secondary fragment interactions---making numerical robustness and computational efficiency a significant challenge. Prior work showed that combining explicit time integration with an extrinsic cohesive-zone model and penalty-based contact leads to severe instabilities [2]. While negligible over short durations, these instabilities worsen over time, introducing artificial fragmentation and energy errors that compromise the reliability of fragment statistics. To address these limitations, we propose an impulse-based contact treatment that accounts for the nonsmooth nature of contact mechanics. This approach is embedded implicitly within an explicit time-integration framework, leveraging the nonsmooth Newmark-β scheme [3]. Benchmarking against dynamic fragmentation scenarios demonstrates that this method restores energy conservation and enhances long-term robustness. Despite an overhead of the implicit contact solver, the increased stability permits larger time steps, yielding computational efficiency comparable to, or exceeding, that of explicit penalty-based methods. This work establishes impulse-based contact as a reliable and efficient alternative for long-term fragmentation modeling. [1] Kessler, D. J., and Cour-Palais, B. G., \emph{Collision frequency of artificial satellites: The creation of a debris belt}, Journal of Geophysical Research: Space Physics, Vol. 83, No. A6, pp. 2637--2646, 1978. [2] Ghesquière-Diérickx T., Molinari J.-F., Anciaux G., Stability of extrinsic cohesive-zone model with penalty-based contact in explicit dynamic fragmentation simulations, Mechanics of Materials, 105581, 2025. [3] Chen Q.-Z., Acary V., Virlez G., Brüls O., A nonsmooth generalized-α scheme for flexible multibody systems with unilateral constraints, International Journal for Numerical Methods in E