Learning accurate and physically consistent models for real-time prediction of multi-body dynamical systems is a central challenge for modern computational mechanics for digital twin applications. While data-driven approaches such as Graph Neural Networks (GNNs) offer a scalable alternative to classical solvers, many existing methods lack explicit physical guaranties, leading to poor long-horizon stability and limited generalization--particularly in systems involving non-central forces, dissipation, constraints, or boundary interactions. We present Dynami-CAL GraphNet, a physics-informed GNN that explicitly enforces the conservation of linear and angular momentum by embedding Newton’s third law directly into the model architecture. The method is designed for dynamical systems with interacting components and full six degrees of freedom, treating each interaction edge as a self-contained dynamical system. To this end, we introduce edge-local reference frames that are equivariant to 3D rotations, invariant to translations, and antisymmetric under node interchange, ensuring equal and opposite internal forces by construction. Within these edge-local reference frames, velocities and angular velocities are projected to form expressive invariant edge embeddings. Interactions are decoded through a physically grounded vectorization scheme that predicts antisymmetric internal forces, antisymmetric angular momentum transfer, and a reference point about which angular momentum is conserved. This formulation enables the correct separation of orbital and spin contributions, allowing the model to preserve total angular momentum even under non-central interactions. This goes beyond what symmetry alone can achieve. Dynami-CAL GraphNet further employs a spatiotemporal message-passing mechanism, in which edge embeddings are iteratively updated across spatial neighbors and multiple sub-time steps, enabling stable long-horizon rollouts despite training with single-step supervision. Boundary interactions are handled in a mesh-free and physically consistent manner using reflected ghost nodes, allowing for a uniform treatment of internal and boundary interactions. We validate the approach on a diverse set of benchmarks, including granular collisions, constrained N-body systems, articulated human motion, protein molecular dynamics, and Euler gas dynamics. Across all settings, Dynami-CAL GraphNet demonstrates improved physical fidelity and long-term stability.