Artificial intelligence presents a transformative opportunity to overcome the computational bottlenecks of traditional particle tracking methods in plasma physics. While existing neural network surrogates—such as Physics-Informed Neural Networks (PINNs) and Fourier Neural Operators—offer efficiency, they rely on soft physical constraints, often compromising strict adherence to fundamental conservation laws. Furthermore, current symmetry-aware models prioritize geometric invariants, neglecting critical spatiotemporal symmetries like translation invariance, which governs momentum conservation. This study pioneers a symmetry-hardened neural operator framework explicitly designed for charged particle dynamics. Our core innovation lies in structurally embedding spatial translation invariance directly into the network architecture, ensuring strict physical consistency without ad-hoc regularization. Methodologically, we: (1) establish the theoretical foundation for translation-invariant neural operators; (2) design a novel architecture that intrinsically satisfies this symmetry; and (3) validate the framework through comprehensive benchmarks against state-of-the-art methods. Results demonstrate that our approach significantly outperforms baseline neural operators and symmetry-regularized Hamiltonian/Lagrangian networks in both accuracy and generalizability to unseen field configurations. By rigorously enforcing a cornerstone physical symmetry at the architectural level, this work bridges a critical gap, offering a new paradigm for efficient, physically consistent simulations of charged particle dynamics.