Recent experiments, atomistic simulations, and theoretical predictions have identified various new types of grain boundary motions that are controlled by the dynamics of underlying microstructure of line defects (dislocations or disconnections), to which the classical motion by mean curvature model does not apply. Although different continuum models have been developed, these continuum models are not in the variational form. We propose a unified variational framework to account for all the underlying line defect mechanisms for the dynamics of both low and high angle grain boundaries and the associated grain rotations. The variational formulation is based on the developed constraints of the dynamic Frank-Bilby equation that governs the microscopic line defect structures. The proposed variational framework is able to recover the available models for different motions under different conditions. The unified variational framework is more efficient to describe the collective behaviors of grain boundary networks at larger length scales. It also provides a mathematically tractable basis for rigorous analysis of these partial differential equation models and for the development of efficient numerical methods.