The training of scientific machine learning (SciML) models is highly resource-intensive due to stiff and ill-conditioned optimization landscapes induced by the spectral properties of underlying differential operators. In this work, we aim to accelerate the training of SciML models through novel domain decomposition (DD)-inspired algorithms that enable model parallelization. To this end, we propose a second-order DD-based AdaGrad optimizer, which divides the model parameters into multiple groups/subdomains. Each subdomain is then trained independently using a variant of the AdaGrad optimizer. Finally, global training steps are performed regularly to synchronize the locally trained parameters. In this talk, we will analyze the impact of different decomposition strategies, the interplay between global and subdomain learning rates, the frequency of global synchronization, and the transition between local and global training phases on the convergence of the proposed DD-AdaGrad algorithm. Special focus will be given to strategies for efficiently combining subdomain corrections obtained in parallel. The empirical performance of our algorithm will be demonstrated through a series of numerical experiments encompassing the training of SciML models, such as physics-informed neural networks or operator learning approaches.