Laser-based additive manufacturing (AM) offers several advantages, such as high precision, material efficiency, and flexibility in designing complex geometries, making the technology ideal for high-performance, customized applications. However, the technology is highly sensitive to process/control parameters, introducing challenges in obtaining consistent quality. A strong causal process-microstructure mechanical property relationship dominates the production, introducing spatial heterogeneity in the properties of the manufactured part. This motivates development of computational tools for predicting such properties (e.g. the local crystallographic texture) that can tackle the ultra-high speeds and extremely fine spatial scales associated with metal AM. Physics-based simulation of microstructures through phase field (PF) and thermal finite element models (FEM) are typical tools for computing the system dynamics; however they are computationally expensive. Deep learning (DL) has achieved remarkable breakthroughs in recent years, with an emerging trend of developing surrogate models for the AM process, e.g., to predict crystallographic properties, temperature gradients, microstructural evolution, and other material properties. In this study, we investigate the potential of DL models with recurrent architectures to predict the spatio-temporal evolution of microstructures in laser-based AM. A calibrated sequentially coupled FEM-PF model was used to generate synthetic data (containing temperature, meltpool information and microstructure characteristics) for training the DL models. We studied three recurrent neural architectures, which are efficient for solving spatiotemporal learning tasks, namely predictive recurrent neural networks, long-term recurrent convolutional networks, and reservoir computing. The results reveal the potential of recurrent tools to generate fast reduced models for temperature and microstructure prediction, which can be applied for online quality control and anomaly detections, as well as off-line for more efficient planning and process optimization.