Hierarchical materials exhibit complex architectures which are built from constituents that are of a size comparable to the structural features [1]. Due to size dependencies and manufacturing variabilities, material properties measured independently from the structure and manufacturing processes do not describe how the material will behave as part of a structure, since their properties do not include the impact of geometric features or presence of defects/variabilities [2]. This challenge is a multi-scale problem, where the complete response of a structure is a function of the structural loads and damage progression on the material architecture-scale. Non-linear multi-scale modelling has emerged as an essential tool for simulating the material and structural behaviour. However, the computational expense of these models is prohibitive as a material discovery tool, which led to the development of several promising multi-scale modelling, AI-Driven surrogates and technologies, aiming to accelerate the simulation process and the material discovery process. This paper presents a novel modular approach to multi-modal spatial-temporal deep learning for multi-scale modelling of material non-linear behaviour. The system is built on an extensive model database of composite materials response including various damage modes (delamination, matrix cracking, plasticity and fibre failure) and covering a number of common manufacturing defects (wrinkles, gaps and voids). This modular approach combines variational autoencoders for material architecture learning, transformer modules for short term time dependencies and spiking neural networks for long term history learning. The modular nature and the built-in explainability of the system make it a considerable step forward from previous technologies based on convolutional recurrent learning [3]. The proposed system provides considerable computational advantages during inference as part of multi-scale finite element simulations, since only the light-weight spiking module needs to be instanced for integration points. Additionally, built-in explainability provides objective metrics for model robustness when used in conjunction with multi-scale simulations.