The modular building, known for the flexibility in fast on-site construction with prefabricated modules, is a highly popular option for constructing high-rise residential housing, emergency hospitals, and business office buildings. However, accurate nonlinear seismic simulations for these systems still remain computationally intensive, especially for higher precision models with over million degrees of freedoms (DOFs), requiring one week on numerical methods to obtain solution fields. To address the spectral bias of conventional deep neural networks and low-frequency filtering mechanism of Fourier Neural Operator (FNO), we propose and develop a novel frequency multi-band neural operator solver that capture both global low-frequency structural responses as well as the localized high-frequency vibrations, which is critical for constructing high-fidelity simulation solvers for modular building systems. A coarse-to-fine learning strategy is adopted to ensure the computational efficiency based on the operator learning capability. The model first is trained on a down-sampled full corner nodal cloud and subsequently transferred to reconstruct the full field solutions on a higher resolution. Our model is trained on seven down-sampled earthquake scenarios (50Hz) and tested on three unseen down-sampled cases of modular buildings. With the pretrained model, higher resolution full field results are validated with the numerical simulations. On the down-sampled case, the developed solver can achieve a relative L2 error below 5%, while extending to more than 200,000 nodes the solver still can achieve top floor seismic structural responses L2 error below 10% through the coarse-to-fine method. Comparing with the precise numerical simulations with one week time cost, the proposed framework can simulate a scenario within 100s, which is offering more than 6,000 magnitude speedup and can be utilized to conduct fast evaluation of modular buildings responses near real time.