Structural damage identification using the topology optimization (TO) method has gained popularity in recent years. However, the iterative nature of TO-based methods often imposes a heavy computational burden. In contrast, surrogate models can expedite the identification process. Once a surrogate model is properly trained, its inference time becomes negligible, thereby significantly accelerating damage identification. More recently, the emergence of operator learning methods has demonstrated remarkable performance in solving parametric partial differential equations, injecting new vitality into the development of high-performance surrogate models. In this work, we propose a Fourier neural operator (FNO)-based method to directly predict the damaged structure from the frequency response data. Given that installing sensors across the entire domain is often cost-prohibitive in practice, full-field response data is unavailable; therefore, we explore scenarios utilizing only a sparse set of fewer than six sensors. We have tested two FNO-based directions to handle this sparse data input. The first direction expands the input FRF data into a 1D space along the frequency dimension, attempting a 1D input, and then uses 1D FNO for prediction. The second direction transforms the sparse FRF into a two-dimensional space using Voronoi embedding, and then uses a 2D version of FNO for prediction. The results demonstrate that while 2D FNO generally outperforms 1D FNO, an exception arises when the sensor count is fewer than three, at which point 1D FNO exhibits superior performance. This is because applying Fourier transforms to 2D embedding fields with low-frequency feature can lead to "zero-place" issues, causing training failure. FNO2D can achieve over 99% identification accuracy with 4 sensors. Additionally, the identification accuracy can be further enhanced through data augmentation or the strategy of perturbations.