Crystal plasticity simulations are computationally prohibitive for materials design applications requiring extensive parametric studies. Spectral-based clustering reduced order models, derived from FFT full-field formulations [1], provide a promising fast alternative to full-field simulations, particularly for large-deformation crystal plasticity. However, their accuracy limitations and dominant sources of error remain insufficiently understood. This work systematically investigates the parameters governing the accuracy of spectral-based cluster models [2] while explicitly identifying their intrinsic limitations and underlying error mechanisms. The effects of cluster resolution, clustering criteria for phase-space grouping, and the choice and evolution of the elastic isotropic reference medium are evaluated. Representative microstructural volume elements are subjected to multiple loading conditions and analyzed using different numerical solution strategies. Systematic comparisons are performed by varying the number of clusters, clustering metrics, and reference medium definitions. The results show that the choice of clustering criteria has a limited influence on the macroscopic response, while increased cluster resolution improves accuracy at a high computational cost with steep scaling. Crucially, the selection and updating of the reference medium is identified as the primary source of error in spectral-based cluster models, strongly affecting both convergence and predictive accuracy. This analysis exposes fundamental limitations of current clustering approaches and delineates clear directions for improving robustness, with direct implications for extensions to damage and strain localization modeling.