Accurate drag prediction for non-spherical particles depends not only on the regression model but also on how particle geometry is represented. Hand-crafted descriptors are interpretable and effective for regular or mildly perturbed shapes, but their sufficiency becomes uncertain for multi-scale irregularities or unseen surface perturbations. This study examines whether learned representations derived from full-field geometry provide a more robust alternative under controlled geometric complexity. Particle geometries are converted into voxelized occupancy fields, and a task-driven variational autoencoder learns compact latent representations for drag prediction. The latent representation is coupled with a multilayer perceptron and evaluated against descriptor-based neural networks and empirical correlations using matched CFD datasets. The datasets include regular ellipsoids, low-order spherical-harmonic perturbations representing global deformation, and high-order perturbations representing small-scale surface irregularity. For regular ellipsoids, the learned latent space organizes aspect ratio and incident angle into interpretable geometric coordinates while achieving descriptor-level accuracy. Under in-distribution low-order and multi-scale perturbations, both learned and descriptor-based neural models remain accurate, but the latent-representation model shows more stable errors as high-order roughness increases. In the out-of-distribution experiment, where models are trained only on smooth particles and tested on unseen high-order perturbations, the descriptor-based model degrades markedly, whereas the learned representation maintains stable accuracy and captures the dominant direction of the roughness-induced drag change. Mahalanobis analysis shows that high-order perturbations create a severe artificial shift in descriptor space. Latent perturbation analysis further reveals that unseen roughness is mapped to a structured displacement that partially overlaps with known geometry-related latent directions, explaining the observed robustness. These results show that task-informed latent representation learning preserves drag-relevant geometric information beyond predefined descriptors and provides a robust framework for geometry-to-physics prediction of irregular particles.