Computational models of cardiac mechanics are limited in their clinical translation by computational cost. Surrogate models based on neural network offer an alternative by using the high-fidelity model only during training and providing cheap evaluation at inference time. However, existing approaches struggle when applied across cohorts with varying anatomical geometries, particularly when point-to-point correspondence cannot be established between different patient anatomies. In this work we present a framework for encoding geometrical variability into a neural operator trained to predict ventricular displacement vector fields under diastolic loading conditions. We consider two different methods for representing geometrical variability. The first one uses principal component analysis (PCA) combined with Universal Ventricular Coordinates (UVCs) to establish correspondence between geometries. In the second we implement a meshless method based on DeepSDF, a neural network model that represents implicitly a geometry by approximating its signed distance function, learning directly from point clouds and without requiring any mesh generation or anatomical correspondence. The DeepSDF model encodes geometry by automatically learning a lower-dimensional latent space that captures the geometrical variability: this enables shape interpolation, inference from sparse data and generation of new synthetic geometries. This geometric encoding conditions the physics surrogate model that predicts finite strain mechanical deformations. We validate the method firstly on 512 idealized left ventricular geometries and then apply it 44 patient-specific anatomies augmenting the dataset with synthetic geometries and demonstrating accurate shape reconstruction and displacement field prediction. We will discuss the difference between the two methods in terms of their data pre-processing constraints and performance scaling with respect to noise and numerosity based on numerical results.