We present IAGO, a framework for design optimization. The method uses latent conditional neural fields trained to represent shapes as signed distance functions (SDFs) that satisfy manufacturability and geometry constraints. After training, Bayesian Optimization (BO) in the latent space identifies high- and low-performing designs by calling a physics solver to compute domain-specific metrics. Thereafter, a refined model is trained with the additional objectives of generating solutions that resemble high-performing designs, while discouraging recurrence of low-performing ones. This two-steps cycle of model training and BO is repeated until performance requirements are met. Building on prior work "[Berzins et al., Geometry-Informed Neural Networks]", IAGO integrates knowledge of prior designs. Further, in contrast to other solver-in-the-loop methods such as TOM "[Radler et al., Diverse Topology Optimization using Modulated Neural Fields]", IAGO requires fewer solver calls and does not rely on gradients of the cost function, as common in topology optimization ("[Radler et al., Diverse Topology Optimization using Modulated Neural Fields]", "[Hoyer et al., Neural reparameterization improves structural optimization]"). Experimental results show that IAGO produces high-performing and manufacturable designs after a few cycles. In this talk, we will provide a few examples of IAGO's application to structural mechanics problems.