Stochastic optimization with high-fidelity, nonlinear models is often prohibitively expensive, as it requires repeated sampling-based estimation of model statistics within an outer optimization loop. Multi-fidelity estimators---such as approximate control variates (ACVs) and multilevel best linear unbiased estimators (MLBLUEs)---can dramatically reduce these costs by leveraging inexpensive low-fidelity models to accelerate estimation of high-fidelity quantities of interest. However, the effectiveness of such estimators depends critically on hyperparameters, including control-variate weights and sample-allocation ratios, which in turn require knowledge of model evaluation costs and cross-model output covariances. In practice, these covariances are rarely known a priori and are typically estimated using independent pilot sampling, which can be inefficient and poorly targeted. Although prior work has studied trade-offs between pilot sampling and multi-fidelity estimation costs, pilot sampling strategies have largely been decoupled from the needs of an outer stochastic optimization loop. We present an adaptive pilot-sampling framework that enables the systematic use of multi-fidelity estimators, such as ACV, within general iterative stochastic optimization algorithms. The approach combines probabilistic covariance emulation with an active-learning strategy that adaptively allocates pilot samples across the design space, focusing effort on regions that are most influential for the optimization process. This yields more accurate covariance estimates where they matter most, leading to improved multi-fidelity hyperparameter selection and overall optimization efficiency. We evaluate several variants of the proposed framework on numerical benchmarks and demonstrate its effectiveness in a multi-fidelity Bayesian optimal experimental design problem, showing gains over static or design-agnostic pilot-sampling strategies.