Machine learning techniques have shown great potential for the active control of fluid flows. In particular, the application of reinforcement learning techniques, which learn from experience in simulation, has yielded promising results in tasks such as the shape optimization of airfoils, drag reduction past cars, and flow control in chaotic regimes. Despite these advances, the high computational cost associated with acquiring sufficient simulation data remains a major challenge. Moreover, practical control policies should be able to quickly adapt to changing system dynamics, motivating the development of more data- and compute-efficient learning approaches. To address these challenges, we investigate the use of differentiable simulators and data-driven surrogate models that provide gradients of objective quantities, such as rewards or control objectives, with respect to control inputs. These gradients enable the incorporation of gradient-based optimization techniques into reinforcement learning and control frameworks. We present several approaches for leveraging such gradients within RL and control algorithms. The methods are evaluated on a set of benchmark problems, including Rayleigh–Bénard convection, for which we develop a differentiable numerical simulator, as well as established benchmark environments. The performance of gradient-based and gradient-free control methods is compared in terms of control effectiveness and training efficiency. In addition, we study how the source of gradient information, obtained either from differentiable simulations or from data-driven surrogate models, affects control performance and training efficiency.