Reinforcement learning (RL) has shown promising results in active flow control (AFC), yet progress in the field remains difficult to assess as existing studies rely on heterogeneous observation and actuation schemes, numerical setups, and evaluation protocols. Current AFC benchmarks attempt to address these issues but heavily rely on external computational fluid dynamics (CFD) solvers, are not fully differentiable, and provide limited 3D and multi-agent support. To overcome these limitations, we introduce FluidGym, the first standalone, fully differentiable benchmark suite for RL in AFC, built on the GPU-accelerated PICT solver. Key features of FluidGym include: (i) plug-and-play benchmark-ing with no external CFD dependencies, (ii) end-to-end differentiability for gradient-based optimization, (iii) multi-agent RL (MARL) for decentralized control, and (iv) high-fidelity 3D simulations for realistic flow scenarios. Our benchmark includes 13 environments spanning 2D and 3D flow scenarios, such as flow past a cylinder, Rayleigh-B´enard convection, airfoil flow, and turbulent channel flow. Each environment is offered at three difficulty levels to capture increasing turbulence and complexity. FluidGym is available at https://github.com/safe-autonomous-systems/fluidgym. Baseline experiments using Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) demonstrate that SAC consistently outperforms PPO in convergence speed and final performance across environments. Results also highlight the potential of MARL for learning coordinated control policies, e.g., as observed in the 3D Rayleigh-Bénard convection task. The standardized protocols in FluidGym, including train/val/test splits and reproducible initial conditions, ensure robust comparisons of algorithms. Significance. FluidGym lowers the entry barrier for RL research in AFC by providing an accessible, reproducible, and scalable platform, enabling systematic comparisons of control methods. All environments, datasets, and trained models are released as public resources.