We discuss how to efficiently construct data- and problem-adapted random feature neural networks to solve partial differential equations. In computational experiments, we demonstrate that the constructed networks are orders of magnitude faster to train and similarly orders of magnitude more accurate than networks trained with gradient descent. We also discuss computational bottlenecks, challenges with higher-dimensional base spaces, and paths towards a better understanding of the similarities of finite element and neural network based discretizations.