In this talk, we investigate the use of single hidden-layer neural networks as a family of ansatz functions for the resolution of partial differential equations (PDEs). We are interested in how known properties of the solution can be enforced on the network, both before and during the training. In particular during the training we optimize only the weights in the output layer, also known as Extreme Learning Machine (ELM), while the weights between input and hidden layer can be sampled randomly from a distribution that reflects the expected regularity of the solution. This allows faster optimization and much better generalization properties. We have investigated how linear constraints on the solution, such as linear boundary conditions, can be enforced through modified activation functions (before training) and integrating techniques from Linear Algebra during the training. In both cases we provide accuracy and convergence results for linear and nonlinear PDEs.