When employing Neural Networks as trial functions for solving Partial Differential Equations, solutions are obtained by minimising loss functions defined through integrals. Unlike trial spaces in Finite Element Methods (FEM), exact quadrature rules are unavailable, making integration errors unavoidable. In particular, Gauss-type rules may yield incorrect results analogous to overfitting. First, we analyse the impact of bias in quadrature choice. We demonstrate that employing biased stochastic quadrature rules can lead to incorrect solutions, even when they are asymptotically exact. To remedy this, we utilise unbiased rules. Second, we focus on variance reduction, which is essential for rapid convergence. We propose new unbiased Gauss-type rules for triangular and tetrahedral elements, giving greater flexibility in complex geometries. They are exact for polynomials of a prescribed order, and substantially improve convergence rates compared to standard Monte Carlo methods. Then, we discuss a quasi-interpolation property that yields automatic variance reduction as the neural network approaches the solution. We compare discretisations that satisfy this property (e.g., Physics-Informed Neural Networks and First-Order System of Least Squares) against those that do not (e.g., the Deep Ritz Method and Variational PINNs), observing that the former attains solutions an order of magnitude more accurate for the same computational cost. Finally, we examine hybrid optimisation schemes combining gradient-based methods with least-squares solvers. These approaches accelerate convergence and reduce spectral bias, extending naturally to linear parametric problems. We analyse how integration errors can introduce bias in these schemes and propose an accumulation strategy for reducing this bias. We consider a parametric two-dimensional transmission problem to illustrate the numerical results, achieving energy-norm errors of 1–2%, with all integrals evaluated using fewer than 200 points, provided that proper integration rules are employed.