Physics-Informed Neural Networks (PINNs) have shown their capability in solving both forward and inverse problems across various fields. In their classical formulation, a neural network approximates the unknown solution such that the strong form, alongside given initial and boundary conditions, is satisfied as closely as possible. This framework has been extended in prior works by incorporating the variational form into the loss function, leading to Variational Physics-Informed Neural Networks (VPINNs). Unlike PINNs, VPINNs theoretically allow both h- and p-refinement, providing greater flexibility for applications such as surrogating high-fidelity finite element models. However, previous studies have primarily relied on Legendre test functions, which have not yielded the expected accuracy gains from p-refinement. This work specifically investigates the role of test functions in VPINNs. Here, we introduce a set of test functions designed to avoid having a zero mean over the integration interval. These test functions are evaluated and compared to Legendre test functions on a specially designed problem requiring a steep solution and an inverse problem. Our study demonstrates that, unlike Legendre-based test functions, the proposed test functions reduce the prediction error without increasing computational cost. Lastly, the proposed test functions enhance the reliability of VPINNs, yielding results with consistent accuracy over repeated network training runs.