We investigate a numerical method for the Vlasov-Poisson (VP) system with asymmetrically-weighted (AW) Hermite bases in velocity space, which is a hyperbolic system. In particular, we focus on spectral methods in velocity space. For the Hermite spectral form of the VP system, we analyze the reason why the form with AW Hermite bases can be unstable. To obtain $L^2$ stability properties, we consider a Galerkin method instead of the classical Petrov-Galerkin method, which naturally ensures stability with respect to the $L^2$ norm. We also present an equivalent form of the method that maintains a computational cost comparable to that of the Petrov-Galerkin method. Finally, we present numerical simulations based on the proposed Hermite spectral method, showcasing its stability and effectiveness.