Physics-Informed Neural Networks (PINNs) have become a powerful framework for solving partial differential equations (PDEs), embedding physical constraints directly into neural network training objectives [1, 2]. Despite their empirical success, PINNs are often difficult to train, and the optimization mechanisms governing their convergence and stability remain poorly understood. Recent work in deep learning has identified the edge-of-stability phenomenon, which can help us understand why it's so hard to train PINNs.[3, 4, 5]. In this work, we investigate how edge-of-stability dynamics manifest in PINN training by analyzing the role of Hessian subspace structure. We compare standard stochastic gradient descent (SGD) with two subspace-restricted variants: dominant-subspace SGD (Dom-SGD), and bulk-subspace SGD (Bulk-SGD). We regularize PINNs Loss by tuning the penalization coefficients of data and residual losses. These methods are evaluated across a suite of elliptic, parabolic, and hyperbolic PDE benchmarks, including problems with high-frequency forcing, boundary layers, limited regularity, and sharp transitions. Our experiments show that Dom-SGD generally fails to reduce the training loss except for the simplest PDEs involving low-order derivatives and smooth solutions. In contrast, Bulk-SGD achieves accuracy comparable to full SGD while exhibiting significantly smoother and more stable loss descent. Yet it sometimes fails in the long run. These findings indicate that effective PINNs learning is governed primarily by dynamics in the bulk of the parameter space, but show an important gap that could boost the performance of PINNs in general. This study provides new insight into the spectral mechanisms underlying stable and efficient PINN training.