Polycrystalline structural materials operating in extreme conditions accumulate point defects that govern long-term microstructural evolution and mechanical degradation. Atomistic simulations are well-suited to studying this defect evolution, particularly under fusion environments, which are challenging to probe experimentally. However, conventional Wigner--Seitz (WS) analysis becomes unreliable for defect detection in polycrystals that undergo severe damage because it relies on initial atomic positions rather than actual lattice sites, leading to spurious vacancies and interstitials near grain boundaries that translate, rotate, or reconstruct. We address this challenge with a scalable defect-analysis workflow that combines common neighbor analysis with graph-theoretical pattern recognition (CNA-GT) to enable type-resolved identification of point defects directly from instantaneous atomic configurations. The workflow (i) filters out atoms in ideal crystalline environments via CNA, (ii) represents the remaining disordered atoms as graphs, (iii) removes extended grain-boundary networks, and (iv) matches the residual local motifs against a library of vacancy and interstitial signatures. This approach eliminates interface-induced artifacts, provides accurate bulk vacancy and interstitial statistics in large-scale molecular dynamics simulations of collision cascades in bicrystalline bcc W. Together, these features enable robust defect mapping in highly distorted polycrystalline microstructures.