In engineering applications, such as solar panels or electronic devices, brittle polycrystalline materials are used. In these materials, fracture can occur both along grain boundaries (intergranular) and through the grain (transgranular); and is strongly affected by the microstructural features of such material. Hence understanding failure mechanisms is of great importance. Obtaining a unified transgranular and intergranular fracture model is challenging for following reasons: (a) the material behaves anisotropic both in terms of elastic material properties and fracture toughness. In addition, the fracture toughness varies along the grain boundaries; (b) requires conforming discretization techniques that are tailored to the complex grain geometries; and (c) crack propagation modeling is associated with difficulties related to crack initiation criteria, propagation direction, re-meshing and corresponding numerical efforts. The phase field method (PFM) for brittle fracture modeling [1] has recently gained popularity as the criteria for crack propagation are inherently handled in the mathematical model. Phase field modeling of fracture in polycrystalline materials has been addressed in [2]. In PFM, a length scale parameter is introduced and this imposes additional constraints on the size of elements in the vicinity of the crack propagation. Hence, a meshing strategy that supports rapid size transition and that conforms to the polycrystal structure is desirable. In this context, polygonal meshes provide great flexibility. This work extends an adaptive phase field approach to fracture based on scaled boundary finite element method (SBFEM) that was previously suggested [3] to model fracture in polycrystalline materials. SBFEM can be applied to elements of any number of edges and can elegantly handle elements with hanging nodes. The anisotropic fracture behaviour and grain orientation are taken into consideration by modifying the phase field equation. A second phase field approximation is introduced by modeling the grain boundaries in a smeared manner. The suggested method will be thoroughly described, and numerous examples will be used to demonstrate accuracy and efficiency.