The Material Point Method (MPM) has emerged as a powerful tool for simulating large deformation problems, characterized by the hybrid Eulerian-Lagrangian strategy. The MPM discretizes the complex material domain into a set of Lagrangian particles, while the processes of solving the momentum equations and imposing boundary conditions are performed on a background grid. However, the use of a conventional structured Cartesian grid struggles to represent complex boundaries, such as those encountered in landslides and debris flows. In this study, we explore the suitability of unstructured background grid composed of polygonal and polyhedral cells for modeling complex boundaries. The Wachspress coordinates are employed as the shape functions for polygonal and polyhedral cells, reducing to standard linear shape functions when the background grid is Cartesian. To accelerate the process of particle localization, we introduce a hash grid linked to the background grid, thereby enhancing computational efficiency. We apply both Dirichlet and frictional boundary conditions at the nodes of the background grid. The frictional boundary condition is specifically implemented via a trial-corrector scheme. The proposed method is validated through several benchmarks, including the vibration of a single particle within a polyhedral cell and the vibrating bar in prismatic cells. Furthermore, it has been successfully applied to practical engineering cases such as the simulation of the Wangjiayan landslide and flood flow over complex terrain, demonstrating its significant potential for real-world applications.