Concrete is the most used construction material worldwide, but its heterogeneous nature makes reliably predicting its structure-property-performance relationships challenging. Many research questions remain about how the shape, size, and orientation of aggregates influence concrete's macroscale properties. In particular, the interfacial transition zone (ITZ) between the aggregate and cement phases is particularly weak and is believed to promote crack propagation around aggregates. In this work, we present a detailed macroscale 3 phase finite element (FE) model (cement, ITZ, and aggregate) to investigate the effects of aggregate on the ballistic properties of concrete. The FE code ALE3D is used to embed the aggregate into a background Arbitrary Lagrangian-Eulerian (ALE) mesh which allowed for the rapid generation, meshing, and simulation of complex aggregate shapes in 2D and 3D. We studied the effects of aggregate size, shape, orientation, strength, and volume fraction and compared the energy dissipation and damage during impact. Furthermore, we investigated the influence and strength of the ITZ to determine how cracks propagate around and through the aggregate. The findings are discussed in terms of generating optimal concrete mixes for enhanced ballistic protection.