This work presents a design optimization method for multi-material structures that undergo contact and account for finite deformations and friction. The method uses level-set functions to describe external boundaries and material interfaces. The structural response is predicted by a finite-strain, hyperelastic model, with contact between components and self-contact idealized as either frictionless or with Coulomb friction. The state variable and level-set fields are discretized using the eXtended IsoGeometric Analysis (XIGA) approach, which employs locally refined meshes. The optimization problems aim to enhance the overall stiffness and/or achieve a desired contact pressure as a function of external load, while imposing constraints on maximum stress, feature size, and curvature. These problems are solved using a nonlinear programming method, with shape sensitivities computed via a semi-analytical version of the adjoint method. Numerical examples in both 2D and 3D demonstrate the effectiveness of the proposed approach for multi-material problems. The results also underscore the importance of high-order shape descriptions, the formulation of the contact conditions in an immersed setting, and the need for feature size and curvature constraints.