The use of multi-materials in structural design has recently increased to achieve both structural efficiency and multifunctionality. Topology optimization is widely applied to handle the resulting complex material distributions. Multi-material additive manufacturing (AM) is a key process for realizing such structures. However, frequent material switching leads to increased production time and material purging waste. It also causes quality issues, such as reduced interlayer bonding due to thermal imbalance. Existing studies have attempted to mitigate these issues by optimizing paths at the slicing stage. However, these post-processing approaches have fundamental limitations in reducing the number of switches because the geometry is already fixed. Therefore, to ensure production efficiency and quality, material switching must be considered during the structural design stage. Moreover, this process requires the simultaneous integration of manufacturability constraints, including overhangs, nozzle collisions, and minimum length scales. To address these limitations, this study proposes a novel multi-material topology optimization method that considers material switching. We utilize a pseudo-time concept to define the fabrication sequence for each material. This approach encourages continuous printing and reduces the number of switches. Furthermore, we achieve the integration of design and manufacturing by formulating the problem to satisfy AM-specific geometric constraints directly within the design phase. Numerical examples demonstrate that the proposed method significantly reduces the number of material switches compared to conventional layer-wise approaches. The methodology is also flexible, applying effectively to both single and multi-build orientations as well as varying numbers of process stages. Simultaneously, the method ensures practical manufacturability by satisfying complex constraints such as material-dependent overhangs. In conclusion, this study contributes to improving both production efficiency and structural performance by realizing the integration of design and manufacturing through a topology optimization formulation that internalizes material switching costs.