Deep reinforcement learning (DRL) is transforming our ability to design, operate, and maintain our engineered world beyond traditional optimization and human-engineered heuristics. This is largely due to the ability of DRL agents to systematically probe digital twins that forecast uncertain future performance dynamics of complex systems, while adaptively updating their latent parametric structures through advanced probabilistic inference engines. This convergence signals a broader paradigm shift for life-cycle engineering, where intervention policies for engineering systems are designed autonomously by leveraging tightly coupled digital models and real-world data. Automating decision support, however, remains constrained by unresolved trade-offs among optimality, scalability, and computational cost in sequential decision-making under uncertainty. At one end of the spectrum, model-free decentralized multi-agent DRL has been shown to improve tractability in large multi-component systems, yet may induce coordination pathologies, limit achievable optimality, and demand extensive sampling from expensive simulators. At the other end, traditional model-based DRL, that could alleviate some of these complexities, relies on implicit representation learning of underlying world models in high-dimensional latent manifolds, leaving little room for engineering interpretation and oversight. Bridging these extremes motivates mechanisms that explicitly exploit problem structure. In this talk, we present recent advances in structure-aware DRL, from model-based influence abstractions that summarize interdependencies into compact engineering-inspired representations, to hierarchical multi-scale decomposition architectures that introduce resource-aware cooperation across system levels and facilitate near-optimal coordination through value-based exploration of combinatorial decision spaces. We argue that these mechanisms outline a pathway toward scalable, near-optimal, and sample-efficient agentic decision-making for engineering systems, where abstraction, hierarchy, and decentralization disentangle the intrinsic complexities of digital twins, physics-based simulation, and uncertainty quantification. Applications across structural, transport, and general reliability systems highlight the computational potential and socioeconomic impact we can achieve. Remaining scientific challenges and future directions for structured DRL-driven intervention planning are finally discussed.