Data-driven decisions for infrastructure management and planning motivate digital twins as a framework that couples data and actions between a physical asset and its digital representation. In a probabilistic digital twin, this loop can be summarized by a belief state: the posterior distribution over conditions given past actions and observations, serving as a sufficient statistic for sequential decision-making. This perspective naturally connects digital twins to partially observable Markov decision processes (POMDPs), a stochastic control framework that supports uncertain data and can balance immediate intervention action costs with long-term expected benefits. Unlike classical stochastic control, which often relies on frequent, accurate sensing and continuous feedback, infrastructure management spans long horizons with latent condition dynamics, uncertain and, often, sparse and costly observations. Consequently, decision-making must jointly plan information gathering and interventions to minimize lifecycle costs. To address this challenge, we introduce a probabilistic digital twin formulation that embeds a dynamic Bayesian network (DBN) within a decentralized POMDP. The DBN models deterioration, exogenous drivers, and sensor health, and dynamically updates the belief state as new data arrive. The decentralized decision layer selects actions based on the current belief state (e.g., inspections, repairs, and sensor replacements), while measurements from the asset refine that belief, yielding explicit bi-directional information flow: “twin to asset” through actions that adapt to uncertainty, and “asset to twin” through data assimilation that updates uncertainty over time. The resulting decentralized control problem is solved using multi-agent deep reinforcement learning, enabling centralized training with decentralized execution. We examine our approach on offshore wind structural management case studies, from a single support structure to a farm-scale setting with shared mobilization costs. Benchmarks against other methods show that the learned strategies provide superior solutions and reduced lifecycle costs by jointly optimizing information gathering and physical interventions.