The geometry of the tracks is measured regularly to ensure traffic safety and passenger comfort. The track monitoring is now mainly carried out by dedicated trains. However, lighter equipment would allow for more regular surveillance. Thus, this paper proposes to identify lateral railway track geometry based on the dynamic response of in-service trains (mainly accelerations). The underlying inverse problem is complex due to its non-linearity, the very large size of the search space, and the presence of uncertainties in the system: variability and lack of knowledge of the adhesion characteristics between the wheel and rail, variability in speed, etc. The first step is to build a simplified analytical model of the lateral dynamics of an axle on a track. Particular attention is paid to the coupling between the different degrees of freedom and to non-linearities (notably those of the wheel/rail contact geometry and the friction laws at the interface). A filter is then identified and used to construct the lateral defects from the simulated accelerations. The third step aims to generalize the identification approach. To assess its performance, we rely on simulations that generate realistic accelerations using a railway dynamics model, and evaluate how well the identification works under these conditions. The encouraging results support the potential deployment of these onboard sensors.