Abstract: State estimation and control are often addressed separately, which can lead to unsafe execution due to sensing noise, execution errors and discrepancies between the planning model and reality. Simultaneous control and trajectory estimation using probabilistic graphical models has been proposed as a unified solution to these challenges. Previous work, however, relies heavily on appropriate Gaussian priors and is limited to holonomic robots with linear time-varying models. The current research extends graphical optimization methods to vehicles with arbitrary dynamical models via Simultaneous Trajectory Estimation and Local Adaptation (STELA). The overall approach first initializes feasible trajectories using a kinodynamic, sampling-based motion planner. Then, STELA simultaneously: (i) estimates the past trajectory based on noisy observations, and (ii) adapts the controls to be executed so as to minimize deviations from the planned, feasible trajectory, while avoiding collisions. The proposed factor graph representation of trajectories in STELA can be applied for any dynamical system given access to first or second-order state update equations, and introduces the duration of execution between two states in the trajectory discretization as another variable to be optimized. These features provide both generalization and flexibility in trajectory following. In addition, and targeting computational efficiency, the proposed strategy allows for the use of incremental updates of the factor graph by using the iSAM algorithm and also introduces a time-window mechanism. This mechanism allows the factor graph to be dynamically updated so as to operate over a limited past history and forward horizon of the planned trajectory. This enables the online update of controls at a minimum of 10Hz. Experiments first demonstrate that STELA achieves at least comparable performance to previous frameworks on idealized vehicles with linear dynamics. More critically, STELA directly applies to and successfully solves trajectory following problems for more complex dynamical models. Beyond generalization, simulations assess STELA’s robustness under varying levels of sensing and execution noise, while ablation studies highlight the importance of different components of the approach. Real-world experiments validate STELA’s practical applicability on a low-cost MuSHR robot, which exhibits high noise and non-trivial dynamics.