Abstract: Position tracking based on bearing measurements and Ultra-wideband (UWB) ranging is widely used in robotic navigation tasks. However, due to variations in the number of robots, anchor configurations, UWB tag layouts, and the presence or absence of anonymous visual observations, existing methods typically rely on specially designed solvers or networks tailored to a particular localization problem. In this work, we introduce AnyAmber, a generalist neural network for versatile anonymous bearing and range based position tracking. To model diverse and dynamic geometric constraints, a heterogeneous EGAT network is employed for unified localization and uncertainty estimation, cascaded with a differentiable hierarchical PGO for improved accuracy. Additionally, we incorporate an Embedded-GRU module for adaptive UWB bias correction and a temporal graph-based matching network for soft assignments between robots and anonymous bearings. By adopting a unified problem formulation, our model is jointly pretrained on a large-scale multi-task dataset encompassing diverse simulated and real-world environments. In the experiments, with only a single trajectory fine-tuning in a target test scenario, the model achieves superior few-shot localization performance than existing methods.