Abstract: Deep learning-based trajectory prediction models have demonstrated promising capabilities in capturing complex interactions. However, their generalization beyond the available training data, limited in both size and diversity, remains a significant challenge. To improve generalization, we propose SHIFT (Spectral Heteroscedastic Informed Forecasting for Trajectories), a novel framework uniquely combining well-calibrated uncertainty modeling with informative priors derived through automated rule extraction. SHIFT reformulates trajectory prediction as a classification task and effectively employs Heteroscedastic Spectral Normalized Gaussian Processes to disentangle epistemic and aleatoric uncertainties. By leveraging the epistemic uncertainty, we learn informative priors from training targets, which are automatically generated from natural language driving rules, such as stop rules and drivability constraints, using a retrieval-augmented generation framework powered by a large language model. Extensive evaluations on the nuScenes dataset, including challenging low-data and cross-location scenarios, demonstrate that SHIFT outperforms state-of-the-art methods, achieving substantial gains in uncertainty calibration and displacement metrics. In particular, our model excels in complex scenarios, such as intersections, where uncertainty is inherently higher. Code and models are publicly available on Anonymous Repo (currently as a supplementary file for the review stage).