Abstract: Training controllers via reinforcement learning (RL) in simulation has emerged as a powerful approach for synthesizing robust and agile robotic behaviors evaluated in reality. We push the envelope of the simulation training paradigm by exposing problems encountered when learning agile behaviors only made possible by dynamic coordination between many joints, such as in the whole-body control of a quadruped robot. We find that training athletic whole-body control behaviors from scratch often fails, and the sim-to-real gap is greatly pronounced, especially on commodity hardware using complex-to-model harmonic drive actuators with limited sensing. We propose general solutions to overcome these issues: (i) leveraging a pre-trained whole-body controller as a robust foundation that can be fine-tuned with RL for a highly dynamic task (ii) a framework for modeling complex actuation mechanisms without requiring access to torque sensors. Along with several other design decisions that we elaborate, we achieve highly-dynamic whole-body control behaviors such as ball throwing, lifting heavy weights, and others.