Particle accelerators are sophisticated scientific facilities that require precise but time-consuming optimisation to achieve optimal performance. Considering benchmark tasks at the ARES and LCLS facilities, this dissertation proposes methods to deploy simulation-trained reinforcement learning (RL) policies for accelerator tuning zero-shot to the real world and novel tuning tasks, while comparing their performance to traditional methods. A high-speed differentiable beam dynamics simulator is developed to make collecting large datasets for RL feasible, and to enable a multitude of novel gradient-based accelerator applications. These contributions lay the groundwork for faster accelerator tuning to better working points, and enable new scientific discoveries.