Reinforcement Learning via Multi-Fidelity Simulators



Project Description  (top)


Reinforcement Learning via Multi-Fidelity Simulators (MFRL) by Mark Cutler


Reinforcement learning (RL) can be a tool for designing policies and controllers for robotic systems. However, the cost of real-world samples remains prohibitive as many RL algorithms require a large number of samples before learning useful policies. Simulators are one way to decrease the number of required real-world samples, but imperfect models make deciding when and how to trust samples from a simulator difficult. This project presents a framework for efficient RL in a scenario where multiple simulators of a target task are available, each with varying levels of fidelity. The framework is designed to limit the number of samples used in each successively higher-fidelity/cost simulator by allowing a learning agent to choose to run trajectories at the lowest level simulator that will still provide it with useful information. Theoretical proofs of the framework's sample complexity are given and empirical results are demonstrated on a remote controlled car with multiple simulators. The approach enables RL algorithms to find near-optimal policies in a physical robot domain with fewer expensive real-world samples than previous transfer approaches or learning without simulators.





Videos  (top)



Related publications  (top)

Journal Papers

  • M. Cutler, T. J. Walsh, J. P. How, "Real-World Reinforcement Learning via Multi-Fidelity Simulators", IEEE Transactions on Robotics, 2014 (submitted)   

Conference Papers

  • M. Cutler, J. P. How, "Efficient Reinforcement Learning for Robots using Informative Simulated Priors", IEEE International Conference on Robotics and Automation (ICRA), Seattle WA, May 2015 (submitted)   

  • M. Cutler, T. J. Walsh, J. P. How, "Reinforcement Learning with Multi-Fidelity Simulators", IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, June 2014   [BibTeX]