Knowledge Transfer in Reinforcement Learning: How agents should benefit from prior knowledge

Abstract

Reinforcement learning (RL) has had many successes in different tasks, but in practice, it often requires significant amounts of data or training time to learn high-performing policies. For complicated tasks, the learning speed may be too slow to be feasible. To speed up the learning, external prior knowledge could be leveraged by the RL agent. Knowledge transfer could be performed to allow a trained (source) agent to assist a new (target) agent. It is also possible that a human demonstrator could behave as the source. With help from the prior, the new agent is expected to learn faster than learning from scratch. In most cases, prior knowledge cannot be directly copied from the source due to the different internal learning representations (e.g., from a human to a virtual agent). We need to re-summarize the usable transferred policy by observing the state-action behavior of the source agent. We propose methods that could analyze the offline confidence in summarized knowledge, which would help the agent better understand the source policy. Given the limited observation data, the agent can estimate which parts of the policy are trustworthy based on the confidence level. However, only measuring the offline confidence based on observed data is not enough. This dissertation extends the confidence analysis to the online learning setting, by further updating the confidence values based on how well the prior policies could adapt to the target tasks. Our online confidence mechanism could also remove manually defined sensitive parameters tuning and let the learner effectively adjust by itself. Confidence measurement could be done by leveraging supervised learning techniques but the demonstrations data could contain more sequential information like spatial or temporal structure. In addition to the confidence measurement of supervised methods, we further explore model-based methods to summarize the policy. Knowledge transfer is more challenging in real-life tasks, this dissertation would discuss techniques that enables the transfer from a model-based simulation to real robots. For extremely complicated learning areas like deep learning (DL), we propose correlated-feature networks that enable the agent to progressively reuse weights considering the feature relations between source and target tasks.