Autonomous vehicles are capable of sensing the environment and moving around with little to no human intervention, enhancing efficiency and safety. Self-driving cars, for instance, will affect our modes of transportation and way of life in the years to come. With rapid advances in hardware and software design, learning-based autonomous driving is becoming a viable and popular solution.

This thesis focuses on data from the front-end and policy from the back-end in learning-based autonomous driving. As commonly known, data is central to all learning-based methods. However, engineers cannot collect all the data from all possible scenarios to train a model due to the great variety of real-world driving scenarios, e.g., different conditions in weather, lighting, roads, traffic, etc. Consequently, the issue of how to train a model with robustness, generalizability, and transferability becomes crucial. In addition, while input data is the key component of autonomous driving in the front-end, policy, which controls the vehicle to navigate safely, is also an essential component in the back-end. Existing methods have made progress on policy learning, but there is room for improvement, e.g., reinforcement learning is not able to utilize expert demos, inverse reinforcement learning can not directly utilize driving domain knowledge, etc. I propose to address these important open research issues by adopting machine learning and deep learning techniques, including adversarial data augmentation and training, auxiliary modality learning, transfer learning, reinforcement learning, and inverse reinforcement learning.

To make autonomous driving more robust against varying weather changes, lighting conditions, or other image corruptions, I propose a gradient-free adversarial training method based on data augmentation and sensitivity analysis. To utilize multi-modal information for good performance with low computational costs, I design an auxiliary modality learning framework that can distill knowledge from multi-modality data to single modality data, with a specific condition that allows the teacher network to stay aware of the student's status for better distillation. I further propose a small-shot cross-modal distillation to solve the problem in a small-shot setting. To overcome the difficulty of collecting data in the real world, I present a transfer learning architecture that is able to transfer knowledge from the simulation domain to the real-world scenarios. To utilize both expert demonstration and real-world driving knowledge, I propose enhanced inverse reinforcement learning with hybrid-weight trust-region optimization and curriculum learning.

In summary, my proposed learning-based frameworks enhance robustness, efficiency, and generalization via adversarial data augmentation and training, auxiliary modality learning, and transfer learning w.r.t. data processing in the front-end; and further improve policy learning via inverse reinforcement learning in the back-end.