Development of a Self-Driving Car using Behavior Cloning and Perception/Planning Stack

Introduction

In this project, I aimed to develop a self-driving car using behavior cloning in the Unity simulator and a separate perception and planning stack in the CARLA simulator. I trained and tested the self-driving car using three NVIDIA models, each with convolutional layers, ELU activation layers, and fully connected layers. Additionally, I implemented a perception stack for object detection and traffic sign prediction, and a planning stack to navigate through traffic using hybrid RRT, A*, and Frenet optimal trajectory algorithms in the CARLA simulator.

Behavior Cloning Approach

I used Udacity's self-driving car simulator to train the models. Behavior cloning involves training a model to imitate a human driver's behavior by learning from a large dataset of expert demonstrations. I experimented with different neural network architectures to improve performance, including increasing the depth of the fully connected layers and adding a skip connection between the input image and the first convolutional layer.

Neural Net Architecture

Training Setup

I drove around the simulator for 4 to 5 laps, collecting a training set of 8000 images and a validation set of 2000 images. I trained all three models for 10 epochs and deployed the trained models in the simulator.

Training Setup

Results

Based on the loss curves, the model with deeper linear layers outperformed the other two models. The base model with deeper linear layers provided the best balance between performance and complexity.

Loss Curves

Validation loss vs epoch

Testing of Base Model.

Testing of Model with deeper linear layers.

Testing of Model with Skip Connection.

Separate Perception and Planning Approach

I used the CARLA simulator to develop a perception stack for object detection and a planning stack for trajectory planning. I implemented and compared Frenet Optimal Trajectory Planner, RRT* planner, and Hybrid A* planner.

Object Detection

In the perception stack, I used Faster-RCNN, SSD-Mobilenet-FPN-640, and SSDLite-Mobilenet-V2 for object detection. These models analyzed image data to provide real-time detection and localization of objects.

Planning

I implemented the Frenet Optimal Trajectory Planner, RRT* planner, and Hybrid A* planner in the planning stack. These planners generated safe and practical trajectories for the autonomous vehicle.

Control

I used PID and MPC controllers to follow waypoints and goal speeds generated by the planning module. Both controllers computed commands for modifying brakes, steering, and throttle.

Results

I evaluated nine combinations of object detection and planning algorithms. The Frenet Optimal Trajectory (FOT) planner combined with Faster-RCNN gave the most accurate and smooth trajectory, while the FOT planner with SSDLite-Mobilenet-V2 was the fastest. The analysis revealed that the FOT planner generated the smoothest trajectory among all planners.

Faster-RCNN + Frenet Optimal Trajectory Planner.

SSD-Mobilenet-FPN-640 + Frenet Optimal Trajectory Planner.

SSD-Mobilenet-FPN-640 + Hybrid A* Planner.

SSDLite-Mobilenet-V2 + Hybrid A* Planner.

SSD-Mobilenet-FPN-640 + RRT* Planner.

Lateral Jerk profiles vs time.

Performance of Various configurations.

Conclusion

My project demonstrated the feasibility of using behavior cloning and a separate perception/planning stack for self-driving cars. I concluded that the combination of Faster-RCNN with the FOT planner is optimal for accuracy and smoothness, while the FOT planner with SSDLite-Mobilenet-V2 is optimal for speed.