I. Learning objectives

• Understand End-to-End Autonomous Driving: distinguish end-to-end from traditional modular architectures and understand its inputs, outputs, strengths, limitations, and role in an autonomous-driving system.
• Learn the UniAD example model: understand the roles of TrackFormer, MapFormer, MotionFormer, OccFormer, and the planning module, as well as query mechanisms, BEV representations, and joint Perception-Prediction-Planning optimization.
• Master the lab workflow: configure Bench2DriveZoo, prepare pretrained weights and datasets, run open-loop evaluation with UniAD-Tiny or UniAD-Base, and conduct CARLA closed-loop testing when resources are available.
• Develop result-analysis skills: use planning L2, collision rate, Driving Score, route completion rate, and RGB/BEV visualizations to analyze why end-to-end models succeed or fail in complex scenes.

II. Key concepts

The lab covers the following main theoretical concepts:

• End-to-end and modular autonomous-driving architectures
• BEV representations, multi-camera perception fusion, and Transformer query mechanisms
• Coordination among detection and tracking, online mapping, Motion Prediction, Occupancy Prediction, and planning in UniAD
• Two-stage training, task-loss weighting, and multi-task learning stability
• Bench2Drive/Bench2DriveZoo datasets, open-loop evaluation, and CARLA closed-loop testing

III. Required environment and tools

• Hardware equipment:Ubuntu 18.04 and above system; NVIDIA independent graphics card, recommended video memory is no less than 16GB; sufficient disk space for data set and model weight storage;
• Development environment:Python 3.8; Anaconda/Miniconda/CondaForge; CUDA 11.8; PyTorch GPU version; ninja, packaging and other compilation auxiliary tools; Bench2DriveZoo, Bench2Drive and CARLA 0.9.15.
• How to run:It is recommended to give priority to using small-scale data packets to complete UniAD-Tiny open-loop verification; when a GPU and CARLA environment are available, closed-loop testing can be completed locally. The full Bench2Drive data set is relatively large, and the subset provided by the official or course can be used for training experiments.

This lab uses UniAD-Tiny in Bench2DriveZoo by default. With sufficient hardware, you may also run UniAD-Base or short training runs. With limited resources, use pretrained weights for open-loop evaluation and visualization analysis.

0. Explore the project structure:

After downloading Bench2DriveZoo, review the following directories to locate configuration, model, data, and closed-loop test code:

1. Create the runtime environment:

Step 1: Create a virtual environment

Step 2: Install the CUDA toolchain

Step 3: Install PyTorch

Step 4: Set environment variables

Step 5: Install ninja and packaging build helpers

Step 6: Clone and install the project

Note: when CUDA, GCC, and PyTorch versions are incompatible, common errors occur during MMCV compilation, CUDA extension loading, and inference.

2. Prepare pretrained weights:

Create a ckpt directory at the project root, then download pretrained visual-encoder weights and the UniAD checkpoint. For this course lab, UniAD-Tiny weights are recommended for open-loop validation.

Official resources are available on the HuggingFace rethinklab/Bench2DriveZoo and rethinklab/Bench2Drive dataset pages. If direct download is slow, use a mirror proxy.

Download links:

3. Prepare the Bench2Drive dataset:

Follow the official dataset directory structure. Bench2Drive-Full is roughly terabyte scale, so a complete download is not required for a standard course lab; the 400 GB version or a course subset is sufficient to understand the training workflow and perform open-loop validation.

4. RunUniADOpen loop verification：

After preparing the data, configuration, and checkpoint, run the UniAD-Tiny open-loop evaluation script. Record planning error, collision-related metrics, and visualization file paths from the terminal output.

To reproduce the training workflow, refer to the two-stage training commands. Stage1 trains tracking and mapping; Stage2 performs end-to-end joint training. Given VRAM and time limits, course work may inspect the configuration only or run a small number of validation epochs.

5. Run the CARLA closed-loop test：

With a CARLA environment, download CARLA 0.9.15 and AdditionalMaps, then configure CARLA_ROOT and the Python API path. Link Bench2DriveZoo to Bench2Drive’s leaderboard test directory and run the closed-loop evaluation script.

For closed-loop testing, record route completion, violations, collision events, Driving Score, and screenshots of failure cases. If closed-loop testing cannot be completed, explain the limitation in the lab report and use open-loop visualizations for analysis.

6. Parameter modification and comparative analysis：

Without substantially changing the framework, select one configuration for a comparison experiment, such as queue_length, planning_steps, the number of motion modes, planning-loss weight, or collision-optimization parameters. Change only one variable at a time and keep the checkpoint, data split, and evaluation script fixed; compare metrics and visualizations.

After open-loop evaluation, organize the model outputs: RGB and BEV views, Object Detection and Tracking results, future trajectories, Occupancy Prediction, and ego planned trajectories. Check whether the plan follows navigation instructions, avoids occupancy-risk areas, and avoids abrupt turns, obstacles, or lane departures.

This project focuses on understanding end-to-end autonomous-driving frameworks and the UniAD example model; full reproduction training of large models is not required. The following questions focus on literature review, code reading, and experimental-result analysis:

1. Review the literature and explain the interface differences between modular end-to-end and fully integrated end-to-end autonomous driving. What problems is each approach suited to solve?
2. Using the UniAD architecture, explain what information is exchanged among TrackFormer, MapFormer, MotionFormer, OccFormer, and the planning module.
3. Why does UniAD train tracking/mapping in Stage1 before end-to-end joint training in Stage2? How does this strategy improve training stability?
4. Does a lower open-loop planning L2 necessarily mean a higher closed-loop Driving Score? Use experimental results to explain why the two may differ.
5. Choose one failure case and propose possible causes from five perspectives: Perception errors, map understanding, Motion Prediction, Occupancy Prediction, and planning optimization.
6. Try modifying one configuration item, such as queue_length or planning_steps, record the changes in metrics and visualizations, and analyze how the parameter affects model performance.