Skip to content

Module 1 — Autonomous Driving Fundamentals

Parent: Phase 5 — Autonomous Driving

Time: 3–4 months

Prerequisites: Phase 3 (Computer Vision, Sensor Fusion), Phase 1 §4 (C++/CUDA).

Why this comes first

Before studying openpilot's codebase or advanced research, you need the foundational vocabulary: how autonomous vehicles perceive, plan, and control. This module covers the algorithms and theory that every ADAS engineer uses daily.

1. Computer Vision for Driving

  • Lane detection:

    • Traditional: Hough transform, sliding window, IPM (inverse perspective mapping).
    • Deep learning: LaneNet, SCNN, U-Net for lane segmentation.
    • Output: polylines or polynomial coefficients for lane boundaries.

  • Object detection:

    • 2D: YOLO family, CenterNet, FCOS — bounding boxes for vehicles, pedestrians, cyclists.
    • 3D: PointPillars, SECOND (LiDAR); FCOS3D, DETR3D (monocular camera).
    • Bird's Eye View (BEV): lifting 2D camera features to BEV space for unified 3D detection.

  • Semantic segmentation:

    • Drivable area, lane markings, curbs, obstacles.
    • Models: DeepLab, SegFormer, lightweight edge variants.
    • Panoptic segmentation: combining semantic + instance for complete scene understanding.

  • Depth estimation:

    • Monocular depth (MiDaS, DepthAnything) — useful when no LiDAR is available.
    • Stereo matching: classic block matching, deep stereo (AANet, RAFT-Stereo).

Projects: * Implement lane detection on a dashcam video using both Hough transform and a U-Net. Compare robustness in curves, shadows, and night. * Train a YOLO model on KITTI or nuScenes for 2D vehicle detection. Measure mAP and inference FPS on Jetson.

2. Planning and Decision Making

  • Behavior planning:

    • High-level decisions: lane keep, lane change, merge, turn, stop.
    • Finite state machines (FSM) and behavior trees.
    • Rule-based vs. learning-based behavior planning.

  • Motion planning:

    • Path planning: A, RRT, RRT, hybrid A* (for non-holonomic vehicles).
    • Trajectory optimization: minimize jerk/acceleration while avoiding collisions.
    • Frenet frame: decompose planning into longitudinal (speed) and lateral (lane offset) components.
    • Lattice planners: discretize trajectory space, evaluate candidates against cost functions.

  • Prediction:

    • Constant velocity / constant turn-rate models (baseline).
    • Learning-based: TNT, LaneGCN, MTR — multi-modal trajectory prediction.
    • Interaction-aware: graph neural networks for modeling agent-agent interactions.
    • Map-conditioned: use lane geometry and traffic rules to constrain predictions.

Projects: * Implement RRT* for path planning in a 2D grid with obstacles. Extend to a bicycle model for non-holonomic constraints. * Build a simple behavior FSM: CRUISE → FOLLOW → LANE_CHANGE → STOP. Test in CARLA with traffic.

3. Control Theory for Autonomous Vehicles

  • Vehicle dynamics:

    • Bicycle model: front/rear axle, slip angle, yaw rate.
    • Tire models: linear cornering stiffness, Pacejka magic formula (overview).
    • Vehicle state estimation: combine IMU + wheel odometry + GPS for localization.

  • Lateral control:

    • Stanley controller: cross-track error + heading error, used in early AV and openpilot.
    • Pure pursuit: geometric path following using a lookahead point.
    • MPC (Model Predictive Control): optimize a sequence of steering commands over a horizon, subject to dynamics constraints.

  • Longitudinal control:

    • PID for speed tracking (adaptive cruise control).
    • MPC for combined speed + following distance.
    • Jerk-limited profiles for passenger comfort.

  • Combined lateral + longitudinal:

    • LQR (Linear Quadratic Regulator) for path tracking with speed control.
    • Cascaded MPC: separate lateral/longitudinal MPC with coordination.

Projects: * Implement a Stanley controller for lane following in CARLA. Tune gains for different speeds. * Implement PID-based ACC (Adaptive Cruise Control): maintain target speed, slow for lead vehicle, stop-and-go. Test in CARLA. * (Advanced) Implement a simple MPC for combined lateral + longitudinal control. Compare smoothness and tracking error vs Stanley + PID.

Resources

Resource Why
CARLA Simulator Test all algorithms in simulation before touching real hardware
Autonomous Driving in the Real World (Shaoshan Liu et al.) Comprehensive AV textbook
Apollo Auto Reference full-stack AV for architectural comparison
KITTI / nuScenes Benchmark datasets
Phase 3 — Computer Vision, Sensor Fusion Prerequisite modules in this roadmap

Next

→ Module 2 — openpilot Reference Stack — Study a real, production-deployed ADAS end-to-end.