Skip to content

Module 2 — openpilot Reference Stack

Parent: Phase 5 — Autonomous Driving

Time: 3–4 months

Prerequisites: Module 1 (Fundamentals), Phase 4 Track B (Jetson — CUDA, Linux BSP, device trees).

Why openpilot

openpilot is comma.ai's open-source ADAS — production-deployed, well-documented, and using tinygrad for inference. It's the best publicly available end-to-end reference for studying how an autonomous driving system actually works: from raw camera pixels to CAN bus steering commands.

1. Architecture Overview

  • End-to-end pipeline:

    • camerad → modeld → plannerd → controlsd → pandad → CAN bus → vehicle actuators.
    • Each process communicates via cereal (msgpack-based IPC) and VisionIpc (shared-memory zero-copy frames).

  • Hardware:

    • comma 3X / comma four: Snapdragon 845/8 Gen 2, 3 cameras (wide road, road, driver), GPS, IMU.
    • panda: CAN-to-USB interface for vehicle communication.

  • Software layers:

    • AGNOS: Custom Linux distro (fork of upstream kernel) for comma devices.
    • openpilot: Python/C++ application layer — perception, planning, control.
    • tinygrad: Inference runtime for neural network models.

  • Flow Diagram — Detailed data flow from camera input to CAN actuation with process-to-source mapping.

Projects: * Clone openpilot. Trace the data flow from camerad to pandad by reading selfdrive/ source. Document the IPC messages between each process. * Run openpilot in simulation using comma's replay tools or CARLA integration.

2. Camera Pipeline (camerad)

Deep dive: camerad Guide

  • Sensor hardware: OX03C10 / OS04C10 image sensors, Qualcomm Spectra ISP.
  • Capture flow: Sensor RAW → CSI → IFE (demosaic, CCM, gamma) → BPS → YUV NV12.
  • Auto exposure (AE): Software algorithm with 3-frame latency, PI-like control loop, DC gain hysteresis for night driving.
  • VisionIpc: Shared-memory IPC for zero-copy frame transport from camerad to modeld.
  • V4L2 integration: Linux Video4Linux2 API for camera device control, request manager, buffer flow.

Projects: * Read system/camerad/ source. Trace a single frame from sensor DQBUF to VisionIpc publish. * Modify the AE target grey fraction. Observe the effect on exposure in different lighting conditions.

3. AGNOS Operating System

Deep dive: AGNOS + OS Course

  • What AGNOS is: Fork of Linux kernel (agnos-kernel-sdm845) + custom build system (agnos-builder) for comma devices.
  • Key kernel customizations: Camera drivers (V4L2/media), device tree for SDM845, SPI (CAN-over-SPI for panda), thermal/power management, cgroups for process isolation.
  • Maps to Phase 1 OS lectures: All 26 OS lectures from Phase 1 are mapped to specific kernel paths and openpilot use cases in the AGNOS guide.

Projects: * Clone agnos-kernel-sdm845. Follow the AGNOS guide to trace how Phase 1 OS concepts (processes, interrupts, scheduling, device tree) manifest in a production ADAS kernel. * Identify the device tree nodes for the 3 camera sensors. Understand how CSI lanes and clocks are configured.

4. Perception (modeld)

  • Model architecture: End-to-end neural network taking multi-camera input, outputting lanes, lead vehicles, pose, plan, and desired path.
  • Warp matrix: Calibration-based image warping applied before inference to normalize camera viewpoint.
  • tinygrad inference: Models run through tinygrad on Snapdragon GPU (Adreno) — see Module 3.
  • Outputs: modelV2 cereal message — lanes, road edges, pose, plan trajectory, action (gas/brake/steer), forward collision warning (FCW).

Projects: * Read selfdrive/modeld/. Trace model input preparation (frame + warp) through tinygrad inference to modelV2 output. * Log modelV2 outputs during a drive replay. Visualize predicted lanes and lead vehicle positions.

5. Planning and Control

  • plannerd:

    • LongitudinalPlanner: speed profile, following distance, stop-and-go.
    • LaneDepartureWarning: lateral safety monitoring.
    • Inputs: modelV2 (perception), radarState, carState.

  • controlsd:

    • LatControl: lateral PID/INDI/torque controller for steering.
    • LongControl: longitudinal PID for gas/brake.
    • Outputs: carControl cereal message → CarInterface → CAN commands.

  • pandad + CAN:

    • pandad translates carControl into raw CAN messages via the panda device.
    • Vehicle-specific CarInterface implementations handle DBC encoding per make/model.

Projects: * Trace a steering command from plannerd's desired path through controlsd's lateral controller to the final CAN message. Document the control loop. * Compare openpilot's lateral control (torque-based) with the Stanley controller from Module 1.

6. Forking and Contributing

  • Fork workflow: Fork openpilot, set up development environment, build and test.
  • Vehicle porting: Add support for a new vehicle — CAN DBC reverse engineering, CarInterface implementation, fingerprinting.
  • Community: Active Discord, community forks, bounty programs for contributions.

Projects: * Fork openpilot. Make a small change (e.g., adjust a UI element or tuning parameter). Build, test in simulation, and submit a PR. * (Advanced) Study the CAN DBC for your vehicle. Implement basic parsing and actuation.

Resources

Resource URL
openpilot source https://github.com/commaai/openpilot
agnos-kernel-sdm845 https://github.com/commaai/agnos-kernel-sdm845
agnos-builder https://github.com/commaai/agnos-builder
comma.ai Blog https://blog.comma.ai/
openpilot community docs https://github.com/commaai/openpilot/wiki

Next

→ Module 3 — tinygrad for Inference — Deep dive into the inference engine that powers openpilot's perception.