Waymo has distilled more than 200 million fully autonomous miles into ten conclusions about how it builds AI, starting with the position that camera-only sensing cannot deliver safe autonomy at scale. Written by Waymo’s Srikanth Thirumalai, the ten fundamental truths span sensing, mapping, model architecture, simulation, and safety governance. The company says the conclusions are validated by its safety data, which it reports shows the Waymo Driver is already making roads safer in the cities where it operates.
Highlights
- More than 200 million fully autonomous miles underpin the ten conclusions.
- Lidar, cameras, and radar each carry a distinct role: 3D geometry at millimeter precision, semantic reading of street signs and traffic-light colors, and velocity tracking through heavy rain, fog, or dust.
- The Waymo Critic reviews millions of on-road miles each week alongside tens of billions of simulated miles.
- The company’s data flywheel operates across exabytes of data, using auto-labelers, model retraining, and simulation validation before safety review.
Sensors and Maps: The Two Oldest AV Debates
Waymo opens with the camera-only question and answers it directly: cameras are not enough. Combining camera, lidar, and radar inputs produces a redundant world view the company says no single sensor can replicate, with each modality contributing a complementary strength rather than a duplicate one.
The second debate is whether to map at all. Waymo treats high-definition maps as a prior — another input alongside the sensors, functioning as a form of memory rather than a fixed script. The company says the approach is most valuable in poor visibility and on complex thoroughfares, and that it frees onboard compute to concentrate on what is dynamic or new, such as a sudden detour or a temporary stop sign. An AI-driven mapping system keeps the reference current.
Fewer Models, but Not One Black Box
Early AV development leaned on specialized modules — one for pedestrian detection, another for vehicle tracking, another for traffic-light state. Waymo describes that architecture as unmaintainable at scale and says it has consolidated into fewer, higher-capacity foundation models, letting data rather than hand-coded priors determine relevance. The company says the approach lets it benefit from the same scaling laws that drove large language model breakthroughs, using teacher-student models to manage onboard compute.
Consolidation stops short of a pure end-to-end system that maps raw pixels to steering commands. Waymo calls an independent onboard validation layer non-negotiable for scaling safely at L4. That layer is a separate AI-based safety system monitoring every trajectory the Driver proposes, checking plans against physics-based constraints and traffic laws using reinforcement learning and generative-AI-inspired reasoning, and acting as a hard backstop when a proposed path violates a limit or risks a collision.
Why Is Closed-Loop Simulation Different From Replay?
Replaying recorded data — open-loop simulation — is insufficient, Waymo says, because surrounding traffic in a replay is indifferent to the vehicle’s actions. In closed-loop simulation, a swerve or a brake application changes how other road users behave, creating the feedback loop the company considers essential for studying complex interactions and applying reinforcement learning.
The Critic Grades the Driver
Waymo built an AI critic to detect undesirable driving behaviors in simulation and on the road, running an automated feedback loop across weekly road miles and tens of billions of simulated miles so engineers can concentrate on the hardest edge cases. Without it, the company writes, the Driver risks “grading its own homework.” The Critic is tuned across safety, traffic-law compliance, turn smoothness, and braking comfort.
Vision-Language Models Do the Slow Thinking
For scenarios outside standard training data — a police officer directing traffic by hand at a collision site — Waymo uses vision-language models trained with Gemini to supply high-level semantic hints. The company says VLMs excel at reasoning but are too slow for real-time control and lack sufficient spatial awareness on their own, so the system splits the work: fast, intuitive sensor fusion handles instantaneous control while the Waymo Foundation Model handles deliberative reasoning over long-tail situations.
Governance, the Flywheel, and the Limits of L2
The final three lessons concern process rather than architecture. Waymo argues that driving, simulation, and evaluation must be built together with safety governance determining readiness, anchored by a quantitative data engine augmented with expert human judgment. Its data flywheel extracts and labels relevant driving data, retrains models, and validates through simulation and the safety framework — the cycle it credits for systematically addressing the long tail.
The tenth lesson is the most pointed: improving an L2 driver-assist system into full autonomy does not work. Waymo says L4 maturity requires a purpose-built system validated on closed courses and hardened by driving without a human in the car, because only full autonomous responsibility exposes the system to consequences that supervised miles and simulation smooth over.
