Autonomous driving models do not learn “driving.” They learn patterns in pixels, point clouds, radar returns, and trajectories. The dataset you choose decides what your model can see, and what it will miss.
This free-first list starts with multimodal sensor stacks (camera + LiDAR/radar/IMU/GPS). Then it rounds things out with widely used image and trajectory benchmarks.
Core Multimodal Driving Datasets
Waymo Open Dataset
Volume: Perception subset: 2,030 segments (20s) and 390K frames
Access: Free with signup
Format: Images + LiDAR point clouds
Task Fit: 3D object detection, tracking, motion forecasting, scenario understanding
Waymo Open is a go-to benchmark when you want strong perception plus motion context. It works well for camera + LiDAR baselines. Many teams already support it in their tooling, so it is easy to reference and compare. Use it when you need a big dataset that can carry detection, tracking, and forecasting. Plan for real-world scale, both in storage and compute.
nuScenes
Volume: 1,000 scenes (~5.5 hours) with 6 cameras, 1 LiDAR, 5 radars, plus GPS/IMU
Access: Free with signup
Format: Sensor streams + labels
Task Fit: 3D object detection, tracking, motion prediction, map-aware perception
nuScenes is a classic multimodal benchmark built for sensor fusion. The sensor suite lets you compare camera-only, LiDAR-only, and fused stacks without switching datasets. It is also a strong choice for reproducibility because tooling and baselines are common. If you only need one task, the full modality stack can slow you down. That is the tradeoff you make for breadth.
Argoverse 2
KITTI (Vision Benchmark Suite)
Volume: Object Detection: 7,481 training images and 7,518 test images
Access: Free
Format: Images + point clouds + labels
Task Fit: 3D object detection, stereo depth, optical flow, odometry
KITTI is the baseline anchor that still shows up everywhere. Teams cite it even when they train on newer datasets. It is compact enough for fast experiments. It is also broad enough to support multiple tasks without rebuilding your evaluation stack. If your audience includes newcomers, KITTI is often the first AV dataset they recognize. The main gotcha is scale: it is smaller than modern benchmarks, so treat generalization carefully.
KITTI-360
Volume: ~320K images and 100K laser scans over 73.7 km
Access: Free (license-based; see page)
Format: Multi-view images + 3D scans + labels
Task Fit: 2D semantic segmentation, 3D semantic segmentation, mapping/SLAM, 3D scene understanding
KITTI-360 is the “more context” answer to classic KITTI. It helps when you need longer-range continuity and richer 3D geometry. It also fits segmentation and mapping workflows where a single frame is not enough. Expect more preprocessing work. Plan your pipeline around the larger footprint.
KITTI Raw Data
Access: Free
Format: Raw synchronized sensor streams
Task Fit: sensor fusion, self-supervised depth, odometry/SLAM, representation learning
KITTI Raw Data is for teams that want raw sync streams. It is not built around benchmark-ready slices. That makes it useful for custom labeling, self-supervised learning, and representation work where “raw” is the point. You get flexibility, but you also take on more engineering. You decide splits, labeling strategy, and evaluation conventions. If your readers build pipelines, this is a solid “build-your-own” option.
A2D2 (Audi Autonomous Driving Dataset)
Volume: 2.3 TB total download size
Access: Free
Format: Camera + LiDAR (and labels where provided)
Task Fit: semantic segmentation, 3D detection, sensor fusion, depth estimation
A2D2 is an industry-origin dataset that adds diversity without leaving the multimodal category. It works well for perception training. It is also useful as a second dataset for validation when you want a generalization check. Annotation types can vary by subset, so read the download options closely before you commit. Treat it as a practical add-on to the usual benchmark trio.
Oxford RobotCar Dataset
Volume: 100+ repetitions; 1,000 km; almost 20 million images (referenced in docs)
Access: Free with signup (license acceptance)
Format: Multi-sensor recordings
Task Fit: localization, long-term mapping, SLAM, robustness evaluation
RobotCar is a strong pick when you care about localization and long-term robustness. It is not only about object detection. The repeated traversals support “same route, different condition” evaluation and mapping tasks. It also helps you tell a clear story about drift and seasonality. The main constraint is size. It is large, so storage planning matters.
ApolloScape
Volume: Released subset cited as 143,906 annotated frames
Access: Free with request/signup (varies by subset)
Format: Images + pixel-level annotations
Task Fit: semantic segmentation, depth estimation, lane/scene parsing, self-localization
ApolloScape is a recognizable dataset family that works well for segmentation-heavy coverage. It balances a list that might otherwise lean too far into detection and tracking. It is also a strong “dataset name” for search, since readers look for Apollo-related data directly. Access can vary across subsets, so treat each download as its own checklist. The dataset is worth it when you want a broader scene parsing angle.
H3D (Honda 3D)
Volume: 160 scenes, 27,721 frames, ~1 million labeled instances (as cited in literature)
Access: Free with request (academic-only workflow; data sharing agreement required)
Format: Full-surround 3D perception data
Task Fit: 3D detection, 3D tracking, multi-object perception
H3D is a credible, research-gated dataset that adds a strong trust signal. It works well as a second-opinion dataset for 3D perception beyond the usual public benchmarks. The access gate is real. This is not a “download in five minutes” option. Still, it is worth including because it is widely recognized in 3D AV research. Use it when you want diversity in sensor setup and collection style.
Driving Video + Street-scene Benchmarks
BDD100K
Volume: Introduced as a 100K driving video dataset for heterogeneous multitask learning
Access: Free with signup
Format: Video + annotations
Task Fit: 2D detection, semantic segmentation, lane detection, drivable area
BDD100K is easy to justify when you want breadth. It is a strong “one dataset, many tasks” option for perception work. It also fits readers who want video-scale data, not only still frames. It supports pipelines that train detection, segmentation, and lane models under one roof. The tradeoff is complexity. Video-scale datasets can slow experimentation if your tooling is not ready.
Cityscapes
Volume: 5,000 fine annotations and 20,000 coarse annotations
Access: Free with signup (research registration)
Format: Images + pixel labels
Task Fit: semantic segmentation, instance segmentation, panoptic-style evaluation
Cityscapes is the canonical urban segmentation benchmark for driving scenes. It is widely recognized and widely reused. It is also one of the quickest ways to anchor a segmentation section with credibility. If you need a clean reference point for street-scene parsing, this is it. Access is free, but registration is required. Plan for that step.
Mapillary Vistas
Volume: 25,000 images and 66 classes
Access: Free with signup
Format: Images + segmentation labels
Task Fit: semantic segmentation, robust street-scene parsing, generalization testing
Mapillary Vistas is a strong choice when you care about diversity and generalization. It pairs well with Cityscapes because it brings broader street-scene variety. It also fits robustness stories where long-tail classes matter. Expect taxonomy differences between datasets. Label mapping is a common integration step.
ACDC (Adverse Conditions Dataset with Correspondences)
Volume: 4,006 images distributed across fog, night, rain, and snow
Access: Free with signup (non-commercial license)
Format: Images + labels/benchmarks
Task Fit: semantic segmentation, uncertainty-aware segmentation, object detection, panoptic segmentation
ACDC targets the moments most datasets avoid: fog, night, rain, and snow. It helps when your model looks good in daylight, then falls apart once conditions change. This dataset fits evaluation-focused work on robustness and uncertainty. It is not only about raw accuracy. The key constraint is licensing. It is non-commercial, so treat it as a research and benchmarking asset.
Lane and road structure
CULane
Volume: 55+ hours of video; 133,235 extracted frames
Access: Free (academic research)
Format: Image frames + lane annotations
Task Fit: lane detection, lane segmentation, robustness by condition buckets
CULane is a practical lane benchmark that still shows up in baselines and repo comparisons. It is especially useful when you want lane detection coverage under harder conditions, not only clean highways. It also works well for quick demos. Lane tasks often train faster than full 3D pipelines. As always with lane datasets, expect format differences across benchmarks.
TuSimple Lane Detection
Volume: About 7,000 one-second clips of 20 frames each
Access: Free
Format: Video frames + lane annotation format
Task Fit: lane detection, lightweight baselines, highway-only evaluation
TuSimple is one of the most searched lane datasets. That makes it a strong, clickable inclusion. The scope is focused, so it is easy to integrate. It works well as a baseline for lane detection pipelines that need to run quickly. It is also useful when a team wants a narrow task before scaling up. The limitation is domain coverage. It is largely highway-focused, not a full urban-lane benchmark.
Traffic Participants and Classic Perception Benchmarks
CityPersons
Volume: 5,000 images with about 35,000 person instances (as described in the paper)
Access: Free with signup (depends on Cityscapes access)
Format: Person bounding boxes
Task Fit: pedestrian detection, occlusion-aware detection, cross-dataset evaluation
CityPersons is a focused pedestrian detection benchmark grounded in real urban scenes. It helps when you want to talk about crowded, occluded pedestrians. That is where detection quality often drops fast. In practice, teams often use it for comparisons and cross-dataset evaluation. It is less common as a sole training source. Because it depends on Cityscapes access, plan for it as a paired dataset.
GTSRB
Volume: 50,000+ images; 40+ classes (distribution record description)
Access: Free
Format: Image files + labels
Task Fit: traffic sign classification, small-object recognition, robust classification
GTSRB is a classic traffic sign classification dataset. Many ML practitioners recognize it instantly. It adds clean classification coverage to a list dominated by detection and segmentation. It is also small enough for fast iteration and teaching. That makes it useful beyond AV research. The main caveat is task clarity. It is classification, not sign detection, so keep evaluation goals aligned.
EuroCity Persons
Volume: 47,300 images; 238,200 persons
Access: Free with signup (license acceptance)
Format: Images + bounding boxes
Task Fit: pedestrian detection, cyclist/rider detection, cross-city generalization
EuroCity Persons pushes generalization across locations. It works well for vulnerable road user perception, including riders and cyclists. It also supports a clean message in a training plan: do not overfit to one city. The download flow is explicit about licensing, which improves clarity in a roundup. Plan for a license acceptance step before you download.
Caltech Pedestrian
Volume: 250,000 frames; 350,000 bounding boxes; ~2,300 unique pedestrians
Access: Free
Format: Video frames + bounding boxes
Task Fit: pedestrian detection, legacy benchmark comparisons, detector evaluation
Caltech Pedestrian is older, but it still matters as a historical reference point. It helps when you want to compare modern detectors against classic evaluation setups. It also works well for teaching the evolution of pedestrian detection benchmarks. The scope is focused, so evaluation pipelines can stay simple. The limitation is distribution shift. Camera setups and environments differ from modern AV fleets.
Traffic behavior and trajectories
highD
Volume: 16.5 hours; ~110,000 vehicles; ~45,000 km; 5,600 lane changes
Access: Freemium (free for non-commercial research; request required)
Format: Trajectory dataset
Task Fit: trajectory prediction, lane-change modeling, scenario mining, safety validation
highD is a clean, structured highway trajectory dataset. It fits prediction and behavior modeling work. It is useful when your roundup needs planning and interaction content, not only perception. The dataset also supports feature engineering and scenario mining, especially around lane changes. Access is typically research-gated. Position it as a trusted benchmark, not a frictionless download.
inD
Volume: 10 hours from four intersections; 11,500+ road users
Access: Freemium (free for non-commercial research; request required)
Format: Trajectory dataset
Task Fit: interaction modeling, trajectory prediction, planning proxies, behavior cloning
inD complements highway datasets by focusing on intersections. That makes it useful for multi-agent prediction. It also helps for behaviors where negotiation and yielding patterns matter. This dataset supports a clear point: driving is not only lane keeping. It is interaction. Like highD, access is request-based and typically non-commercial.
rounD
Access: Freemium (free for non-commercial research; request required)
Format: Trajectory files + maps/background images (as described in format PDF)
Task Fit: trajectory prediction, negotiation behavior, multi-agent interaction at roundabouts
rounD adds a different topology: roundabouts. Interaction patterns can look very different here than at intersections or on highways. That makes it a useful stress test dataset for prediction and negotiation behavior. The official format documentation helps integration. You can plan around a known schema. Access is typically research-only, so it fits best as a trusted benchmark dataset card.
NGSIM Vehicle Trajectories
Volume: Vehicle positions recorded every 0.1 seconds (sampling statement on catalog page)
Access: Free
Format: Trajectory data + supporting metadata
Task Fit: trajectory prediction, lane-change prediction, traffic flow modeling, behavior baselines
NGSIM is a classic baseline for traffic trajectory research. It helps when your audience wants a well-known dataset for behavior modeling and flow analysis. The data is older than modern AV stacks, but it remains a common reference point. It also works well for building traditional prediction baselines before moving to newer datasets. As with any historical dataset, validate how well the patterns match current driving distributions.
Synthetic and Simulation
Virtual KITTI 2
Volume: 5 sequence clones from KITTI tracking with multiple variants (weather/camera) and multiple output modalities
Access: Free
Format: RGB/depth/segmentation/flow/scene flow image sets
Task Fit: domain adaptation, optical/scene flow, segmentation, depth estimation
Virtual KITTI 2 gives you controlled variation without running a full simulator pipeline. It is practical for robustness testing, especially for weather and camera changes. It also provides rich ground-truth modalities for multi-task learning. This makes it a good fit for domain adaptation stories. Synthetic realism is still synthetic, so validate against real datasets before you draw strong conclusions.
CARLA Simulator
Volume: generator, not a fixed dataset
Access: Free
Format: Simulator-generated sensor outputs
Task Fit: scenario generation, planning, perception data synthesis, closed-loop evaluation
CARLA is best treated as a controllable data generator. It is not a benchmark dataset with fixed splits. It helps with rare scenarios, targeted ablations, and closed-loop evaluation. It also supports scenario design and testing infrastructure work. The main constraint is the synthetic-to-real transfer. Pair CARLA-derived data with real-world validation whenever possible.
Conclusion
Autonomous driving datasets are not interchangeable. Some are built for sensor fusion with camera and LiDAR. Others are tuned for street-scene segmentation, lane detection, or trajectory prediction. The best shortlist is the one that matches your model’s input stack and your evaluation plan.
Start with a multimodal benchmark if you are training 3D perception or motion models. Then add a street-scene dataset to stress-test semantic segmentation and long-tail classes. Finally, bring in trajectories or simulation when you need planning signals, interaction patterns, or controlled edge cases. If you can reproduce results across at least two datasets, you are much closer to a model that works outside a single benchmark.
Frequently Asked Questions (FAQ)
If you want camera and LiDAR in the same stack, start with multimodal benchmarks like Waymo Open Dataset and nuScenes. They are widely used for 3D object detection, tracking, and motion forecasting. They also tend to come with devkits and common evaluation setups, which makes comparisons easier. Add a second dataset, such as Argoverse 2 or A2D2, when you want diversity and a more realistic generalization check.
Most “free” autonomous driving datasets fall into one of two access types: open download or free with signup and license acceptance. For example, some datasets provide immediate downloads, while others require an account or an academic registration step. Always confirm the license and the allowed use before you build a training pipeline, especially if you plan commercial deployment. If access is gated, treat the dataset as a benchmarking asset and keep an alternative dataset ready for iteration speed.
For lane detection, dedicated lane datasets like CULane and TuSimple are practical starting points. They are focused, so you can train and evaluate quickly, and they map well to lane detection baselines. For road understanding beyond lanes, street-scene datasets like Cityscapes and Mapillary Vistas are strong for semantic segmentation and instance-level scene parsing. If you care about adverse conditions, add ACDC to evaluate performance in fog, rain, snow, and night.
Choose image-only datasets when your model is vision-first: 2D detection, semantic segmentation, lane detection, and driving-scene classification. They are also easier to store and faster to iterate on. Choose LiDAR-based datasets when geometry matters: 3D object detection, 3D tracking, depth reasoning, and sensor fusion pipelines. If your end system uses both cameras and LiDAR, train with both early, because the fusion behavior is part of what you are trying to learn.
