odern robotics systems don't rely on a single sensor. They fuse LiDAR, RGB cameras, thermal, and IMU data into a single perception pipeline. Annotating this multi-modal data is straightforward if your sensors are perfectly calibrated; it's a nightmare if they're not.
The sensor fusion annotation problem
LiDAR gives you sparse 3D geometry. RGB gives you rich visual features — colour, texture, edges. To train a fusion model, you need labels that are consistent across modalities: a point cloud label and the corresponding image region must refer to the same object.
If your LiDAR points misalign with image pixels by 2–3 pixels, annotators will see a "car" label on the point cloud that corresponds to empty road in the RGB image. The model learns to distrust the fused representation, and performance suffers.
This alignment problem is often overlooked in annotation planning. Vendors quote a cost-per-frame without mentioning calibration, and projects discover the problem 30% through annotation.
Pre-annotation calibration: non-negotiable
Before a single annotation happens, you need:
Intrinsic calibration: Each camera and LiDAR must be internally calibrated (focal length, principal point, distortion). Use checkerboard patterns or aruco markers in 5–10 frames per sensor per environment.
Extrinsic calibration: The relative pose (translation, rotation) between LiDAR and cameras must be known to submillimetre accuracy. Use a calibration target (checkerboard or April tags) visible to both sensors. Run this in each new environment (different rooms, outdoor vs. indoor).
Temporal synchronisation: LiDAR, camera, and IMU run at different frame rates and may have clock skew. Synchronise on a known event (hardware trigger or visual pulse) and compute precise inter-sensor delays.
Skip any of these, and you'll spend 20–30% of annotation time chasing alignment issues rather than labelling objects.
Annotation workflow for fused data
Display strategy: Show the LiDAR point cloud rendered as a 3D scene, with the RGB image projected onto it (or vice versa). Annotators label in one view; the tool propagates labels to the other.
Example (LiDAR-primary workflow):
- Render point cloud as 3D mesh, coloured by RGB projection
- Annotator places a 3D cuboid around a car in the point cloud
- Tool automatically converts cuboid to 2D bounding box in RGB image
- Annotator visually verifies: does the 2D box match the car in the image?
- If misalignment >2 pixels, flag for calibration check
Quality check: For every Nth frame (e.g., every 10th), project cuboid boundaries from LiDAR onto RGB and vice versa. Annotators verify alignment. Misalignment indicates calibration drift.
Cross-modal consistency checks
Same-object detection across views: An object visible in both LiDAR and RGB must get the same label and object ID. Common error: the same car labeled as two different IDs (one from point cloud, one from image).
Occlusion consistency: If an object is occluded in RGB but visible in LiDAR (or vice versa), annotators must handle this consistently. Define a rule: "If occluded in >80% of visible sensors, label as occluded. Otherwise label normally."
Confidence scoring: Some regions of the scene are easier to annotate in one modality than the other. A distant car is sharp in LiDAR but blurry in RGB; a reflective object is visible in RGB but appears as noise in LiDAR. Allow annotators to mark confidence per modality.
Common pitfalls and fixes
Calibration drift over time: Sensors shift slightly with temperature, vibration, or impact. Recalibrate every 100–200 frames of continuous operation. If you see a pattern of misalignment (all labels drift right by 3 pixels), stop annotation and recalibrate.
Annotation tool limitations: Many 3D annotation tools don't handle sensor fusion well. They show point clouds and images separately, requiring annotators to mentally align them. Invest in or build a tool that shows fused data natively (web-based viewer with WebGL rendering).
Object boundary ambiguity at the edge: Near the LiDAR's range limit, points become sparse and noisy. Object boundaries are hard to define. Use a rule: "If <50% of object surface is visible in point cloud, don't label it." This prevents low-confidence labels.
Inconsistent frame rates: If LiDAR runs at 10 Hz and camera at 30 Hz, which frames do you annotate? Define a strategy: "Annotate LiDAR frames only; image is auxiliary for visual confirmation." Or: "Annotate on a 10 fps resampled sequence aligned to LiDAR."
Handling partial observability
In real scenes, some objects are visible only in one sensor:
- Highly reflective surfaces (metal, glass) show up in LiDAR but not RGB, or vice versa
- Dynamic objects (moving people) may be captured by one sensor but not the other due to timing
- Shadows and glare obscure objects in one modality
Define explicit rules:
- "Label an object if visible in >50% of active sensors"
- "If visible in LiDAR only, mark as 'LiDAR-only'; if visible in RGB only, mark as 'RGB-only'"
- "If visible in both but attributes differ (size, pose), use majority vote or annotator discretion"
Cost implications and dataset size
Multi-sensor fusion annotation requires significant upfront investment in calibration and cross-modal consistency verification. This cost is justified by improved model performance, as fusion models can resolve ambiguities that single-modality models cannot. The upfront investment pays back through faster convergence and fewer model refinement cycles.
What this means for you
If you're building a multi-sensor robot, sensor fusion annotation is essential. The upfront cost is high, but you can't skip calibration — it's the foundation of everything downstream.
Plan 2–3 weeks for initial calibration, ongoing 1–2 days per 500 frames for recalibration. Use a tool that displays fused data natively. Hire annotators who understand 3D geometry and sensor physics, not generalists.
And remember: a misaligned dataset won't fail obviously. Your model will train, but it will be 20–30% worse than expected. By the time you discover the issue, you've invested months.
This is why multi-sensor annotation works best with stable, experienced teams. A specialist who has calibrated and annotated 500 fusion datasets will spot misalignment errors in seconds. New annotators take weeks to develop this intuition. Retention is your biggest lever on calibration quality and overall data quality.
FAQ
Q: How often does sensor calibration drift, and when should I recalibrate? A: Extrinsic calibration typically drifts within 100–200 frames of continuous operation due to temperature, vibration, or impact. Recalibrate every 100–200 frames, or immediately if you spot a systematic pattern (e.g. all labels drifting right by 3 pixels). Intrinsic calibration is more stable but should be verified monthly if the system is subject to environmental stress.
Q: What's the minimum alignment tolerance before I should stop annotation and recalibrate? A: More than 2 pixels of projected misalignment between LiDAR and RGB. If a 3D cuboid in the point cloud projects to a 2D box that's consistently off by >2 pixels, your calibration has drifted. Stop annotation and recalibrate. Allowing larger tolerances corrupts the training data and wastes annotation effort.
Q: Can I annotate LiDAR and RGB separately, then merge the labels? A: Not reliably. Separate annotation workflows create consistency problems: the same object gets different object IDs, or attributes diverge. Label in a fused view where both modalities are visible simultaneously. This enforces cross-modal consistency and reduces re-work.
Q: What should I do if an object is visible in LiDAR but invisible in RGB (or vice versa)? A: Define an explicit rule upfront: "Label objects visible in >50% of active sensors" or "Label in both modalities independently, then flag discrepancies for review." Common cases include reflective surfaces (visible in LiDAR, not RGB) and shadows (visible in RGB, sparse in LiDAR). Your rule should be documented in the annotation rubric.
Q: How do I handle occlusions that differ between LiDAR and RGB? A: A car might be partially occluded in RGB but fully visible in the point cloud (behind transparent material or glass). Define rules: "If occluded in >80% of visible sensors, mark as occluded; otherwise label normally." Or use a confidence score per modality if your model can consume it. Consistency across modalities matters more than perfect occlusion handling.
Q: Is a web-based annotation tool necessary for multi-sensor fusion, or can I use desktop tools? A: A web-based tool with native multi-modal display (WebGL rendering of fused point clouds + images) is strongly preferred. Desktop tools often show modalities separately, forcing annotators to mentally align them—this introduces errors. If your tool doesn't support fused display, you will spend 20–30% of annotation time chasing alignment issues instead of labelling.
Q: Should I annotate at the sensor's native frame rate, or downsample? A: Downsample if sensors run at different rates. If LiDAR is 10 Hz and cameras are 30 Hz, annotate at 10 Hz (LiDAR timebase) and use images as auxiliary visual confirmation. This avoids frame-rate mismatches and reduces annotation volume without losing temporal resolution. Document the resampling strategy in your rubric.
Learn more about robotics data pipelines or explore our data enrichment services.