The Real Robotics Stack: Where Sensors, Compute, and Middleware Actually Break

The robotics stack is a integration problem disguised as an engineering problem. Each layer — sensors, compute, middleware, actuators — can be made to work in isolation, demonstrated cleanly, and benchmarked convincingly. The failure happens at the handoffs: the moment a LiDAR packet arrives 3 milliseconds late, the moment a DDS topic floods a constrained network, the moment a firmware update on a motor controller breaks an assumption baked into the abstraction layer six months ago. Teams that ship production robots have internalized one thing that demo teams haven't: the stack is only as strong as its worst seam.

The Sensor Layer: Timestamps, Dropouts, and the Illusion of Clean Data#

The first lie the sensor layer tells you is that data arrives cleanly and on time. In practice, every sensor class has a distinct failure personality.

LiDAR produces high-bandwidth point clouds — easily tens of megabytes per second for a dense spinning unit — and its failure mode is subtle: scan timestamps are often assigned at the start of a full rotation, not per-point. That means a single "frame" spans anywhere from 50 to 100 milliseconds of real time, during which the robot has moved. If you don't correct for this — called motion distortion compensation — your map is smeared. The fix requires knowing the robot's pose at every point in the scan, which means tight coupling with the IMU before you even start fusion.

IMUs run fast, often at hundreds of Hz, and feel like the cleanest data source. They aren't. IMU data drifts — gyroscopes accumulate error over time in ways that are predictable in direction but not in magnitude without calibration. The timestamp accuracy is usually good, but the data quality degrades with temperature, vibration, and age. Worse, the noise characteristics that you measured in the lab don't match what the sensor sees bolted next to a brushless motor.

RGB-D cameras (structured light or stereo depth) introduce a different problem: the depth map is only valid at a fixed range and under certain lighting conditions. Outdoors or near highly reflective surfaces, depth readings either fail silently — returning maximum-range values that look plausible — or produce structured noise that naive fusion treats as real geometry. Their bandwidth footprint is large, and on embedded hardware, just getting raw data off the sensor without dropping frames requires careful DMA configuration and bus management.

Proprioception — joint encoders, force-torque sensors, current sensing — seems low-stakes until you realize it's the only feedback the controller has about what the robot's body is actually doing. Encoder resolution matters, but what matters more is latency. A joint position reading that is even a few milliseconds stale can destabilize a high-bandwidth controller. Timing here isn't a software problem; it's a hardware problem that software has to compensate for.

The naive approach to all of this is to timestamp everything on receipt and assume synchrony. Production teams timestamp at the hardware interrupt level where possible, propagate uncertainty explicitly, and treat any sensor that misses more than a small number of consecutive packets as degraded, not absent.

Middleware in the Middle: ROS 2, DDS, and the Real-Time Ceiling#

ROS 2 is the closest thing robotics has to a lingua franca. It solves real problems: process isolation, a topic/service/action abstraction that maps naturally to robot architectures, a growing ecosystem of drivers and tools. If you're building a new robot system and you're not starting with ROS 2, you need a specific reason.

Under the hood, ROS 2 is built on DDS — Data Distribution Service — a publish-subscribe middleware standard from the Object Management Group. DDS handles discovery, serialization, transport, and quality-of-service policies. The ROS 2 abstraction layer sits on top of a DDS vendor implementation: Fast DDS (formerly FastRTPS) is the default, Cyclone DDS is widely used and generally considered lower-latency on local networks, and RTI Connext is the enterprise option with the strongest real-time story.

The choice of DDS vendor is not cosmetic. On a robot with a dozen nodes exchanging data at high frequency over localhost, the difference between vendors in latency, CPU overhead, and discovery time is measurable in ways that affect control loop stability. Cyclone DDS tends to perform better in the scenarios most robotic systems actually face — high message rates, small payloads, shared-memory transport on a single machine. Fast DDS has more configuration surface area, which is both its strength and its trap: misconfigured QoS policies are a common source of mysterious dropped messages and discovery failures that look like sensor problems.

The real-time ceiling is where ROS 2's architectural compromise becomes unavoidable. ROS 2 nodes run in standard Linux processes. Linux is not a real-time operating system. Even with SCHED_FIFO scheduling, kernel preemption patches, and CPU isolation, you cannot guarantee sub-millisecond jitter in a ROS 2 node. For a path planner or a state estimator, this is acceptable. For a motor controller that needs to close a current loop at several kilohertz, it is not.

This is why micro-ROS exists. It runs a stripped-down ROS 2 client on a microcontroller — a Cortex-M4 or similar — using a bridge to connect to the main ROS 2 graph. The microcontroller runs an RTOS like FreeRTOS or Zephyr, which can meet hard real-time deadlines. The bridge (micro-ROS agent) runs on the host system and translates between the two worlds. It works, but the bridge is itself a point of failure: if it crashes or stalls, the microcontroller is isolated from the planner.

The Hard Boundary: Real-Time Control vs. High-Level Planning#

Every serious robot system has an architectural split between a low-level real-time controller and a high-level non-real-time planner. This isn't a design choice so much as a physical reality imposed by timing requirements.

The low-level controller — typically on a microcontroller or dedicated FPGA — runs at kilohertz rates and is responsible for: reading encoder feedback, running PID or more sophisticated control laws, generating PWM signals, and enforcing hardware safety limits. It must not miss its deadline. The consequence of a missed deadline isn't a slower response; it's instability, a jerk, or a fault condition.

The high-level planner — running on an SBC like a Raspberry Pi-class board, a Jetson, or an x86 compute unit — handles perception, planning, state estimation, and user interface. It operates on a much slower timescale: tens to hundreds of milliseconds. It is inherently non-deterministic in its timing.

The interface between these two worlds is where teams get into trouble. A common failure mode: the high-level planner sends a velocity command to the low-level controller, the command is delayed due to system load, and the low-level controller either holds the last command (which may now be wrong) or goes to a safe stop. Teams that have shipped reliable systems define this interface contract explicitly: what is the command rate, what is the timeout before the low-level system declares the high-level dead and enters a safe state, and what "safe" means for that specific platform.

The watchdog is not optional. Any low-level controller that can receive commands from a non-real-time system needs a hardware or firmware-level watchdog that triggers a known safe behavior — zero velocity, hold position, power-off — if commands stop arriving. This is not complex to implement but is frequently missing in systems that move from lab to deployment.

Fusion Without Fiction: Making Asynchronous Sensors Agree#

Sensor fusion is the process of combining multiple noisy, asynchronous data streams into a single consistent state estimate. The two dominant approaches in production systems are the Extended Kalman Filter (EKF) and factor graphs.

An EKF maintains a running estimate of state (pose, velocity, etc.) and updates it as new sensor measurements arrive. Its strength is computational efficiency — an EKF update is cheap and can run in real time on modest hardware. Its weakness is that it processes measurements sequentially and doesn't handle out-of-order data gracefully. If a measurement arrives late — as LiDAR measurements frequently do — incorporating it correctly requires either holding a buffer of state history (computationally expensive) or discarding it (losing information).

Factor graphs (used in systems like GTSAM and g2o) frame the estimation problem differently: as a graph of variables (poses, landmarks) connected by factors (measurements). Optimization over the graph can incorporate measurements at any timestamp, handle loop closures, and produce globally consistent estimates. The cost is computational: factor graph optimization is batch, and while incremental solvers like iSAM2 make it tractable in real time, it still demands more CPU than an EKF and requires more careful management of graph growth.

The engineering tradeoff is explicit: EKF for low-latency applications where you can tolerate occasional inconsistency; factor graphs for applications where global consistency matters more than instantaneous latency — mapping, long-horizon localization, manipulation with contact.

The cost of getting fusion wrong isn't just inaccurate state estimates. A fused pose that's wrong by a few centimeters at the wrong moment means a manipulator hits an obstacle the planner thought it had cleared, or a mobile robot chooses a path that doesn't exist. Silent fusion failures — where the filter remains confident while diverging from reality — are harder to detect than outright crashes and more dangerous in deployment.

Hardware Abstraction or Hardware Fragility#

Every sensor and actuator ships with a vendor API — a SDK, a ROS driver, or both. Every one of those APIs will change. The question is whether your system is designed so that change is contained.

Teams that don't think about this explicitly end up with vendor APIs threaded through their codebase. When a motor controller manufacturer pushes a firmware update that changes the command packet structure, or when a LiDAR vendor releases a new SDK that renames a function, the effect propagates everywhere.

The production pattern is a hardware abstraction layer (HAL): a thin interface your system code talks to, with a vendor-specific implementation behind it. Your sensor fusion node doesn't know it's talking to a Velodyne or a Hesai — it talks to your LidarSource interface. Your motion controller doesn't know it's driving a specific motor driver — it calls your JointController interface. Firmware updates require updating the vendor implementation, not touching the rest of the stack.

This sounds obvious but requires discipline to maintain. The temptation — especially in early development — is to call vendor APIs directly because it's faster. The cost surfaces at exactly the wrong time: during integration testing at a customer site, or in a fleet of deployed robots that can't all be updated simultaneously.

A related practice: version-pin your driver layer and test firmware updates explicitly before pushing to production hardware. Vendor firmware updates have broken production robots. This is not rare.

From Demo to Deployment: What a Stack That Actually Ships Looks Like#

A robot that runs for twenty minutes in a controlled demo and a robot that runs an eight-hour shift in a warehouse are different systems. The gap is mostly in instrumentation, failure handling, and update infrastructure.

Operational telemetry is the first thing that separates shipped systems from research systems. Every node logs its cycle time and message latency. Every sensor logs its packet drop rate. The low-level controller logs its watchdog resets. None of this is sent to the cloud in real time — it's buffered locally and synced during downtime — but it means that when something goes wrong in the field, you have data to reason from instead of guessing.

Graceful degradation means the system has explicit policies for sensor failure. If the primary LiDAR drops out, the robot falls back to camera-only navigation at reduced speed and flags the condition to the fleet manager. It doesn't freeze. It doesn't crash. It doesn't continue at full speed with a blind spot. Each failure mode has a defined degraded operating mode, and those modes are tested explicitly — not just in simulation.

OTA update pipelines for robots are harder than for phones because the system being updated may be physically engaged with the world. A standard practice is staged rollouts: update one robot, run it for a shift, validate telemetry, then roll to the fleet. The update system needs to be able to roll back atomically — not just the application layer, but firmware on microcontrollers, which requires either dual-partition firmware storage or a robust recovery mode.

Watchdog logic extends beyond the low-level controller. The high-level system needs watchdogs too: if the localization module hasn't published a pose estimate in more than some threshold, the planner should stop requesting motion. If the planner hasn't sent a command in longer than expected, the mission manager should flag the robot as stalled. These aren't exotic features — they're the minimum operational infrastructure for a system that runs unsupervised.

Key Takeaways#

• The stack fails at seams, not centers. Every layer works in isolation. Invest engineering effort in the interfaces between them.
• Timestamp everything at the hardware level. Software-assigned timestamps introduce jitter that fusion algorithms can't distinguish from real sensor noise.
• ROS 2 is the right starting point, but its real-time ceiling is real — plan the microcontroller/SBC split before you need it, not after.
• DDS vendor choice matters. Benchmark on your actual hardware and message patterns before assuming defaults are adequate.
• Build a HAL from day one. Vendor APIs will change; whether that change breaks your system is an architecture decision you make early.
• Graceful degradation and watchdog logic are not polish. They're the difference between a robot that demos and a robot that ships.
• Operational telemetry is how you learn from the field. You cannot debug a robot you can't observe.