A robotic arm with a gripper attachment positioned over an electronics workbench scattered with components and cables

ArticleroboticsDeep read

The Real Robotics Stack: Where Sensors, Compute, and Middleware Actually Break

Ahmad JAug 8, 202611 minUpdated Sep 15, 2026

Robotics doesn't fail in the lab: it fails at the seams. Here's an honest map of where the stack breaks and what shipping teams do differently.

A deep read: the full picture, with the receipts.

Key takeaways

LiDAR scan timestamps are assigned at the start of a full rotation rather than per point, meaning a single frame spans 50 to 100 milliseconds of real time and requires motion distortion compensation before sensor fusion.
The choice of DDS vendor under ROS 2 has measurable effects on latency, CPU overhead, and discovery time, with Cyclone DDS generally performing better for high message rates and small payloads on a single machine.
A hardware or firmware watchdog on the low-level controller is required for any system that receives commands from a non-real-time high-level planner, triggering a defined safe state if commands stop arriving.
Extended Kalman Filters handle real-time fusion efficiently but struggle with out-of-order data, while factor graph approaches such as GTSAM handle late measurements at higher computational cost.

Signaldefinitive2independent sources

The robotics stack is a integration problem disguised as an engineering problem. Each layer (sensors, compute, middleware, actuators) can be made to work in isolation, demonstrated cleanly, and benchmarked convincingly. The failure happens at the handoffs: the moment a LiDAR packet arrives 3 milliseconds late, the moment a DDS topic floods a constrained network, the moment a firmware update on a motor controller breaks an assumption baked into the abstraction layer six months ago. Teams that ship production robots have internalized one thing that demo teams haven't: the stack is only as strong as its worst seam.

The Sensor Layer: Timestamps, Dropouts, and the Illusion of Clean Data#

The first lie the sensor layer tells you is that data arrives cleanly and on time. In practice, every sensor class has a distinct failure personality.

The sensor	Its failure personality
LiDAR	Produces high-bandwidth point clouds, easily tens of megabytes per second for a dense spinning unit, and its failure mode is subtle: scan timestamps are often assigned at the start of a full rotation, not per-point. A single "frame" spans anywhere from 50 to 100 milliseconds of real time, during which the robot has moved. If you don't correct for this, called motion distortion compensation, your map is smeared. The fix requires knowing the robot's pose at every point in the scan, which means tight coupling with the IMU before you even start fusion
IMUs	Run fast, often at hundreds of Hz, and feel like the cleanest data source. They aren't. IMU data drifts: gyroscopes accumulate error over time in ways that are predictable in direction but not in magnitude without calibration. The timestamp accuracy is usually good, but the data quality degrades with temperature, vibration and age. Worse, the noise characteristics you measured in the lab don't match what the sensor sees bolted next to a brushless motor
RGB-D cameras	Structured light or stereo depth, and a different problem: the depth map is only valid at a fixed range and under certain lighting conditions. Outdoors or near highly reflective surfaces, depth readings either fail silently, returning maximum-range values that look plausible, or produce structured noise that naive fusion treats as real geometry. Their bandwidth footprint is large, and on embedded hardware, just getting raw data off the sensor without dropping frames requires careful DMA configuration and bus management
Proprioception	Joint encoders, force-torque sensors, current sensing. Seems low-stakes until you realise it is the only feedback the controller has about what the robot's body is actually doing. Encoder resolution matters, but latency matters more. A joint position reading that is even a few milliseconds stale can destabilise a high-bandwidth controller. Timing here isn't a software problem; it's a hardware problem that software has to compensate for

The naive approach to all of this is to timestamp everything on receipt and assume synchrony. Production teams timestamp at the hardware interrupt level where possible, propagate uncertainty explicitly, and treat any sensor that misses more than a small number of consecutive packets as degraded, not absent.

Middleware in the Middle: ROS 2, DDS, and the Real-Time Ceiling#

ROS 2 is the closest thing robotics has to a lingua franca. It solves real problems: process isolation, a topic/service/action abstraction that maps naturally to robot architectures, a growing ecosystem of drivers and tools. If you're building a new robot system and you're not starting with ROS 2, you need a specific reason.

Under the hood, ROS 2 is built on DDS, Data Distribution Service, a publish-subscribe middleware standard from the Object Management Group. DDS handles discovery, serialization, transport, and quality-of-service policies. The ROS 2 abstraction layer sits on top of a DDS vendor implementation, and the choice is not cosmetic.

The vendor	What it is, and where it bites
Fast DDS (formerly FastRTPS)	The default. It has more configuration surface area, which is both its strength and its trap: misconfigured QoS policies are a common source of mysterious dropped messages and discovery failures that look like sensor problems
Cyclone DDS	Widely used and generally considered lower-latency on local networks. It tends to perform better in the scenarios most robotic systems actually face: high message rates, small payloads, shared-memory transport on a single machine
RTI Connext	The enterprise option, with the strongest real-time story

On a robot with a dozen nodes exchanging data at high frequency over localhost, the difference between vendors in latency, CPU overhead and discovery time is measurable in ways that affect control loop stability.

The real-time ceiling is where ROS 2's architectural compromise becomes unavoidable. ROS 2 nodes run in standard Linux processes. Linux is not a real-time operating system. Even with SCHED_FIFO scheduling, kernel preemption patches, and CPU isolation, you cannot guarantee sub-millisecond jitter in a ROS 2 node. For a path planner or a state estimator, this is acceptable. For a motor controller that needs to close a current loop at several kilohertz, it is not.

This is why micro-ROS exists. It runs a stripped-down ROS 2 client on a microcontroller, a Cortex-M4 or similar, using a bridge to connect to the main ROS 2 graph. The microcontroller runs an RTOS like FreeRTOS or Zephyr, which can meet hard real-time deadlines. The bridge (micro-ROS agent) runs on the host system and translates between the two worlds. It works, but the bridge is itself a point of failure: if it crashes or stalls, the microcontroller is isolated from the planner.

The Hard Boundary: Real-Time Control vs. High-Level Planning#

Every serious robot system has an architectural split between a low-level real-time controller and a high-level non-real-time planner. This isn't a design choice so much as a physical reality imposed by timing requirements.

	Low-level controller	High-level planner
Runs on	A microcontroller or dedicated FPGA	An SBC like a Raspberry Pi-class board, a Jetson, or an x86 compute unit
Rate	Kilohertz	Tens to hundreds of milliseconds
Responsible for	Reading encoder feedback, running PID or more sophisticated control laws, generating PWM signals, and enforcing hardware safety limits	Perception, planning, state estimation and user interface
Timing	It must not miss its deadline. The consequence of a missed deadline isn't a slower response; it's instability, a jerk, or a fault condition	Inherently non-deterministic

The interface between these two worlds is where teams get into trouble. A common failure mode: the high-level planner sends a velocity command to the low-level controller, the command is delayed due to system load, and the low-level controller either holds the last command (which may now be wrong) or goes to a safe stop. Teams that have shipped reliable systems define this interface contract explicitly: what is the command rate, what is the timeout before the low-level system declares the high-level dead and enters a safe state, and what "safe" means for that specific platform.

The watchdog is not optional. Any low-level controller that can receive commands from a non-real-time system needs a hardware or firmware-level watchdog that triggers a known safe behavior (zero velocity, hold position, power-off) if commands stop arriving. This is not complex to implement but is frequently missing in systems that move from lab to deployment.

Fusion Without Fiction: Making Asynchronous Sensors Agree#

Sensor fusion is the process of combining multiple noisy, asynchronous data streams into a single consistent state estimate. The two dominant approaches in production systems are the Extended Kalman Filter (EKF) and factor graphs.

	Extended Kalman Filter	Factor graphs (GTSAM, g2o)
How it works	Maintains a running estimate of state — pose, velocity — and updates it as new sensor measurements arrive	Frames the estimation problem as a graph of variables (poses, landmarks) connected by factors (measurements)
Strength	Computational efficiency: an EKF update is cheap and can run in real time on modest hardware	Optimisation over the graph can incorporate measurements at any timestamp, handle loop closures, and produce globally consistent estimates
Weakness	Processes measurements sequentially and doesn't handle out-of-order data gracefully. A late measurement, as LiDAR measurements frequently are, requires either holding a buffer of state history (expensive) or discarding it (losing information)	Cost is computational: optimisation is batch, and while incremental solvers like iSAM2 make it tractable in real time, it still demands more CPU than an EKF and more careful management of graph growth
Use it for	Low-latency applications where you can tolerate occasional inconsistency	Applications where global consistency matters more than instantaneous latency: mapping, long-horizon localisation, manipulation with contact

The cost of getting fusion wrong isn't just inaccurate state estimates. A fused pose that's wrong by a few centimeters at the wrong moment means a manipulator hits an obstacle the planner thought it had cleared, or a mobile robot chooses a path that doesn't exist. Silent fusion failures (where the filter remains confident while diverging from reality) are harder to detect than outright crashes and more dangerous in deployment.

Hardware Abstraction or Hardware Fragility#

Every sensor and actuator ships with a vendor API: a SDK, a ROS driver, or both. Every one of those APIs will change. The question is whether your system is designed so that change is contained.

The pattern What happens when the vendor changes something

Vendor APIs threaded through the codebase When a motor controller manufacturer pushes a firmware update that changes the command packet structure, or a LiDAR vendor releases a new SDK that renames a function, the effect propagates everywhere

A hardware abstraction layer (HAL) A thin interface your system code talks to, with a vendor-specific implementation behind it. Your sensor fusion node doesn't know it's talking to a Velodyne or a Hesai: it talks to your LidarSource interface. Your motion controller doesn't know it's driving a specific motor driver: it calls your JointController interface. Firmware updates require updating the vendor implementation, not touching the rest of the stack

The pattern	What happens when the vendor changes something
Vendor APIs threaded through the codebase	When a motor controller manufacturer pushes a firmware update that changes the command packet structure, or a LiDAR vendor releases a new SDK that renames a function, the effect propagates everywhere
A hardware abstraction layer (HAL)	A thin interface your system code talks to, with a vendor-specific implementation behind it. Your sensor fusion node doesn't know it's talking to a Velodyne or a Hesai: it talks to your `LidarSource` interface. Your motion controller doesn't know it's driving a specific motor driver: it calls your `JointController` interface. Firmware updates require updating the vendor implementation, not touching the rest of the stack

This sounds obvious but requires discipline to maintain. The temptation, especially in early development, is to call vendor APIs directly because it's faster. The cost surfaces at exactly the wrong time: during integration testing at a customer site, or in a fleet of deployed robots that can't all be updated simultaneously.

A related practice: version-pin your driver layer and test firmware updates explicitly before pushing to production hardware. Vendor firmware updates have broken production robots. This is not rare.

From Demo to Deployment: What a Stack That Actually Ships Looks Like#

A robot that runs for twenty minutes in a controlled demo and a robot that runs an eight-hour shift in a warehouse are different systems. The gap is mostly in instrumentation, failure handling, and update infrastructure.

What ships adds	What it means in practice
Operational telemetry	The first thing that separates shipped systems from research systems. Every node logs its cycle time and message latency. Every sensor logs its packet drop rate. The low-level controller logs its watchdog resets. None of this is sent to the cloud in real time — it is buffered locally and synced during downtime — but it means that when something goes wrong in the field, you have data to reason from instead of guessing
Graceful degradation	The system has explicit policies for sensor failure. If the primary LiDAR drops out, the robot falls back to camera-only navigation at reduced speed and flags the condition to the fleet manager. It doesn't freeze. It doesn't crash. It doesn't continue at full speed with a blind spot. Each failure mode has a defined degraded operating mode, and those modes are tested explicitly, not just in simulation
OTA update pipelines	Harder for robots than for phones, because the system being updated may be physically engaged with the world. A standard practice is staged rollouts: update one robot, run it for a shift, validate telemetry, then roll to the fleet. The update system needs to roll back atomically — not just the application layer but firmware on microcontrollers, which requires either dual-partition firmware storage or a robust recovery mode
Watchdog logic	Extends beyond the low-level controller. If the localisation module hasn't published a pose estimate in more than some threshold, the planner should stop requesting motion. If the planner hasn't sent a command in longer than expected, the mission manager should flag the robot as stalled. These aren't exotic features: they're the minimum operational infrastructure for a system that runs unsupervised

Key Takeaways#

The stack fails at seams, not centers. Every layer works in isolation. Invest engineering effort in the interfaces between them.
Timestamp everything at the hardware level. Software-assigned timestamps introduce jitter that fusion algorithms can't distinguish from real sensor noise.
ROS 2 is the right starting point, but its real-time ceiling is real: plan the microcontroller/SBC split before you need it, not after.
DDS vendor choice matters. Benchmark on your actual hardware and message patterns before assuming defaults are adequate.
Build a HAL from day one. Vendor APIs will change; whether that change breaks your system is an architecture decision you make early.
Graceful degradation and watchdog logic are not polish. They're the difference between a robot that demos and a robot that ships.
Operational telemetry is how you learn from the field. You cannot debug a robot you can't observe.