CUDA Lock-In Is Real: A Precise Cost Accounting of What Switching GPU Vendors Actually Breaks

CUDA lock-in is the compounded result of a decade of legitimate engineering decisions, not a vendor conspiracy — and understanding it at that level of precision is the only way to make a defensible GPU infrastructure choice. The question isn't whether you're locked in; you are. The question is which layers of that lock-in are load-bearing for your workload, which escape routes are real, and what each one actually costs.

The Five Software Layers That Are Each a Separate Porting Problem#

Most teams think of CUDA lock-in as a single thing. It isn't. It's a stack of interdependencies that each require independent attention:

CUDA Runtime and driver API. The foundational layer — kernel launches, memory allocation, streams, events, unified memory. Everything above it assumes specific pointer semantics and synchronisation behaviour. HIP (the portability layer in AMD's ROCm stack) covers a large fraction of the runtime API syntactically, and hipify can mechanically translate most calls. But mechanical translation and behavioural equivalence are different things. Warp-level primitives, cooperative groups, and CUDA graph APIs have partial or inconsistent HIP analogues depending on the ROCm version and the target GPU generation.

cuDNN. NVIDIA's deep neural network library is the silent load-bearing wall in almost every production training stack. It provides hand-tuned implementations of convolutions, attention, normalisation, and activation functions that are not just fast but numerically characterised — teams build regression tolerances around its specific floating-point behaviour. AMD's MIOpen is the functional counterpart, but library coverage, operator fusion support, and the version-to-version stability of numerics differ. Porting cuDNN-dependent code to MIOpen is a revalidation project, not a drop-in swap.

cuBLAS and cuSPARSE. Matrix multiplication throughput is where most compute time actually goes. cuBLAS has been tuned for NVIDIA tensor core generations through multiple architecture cycles. rocBLAS covers the core GEMM surface, but tiling strategies, batching behaviour, and edge-case precision may diverge. Any model that has been hand-tuned with specific cuBLAS workspace allocations or algorithm selection hints needs those decisions revisited.

Thrust. The parallel algorithms library is embedded in a surprising amount of GPU-adjacent data pipeline code — sorting, prefix sums, transforms on device vectors. Its ROCm counterpart is rocThrust, which is a genuine port rather than a reimplementation, but code that mixes Thrust with raw CUDA kernels or CUDA streams creates coupling that hipify doesn't cleanly resolve.

NCCL. Multi-GPU and multi-node collective communication is where distributed training either scales or doesn't. NCCL is deeply integrated with NVIDIA's NVLink topology awareness. RCCL is AMD's fork of NCCL and has improved substantially, but ring-topology optimisations, algorithm selection for specific collective sizes, and behaviour on heterogeneous network fabrics remain areas where practical production experience is thinner than vendor documentation implies.

The compounding effect is the point: a training job that uses all five of these — which describes most production deep learning workloads — isn't facing one porting decision; it's facing five separate porting decisions with validation requirements that interact.

A Realistic Switching-Cost Taxonomy#

Developer re-tooling time. A competent CUDA engineer is not automatically a competent ROCm or oneAPI engineer. The GPU programming model is similar, but the toolchain (profilers, debuggers, compiler flags, occupancy calculators) is different enough that productivity drops measurably during ramp-up. Budget for this explicitly — teams that don't typically underestimate the calendar impact of context switching.

Numerical revalidation. This is the cost most migration estimates ignore. Modern ML pipelines have accumulated tolerance assumptions about specific library implementations. Revalidating that a model trains to the same loss curve on a different hardware stack, using different library kernels, can take weeks per model and requires regression infrastructure that many teams don't have in a hardware-agnostic form. Silent numerical divergence — where outputs look plausible but differ from reference — is the specific risk that makes experienced engineers cautious about half-finished migrations.

CI/CD pipeline changes. Most ML CI pipelines assume NVIDIA container toolkits, specific driver versions, and CUDA-tagged base images. Rebuilding these for ROCm or oneAPI isn't architecturally complex, but it consumes real engineering time, especially when third-party testing tools (profiling dashboards, experiment tracking integrations) have NVIDIA-specific code paths.

Third-party library access. This is the hidden cost that compounds fastest. Flash Attention's highly optimised CUDA kernels have ROCm ports of varying maturity. Many specialised libraries in the production ML ecosystem — certain quantisation libraries, custom attention variants, inference engines — exist only with CUDA backends or have ROCm support that lags the CUDA version by one or more releases. A switch doesn't just require porting your code; it requires auditing every dependency and accepting that some won't follow.

ROCm and HIP: Honest Capability Assessment#

ROCm (the software platform) and HIP (the C++ runtime API within it) are distinct things. Conflating them produces bad migration estimates. ROCm is a full platform stack including drivers, compilers, math libraries, and profiling tools. HIP is specifically the runtime API and kernel language that allows source-level portability between AMD and NVIDIA hardware.

Recent ROCm major releases have made genuine progress on MI300-series hardware support, improved the hipify tooling, and expanded the MIOpen operator set. For workloads that map cleanly to standard transformer training — large GEMM-heavy forward and backward passes, straightforward collective communications — ROCm on current AMD data-centre hardware is meaningfully competitive in practice, not just on paper.

The honest problem areas: CUDA features introduced after roughly Ampere's cooperative group extensions have uneven HIP support. Warp-level intrinsics that CUDA kernels use for custom attention or custom normalisation may translate syntactically but produce wrong results on certain AMD GPU generations due to warp-size differences (AMD's wavefront is 64 lanes, not 32) — and this is a correctness problem, not a performance problem. Custom CUDA kernels written with warp-size assumptions hardcoded are a frequent source of silent divergence after hipify. Any migration plan must include explicit wavefront-size audits.

Portability Frameworks: What Has Actually Shipped at Scale#

OpenCL remains the oldest portable GPU compute standard and is genuinely cross-vendor, but its programming model is verbose, its performance portability is limited (you still need vendor-specific tuning for each backend), and mainstream ML framework development has largely moved past it. It's a viable option for certain compute workloads, not a practical path for production deep learning.

SYCL/oneAPI is Intel's bet — a C++ abstraction over heterogeneous compute with DPC++ as the compiler. The portability claim is real in the sense that SYCL code can target Intel, AMD, and NVIDIA hardware. The practice is that each backend requires target-specific tuning to reach competitive performance, and the Intel Gaudi (formerly Habana) line uses a separate optimised software path rather than oneAPI for its production workloads. Intel's portability-first messaging accurately describes an architecture choice; it doesn't automatically mean write-once-perform-everywhere.

OpenAI's Triton compiler is the most interesting near-term option for teams writing custom GPU kernels. Triton operates at a tile abstraction level above raw CUDA/HIP and generates reasonably efficient code for multiple backends including NVIDIA and AMD. Production ML workloads have shipped on Triton-generated kernels — PyTorch 2.x's torch.compile uses Triton as a backend, and this is real, at-scale production usage. The limitation is that Triton targets the kernel layer; it doesn't replace NCCL, doesn't abstract memory management, and doesn't solve the library dependency problem. It reduces the kernel-porting burden significantly for teams writing custom operators, but it's one piece, not a complete solution.

JAX with pluggable backends offers genuine portability for ML research code, and there is production deployment experience on TPUs and increasingly on AMD hardware through the XLA compiler stack. For teams already in the JAX ecosystem, the backend abstraction is architecturally cleaner than PyTorch's. For teams migrating PyTorch production code to JAX, the migration cost is non-trivial in its own right.

The Organisational Layer: Harder to Fix Than the Code#

CUDA expertise functions as an implicit hiring filter across the ML engineering market. Job descriptions, interview rubrics, and senior engineer mentorship pipelines are all calibrated around CUDA knowledge. A migration project doesn't just need code changes; it needs engineers who can reason about AMD's GCN/RDNA/CDNA architecture in the same detail that CUDA engineers reason about SM occupancy and shared memory bank conflicts. That expertise is thinner on the market and takes time to build internally.

Internal tooling debt compounds this. Many mature ML infrastructure teams have profiling dashboards, training monitoring tools, and debugging workflows that have NVIDIA-specific assumptions baked in — Nsight integration, NVML for hardware metrics, specific CUDA event timing patterns. These tools don't break loudly when you switch hardware; they quietly provide less useful data, which degrades your ability to diagnose performance regressions.

Enterprise re-skilling typically happens in one of two ways: a dedicated tiger team that owns the migration and builds knowledge with explicit mandate, or a distributed model where each team is expected to maintain both CUDA and alternative-hardware expertise simultaneously. The latter consistently underestimates time and produces partial migrations that introduce more risk than a clean cut.

Concrete Mitigation Strategies with Honest Trade-offs#

Abstraction layers at the framework level, not the kernel level. If your training code stays in PyTorch or JAX and avoids custom CUDA extensions, portability is substantially higher. The trade-off is that some optimisations — particularly custom attention kernels, custom quantisation, custom collective operations — require dropping to a lower level. Decide explicitly which optimisations are worth the portability debt before writing them.

Containerisation with hardware-abstracted entrypoints. Structuring your container builds so that the CUDA/ROCm runtime layer is injected at deployment time rather than baked into application logic is achievable today and reduces the blast radius of a hardware switch. It doesn't eliminate library compatibility issues, but it isolates them.

Workload segmentation by portability risk. Not all workloads carry equal lock-in. Inference on standard transformer architectures using framework-level ops is more portable than custom training loops with hand-written CUDA kernels. Running commodity inference workloads on alternative hardware while keeping custom training on CUDA is a real middle path that some organisations are executing, not a theoretical option.

Invest in Triton for new custom kernels. If your team writes custom GPU kernels, writing them in Triton rather than CUDA is the highest-leverage portability investment available today. You accept some performance ceiling uncertainty on non-NVIDIA hardware, but you preserve the option to port without a rewrite.

Explicit numerical regression infrastructure. Before any migration, build hardware-agnostic numerical regression tests that capture acceptable tolerance ranges for your models. This investment pays off regardless of whether you switch vendors — it also catches regressions within CUDA version upgrades — and it's a prerequisite for making any hardware migration defensible.

Key Takeaways#

• CUDA lock-in is five separate porting problems (runtime, cuDNN, cuBLAS, Thrust, NCCL), each with independent validation requirements.
• The largest underestimated cost is numerical revalidation, not code translation.
• ROCm/HIP has made real progress; the specific risk is warp-size-dependent correctness bugs post-hipify, not general incompetence.
• Triton is the most actionable portability investment for teams writing custom kernels today.
• Organisational lock-in — expertise, tooling assumptions, hiring filters — outlasts code migrations and needs an explicit plan.
• The practical path is workload segmentation and abstraction-layer discipline, not a single flag-day cutover.
• Vendor portability claims describe architectural intent; performance portability on production workloads requires target-specific work regardless of the framework.