When writing low-latency device drivers or user-space PCIe drivers (such as with DPDK, SPDK, or custom VFIO mappings), a common design pattern is the payload-and-doorbell mechanism. You write a command payload into memory-mapped I/O (MMIO) BAR space and subsequently ring a doorbell register to tell the device hardware to process the payload.

A critical question often arises: Do you need to mark the payload as volatile, or is marking only the doorbell register as volatile sufficient?

The short answer is: You must treat both the payload and the doorbell as volatile (or use explicit MMIO accessors) AND you must use the proper memory fence. Relying only on a volatile doorbell while leaving the payload as standard non-volatile memory can introduce silent corruption and subtle race conditions. Let’s break down why this happens and how to implement it correctly on x86-64 and AArch64 in C++20.

1. Why the Payload Must Also Be Accessed as Volatile

In standard C++, compiler optimizations operate under the "as-if" rule. The compiler is free to reorder, coalesce, or eliminate memory operations as long as observable side effects within the C++ abstract machine are preserved.

If your Payload struct is accessed without volatile semantics, several issues arise:

  • Compiler Reordering Across Volatiles: According to the ISO C++ standard, a non-volatile store can be reordered across a volatile access. The compiler is entirely within its rights to hoist or sink non-volatile writes across the doorbell write.
  • Dead Store Elimination & Merging: If you write consecutive commands to the same payload slot, the compiler may optimize away earlier writes, assuming normal RAM semantics where intermediate values are never read.
  • Unintended Vectorization and Bus Split Transactions: For plain C++ structs, compilers may emit wide SSE/AVX vector instructions (like vmovups) to copy Payload. Many PCIe endpoints cannot handle 128-bit or 256-bit unaligned MMIO bus transactions, triggering bus errors or silent drops. volatile forces scalar, predictable access sizes.

2. Compiler Ordering vs. Hardware Bus Ordering

Marking accesses as volatile guarantees compiler-level ordering among volatile operations. However, volatile in standard C++ generates no CPU-level hardware memory barriers (with the exception of MSVC with /volatile:ms, which is non-portable).

Hardware ordering depends heavily on the memory mapping type configured during mmap:

  • Uncacheable (UC): Typical for MMIO registers. On x86-64, writes to UC memory are strongly ordered by the processor pipeline. A simple compiler barrier is often sufficient. On AArch64 (Device-nGnRE), writes to the same peripheral are ordered, but writes across different endpoints or memory types require explicit barriers.
  • Write-Combining (WC): Frequently used for high-performance write queues. Under WC, the CPU buffers and can reorder writes before flushing them to the PCIe fabric as Transaction Layer Packets (TLPs). In this scenario, without a hardware store fence, the doorbell write can reach the PCIe device before the payload data.

3. The Correct Pattern in Modern C++

Instead of casting an entire struct to a volatile pointer (which was partially deprecated in C++20 for compound assignments, though still legal for simple member assignment), the safest and cleanest approach is using dedicated MMIO accessor functions or explicit volatile wrappers.

Here is a robust, portable implementation targeting GCC/Clang on x86-64 and AArch64:

#include <cstdint>]n#include <atomic>]n#if defined(__x86_64__) || defined(_M_X64)]n#include <immintrin.h>]n#endif]n]nstruct Payload {]n    uint64_t addr;]n    uint32_t length;]n    uint32_t flags;]n};]n]nstruct Bar1 {]n    Payload  payload;]n    uint32_t doorbell;]n};]n]n// Helper to write to MMIO space predictably]ntemplate <typename T>]ninline void mmio_write(volatile T* addr, T value) noexcept {]n    *addr = value;]n}]n]n// Enforce CPU and bus store ordering between payload and doorbell]ninline void mmio_wc_barrier() noexcept {]n#if defined(__x86_64__) || defined(_M_X64)]n    // If the BAR mapping is Write-Combining (WC), _mm_sfence() is mandatory.]n    // If it is guaranteed to be Uncacheable (UC), a compiler barrier suffices.]n    _mm_sfence();]n#elif defined(__aarch64__)]n    // Outer Shareable Store barrier for Device memory]n    asm volatile("dmb oshst" ::: "memory");]n#else]n    // Fallback: standard atomic thread fence]n    std::atomic_thread_fence(std::memory_order_seq_cst);]n#endif]n}]n]nvoid submit(Bar1* bar, const Payload& p, uint32_t seq) {]n    // 1. Write the payload using volatile writes]n    volatile Bar1* vbar = bar;]n    mmio_write(&vbar->payload.addr,   p.addr);]n    mmio_write(&vbar->payload.length, p.length);]n    mmio_write(&vbar->payload.flags,  p.flags);]n]n    // 2. Hardware and compiler barrier]n    mmio_wc_barrier();]n]n    // 3. Ring the doorbell]n    mmio_write(&vbar->doorbell, seq);]n}

4. Can We Use std::atomic Instead?

In standard C++, std::atomic<T> is designed for inter-thread communication in normal system RAM. Using std::atomic over an mmap'd PCIe BAR is technically undefined behavior (UB) according to the ISO standard. MMIO memory cannot support hardware cache coherence protocols or compare-and-swap bus transactions that standard atomics may compile to.

Standard practice in production systems (such as the Linux kernel’s writeq()/writel() macros or DPDK’s rte_write32()) is to use raw pointers with volatile qualifiers accompanied by inline assembly barriers (or intrinsics like _mm_sfence).

Summary Checklist

  • Always use volatile for both: Do not rely on compiler luck. Both the payload registers and the doorbell must be treated as volatile to prevent store elimination and unwanted vectorization.
  • Include an explicit memory fence: Compiler barriers (asm volatile("" ::: "memory")) prevent compiler instruction scheduling, but a hardware fence (such as _mm_sfence() on x86 or dmb oshst on ARM64) is essential if the mapping uses write-combining buffers.
  • Respect register widths: Ensure your writes match the bus sizes supported by the target PCIe peripheral (e.g., distinct 32-bit or 64-bit aligned writes).