When developing custom hardware models or virtual devices in QEMU, managing synchronous I/O operations without stalling the entire hypervisor is a classic challenge. A common scenario arises when an MMIO read/write operation must perform a blocking network or file descriptor (FD) transaction, such as writing a payload and waiting for an external response before completing the guest memory transaction.

If you execute blocking system calls directly inside your device's MMIO callbacks, you inadvertently block the Big QEMU Lock (BQL) and the Main Event Loop. This freezes other virtual cores, timers, graphical interfaces, and device updates. Here is an in-depth guide on how to safely offload synchronous FD read/writes to a worker thread while halting only the calling vCPU.

Understanding the Concurrency Architecture in QEMU

To grasp why MMIO calls freeze QEMU, it helps to review the thread architecture:

  • vCPU Threads: Execute guest code. When an MMIO access occurs, the vCPU exits the guest context into QEMU core code while holding the Big QEMU Lock (BQL).
  • Main Event Loop (iothread): Runs timers, watches file descriptors, and dispatches bottom halves (BHs). In older or single-threaded models, the main loop and vCPU share significant state protected by the BQL.
  • Worker Threads: Auxiliary threads created to offload blocking tasks (using facilities like qemu_thread_create or the thread pool API).

If your MMIO handler performs qemu_chr_fe_read_all() synchronously, the vCPU holds the BQL for the entire duration of the socket exchange, blocking every other subsystem.

The Core Strategy: Release BQL and Wait on a Condition Variable

Because hardware architectures dictate that a synchronous MMIO read must block until data is placed into the target register, the calling vCPU must stall. However, you do not want other vCPUs or the main event loop to stall alongside it.

The canonical pattern in QEMU involves:

  1. Packaging the request details (data buffer, target address, size).
  2. Dispatching the job to a worker thread (or using QEMU's thread pool API).
  3. Releasing the Big QEMU Lock (bql_unlock() / qemu_mutex_unlock_iothread()).
  4. Putting the vCPU thread to sleep using a condition variable (QemuCond) and dedicated mutex (QemuMutex).
  5. Re-acquiring the BQL once the worker notifies that the transaction is complete, returning the MMIO result.

Step-by-Step Implementation

Here is an architectural example demonstrating how to implement this safely:

1. Define the Async Work Context

#include "qemu/osdep.h"
#include "qemu/thread.h"
#include "exec/memattrs.h"

typedef struct FdMmioTask {
    CharFrontend *fe;
    uint64_t wdata;
    uint64_t rdata;
    int bytes;
    bool done;
    MemTxResult result;

    QemuMutex lock;
    QemuCond cond;
} FdMmioTask;

2. Implement the Worker Thread Function

The worker thread performs the blocking socket write/read independently from the BQL:

static void *fd_worker_thread(void *opaque)
{
    FdMmioTask *task = (FdMmioTask *)opaque;
    int ret;

    /* Perform the blocking operations */
    ret = qemu_chr_fe_write_all(task->fe, (uint8_t *)&task->wdata, task->bytes);
    if (ret < 0) {
        task->result = MEMTX_ERROR;
        goto finish;
    }

    ret = qemu_chr_fe_read_all(task->fe, (uint8_t *)&task->rdata, task->bytes);
    if (ret < 0) {
        task->result = MEMTX_ERROR;
        goto finish;
    }

    task->result = MEMTX_OK;

finish:
    qemu_mutex_lock(&task->lock);
    task->done = true;
    qemu_cond_signal(&task->cond);
    qemu_mutex_unlock(&task->lock);

    return NULL;
}

3. Handle the MMIO Operation by Dropping the BQL

Inside the MMIO handler, unlock the BQL so the main loop keeps running while the vCPU waits on the worker thread:

MemTxResult fd_mmio_rmw(hwaddr addr, bool is_write, int bytes,
                        uint64_t wdata, uint64_t *rdata,
                        CharFrontend *fe)
{
    if (!qemu_chr_fe_backend_open(fe)) {
        return MEMTX_ERROR;
    }

    FdMmioTask task = {
        .fe = fe,
        .wdata = wdata,
        .rdata = 0,
        .bytes = bytes,
        .done = false,
        .result = MEMTX_ERROR,
    };

    qemu_mutex_init(&task.lock);
    qemu_cond_init(&task.cond);

    QemuThread thread;
    qemu_thread_create(&thread, "fd-io-worker", fd_worker_thread, &task, QEMU_THREAD_JOINABLE);

    /*
     * Release the Big QEMU Lock (BQL) so the main event loop and other 
     * vCPUs continue processing events while this vCPU sleeps.
     */
    bql_unlock(); /* In older versions: qemu_mutex_unlock_iothread() */

    qemu_mutex_lock(&task.lock);
    while (!task.done) {
        qemu_cond_wait(&task.cond, &task.lock);
    }
    qemu_mutex_unlock(&task.lock);

    /* Re-acquire the BQL before re-entering QEMU internal state */
    bql_lock(); /* In older versions: qemu_mutex_lock_iothread() */

    qemu_thread_join(&thread);
    qemu_mutex_destroy(&task.lock);
    qemu_cond_destroy(&task.cond);

    if (task.result == MEMTX_OK && !is_write) {
        *rdata = task.rdata;
    }

    return task.result;
}

Key Considerations and Best Practices

  • Re-entrance and Chardev Safety: Ensure that the underlying CharFrontend backend is safe to access across arbitrary threads. If your backend relies on the main event loop context, avoid direct FD access on worker threads without proper locking or explicit non-BQL event loops.
  • Alternative: Split Transactions: If your guest architecture allows it, a split transaction is often more performant than holding the vCPU. Have the MMIO write queue a task that immediately returns a status code, and emit a hardware interrupt (IRQ) when the I/O completion read finishes.
  • Thread Pool Reuse: Creating a thread per access via qemu_thread_create incurs overhead. For production implementations, consider using QEMU's thread pool infrastructure (thread-pool.c) to dispatch worker jobs seamlessly.

Conclusion

Halting a specific vCPU without stalling the entire emulator comes down to unlocking the BQL before waiting on a condition variable. By yielding the lock, the main loop can handle events normally, ensuring high responsiveness while the guest core remains halted waiting for the worker thread to finish.