Chapter 31: Complete VM Write To SSD Walkthrough

Chapter Goal

This chapter follows one guest write from a VM down to a physical SSD and then follows completion back up. For current diskengine, the primary path is vhost-blk, RAID, NVMe-oF initiator, NVMe-oF target, lvol/blobstore, and physical NVMe. SPDK also supports vfio-user NVMe exposure, but the inspected diskengine baremetal path exposes VM volumes through vhost-blk.

The key idea: diskengine constructs and repairs the graph, but the write itself flows through SPDK data-plane code.

flowchart TB guest[VM guest block write] --> qemu[QEMU virtio-blk device] qemu --> export[vhost-blk endpoint] export --> raid[RAID bdev on baremetal] raid --> nvmeInit[bdev_nvme initiator] nvmeInit --> fabric[NVMe-oF fabric] fabric --> nvmf[NVMe-oF target on storage node] nvmf --> lvol[lvol bdev] lvol --> blob[blobstore cluster map] blob --> phys[physical NVMe bdev] phys --> ssd[NVMe SSD] ssd --> phys phys --> blob blob --> lvol lvol --> nvmf nvmf --> fabric fabric --> nvmeInit nvmeInit --> raid raid --> export export --> qemu qemu --> guest

VFIOUSER variant: SPDK's NVMf VFIOUSER transport can expose an NVMe controller through a vfio-user socket, as covered in Chapter 25. That is a different guest device model from virtio-blk/vhost-user. It is not the current diskengine VM volume exposure path reviewed here.

Scenario

Assume a VM writes 4 KiB to a disk backed by a replicated excloud volume:

The VM is running on a baremetal node.
QEMU uses a vhost-blk socket created by SPDK on that baremetal node.
The vhost-blk controller exposes raid_123.
raid_123 has base bdevs produced by bdev_nvme connections to storage-node NQNs.
Each storage-node NQN exports one lvol as an NVMe-oF namespace.
Each lvol is backed by an SPDK lvstore/blobstore on a local physical NVMe SSD.

Stage 0: Control Plane Already Ran

Before the write, these diskengine loops prepared the graph.

Storage node:

internal/storagenode/disk_init.go: initialiseDisk
internal/storagenode/provisionlvol.go: provisionLvol
internal/storagenode/nvmeofexport.go: reconcileExports

Baremetal:

internal/baremetal/nvme_attach.go: reconcileNVMeConnections
internal/baremetal/raidensure.go: ensureRaid
internal/baremetal/attach.go: ensureVhost

This distinction matters. If you are debugging one write, first decide whether the graph exists. If it does not, debug control plane. If it exists but I/O fails, debug data path.

Stage 1: Guest Submits virtio-blk Request

Inside the VM, the guest kernel submits a block write to its virtio-blk device. The guest fills virtqueue descriptors that point to guest memory containing the request header and payload. QEMU and SPDK have negotiated vhost-user memory mappings, so SPDK can translate those guest physical addresses.

SPDK anchors:

lib/vhost/vhost_blk.c: process_blk_task
lib/vhost/vhost_blk.c: process_packed_blk_task
lib/vhost/vhost_blk.c: blk_iovs_split_queue_setup
lib/vhost/vhost_blk.c: blk_iovs_packed_queue_setup
lib/vhost/vhost_internal.h: vhost_gpa_to_vva
lib/vhost/vhost_internal.h: vhost_vring_desc_to_iov

The output of this stage is a spdk_vhost_blk_task with iovs and request metadata.

Stage 2: vhost-blk Submits bdev Write To RAID

The request reaches:

lib/vhost/vhost_blk.c: vhost_user_process_blk_request
lib/vhost/vhost_blk.c: virtio_blk_process_request

For a write, virtio_blk_process_request submits bdev I/O to the backing bdev, which is raid_123 in this scenario. The completion callback is:

lib/vhost/vhost_blk.c: blk_request_complete_cb

Nothing has reached the SSD yet. At this moment SPDK has created an asynchronous bdev write against a RAID bdev.

Stage 3: RAID Maps The Write To Base bdevs

The RAID module receives the bdev write through its module submit path:

module/bdev/raid/bdev_raid.c: raid_bdev_submit_request
module/bdev/raid/raid1.c

For RAID1, the write must be propagated to mirror bases according to the module logic. Each base bdev is an NVMe bdev created by bdev_nvme_attach_controller.

Useful RAID RPC/debug anchors:

module/bdev/raid/bdev_raid_rpc.c: rpc_bdev_raid_get_bdevs
module/bdev/raid/bdev_raid_rpc.c: rpc_bdev_raid_add_base_bdev
module/bdev/raid/bdev_raid_rpc.c: rpc_bdev_raid_remove_base_bdev

If RAID is rebuilding or degraded, behavior depends on current base state. This is why diskengine's health loop watches RAID processes:

internal/baremetal/baremetal_health.go: runHealthIteration

Stage 4: `bdev_nvme` Converts Base Writes To NVMe Commands

Each RAID base bdev receives a write:

module/bdev/nvme/bdev_nvme.c: bdev_nvme_submit_request
module/bdev/nvme/bdev_nvme.c: _bdev_nvme_submit_request
module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev

The bdev module submits NVMe namespace write commands through the NVMe library:

lib/nvme/nvme_ns_cmd.c: spdk_nvme_ns_cmd_writev
lib/nvme/nvme_ns_cmd.c: spdk_nvme_ns_cmd_write_ext
lib/nvme/nvme_qpair.c
lib/nvme/nvme_rdma.c

Because this is a remote storage-node export, the NVMe transport is RDMA in the typical diskengine path. The write is now an NVMe-oF command traveling from baremetal to storage node.

Stage 5: Storage Node NVMf Target Receives The Write

On the storage node, the RDMA transport receives the request and calls the common NVMf execution path:

lib/nvmf/rdma.c: spdk_nvmf_request_exec call sites
lib/nvmf/ctrlr.c: spdk_nvmf_request_exec
lib/nvmf/ctrlr.c: nvmf_ctrlr_process_io_cmd
lib/nvmf/ctrlr.c: spdk_nvmf_request_get_bdev
lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_write_cmd

spdk_nvmf_request_get_bdev resolves the namespace ID to the lvol bdev attached to that subsystem. This is where the NQN/namespace created by storage-node control plane becomes a real bdev operation.

Stage 6: lvol/blobstore Writes To Physical NVMe bdev

The lvol bdev maps guest-visible logical blocks to blobstore clusters. Source anchors for deeper reading:

module/bdev/lvol/vbdev_lvol.c
module/bdev/lvol/vbdev_lvol_rpc.c
lib/blob/blobstore.c
module/blob/bdev/blob_bdev.c

The lvol ultimately submits I/O to its base bdev, which is the physical NVMe namespace attached on the storage node:

module/bdev/nvme/bdev_nvme.c: bdev_nvme_submit_request
module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev
lib/nvme/nvme_pcie.c
lib/nvme/nvme_qpair.c

At the physical device boundary, the write is an NVMe command on a PCIe qpair. The SSD controller writes data into NAND through its internal flash translation layer. SPDK does not manage NAND pages directly.

Stage 7: Completion Returns Upward

Completion reverses the path:

Physical SSD completes NVMe command.
Storage-node NVMe driver polls completion.
lvol/blobstore completes its bdev I/O.
NVMf target completes request with spdk_nvmf_request_complete.
RDMA response reaches baremetal initiator.
Baremetal bdev_nvme_writev_done completes base bdev write.
RAID completes when required base writes finish.
vhost blk_request_complete_cb sets virtio status.
vhost_user_blk_request_finish updates the used ring.
Guest sees the virtio-blk completion.

SPDK completion anchors:

module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev_done
lib/nvmf/ctrlr.c: spdk_nvmf_request_complete
lib/nvmf/transport.c: nvmf_transport_req_complete
lib/vhost/vhost_blk.c: blk_request_complete_cb
lib/vhost/vhost_blk.c: vhost_user_blk_request_finish

Prose Diagram: Complete Write Path

Draw two machines: baremetal node on the left, storage node on the right.

Baremetal stack, top to bottom:

VM guest filesystem -> guest virtio-blk driver -> QEMU -> SPDK vhost-blk -> raid_123 -> bdev_nvme remote base bdevs -> RDMA NIC.

Storage-node stack:

RDMA NIC -> SPDK NVMf target -> subsystem namespace -> lvol bdev -> blobstore/lvstore -> physical NVMe bdev -> PCIe SSD.

Draw the write arrow left-to-right across RDMA between the bdev_nvme layer and NVMf target. Draw completion right-to-left all the way back to the guest. Use dashed boxes around diskengine loops above both machines to show they create the graph but are not on the per-I/O arrow.

Debugging By Layer

Guest layer:

Is the disk visible?
Are writes hanging or failing with I/O errors?

vhost layer:

vhost_get_controllers
thread_get_pollers
Source: lib/vhost/vhost_blk.c

RAID layer:

bdev_raid_get_bdevs
base status, rebuild process, online/configuring/offline.
Source: module/bdev/raid/bdev_raid_rpc.c.

Baremetal NVMe initiator:

bdev_nvme_get_controllers
bdev_nvme_get_io_paths
Source: module/bdev/nvme/bdev_nvme_rpc.c.

Storage-node NVMf target:

nvmf_get_subsystems
nvmf_get_transports
Source: lib/nvmf/nvmf_rpc.c.

Storage-node bdev/lvol:

bdev_get_bdevs
bdev_lvol_get_lvstores
Source: module/bdev/lvol/vbdev_lvol_rpc.c.

Physical NVMe:

bdev_nvme_get_controller_health_info
SMART/temperature/media errors.

Edge Cases And Misleading Symptoms

Guest hang can be vhost completion, not SSD failure:

If bdev I/O completes but the used ring is not updated or interrupt is missed, the guest waits even though lower storage is fine.

RAID online can hide one bad replica:

A degraded RAID can still serve I/O. Check rebuild and base status, not just presence.

NVMe controller enabled can still have path issues:

Use I/O path RPCs and stats, not just controller list.

Storage-node lvol exists but is not exported:

Baremetal attach fails at NVMf connect/discovery even though storage capacity exists.

Physical SSD healthy does not prove export healthy:

Network, target subsystem, namespace mapping, and bdev graph can fail above the SSD.

Control-plane race can look like data-plane failure:

If the VM starts before vhost or RAID is ready, the symptom may be a missing or failed disk. Check diskengine state transitions.

Misconceptions To Kill

"The write goes through diskengine Go code."

No. diskengine created the objects. The write flows through SPDK.

"There is one queue."

No. There are guest virtqueues, RAID bdev queues, NVMe initiator qpairs, NVMf target qpairs, lvol/blobstore work, and physical NVMe queues.

"Completion means durable on NAND."

Completion means the storage stack and device reported command completion according to their semantics. Durability depends on flush/FUA, volatile caches, SSD power-loss protection, and the protocol command used.

"A single RPC can diagnose the whole path."

No. Use layer-specific checks.

Lab: Build A Trace Checklist

For a volume 123, write a checklist with expected object names:

vhost controller: vhost<volume_vm_mapping_id>
RAID bdev: raid_123
base bdevs: derived from NQNs using baseBdevNameFromNQN
storage-node NQNs: from volume_lvol_mapping
namespace bdev names: lvol UUIDs
physical controller names: storage-node NvmeDisk<disk_id>

Then map each object to one SPDK RPC that can prove it exists.

Source Reading Exercise

Read in this order, stopping at the first async submission/completion pair in each layer:

lib/vhost/vhost_blk.c: virtio_blk_process_request
module/bdev/raid/bdev_raid.c: raid_bdev_submit_request
module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev
lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_write_cmd
module/bdev/lvol/vbdev_lvol.c

For each layer, identify:

input object,
output object,
async completion callback.

Operational Debug Exercise

Symptom: write latency spikes every few minutes.

Investigate:

Guest/vhost queue depth and session state.
RAID rebuild activity from bdev_raid_get_bdevs.
NVMe-oF path reconnects or disabled paths.
Storage-node disk health and temperature.
lvol/blobstore free space and snapshots/clones.
CPU reactor saturation and poller stats.
Network RDMA counters outside SPDK if available.

Tie each observation to a layer in the diagram. Avoid jumping straight from "VM slow" to "SSD bad."

Self-Check

Which layer translates guest descriptors into iovs?
Which layer turns a RAID base write into an NVMe command?
Which layer maps NVMf namespace ID to storage-node lvol bdev?
Why is diskengine not in the write hot path?
Name three places where completion can be delayed after the physical SSD has accepted the command.

References

Local SPDK: lib/vhost/vhost_blk.c
Local SPDK: module/bdev/raid/bdev_raid.c
Local SPDK: module/bdev/nvme/bdev_nvme.c
Local SPDK: lib/nvmf/ctrlr.c
Local SPDK: lib/nvmf/ctrlr_bdev.c
Local SPDK: module/bdev/lvol/vbdev_lvol.c
Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode
Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal
SPDK bdev documentation: https://spdk.io/doc/bdev.html
SPDK NVMe-oF documentation: https://spdk.io/doc/nvmf.html

Source Walkthrough: The vhost Request Becomes A bdev I/O

The vhost layer is where a guest block request becomes an SPDK bdev request. The guest gives SPDK descriptors. SPDK validates them, interprets the virtio block header, then submits read/write/flush/unmap to the backing bdev.

From lib/vhost/vhost_blk.c:

virtio_blk_process_request(struct spdk_vhost_dev *vdev, struct spdk_io_channel *ch,
			   struct spdk_vhost_blk_task *task, virtio_blk_request_cb cb, void *cb_arg)
{
	struct spdk_vhost_blk_dev *bvdev = to_blk_dev(vdev);
	struct virtio_blk_outhdr req;
	struct iovec *iov;
	uint32_t type;
	uint64_t flush_bytes;
	uint32_t payload_len;
	uint16_t iovcnt;
	int rc;

vdev is the guest-facing controller. ch is the SPDK I/O channel bound to the thread handling this queue. task is the per-request object that carries guest iovs, status byte, callback, and retry state. bvdev is the vhost block device object; it points to the backing bdev descriptor.

iov = &task->iovs[0];
if (spdk_unlikely(iov->iov_len != sizeof(req))) {
	blk_request_finish(VIRTIO_BLK_S_UNSUPP, task);
	return -1;
}

memcpy(&req, iov->iov_base, sizeof(req));

The first descriptor must contain the virtio block header. SPDK copies it to the stack because some firmware does not align it the way C sanitizers expect. This is a small but important lesson: high-performance code still contains defensive compatibility details.

payload_len = task->payload_size;
task->status = iov->iov_base;
payload_len -= sizeof(req) + sizeof(*task->status);
iovcnt = task->iovcnt - 2;
type = req.type;

The task payload includes the header, data buffers, and final one-byte virtio status. SPDK subtracts the metadata and treats the middle descriptors as the data vector. This is why a guest write can stay scatter/gather instead of being copied into one flat buffer.

For a normal write:

rc = spdk_bdev_writev(bvdev->bdev_desc, ch,
		      &task->iovs[1], iovcnt, req.sector * 512,
		      payload_len, blk_request_complete_cb, task);

This is the main handoff. bvdev->bdev_desc identifies the opened backing bdev, which may be a RAID bdev in diskengine baremetal mode. ch chooses the per-thread channel. &task->iovs[1] skips the virtio header and points at guest data buffers. req.sector * 512 converts the virtio sector number into byte offset. payload_len is the write length. blk_request_complete_cb is the function that will eventually finish the guest request.

The write has not completed when spdk_bdev_writev returns zero. A zero return means accepted for asynchronous execution. Completion comes later.

Edge Case: Queue Full And `-ENOMEM`

The vhost code handles a temporary no-memory result differently from a hard I/O error:

if (rc) {
	if (rc == -ENOMEM) {
		SPDK_DEBUGLOG(vhost_blk, "No memory, start to queue io.\n");
		blk_request_queue_io(vdev, ch, task);
	} else {
		blk_request_finish(VIRTIO_BLK_S_IOERR, task);
		return -1;
	}
}

-ENOMEM here does not automatically mean the host is out of RAM. It can mean SPDK cannot allocate a bdev I/O object or buffer right now. The request is queued for resubmission. Other errors become guest-visible I/O errors.

This is an important debugging branch. If a VM write hangs but does not fail, check whether the request is waiting on SPDK internal resources. The guest may see latency, not an error. If SPDK eventually gets resources and resubmits, the write can complete successfully.

Source Walkthrough: Completion Back To The Guest

The vhost completion callback is short:

blk_request_complete_cb(struct spdk_bdev_io *bdev_io, bool success, void *cb_arg)
{
	struct spdk_vhost_blk_task *task = cb_arg;

	spdk_bdev_free_io(bdev_io);
	blk_request_finish(success ? VIRTIO_BLK_S_OK : VIRTIO_BLK_S_IOERR, task);
}

bdev_io is the completed lower-layer SPDK I/O. The vhost layer frees it first. Then blk_request_finish writes the virtio status byte: success maps to VIRTIO_BLK_S_OK, failure maps to VIRTIO_BLK_S_IOERR. blk_request_finish then invokes the task callback. For vhost-user blk, that callback is vhost_user_blk_request_finish, which calls blk_task_enqueue; blk_task_enqueue publishes completion through vhost_vq_used_ring_enqueue for split rings or vhost_vq_packed_ring_enqueue for packed rings.

This is why the completion path has two separate questions:

Did lower bdev I/O complete?
Did vhost publish completion back to the guest?

If bdev stats show completion but the guest still waits, the likely bug is above the bdev layer: used ring update, interrupt/eventfd, QEMU interaction, or guest driver behavior.

Source Walkthrough: The NVMe-oF Target Validates A Write

On the storage node, the NVMf target receives an NVMe command and maps it to a bdev operation. From lib/nvmf/ctrlr_bdev.c:

nvmf_bdev_ctrlr_write_cmd(struct spdk_bdev *bdev, struct spdk_bdev_desc *desc,
			  struct spdk_io_channel *ch, struct spdk_nvmf_request *req)
{
	struct spdk_bdev_ext_io_opts opts = {
		.size = SPDK_SIZEOF(&opts, nvme_cdw13),
		.memory_domain = req->memory_domain,
		.memory_domain_ctx = req->memory_domain_ctx,
		.accel_sequence = req->accel_sequence,
	};
	uint64_t bdev_num_blocks = spdk_bdev_get_num_blocks(bdev);
	uint32_t block_size = spdk_bdev_desc_get_block_size(desc);
	struct spdk_nvme_cmd *cmd = &req->cmd->nvme_cmd;
	struct spdk_nvme_cpl *rsp = &req->rsp->nvme_cpl;
	uint64_t start_lba;
	uint64_t num_blocks;

bdev is the namespace backing device, which is usually an lvol bdev in the storage-node part of diskengine. desc is the opened descriptor. ch is the I/O channel on the NVMf poll group thread. req is the fabric request containing the NVMe command, data buffers, and completion response.

The options carry memory-domain and acceleration metadata. This matters for RDMA and zero-copy paths: the data buffer may live in a memory domain that the transport and bdev path understand.

nvmf_bdev_ctrlr_get_rw_params(cmd, &start_lba, &num_blocks);
nvmf_bdev_ctrlr_get_rw_ext_params(cmd, &opts, !req->dif_enabled);

The target decodes the NVMe command into bdev parameters. NVMe speaks in LBAs and command dwords. The bdev layer needs offset/length plus options.

if (spdk_unlikely(!nvmf_bdev_ctrlr_lba_in_range(bdev_num_blocks, start_lba, num_blocks))) {
	SPDK_ERRLOG("end of media\n");
	rsp->status.sct = SPDK_NVME_SCT_GENERIC;
	rsp->status.sc = SPDK_NVME_SC_LBA_OUT_OF_RANGE;
	return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
}

This is a critical edge case. A guest or initiator can submit an out-of-range write. The target does not pass it downward and hope the SSD catches it. It sets an NVMe completion status and completes the fabric request. That status travels back to the initiator, then eventually to the VM.

if (spdk_unlikely(num_blocks * block_size > req->length)) {
	SPDK_ERRLOG("Write NLB ... > SGL length ...\n");
	rsp->status.sct = SPDK_NVME_SCT_GENERIC;

The target also checks that the command's requested number of blocks fits the data payload length. This catches malformed or inconsistent requests. For debugging, this tells you to compare command LBAs, namespace block size, and SGL length instead of blaming the SSD.

Flush, FUA, And Durability

A normal write completion is not always the same as "data is physically safe in NAND forever". You need to separate three ideas:

Command completion: the stack reports the write command completed.
Volatile cache behavior: a controller may have write cache semantics.
Durability request: flush is the generic bdev durability primitive; FUA is protocol-specific, such as an NVMe write flag, and matters only when the path exposes and preserves that protocol flag.

In the vhost source, flush has a separate path from ordinary read/write. In NVMe, flush is a command, and FUA is a write-command flag rather than a portable bdev I/O type. In bdev, flush is a distinct I/O type. In diskengine, durability depends on every layer preserving the relevant semantic: guest, QEMU/vhost, RAID/lvol/blobstore, NVMe-oF, and physical SSD.

Edge cases:

A filesystem may issue writes and later flush. If the flush fails, earlier writes may already have completed from the guest's point of view.
RAID1 write completion may require the policy's required replicas. Read the RAID module for exact semantics.
An SSD without power-loss protection may acknowledge writes according to its advertised cache rules, not your hopes.
A transport disconnect after write submit but before completion can leave the initiator unsure. Recovery logic must use protocol status and upper-layer consistency, not guesses.

Discard, Unmap, And Write Zeroes

Not every guest operation carries data. Discard/unmap says ranges no longer need to be preserved. Write zeroes asks the backend to logically zero a range. These operations can be cheaper than writes, but only when every layer supports them.

When reading source, follow the operation type:

virtio request type
  -> vhost branch
  -> bdev I/O type
  -> virtual bdev support check
  -> lower bdev or NVMe command
  -> completion status

Edge cases:

A layer may split a large unmap into smaller operations.
A virtual bdev may reject an operation even if the physical device supports it.
Alignment can matter. A guest byte range may need to map to block boundaries.
A discard completion does not mean NAND was erased immediately; it means the logical contract completed.

Queueing Locations In One Write

A single write can queue in many places:

guest virtqueue
vhost task pool
bdev I/O object pool
RAID base-device fanout
bdev_nvme request pool
NVMe qpair submission queue
RDMA send/receive resources
NVMf target request buffers
lvol/blobstore metadata work
physical NVMe submission queue
SSD internal controller queues

When latency spikes, do not ask "where is the queue?" Ask "which queue's depth or wait time changed?" That points you to the layer.

Useful correlations:

vhost accepted request but bdev stats do not move: vhost-to-bdev issue.
RAID stats move but one base bdev does not: base path or degraded RAID issue.
baremetal NVMe initiator stats move but storage-node NVMf stats do not: fabric/network/connectivity issue.
storage-node lvol stats move but physical NVMe stats lag: blobstore/base-device issue.
physical NVMe completes but guest waits: completion publication issue.

Error Propagation Back To The VM

SPDK often stores rich internal status but the guest may see a small status code. For vhost-blk, many failures collapse to VIRTIO_BLK_S_IOERR. For NVMe, the initiator can receive status code type/status code pairs. For bdev, internal completion status may be SUCCESS, FAILED, NOMEM, or a protocol-specific status.

This compression is why source reading matters. If a guest sees I/O error, the cause may be:

out-of-range LBA at NVMf target,
base bdev removed,
no memory retry eventually failed,
RAID base write failed,
NVMe qpair reset,
lvol/blobstore metadata failure,
vhost request malformed,
readonly vhost bdev.

The guest error is the end of the story, not the diagnosis.

Diskengine Boundary: Control Plane Versus Data Plane

diskengine appears heavily in setup chapters, but a VM write does not call diskengine Go code for every I/O. diskengine's role is to make sure the graph exists:

storage-node loop creates lvol and NVMf export
baremetal loop attaches NVMf controller
baremetal loop creates RAID bdev
baremetal loop creates guest-facing vhost-blk device
then SPDK pollers move I/O

SPDK vfio-user NVMe can be substituted as a design variant, but that is not the current diskengine path. The current diskengine source creates vhost_create_blk_controller controllers for VM volumes and has no inspected path that creates NVMf VFIOUSER listeners or generic vfu endpoints for VM disk exposure.

This separation is powerful and dangerous. It is powerful because the hot path stays in userspace C pollers. It is dangerous because control-plane bugs can masquerade as data-plane bugs. If diskengine created the wrong graph, SPDK will faithfully move I/O through the wrong graph.

Source-reading habit:

If the object is missing, read diskengine reconciliation and SPDK RPC creation code.
If the object exists but I/O fails, read SPDK data path.
If the object exists but is stale/wrong, read both: diskengine desired-state logic and SPDK lifetime rules.

Mini Lab: Trace One 4 KiB Write On Paper

For one 4 KiB guest write at sector S, write down:

guest byte offset: S * 512.
vhost payload length: 4096.
backing bdev name: raid_<volume>.
RAID base bdev names.
NVMe-oF controller name for each base.
storage-node NQN.
namespace ID.
lvol bdev UUID/name.
physical NVMe bdev.
completion callback at each layer.

Then answer:

Which layer first validates payload alignment?
Which layer first validates namespace LBA range?
Which layer turns fabric request into bdev I/O?
Which layer publishes the final guest status byte?

If you can answer those without hand-waving, the black box is shrinking.