Chapter 24: vhost-blk And QEMU Exposure

Chapter Goal

This chapter explains how SPDK exposes a bdev to a VM through vhost-blk. By the end, the reader should know what the vhost-user socket represents, how QEMU virtio-blk requests are translated into SPDK bdev I/O, why controller and session teardown ordering matters, and how diskengine uses vhost controllers in baremetal mode.

The key idea is simple but easy to blur: vhost-blk is not a disk format and it is not an NVMe device. It is a virtio-blk back-end. QEMU presents the guest-visible virtio PCI device, while SPDK services the virtqueues and submits ordinary SPDK bdev I/O to whatever bdev graph sits underneath.

Beginner Mental Model

Virtio is the guest-visible device model. The guest kernel has a virtio-blk driver, so it sees a block disk such as /dev/vda. vhost is the acceleration model that lets the virtqueue back-end live outside the guest-facing device model. vhost-user moves that back-end into a separate userspace process and connects QEMU to it over a Unix domain socket.

In this setup:

• The guest sees a virtio-blk disk.
• QEMU owns guest emulation and the virtio PCI front-end.
• QEMU passes guest memory mappings, vring addresses, and event file descriptors to SPDK through vhost-user messages.
• SPDK owns the vhost-blk back-end.
• SPDK polls the virtqueues, translates descriptors into struct iovec arrays backed by guest memory, submits bdev I/O, and writes used-ring completions.

The vhost-user socket is therefore a control and setup endpoint, not the data pipe for every sector. The official QEMU vhost-user protocol describes the front-end as the application sharing virtqueues, normally QEMU, and the back-end as the process consuming them. It also states that the protocol uses Unix domain sockets and ancillary file descriptor passing. That is why QEMU must launch the VM with shareable memory: the actual read and write payload lives in guest memory that SPDK can map.

The guest does not know about SPDK bdev names, RAID bdevs, NVMe-oF, lvols, or storage nodes. It sees a block disk. That is the entire point.

Why This Matters For diskengine/excloud

In baremetal mode, diskengine builds a local bdev graph first:

remote lvol namespaces -> bdev_nvme bdevs -> RAID bdev -> optional QoS -> vhost-blk controller.

The VM attachment edge is:

• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/attach.go: startAttachLoop
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/attach.go: attach
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/attach.go: ensureVhost
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: VhostCreateBlkController
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: VhostGetControllers
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: VhostDeleteController

The teardown edge is:

• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/vhost_detach.go: startVhostDetachLoop
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/vhost_detach.go: detachVhost
• /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/raid_detach.go: finalizeVolumeDetach

The vhost controller name is part of the contract between diskengine and VM launch orchestration. diskengine names the controller from the VM mapping ID, for example vhost123. The SPDK vhost app turns that controller name into a socket path under its socket directory, while QEMU must point its -chardev socket,...,path=... at the same path. If QEMU points at the wrong socket or a stale socket, the guest disk will not appear even if RAID is healthy.

This is the attachment gate in diskengine. It refuses to create vhost exposure until the RAID bdev exists and is online:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/attach.go
func ensureVhost(client *spdkclient.Client, snap *spdkSnapshot, volumeID int64, volumeVMMappingID int64) error {
	vhostCtrlrName := fmt.Sprintf("vhost%d", volumeVMMappingID)
	raidName := fmt.Sprintf("raid_%d", volumeID)

	if !snap.raidOnline(raidName) {
		info := snap.raidInfo(raidName)
		if info == nil {
			logger.Warn.Printf("ensure vhost: raid %s not ready yet; deferring vhost create", raidName)
		} else {
			logger.Warn.Printf("ensure vhost: raid %s not ready yet (state=%s discovered=%d); deferring vhost create", raidName, info.State, info.NumBaseBdevsDiscovered)
		}
		return nil
	}

	_, found := snap.vhostController(vhostCtrlrName)
	if found {
		return nil
	}

	if !found {
		blkParams := spdkclient.VhostCreateBlkControllerParams{Ctrlr: vhostCtrlrName, BdevName: raidName}
		if config.Value.VHOST_CPUMASK != "" {
			mask := config.Value.VHOST_CPUMASK
			blkParams.Cpumask = &mask
		}
		if err := client.VhostCreateBlkController(blkParams); err != nil {

The important part is not the logging. It is the ordering. Creating a vhost-blk controller before the RAID bdev exists would expose an endpoint whose backing device cannot be opened. Waiting for RAID online status also makes retries cheap: the loop can defer without turning transient storage discovery into a permanent VM attach failure.

RPC And Controller Creation

The JSON-RPC method for the block controller is vhost_create_blk_controller. The request names two things: ctrlr, the vhost controller/socket identity, and dev_name, the backing SPDK bdev. Optional fields such as cpumask and transport tune placement and transport selection.

// lib/vhost/vhost_rpc.c
static const struct spdk_json_object_decoder rpc_vhost_create_blk_controller_decoders[] = {
	{"ctrlr", offsetof(struct rpc_vhost_blk_ctrlr, ctrlr), spdk_json_decode_string },
	{"dev_name", offsetof(struct rpc_vhost_blk_ctrlr, dev_name), spdk_json_decode_string },
	{"cpumask", offsetof(struct rpc_vhost_blk_ctrlr, cpumask), spdk_json_decode_string, true},
	{"transport", offsetof(struct rpc_vhost_blk_ctrlr, transport), spdk_json_decode_string, true},
};

static void
rpc_vhost_create_blk_controller(struct spdk_jsonrpc_request *request,
				const struct spdk_json_val *params)
{
	struct rpc_vhost_blk_ctrlr req = {0};
	int rc;

	if (spdk_json_decode_object_relaxed(params, rpc_vhost_create_blk_controller_decoders,
					    SPDK_COUNTOF(rpc_vhost_create_blk_controller_decoders),
					    &req)) {
		SPDK_DEBUGLOG(vhost_rpc, "spdk_json_decode_object failed\n");
		rc = -EINVAL;
		goto invalid;
	}

	rc = spdk_vhost_blk_construct(req.ctrlr, req.cpumask, req.dev_name, req.transport, params);
	if (rc < 0) {
		goto invalid;
	}

This is deliberately thin. The RPC layer validates and decodes the user request, then hands ownership decisions to the block controller constructor.

The constructor opens the backing bdev first. That matters because an SPDK vhost-blk controller is not useful unless it can hold an open descriptor to the bdev it will submit I/O to. After the bdev is open, SPDK derives virtio features from bdev capabilities: discard depends on UNMAP support, write-zeroes depends on WRITE_ZEROES support, and flush depends on FLUSH support.

// lib/vhost/vhost_blk.c
int
spdk_vhost_blk_construct(const char *name, const char *cpumask, const char *dev_name,
			 const char *transport, const struct spdk_json_val *params)
{
	struct spdk_vhost_blk_dev *bvdev = NULL;
	struct spdk_vhost_dev *vdev;
	struct spdk_bdev *bdev;
	const char *transport_name = VIRTIO_BLK_DEFAULT_TRANSPORT;
	int ret = 0;

	bvdev = calloc(1, sizeof(*bvdev));
	if (bvdev == NULL) {
		ret = -ENOMEM;
		goto out;
	}

	if (transport != NULL) {
		transport_name = transport;
	}

	bvdev->ops = virtio_blk_get_transport_ops(transport_name);
	if (!bvdev->ops) {
		ret = -EINVAL;
		SPDK_ERRLOG("Transport type '%s' unavailable.\n", transport_name);
		goto out;
	}

	ret = spdk_bdev_open_ext(dev_name, true, bdev_event_cb, bvdev, &bvdev->bdev_desc);
	if (ret != 0) {
		SPDK_ERRLOG("%s: could not open bdev '%s', error=%d\n",
			    name, dev_name, ret);
		goto out;
	}
	bdev = spdk_bdev_desc_get_bdev(bvdev->bdev_desc);

Later in the same constructor, the vhost device records the derived virtio features and registers the controller:

// lib/vhost/vhost_blk.c
	vdev = &bvdev->vdev;
	vdev->virtio_features = SPDK_VHOST_BLK_FEATURES_BASE;
	vdev->disabled_features = SPDK_VHOST_BLK_DISABLED_FEATURES;
	vdev->protocol_features = SPDK_VHOST_BLK_PROTOCOL_FEATURES;

	if (spdk_bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_UNMAP)) {
		vdev->virtio_features |= (1ULL << VIRTIO_BLK_F_DISCARD);
	}
	if (spdk_bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_WRITE_ZEROES)) {
		vdev->virtio_features |= (1ULL << VIRTIO_BLK_F_WRITE_ZEROES);
	}

	if (spdk_bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_FLUSH)) {
		vdev->virtio_features |= (1ULL << VIRTIO_BLK_F_FLUSH);
	}

	bvdev->bdev = bdev;
	bvdev->readonly = false;
	ret = vhost_dev_register(vdev, name, cpumask, params, &vhost_blk_device_backend,
				 &vhost_blk_user_device_backend, false);

That is the main ownership transition. spdk_vhost_blk_dev owns the bdev descriptor and embeds the generic spdk_vhost_dev. The generic vhost layer owns socket registration, session tracking, and queue dispatch, while the block-specific layer owns virtio-blk request semantics.

The vhost-user transport creates and starts the Unix socket endpoint:

// lib/vhost/rte_vhost_user.c
int
vhost_user_dev_start(struct spdk_vhost_dev *vdev)
{
	return vhost_register_unix_socket(vdev->path, vdev->name, vdev->virtio_features,
					  vdev->disabled_features,
					  vdev->protocol_features);
}

int
vhost_user_dev_create(struct spdk_vhost_dev *vdev, const char *name, struct spdk_cpuset *cpumask,
		      const struct spdk_vhost_user_dev_backend *user_backend, bool delay)
{
	int rc;
	struct spdk_vhost_user_dev *user_dev;

	rc = vhost_user_dev_init(vdev, name, cpumask, user_backend);
	if (rc != 0) {
		return rc;
	}

	if (delay == false) {
		rc = vhost_user_dev_start(vdev);
		if (rc != 0) {

The SPDK vhost documentation shows the corresponding QEMU side:

-m 1G -object memory-backend-file,id=mem0,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem0
-chardev socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.1
-device vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=2

share=on is not cosmetic. QEMU must be able to share VM memory with the back-end. The socket path must match the SPDK controller's socket path. The vhost-user-blk-pci device is the guest-visible virtio-blk device that drives the queue traffic.

Controller, Device, Session, And Queue Objects

The code uses three related object levels:

• spdk_vhost_dev is the controller-level object. It has a name, socket path, SPDK thread, feature bits, and backend callbacks.
• spdk_vhost_user_dev is the vhost-user transport wrapper. It tracks current sessions and protects them with a lock.
• spdk_vhost_session is a live QEMU connection. It owns negotiated features, the memory table, and the virtqueues for that connection.

// lib/vhost/vhost_internal.h
struct spdk_vhost_session {
	struct spdk_vhost_dev *vdev;

	/* rte_vhost connection ID. */
	int vid;

	/* Unique session ID. */
	uint64_t id;
	/* Unique session name. */
	char *name;

	bool started;
	bool starting;
	bool needs_restart;

	struct rte_vhost_memory *mem;

	int task_cnt;

	uint16_t max_queues;
	/* Maximum number of queues before restart, used with 'needs_restart' flag */
	uint16_t original_max_queues;

	uint64_t negotiated_features;

A controller can exist with no live session. That means vhost_get_controllers proving that a controller exists is not the same as proving a VM is actively using it. Conversely, a session means QEMU has connected and the negotiated memory and queue state may be active. That distinction is why teardown code should inspect sessions, not just socket files.

The QEMU vhost-user documentation also matters for multiqueue reasoning. It says multiple queue support is negotiated through VHOST_USER_PROTOCOL_F_MQ, queried with VHOST_USER_GET_QUEUE_NUM, and enabled with VHOST_USER_SET_VRING_ENABLE. On the SPDK side, the visible result is one session with multiple virtqueues to poll. The queue count chosen in QEMU must be something the back-end can support, and operationally it should be sized for the VM and host cores rather than blindly maximized.

Request Path

Once QEMU and SPDK have negotiated memory and virtqueue state, the fast path is mostly poller work. The guest virtio-blk driver places a request descriptor chain in a virtqueue. SPDK polls the virtqueue, converts the descriptor chain into iovs, validates the virtio-blk header and status byte, submits bdev I/O, then marks the descriptor used.

For split rings, process_vq gets available request indices and calls process_blk_task:

// lib/vhost/vhost_blk.c
static int
process_vq(struct spdk_vhost_blk_session *bvsession, struct spdk_vhost_virtqueue *vq)
{
	struct spdk_vhost_session *vsession = &bvsession->vsession;
	uint16_t reqs[SPDK_VHOST_VQ_MAX_SUBMISSIONS];
	uint16_t reqs_cnt, i;
	int resubmit_cnt = 0;

	resubmit_cnt = submit_inflight_desc(bvsession, vq);

	reqs_cnt = vhost_vq_avail_ring_get(vq, reqs, SPDK_COUNTOF(reqs));
	if (!reqs_cnt) {
		return resubmit_cnt;
	}

	for (i = 0; i < reqs_cnt; i++) {
		SPDK_DEBUGLOG(vhost_blk, "====== Starting processing request idx %"PRIu16"======\n",
			      reqs[i]);

		if (spdk_unlikely(reqs[i] >= vq->vring.size)) {
			SPDK_ERRLOG("%s: request idx '%"PRIu16"' exceeds virtqueue size (%"PRIu16").\n",
				    vsession->name, reqs[i], vq->vring.size);
			vhost_vq_used_ring_enqueue(vsession, vq, reqs[i], 0);
			continue;
		}

		rte_vhost_set_inflight_desc_split(vsession->vid, vq->vring_idx, reqs[i]);

		process_blk_task(vq, reqs[i]);
	}

The queue handler protects against bad indices before it ever tries to parse a request. It also records inflight state before processing. That is part of making reconnect and recovery less ambiguous: a descriptor that was visible to SPDK but not completed is not silently forgotten.

process_blk_task is the split-ring version of "turn one available descriptor chain into one block task":

// lib/vhost/vhost_blk.c
static void
process_blk_task(struct spdk_vhost_virtqueue *vq, uint16_t req_idx)
{
	struct spdk_vhost_user_blk_task *task;
	struct spdk_vhost_blk_task *blk_task;
	int rc;

	assert(vq->packed.packed_ring == false);

	task = &((struct spdk_vhost_user_blk_task *)vq->tasks)[req_idx];
	blk_task = &task->blk_task;
	if (spdk_unlikely(task->used)) {
		SPDK_ERRLOG("%s: request with idx '%"PRIu16"' is already pending.\n",
			    task->bvsession->vsession.name, req_idx);
		blk_task->used_len = 0;
		blk_task_enqueue(task);
		return;
	}

	blk_task_inc_task_cnt(task);

	blk_task_init(task);

	rc = blk_iovs_split_queue_setup(task->bvsession, vq, task->req_idx,
					blk_task->iovs, &blk_task->iovcnt, &blk_task->payload_size);

	if (rc) {
		SPDK_DEBUGLOG(vhost_blk, "Invalid request (req_idx = %"PRIu16").\n", task->req_idx);
		/* Only READ and WRITE are supported for now. */
		vhost_user_blk_request_finish(VIRTIO_BLK_S_UNSUPP, blk_task, NULL);
		return;
	}

Descriptor translation is where the socket mental model usually breaks down. SPDK is not receiving a byte stream containing the write payload. It is walking a descriptor chain that points into mapped guest memory. The split-ring setup helper asks the vhost layer for the descriptor, converts each descriptor to an iov, accumulates the total length, and detects malformed chains.

// lib/vhost/vhost_blk.c
static int
blk_iovs_split_queue_setup(struct spdk_vhost_blk_session *bvsession,
			   struct spdk_vhost_virtqueue *vq,
			   uint16_t req_idx, struct iovec *iovs, uint16_t *iovs_cnt, uint32_t *length)
{
	struct spdk_vhost_session *vsession = &bvsession->vsession;
	struct spdk_vhost_dev *vdev = vsession->vdev;
	struct vring_desc *desc, *desc_table;
	uint16_t out_cnt = 0, cnt = 0;
	uint32_t desc_table_size, len = 0;
	uint32_t desc_handled_cnt;
	int rc;

	rc = vhost_vq_get_desc(vsession, vq, req_idx, &desc, &desc_table, &desc_table_size);
	if (rc != 0) {
		SPDK_ERRLOG("%s: invalid descriptor at index %"PRIu16".\n", vdev->name, req_idx);
		return -1;
	}

	desc_handled_cnt = 0;
	while (1) {
		if (spdk_unlikely(cnt == *iovs_cnt)) {
			SPDK_DEBUGLOG(vhost_blk, "%s: max IOVs in request reached (req_idx = %"PRIu16").\n",
				      vsession->name, req_idx);
			return -1;
		}

		if (spdk_unlikely(vhost_vring_desc_to_iov(vsession, iovs, &cnt, desc))) {
			SPDK_DEBUGLOG(vhost_blk, "%s: invalid descriptor %" PRIu16" (req_idx = %"PRIu16").\n",
				      vsession->name, req_idx, cnt);
			return -1;
		}

Packed queues use a different ring layout and cannot use the split-ring request index as the stable task index. SPDK uses the packed descriptor's buffer ID to select the task object:

// lib/vhost/vhost_blk.c
static void
process_packed_blk_task(struct spdk_vhost_virtqueue *vq, uint16_t req_idx)
{
	struct spdk_vhost_user_blk_task *task;
	struct spdk_vhost_blk_task *blk_task;
	uint16_t task_idx = req_idx, num_descs;
	int rc;

	assert(vq->packed.packed_ring);

	/* Packed ring used the buffer_id as the task_idx to get task struct. */
	task_idx = vhost_vring_packed_desc_get_buffer_id(vq, req_idx, &num_descs);

	task = &((struct spdk_vhost_user_blk_task *)vq->tasks)[task_idx];
	blk_task = &task->blk_task;
	if (spdk_unlikely(task->used)) {
		SPDK_ERRLOG("%s: request with idx '%"PRIu16"' is already pending.\n",
			    task->bvsession->vsession.name, task_idx);
		blk_task->used_len = 0;
		blk_task_enqueue(task);
		return;
	}

	task->req_idx = req_idx;
	task->num_descs = num_descs;
	task->buffer_id = task_idx;

This is why debugging only split-ring code can be misleading on modern guests. The request semantics are still virtio-blk, but the ring bookkeeping is different.

After descriptor setup, virtio_blk_process_request decodes the request. The first descriptor must contain struct virtio_blk_outhdr; the last descriptor must be a one-byte status buffer. The middle descriptors are payload. Reads write data into guest memory and therefore return payload_len + status as used length. Writes consume guest memory and only write the status byte.

// lib/vhost/vhost_blk.c
virtio_blk_process_request(struct spdk_vhost_dev *vdev, struct spdk_io_channel *ch,
			   struct spdk_vhost_blk_task *task, virtio_blk_request_cb cb, void *cb_arg)
{
	struct spdk_vhost_blk_dev *bvdev = to_blk_dev(vdev);
	struct virtio_blk_outhdr req;
	struct virtio_blk_discard_write_zeroes *desc;
	struct iovec *iov;
	uint32_t type;
	uint64_t flush_bytes;
	uint32_t payload_len;
	uint16_t iovcnt;
	int rc;

	assert(bvdev != NULL);

	task->cb = cb;
	task->cb_arg = cb_arg;

	iov = &task->iovs[0];
	if (spdk_unlikely(iov->iov_len != sizeof(req))) {
		SPDK_DEBUGLOG(vhost_blk,
			      "First descriptor size is %zu but expected %zu (task = %p).\n",
			      iov->iov_len, sizeof(req), task);
		blk_request_finish(VIRTIO_BLK_S_UNSUPP, task);
		return -1;
	}

The read/write branch is the core translation point from virtio-blk into SPDK bdev I/O:

// lib/vhost/vhost_blk.c
	case VIRTIO_BLK_T_IN:
	case VIRTIO_BLK_T_OUT:
		if (spdk_unlikely(payload_len == 0 || (payload_len & (512 - 1)) != 0)) {
			SPDK_ERRLOG("%s - passed IO buffer is not multiple of 512b (task = %p).\n",
				    type ? "WRITE" : "READ", task);
			blk_request_finish(VIRTIO_BLK_S_UNSUPP, task);
			return -1;
		}

		if (type == VIRTIO_BLK_T_IN) {
			task->used_len = payload_len + sizeof(*task->status);
			rc = spdk_bdev_readv(bvdev->bdev_desc, ch,
					     &task->iovs[1], iovcnt, req.sector * 512,
					     payload_len, blk_request_complete_cb, task);
		} else if (!bvdev->readonly) {
			task->used_len = sizeof(*task->status);
			rc = spdk_bdev_writev(bvdev->bdev_desc, ch,
					      &task->iovs[1], iovcnt, req.sector * 512,
					      payload_len, blk_request_complete_cb, task);
		} else {
			SPDK_DEBUGLOG(vhost_blk, "Device is in read-only mode!\n");
			rc = -1;
		}

The same function also handles DISCARD, WRITE_ZEROES, FLUSH, and GET_ID. Those become bdev unmap, write-zeroes, flush, and metadata-style responses when supported. Unsupported or malformed requests complete with an error or unsupported virtio status rather than becoming undefined bdev operations.

If bdev submission returns -ENOMEM, SPDK does not immediately fail the guest request. It queues an I/O wait and resubmits when the bdev layer reports resources are available:

// lib/vhost/vhost_blk.c
static void
blk_request_resubmit(void *arg)
{
	struct spdk_vhost_blk_task *task = arg;
	int rc = 0;

	rc = virtio_blk_process_request(task->bdev_io_wait_vdev, task->bdev_io_wait_ch, task,
					task->cb, task->cb_arg);
	if (rc == 0) {
		SPDK_DEBUGLOG(vhost_blk, "====== Task %p resubmitted ======\n", task);
	} else {
		SPDK_DEBUGLOG(vhost_blk, "====== Task %p failed ======\n", task);
	}
}

static inline void
blk_request_queue_io(struct spdk_vhost_dev *vdev, struct spdk_io_channel *ch,
		     struct spdk_vhost_blk_task *task)
{
	int rc;
	struct spdk_bdev *bdev = vhost_blk_get_bdev(vdev);

	task->bdev_io_wait.bdev = bdev;
	task->bdev_io_wait.cb_fn = blk_request_resubmit;
	task->bdev_io_wait.cb_arg = task;

That behavior is important during memory pressure. From the guest's point of view, the request is outstanding for longer. From SPDK's point of view, the task remains owned by the vhost queue path until it can submit or fail decisively.

Completion returns from the bdev layer to the vhost-blk task. The completion callback frees the bdev I/O object and writes a virtio status:

// lib/vhost/vhost_blk.c
static void
blk_request_complete_cb(struct spdk_bdev_io *bdev_io, bool success, void *cb_arg)
{
	struct spdk_vhost_blk_task *task = cb_arg;

	spdk_bdev_free_io(bdev_io);
	blk_request_finish(success ? VIRTIO_BLK_S_OK : VIRTIO_BLK_S_IOERR, task);
}

static void
blk_request_finish(uint8_t status, struct spdk_vhost_blk_task *task)
{

	if (task->status) {
		*task->status = status;
	}

	task->cb(status, task, task->cb_arg);
}

For the vhost-user transport, the task callback enqueues the task back to the queue completion path:

// lib/vhost/vhost_blk.c
static void
vhost_user_blk_request_finish(uint8_t status, struct spdk_vhost_blk_task *task, void *cb_arg)
{
	struct spdk_vhost_user_blk_task *user_task;

	user_task = SPDK_CONTAINEROF(task, struct spdk_vhost_user_blk_task, blk_task);

	blk_task_enqueue(user_task);

	SPDK_DEBUGLOG(vhost_blk, "Finished task (%p) req_idx=%d\n status: %" PRIu8"\n",
		      user_task, user_task->req_idx, status);
	blk_task_finish(user_task);
}

The used-ring update is the guest-visible completion. For split rings, SPDK writes the used element, orders memory with a barrier, updates used->idx, clears inflight tracking, and signals if interrupt mode requires it:

// lib/vhost/rte_vhost_user.c
void
vhost_vq_used_ring_enqueue(struct spdk_vhost_session *vsession,
			   struct spdk_vhost_virtqueue *virtqueue,
			   uint16_t id, uint32_t len)
{
	struct rte_vhost_vring *vring = &virtqueue->vring;
	struct vring_used *used = vring->used;
	uint16_t last_idx = virtqueue->last_used_idx & (vring->size - 1);
	uint16_t vq_idx = virtqueue->vring_idx;

	vhost_log_req_desc(vsession, virtqueue, id);

	virtqueue->last_used_idx++;
	used->ring[last_idx].id = id;
	used->ring[last_idx].len = len;

	/* Ensure the used ring is updated before we log it or increment used->idx. */
	spdk_smp_wmb();

	rte_vhost_set_last_inflight_io_split(vsession->vid, vq_idx, id);

	vhost_log_used_vring_elem(vsession, virtqueue, last_idx);
	* (volatile uint16_t *) &used->idx = virtqueue->last_used_idx;
	vhost_log_used_vring_idx(vsession, virtqueue);

A completed guest write is therefore not "done" when QEMU sends a message. It is done after the bdev graph completes, the status byte is written to guest memory, and the used ring makes that completion visible to the guest driver.

Prose Diagram: Guest Write Through vhost-blk

Think of guest memory as a shared region beside QEMU and SPDK. QEMU sets up access to that region through vhost-user. SPDK then uses the vring metadata and memory table to find the buffers directly.

Teardown Ordering

vhost teardown is a common source of data-plane bugs. A controller with an active VM session is not just a config object. The guest may still have outstanding writes, SPDK may still have inflight descriptors, and a bdev I/O may still be waiting for resources. Deleting the RAID or detaching NVMe controllers under an active vhost can turn a clean detach into guest-visible I/O errors or retry loops.

The SPDK RPC delete path first finds the controller, then asks the vhost layer to remove it. If removal returns -EBUSY, the RPC schedules itself to retry on the SPDK thread instead of pretending the controller is gone:

// lib/vhost/vhost_rpc.c
	rc = spdk_vhost_dev_remove(vdev);
	if (rc < 0) {
		if (rc == -EBUSY) {
			struct vhost_delete_ctrlr_context *ctx;

			ctx = calloc(1, sizeof(*ctx));
			if (ctx == NULL) {
				SPDK_ERRLOG("Failed to allocate memory for vhost_delete_ctrlr context\n");
				rc = -ENOMEM;
				goto invalid;
			}
			ctx->request = request;
			ctx->params = params;
			spdk_thread_send_msg(spdk_get_thread(), _rpc_vhost_delete_controller, ctx);
			free_rpc_delete_vhost_ctrlr(&req);
			return;
		}
		goto invalid;
	}

A QEMU disconnect destroys the vhost session and unregisters the memory table:

// lib/vhost/rte_vhost_user.c
static void
destroy_connection(int vid)
{
	struct spdk_vhost_session *vsession;
	struct spdk_vhost_user_dev *user_dev;

	vsession = vhost_session_find_by_vid(vid);
	if (vsession == NULL) {
		SPDK_ERRLOG("Couldn't find session with vid %d.\n", vid);
		return;
	}
	user_dev = to_user_dev(vsession->vdev);

	pthread_mutex_lock(&user_dev->lock);
	if (vsession->started || vsession->starting) {
		if (_stop_session(vsession) != 0) {
			pthread_mutex_unlock(&user_dev->lock);
			return;
		}
	}

	if (vsession->mem) {
		vhost_session_mem_unregister(vsession->mem);
		free(vsession->mem);
	}

diskengine adds its own gates above SPDK. Its vhost detach loop checks active sessions and defers when QEMU is still connected:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/vhost_detach.go
func detachVhost(ctx context.Context, client *spdkclient.Client, ctrlSet map[string]spdkclient.VhostController, volumeVMMappingID, volumeID int64) error {
	vhostName := fmt.Sprintf("vhost%d", volumeVMMappingID)

	ctrl, present := ctrlSet[vhostName]
	if !present {
		return nil // already gone, idempotent
	}

	if len(ctrl.Sessions) > 0 {
		logger.Warn.Printf("vhost detach: %s has %d active session(s); deferring", vhostName, len(ctrl.Sessions))
		return nil
	}

	hasOther, err := repository.HasOtherNonDetachedVMMappingsOnBaremetal(ctx, volumeID, config.Value.BAREMETAL_ID, volumeVMMappingID)
	if err != nil {
		return fmt.Errorf("check other non-detached mappings for vol %d: %w", volumeID, err)
	}

The RAID finalization loop also refuses to delete lower storage while any mapping's vhost controller is still present:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/raid_detach.go
	ctrlSet := map[string]struct{}{}
	for _, c := range allControllers {
		ctrlSet[c.Ctrlr] = struct{}{}
	}
	for _, id := range mappingIDs {
		name := fmt.Sprintf("vhost%d", id)
		if _, ok := ctrlSet[name]; ok {
			logger.Warn.Printf("raid detach: vol %d spdk gate failed (vhost %s still present); deferring", volumeID, name)
			return nil
		}
	}

	// Delete RAID bdev
	raidName := fmt.Sprintf("raid_%d", volumeID)
	if er := client.BdevRaidDelete(spdkclient.BdevRaidDeleteParams{Name: raidName}); er != nil {
		if !isSPDKNotFoundErr(er) {
			// Real error: do NOT transition to DETACHED. Retry next tick.
			return fmt.Errorf("delete raid %s: %w", raidName, er)
		}
	}

The strict safety invariant is: delete, or prove absent in vhost_get_controllers, every vhost controller for the volume before deleting the backing RAID bdev or detaching lower NVMe controllers. A controller with zero sessions is still a VM exposure object and QEMU can reconnect to it.

Current diskengine has a correctness hazard here. detachVhost checks that there are no active sessions, then deletes the RAID and NVMe controllers before calling VhostDeleteController. That can temporarily leave a sessionless vhost controller/socket pointing at removed lower storage if vhost delete is delayed or fails. finalizeVolumeDetach is stricter because it gates lower deletion on the relevant vhost controllers being absent. Treat the finalizeVolumeDetach ordering as the intended invariant, and treat the current detachVhost ordering as a risky path that should be fixed rather than copied.

Edge Cases And Failure Modes

Socket exists but controller is stale:

A Unix socket path can remain from an older run or a controller can exist without the VM currently being attached. Use vhost_get_controllers, not filesystem checks alone. Also inspect sessions; a controller with zero sessions is exposure state, not proof of guest use.

Guest still connected:

Deleting the controller may fail or be unsafe if a vhost session is active. SPDK tracks sessions inside spdk_vhost_user_dev, and diskengine explicitly defers vhost detach when len(ctrl.Sessions) > 0.

Descriptor is invalid:

Bad or unexpected virtqueue descriptors fail during descriptor-to-iov setup. Common failures include invalid descriptor index, too many iovs, cyclic descriptor chains, missing status byte, or payload lengths that are not sector-aligned.

Read-only device:

virtio_blk_process_request rejects writes when the vhost-blk controller is read-only. This is a virtio status failure, not a QEMU socket failure.

NOMEM:

vhost-blk can queue an I/O wait when bdev submission returns no memory. The request remains owned by the vhost task and is retried through blk_request_resubmit.

Packed versus split queues:

Modern virtio can use packed queues. Debugging only split-ring code can miss the active path. Read both process_blk_task and process_packed_blk_task, and remember that packed queues use buffer_id to find the task.

Feature mismatch:

Discard, write-zeroes, and flush are exposed only when the backing bdev supports the corresponding bdev I/O type. If the guest expects a feature that was not negotiated, SPDK should not receive that command as supported traffic.

Wrong QEMU memory setup:

If QEMU memory is not shareable, the back-end cannot safely map guest memory. In SPDK's documented QEMU launch, this is why memory-backend-file uses share=on.

Too many queues:

The official SPDK docs warn that adding too many queues can degrade vhost performance when many vhost devices are used, because each device requires additional queues to be polled. Queue count should match the VM and host placement plan.

Misconceptions To Kill

"vhost-blk is an NVMe device."

No. The guest sees virtio-blk. The backing bdev may eventually hit NVMe, RAID, or lvol, but that is hidden.

"The vhost socket contains the data."

No. The socket carries control messages and file descriptors. Bulk data is in shared guest memory referenced by descriptors.

"Deleting a vhost controller deletes the volume."

No. It removes a VM exposure endpoint. The underlying RAID, NVMe bdevs, and storage-node lvols are separate objects unless higher-level orchestration deletes them too.

"QEMU can use any bdev name directly."

QEMU uses a vhost-user socket. diskengine and SPDK map that socket/controller to a bdev.

"A controller means a VM is connected."

No. A controller can exist with zero sessions. Use controller presence, session presence, and the diskengine mapping state together.

Lab: Trace One Write Request

Open lib/vhost/vhost_blk.c and follow one request:

1. process_vq or process_packed_vq
2. process_blk_task or process_packed_blk_task
3. blk_iovs_split_queue_setup or blk_iovs_packed_queue_setup
4. vhost_user_process_blk_request
5. virtio_blk_process_request
6. the VIRTIO_BLK_T_OUT case
7. spdk_bdev_writev
8. blk_request_complete_cb
9. vhost_user_blk_request_finish
10. vhost_vq_used_ring_enqueue or vhost_vq_packed_ring_enqueue

For each step, write whether it is parsing guest descriptors, submitting SPDK I/O, handling backpressure, or completing guest-visible status.

Operational Debug Exercise

Symptom: VM boots but disk is missing.

Check:

1. Does diskengine think the VM mapping is ATTACHING or ATTACHED?
2. Does vhost_get_controllers show the expected vhost<volume_vm_mapping_id>?
3. Does that controller point at the expected RAID bdev?
4. Does vhost_get_controllers show an active session after QEMU starts?
5. Does QEMU reference the same socket path/name?
6. Did QEMU launch with shared memory, for example memory-backend-file and share=on?
7. Does bdev_raid_get_bdevs show the RAID online?
8. Are remote NVMe bdevs enabled underneath the RAID?

Do not start by debugging the SSD. A missing guest disk is often a vhost or QEMU socket wiring problem. A controller with no session points toward QEMU/socket launch. No controller points toward diskengine attach gating or SPDK RPC failure. A controller with a session but failing I/O points toward descriptor, feature, bdev, or lower storage behavior.

Source Reading Path

Read in this order:

1. doc/vhost.md for the operational setup and QEMU command-line shape.
2. lib/vhost/vhost_rpc.c: rpc_vhost_create_blk_controller, rpc_vhost_get_controllers, and rpc_vhost_delete_controller.
3. lib/vhost/vhost_blk.c: spdk_vhost_blk_construct.
4. lib/vhost/rte_vhost_user.c: vhost_user_dev_create, vhost_user_dev_start, new_connection, and destroy_connection.
5. lib/vhost/vhost_internal.h: struct spdk_vhost_dev, struct spdk_vhost_user_dev, and struct spdk_vhost_session.
6. lib/vhost/vhost_blk.c: process_vq, process_packed_vq, process_blk_task, and process_packed_blk_task.
7. lib/vhost/vhost_blk.c: blk_iovs_split_queue_setup, blk_iovs_packed_queue_setup, and virtio_blk_process_request.
8. lib/vhost/vhost_blk.c: blk_request_complete_cb, blk_request_queue_io, and vhost_user_blk_request_finish.
9. lib/vhost/rte_vhost_user.c: vhost_vq_used_ring_enqueue and vhost_vq_packed_ring_enqueue.
10. diskengine attach.go, vhost_detach.go, and raid_detach.go for orchestration around SPDK state.

Self-Check

1. What object does QEMU connect to?
2. What object does SPDK submit bdev I/O to?
3. Why can a vhost controller exist without proving a guest is currently using it?
4. Where does SPDK translate guest descriptors into iovs?
5. Why does share=on matter in the QEMU command line?
6. How does the split-ring request path differ from the packed-ring path?
7. Why should RAID deletion wait until vhost exposure and active sessions are gone?
8. What happens when bdev submission returns -ENOMEM?

References

• Local SPDK: doc/vhost.md
• Local SPDK: doc/virtio.md
• Local SPDK: lib/vhost/vhost_rpc.c
• Local SPDK: lib/vhost/vhost_blk.c
• Local SPDK: lib/vhost/rte_vhost_user.c
• Local SPDK: lib/vhost/vhost_internal.h
• Local SPDK: include/spdk/vhost.h
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/attach.go
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/vhost_detach.go
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/raid_detach.go
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/types.go
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go
• SPDK vhost documentation: https://spdk.io/doc/vhost.html
• QEMU vhost-user documentation: https://www.qemu.org/docs/master/interop/vhost-user.html