Chapter 25: vfio-user NVMe Exposure

Chapter Goal

This chapter explains vfio-user as a way to expose an emulated NVMe controller through a Unix socket, with guest-visible PCI/NVMe semantics and SPDK bdev-backed storage. The reader should understand how this differs from vhost-blk, how queue memory and doorbells enter the design, and where to look when a vfio-user endpoint wedges.

The important shift is that vfio-user is not a storage command set by itself. It is a userspace way to model a VFIO-like device. QEMU's vfio-user documentation describes the client as a way to implement PCI devices in userspace outside QEMU, and notes that unlike vhost-user it can emulate arbitrary PCI devices, not only virtio devices. In SPDK's NVMe-oF target, that arbitrary PCI device is shaped as an NVMe controller.

SPDK's implementation therefore has two layers at once:

• At the guest-facing side, it looks like PCI config space, BARs, NVMe registers, doorbells, SQs, CQs, MSI-X, DMA mappings, and reset/quiesce callbacks.
• At the storage-facing side, it becomes ordinary SPDK NVMf controller, qpair, request, and bdev execution.

That combination is what makes this chapter easy to misread. The transport is local and socket-backed, but the protocol surface is not a block socket protocol. It is closer to a userspace PCIe NVMe controller whose control plane is negotiated over a Unix socket and whose data path uses mapped guest memory.

Beginner Mental Model

vhost-blk gives a VM a virtio-blk device. vfio-user can give a VM something that behaves like a PCI device. In SPDK's NVMe-oF vfio-user transport, the exported PCI device presents NVMe controller semantics. The guest or client sees NVMe queues, doorbells, admin commands, I/O commands, completions, and interrupts.

The useful beginner comparison is:

• vhost-blk: virtio request -> SPDK vhost target -> SPDK bdev I/O.
• vfio-user NVMe: NVMe SQE/CQE and doorbells -> SPDK NVMf request -> SPDK bdev I/O.

The socket is still central, but not because every I/O payload is copied through it. The socket is used for vfio-user protocol messages such as version negotiation, device/region discovery, DMA map/unmap, region reads/writes when a region is not directly mappable, IRQ setup, reset, and migration-related control. When the client shares memory through mappable file descriptors, the server can map that memory and access queues and data buffers directly.

QEMU's vfio-user protocol documentation makes this distinction explicit: DMA map/unmap messages tell the server which client memory ranges are valid, and direct memory access is possible when the client provides file descriptors the server can mmap(). SPDK's target-side vfio-user transport is built around that mapped-memory model.

Where This Lives In SPDK

The target-side NVMe-over-vfio-user implementation is concentrated in lib/nvmf/vfio_user.c. It registers as an NVMf transport named muser, but its public transport name is VFIOUSER.

const struct spdk_nvmf_transport_ops spdk_nvmf_transport_vfio_user = {
	.name = "VFIOUSER",
	.type = SPDK_NVME_TRANSPORT_VFIOUSER,
	.opts_init = nvmf_vfio_user_opts_init,
	.create = nvmf_vfio_user_create,
	.destroy = nvmf_vfio_user_destroy,

	.listen = nvmf_vfio_user_listen,
	.stop_listen = nvmf_vfio_user_stop_listen,
	.cdata_init = nvmf_vfio_user_cdata_init,
	.listen_associate = nvmf_vfio_user_listen_associate,

	.listener_discover = nvmf_vfio_user_discover,

	.poll_group_create = nvmf_vfio_user_poll_group_create,
	.get_optimal_poll_group = nvmf_vfio_user_get_optimal_poll_group,
	.poll_group_destroy = nvmf_vfio_user_poll_group_destroy,
	.poll_group_add = nvmf_vfio_user_poll_group_add,
	.poll_group_remove = nvmf_vfio_user_poll_group_remove,
	.poll_group_poll = nvmf_vfio_user_poll_group_poll,

	.req_free = nvmf_vfio_user_req_free,
	.req_complete = nvmf_vfio_user_req_complete,
};

SPDK_NVMF_TRANSPORT_REGISTER(muser, &spdk_nvmf_transport_vfio_user);

This is the bridge between "PCI device on a socket" and "SPDK NVMf target." The NVMf core calls transport operations such as listen, poll_group_add, poll_group_poll, and req_complete. The vfio-user implementation fills those hooks by accepting a libvfio-user connection, mapping NVMe queues, polling SQ doorbells, and posting NVMe CQEs.

The common NVMf request execution still ends up in the same layer used by other transports:

• lib/nvmf/ctrlr.c: spdk_nvmf_request_exec
• lib/nvmf/ctrlr.c: nvmf_ctrlr_process_admin_cmd
• lib/nvmf/ctrlr.c: nvmf_ctrlr_process_io_cmd
• lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_write_cmd
• lib/nvmf/ctrlr.c: spdk_nvmf_request_complete

That reuse is the reason this implementation is a transport instead of a one-off NVMe emulator. The guest-visible mechanism is special; the storage command execution path is not.

Endpoint Setup: Socket, PCI Identity, BARs, DMA, Interrupts

An NVMf listener for VFIOUSER creates a vfio-user endpoint. In SPDK terms, the endpoint is the socket-backed PCI endpoint. In libvfio-user terms, it is the vfu_ctx_t plus callbacks and regions.

nvmf_vfio_user_listen() constructs the endpoint, creates a small file-backed BAR0 area for doorbells, maps it into SPDK, then creates the libvfio-user context at a socket path under the transport address.

endpoint = calloc(1, sizeof(*endpoint));
if (!endpoint) {
	return -ENOMEM;
}

pthread_mutex_init(&endpoint->lock, NULL);
endpoint->devmem_fd = -1;
memcpy(&endpoint->trid, trid, sizeof(endpoint->trid));
endpoint->transport = vu_transport;

ret = snprintf(path, PATH_MAX, "%s/bar0", endpoint_id(endpoint));
if (ret < 0 || ret >= PATH_MAX) {
	SPDK_ERRLOG("%s: error to get socket path: %s.\n", endpoint_id(endpoint), spdk_strerror(errno));
	ret = -1;
	goto out;
}

ret = open(path, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
if (ret == -1) {
	SPDK_ERRLOG("%s: failed to open device memory at %s: %s.\n",
		    endpoint_id(endpoint), path, spdk_strerror(errno));
	goto out;
}
unlink(path);
endpoint->devmem_fd = ret;
ret = ftruncate(endpoint->devmem_fd,
		NVME_DOORBELLS_OFFSET + NVMF_VFIO_USER_DOORBELLS_SIZE);
if (ret != 0) {
	SPDK_ERRLOG("%s: error to ftruncate file %s: %s.\n", endpoint_id(endpoint), path,
		    spdk_strerror(errno));
	goto out;
}
endpoint->bar0 = mmap(NULL, NVME_REG_BAR0_SIZE, PROT_READ | PROT_WRITE,
		      MAP_SHARED, endpoint->devmem_fd, 0);
if (endpoint->bar0 == MAP_FAILED) {
	SPDK_ERRLOG("%s: error to mmap file %s: %s.\n", endpoint_id(endpoint), path, spdk_strerror(errno));
	endpoint->bar0 = NULL;
	ret = -1;
	goto out;
}
endpoint->bar0_doorbells = (uint32_t *)(((unsigned long)endpoint->bar0) + NVME_DOORBELLS_OFFSET);

ret = snprintf(uuid, PATH_MAX, "%s/cntrl", endpoint_id(endpoint));
if (ret < 0 || ret >= PATH_MAX) {
	SPDK_ERRLOG("%s: error to get ctrlr file path: %s\n", endpoint_id(endpoint), spdk_strerror(errno));
	ret = -1;
	goto out;
}
endpoint->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, uuid, LIBVFIO_USER_FLAG_ATTACH_NB,
				   endpoint, VFU_DEV_TYPE_PCI);

The fixed NVME_DOORBELLS_OFFSET matters because NVMe BAR0 has controller registers first and doorbells starting at offset 0x1000. SPDK maps enough local memory to expose the doorbell area, then tells libvfio-user how BAR0 can be accessed.

After the context exists, vfio_user_dev_info_fill() describes the device to libvfio-user: PCI type, vendor/device IDs, NVMe class code, capabilities, BAR regions, DMA callbacks, reset/quiesce callbacks, and interrupt counts.

ret = vfu_pci_init(vfu_ctx, VFU_PCI_TYPE_EXPRESS, PCI_HEADER_TYPE_NORMAL, 0);
if (ret < 0) {
	SPDK_ERRLOG("vfu_ctx %p failed to initialize PCI\n", vfu_ctx);
	return ret;
}
vfu_pci_set_id(vfu_ctx, SPDK_PCI_VID_EXCLOUD, 0x0001, SPDK_PCI_VID_EXCLOUD, 0);
/*
 * 0x02, controller uses the NVM Express programming interface
 * 0x08, non-volatile memory controller
 * 0x01, mass storage controller
 */
vfu_pci_set_class(vfu_ctx, 0x01, 0x08, 0x02);

ret = vfu_setup_region(vfu_ctx, VFU_PCI_DEV_CFG_REGION_IDX, NVME_REG_CFG_SIZE,
		       access_pci_config, VFU_REGION_FLAG_RW, NULL, 0, -1, 0);
if (ret < 0) {
	SPDK_ERRLOG("vfu_ctx %p failed to setup cfg\n", vfu_ctx);
	return ret;
}

ret = vfu_setup_region(vfu_ctx, VFU_PCI_DEV_BAR0_REGION_IDX, NVME_REG_BAR0_SIZE,
		       access_bar0_fn, VFU_REGION_FLAG_RW | VFU_REGION_FLAG_MEM,
		       sparse_mmap, 1, endpoint->devmem_fd, 0);
if (ret < 0) {
	SPDK_ERRLOG("vfu_ctx %p failed to setup bar 0\n", vfu_ctx);
	return ret;
}

ret = vfu_setup_device_dma(vfu_ctx, memory_region_add_cb, memory_region_remove_cb);
if (ret < 0) {
	SPDK_ERRLOG("vfu_ctx %p failed to setup dma callback\n", vfu_ctx);
	return ret;
}

ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_MSIX_IRQ, NVMF_VFIO_USER_MSIX_NUM);
if (ret < 0) {
	SPDK_ERRLOG("vfu_ctx %p failed to setup MSIX\n", vfu_ctx);
	return ret;
}

vfu_setup_device_quiesce_cb(vfu_ctx, vfio_user_dev_quiesce_cb);

This excerpt shows why debugging vfio-user exposure is not only "is the NVMf subsystem configured?" A device can fail before any bdev I/O exists: PCI setup, BAR setup, sparse mmap setup, DMA callback setup, IRQ setup, or vfu_realize_ctx() can all fail. A guest that sees no PCI device, a guest that sees a PCI device but cannot enable it, and a guest that enables the device but hangs on I/O are different failure classes.

The Object Model: Request, SQ, CQ, Controller, Endpoint

The core vfio-user NVMe objects mirror the NVMe queue machine:

• nvmf_vfio_user_endpoint: one socket-backed PCI endpoint.
• nvmf_vfio_user_ctrlr: one active controller connection for that endpoint.
• nvmf_vfio_user_sq: one NVMe submission queue and its NVMf qpair.
• nvmf_vfio_user_cq: one NVMe completion queue.
• nvmf_vfio_user_req: one SPDK NVMf request plus vfio-user/NVMe state.
• nvmf_vfio_user_poll_group: SPDK poll group state plus vfio-user wake statistics.

The request object is deliberately a wrapper around struct spdk_nvmf_request. The embedded NVMf request is what lets the common NVMf controller path process a command after vfio-user has translated guest queue state into SPDK request state.

struct nvmf_vfio_user_req  {
	struct spdk_nvmf_request		req;
	struct spdk_nvme_cpl			rsp;
	struct spdk_nvme_cmd			cmd;

	enum nvmf_vfio_user_req_state		state;
	nvmf_vfio_user_req_cb_fn		cb_fn;
	void					*cb_arg;

	TAILQ_ENTRY(nvmf_vfio_user_req)		link;

	struct iovec				iov[NVMF_VFIO_USER_MAX_IOVECS];
	uint8_t					iovcnt;

	/* NVMF_VFIO_USER_MAX_IOVECS worth of dma_sg_t. */
	uint8_t					sg[];
};

The SQ and CQ store the queue mapping plus the queue indices that the backend owns. For an SQ, SPDK owns the head pointer and reads the guest-updated tail doorbell. For a CQ, SPDK owns the tail pointer and reads the guest-updated head doorbell. That is the same ownership split taught in the NVMe queue chapter, but here the queue memory is guest/client memory mapped into the SPDK process.

	uint32_t				qid;
	/* Number of entries in queue. */
	uint32_t				size;
	struct nvme_q_mapping			mapping;
	enum nvmf_vfio_user_sq_state		sq_state;

	uint32_t				head;
	volatile uint32_t			*dbl_tailp;

	/* Whether a shadow doorbell eventidx needs setting. */
	bool					need_rearm;

	/* multiple SQs can be mapped to the same CQ */
	uint16_t				cqid;

	uint32_t				qid;
	/* Number of entries in queue. */
	uint32_t				size;
	struct nvme_q_mapping			mapping;
	enum nvmf_vfio_user_cq_state		cq_state;

	uint32_t				tail;
	volatile uint32_t			*dbl_headp;

	bool					phase;

	uint16_t				iv;
	bool					ien;

The controller object ties all queues to one endpoint connection and records both possible doorbell locations: BAR0 doorbells and optional shadow doorbells.

struct nvmf_vfio_user_ctrlr {
	struct nvmf_vfio_user_endpoint		*endpoint;
	struct nvmf_vfio_user_transport		*transport;

	TAILQ_HEAD(, nvmf_vfio_user_sq)		connected_sqs;
	enum nvmf_vfio_user_ctrlr_state		state;

	struct spdk_thread			*thread;
	struct spdk_poller			*vfu_ctx_poller;
	struct spdk_interrupt			*intr;
	int					intr_fd;

	uint16_t				cntlid;
	struct spdk_nvmf_ctrlr			*ctrlr;

	struct nvmf_vfio_user_sq		*sqs[NVMF_VFIO_USER_MAX_QPAIRS_PER_CTRLR];
	struct nvmf_vfio_user_cq		*cqs[NVMF_VFIO_USER_MAX_QPAIRS_PER_CTRLR];

	volatile uint32_t			*bar0_doorbells;
	struct nvmf_vfio_user_shadow_doorbells	*sdbl;
	uint64_t				shadow_doorbell_buffer;
	uint64_t				eventidx_buffer;
};

When you debug this transport, keep the ownership map in your head:

• Guest writes SQEs and SQ tail doorbells.
• SPDK reads SQ tail doorbells, consumes SQEs, and advances SQ head.
• SPDK writes CQEs and advances CQ tail.
• Guest reads CQEs and writes CQ head doorbells.
• Optional shadow doorbell buffers can replace the BAR0 doorbell pointers for I/O queues.

Guest Memory Mapping

vfio-user only works if the server can translate the addresses that the guest/client put into NVMe queue setup commands and data PRPs/SGLs. QEMU's protocol docs describe VFIO_USER_DMA_MAP as the client informing the server of memory regions it can access before DMA. In libvfio-user, SPDK receives those regions through callbacks registered by vfu_setup_device_dma().

SPDK's local helper for a single mapping is small but central:

map_one(vfu_ctx_t *ctx, uint64_t addr, uint64_t len, dma_sg_t *sg,
	struct iovec *iov, int32_t flags)
{
	int prot = PROT_READ;
	int ret;

	if (flags & MAP_RW) {
		prot |= PROT_WRITE;
	}

	ret = vfu_addr_to_sgl(ctx, (void *)(uintptr_t)addr, len, sg, 1, prot);
	if (ret < 0) {
		SPDK_ERRLOG("failed to translate IOVA [%#lx, %#lx) (prot=%d) to local VA: %m\n",
			    addr, addr + len, prot);
		return NULL;
	}

	ret = vfu_sgl_get(ctx, sg, iov, 1, 0);
	if (ret != 0) {
		SPDK_ERRLOG("failed to get iovec for IOVA [%#lx, %#lx): %m\n",
			    addr, addr + len);
		return NULL;
	}

	return iov->iov_base;
}

The address here is an IOVA/GPA-style address from the client side. vfu_addr_to_sgl() asks libvfio-user to translate it into a scatter-gather entry, and vfu_sgl_get() pins or retrieves a local iovec for SPDK to touch. That is why a queue can be represented as an iovec in struct nvme_q_mapping.

map_q(struct nvmf_vfio_user_ctrlr *vu_ctrlr, struct nvme_q_mapping *mapping,
      uint32_t flags)
{
	void *ret;

	assert(mapping->len != 0);
	assert(q_addr(mapping) == NULL);

	ret = map_one(vu_ctrlr->endpoint->vfu_ctx, mapping->prp1, mapping->len,
		      mapping->sg, &mapping->iov, flags);
	if (ret == NULL) {
		return -EFAULT;
	}

	if (flags & MAP_INITIALIZE) {
		memset(q_addr(mapping), 0, mapping->len);
	}

	return 0;
}

The memory-region callback adds one more operational detail: SPDK ignores non-mappable memory. This checkout's vfio-user NVMe target expects clients that share mappable memory. If a client only supports message-based DMA, this target-side code path is not enough.

memory_region_add_cb(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
{
	struct nvmf_vfio_user_endpoint *endpoint = vfu_get_private(vfu_ctx);
	struct nvmf_vfio_user_ctrlr *ctrlr;
	struct nvmf_vfio_user_sq *sq;
	struct nvmf_vfio_user_cq *cq;
	void *map_start, *map_end;
	int ret;

	/*
	 * We're not interested in any DMA regions that aren't mappable (we don't
	 * support clients that don't share their memory).
	 */
	if (!info->vaddr) {
		return;
	}

	map_start = info->mapping.iov_base;
	map_end = info->mapping.iov_base + info->mapping.iov_len;

	if (((uintptr_t)info->mapping.iov_base & MASK_2MB) ||
	    (info->mapping.iov_len & MASK_2MB)) {
		SPDK_DEBUGLOG(nvmf_vfio, "Invalid memory region vaddr %p, IOVA %p-%p\n",
			      info->vaddr, map_start, map_end);
		return;
	}

That explains a common hang: the endpoint can exist and the controller can be visible, but queue PRPs or data PRPs may not translate. In that case commands fail before they become meaningful bdev I/O.

Doorbells, SQs, CQs, And Shadow Doorbells

The SPDK NVMe learning doc summarizes normal NVMe submission as: build a 64-byte command, place it at the submission queue tail, then write the new tail to the SQ tail doorbell. vfio-user preserves that model. The difference is that the "device" reading the queue is an SPDK thread in a userspace process.

The queue helper functions make index ownership explicit:

static inline uint32_t *
sq_headp(struct nvmf_vfio_user_sq *sq)
{
	assert(sq != NULL);
	return &sq->head;
}

static inline volatile uint32_t *
sq_dbl_tailp(struct nvmf_vfio_user_sq *sq)
{
	assert(sq != NULL);
	return sq->dbl_tailp;
}

static inline volatile uint32_t *
cq_dbl_headp(struct nvmf_vfio_user_cq *cq)
{
	assert(cq != NULL);
	return cq->dbl_headp;
}

static inline volatile uint32_t *
cq_tailp(struct nvmf_vfio_user_cq *cq)
{
	assert(cq != NULL);
	return &cq->tail;
}

The advance helpers are intentionally boring. Ring queues are supposed to wrap. If they do not, an ordinary high-load workload becomes an out-of-bounds access.

static inline void
sq_head_advance(struct nvmf_vfio_user_sq *sq)
{
	assert(*sq_headp(sq) < sq->size);
	(*sq_headp(sq))++;

	if (spdk_unlikely(*sq_headp(sq) == sq->size)) {
		*sq_headp(sq) = 0;
	}
}

static inline void
cq_tail_advance(struct nvmf_vfio_user_cq *cq)
{
	assert(*cq_tailp(cq) < cq->size);
	(*cq_tailp(cq))++;

	if (spdk_unlikely(*cq_tailp(cq) == cq->size)) {
		*cq_tailp(cq) = 0;
		cq->phase = !cq->phase;
	}
}

Shadow doorbells add a second possible doorbell storage location. They exist so the host and controller can avoid trapping every doorbell write and support migration/stop-and-copy flows. SPDK switches I/O queue doorbell pointers between BAR0 and the mapped shadow doorbell buffer. Admin queue doorbells stay on BAR0.

vfio_user_ctrlr_switch_doorbells(struct nvmf_vfio_user_ctrlr *ctrlr, bool shadow)
{
	volatile uint32_t *doorbells = shadow ? ctrlr->sdbl->shadow_doorbells :
				       ctrlr->bar0_doorbells;

	assert(doorbells != NULL);

	for (size_t i = 1; i < NVMF_VFIO_USER_DEFAULT_MAX_QPAIRS_PER_CTRLR; i++) {
		struct nvmf_vfio_user_sq *sq = ctrlr->sqs[i];
		struct nvmf_vfio_user_cq *cq = ctrlr->cqs[i];

		if (sq != NULL) {
			sq->dbl_tailp = doorbells + queue_index(sq->qid, false);
			ctrlr->sqs[i]->need_rearm = shadow;
		}

		if (cq != NULL) {
			cq->dbl_headp = doorbells + queue_index(cq->qid, true);
		}
	}
}

For debugging, always ask: which doorbell is live now? BAR0 and shadow doorbells can hold different stale-looking values during migration, reset, or after older guest drivers skip reinitialization. The pointer in sq->dbl_tailp or cq->dbl_headp tells you what SPDK will actually read.

Doorbell To Request Execution

The I/O path begins when SPDK notices that the SQ tail doorbell differs from SPDK's SQ head. That can happen because a BAR0 region write path calls into doorbell handling, because a mappable doorbell is polled, or because interrupt mode wakes the relevant poller and it checks queue state.

nvmf_vfio_user_sq_poll() is the compact read-side version of the path. It refuses to process commands while the controller is not running, invalidates/refreshes the doorbell cache for platforms that need it, reads the tail, validates it, then drains commands up to the new tail.

nvmf_vfio_user_sq_poll(struct nvmf_vfio_user_sq *sq)
{
	struct nvmf_vfio_user_ctrlr *ctrlr;
	uint32_t new_tail;
	int count = 0;

	ctrlr = sq->ctrlr;

	/*
	 * A quiesced, or migrating, controller should never process new
	 * commands.
	 */
	if (ctrlr->state != VFIO_USER_CTRLR_RUNNING) {
		return SPDK_POLLER_IDLE;
	}

	if (ctrlr->adaptive_irqs_enabled) {
		handle_suppressed_irq(ctrlr, sq);
	}

	spdk_ivdt_dcache(sq_dbl_tailp(sq));

	/* Load-Acquire. */
	new_tail = *sq_dbl_tailp(sq);

	new_tail = new_tail & 0xffffu;
	if (spdk_unlikely(new_tail >= sq->size)) {
		spdk_nvmf_ctrlr_async_event_error_event(ctrlr->ctrlr, SPDK_NVME_ASYNC_EVENT_INVALID_DB_WRITE);
		return -1;
	}

	if (*sq_headp(sq) == new_tail) {
		return 0;
	}

	spdk_rmb();

	count = handle_sq_tdbl_write(ctrlr, new_tail, sq);
	if (spdk_unlikely(count < 0)) {
		fail_ctrlr(ctrlr);
	}

	return count;
}

The barrier matters. The guest is expected to write queue entries first, then publish the new tail. SPDK reads the published tail, then uses a read barrier before consuming SQEs. If a command appears as garbage, suspect memory ordering or memory mapping before suspecting the bdev.

handle_sq_tdbl_write() is the drain loop. It enforces CQ flow control before taking an SQE, advances the SQ head before command execution so completions report the correct SQHD, and calls consume_cmd().

static int
handle_sq_tdbl_write(struct nvmf_vfio_user_ctrlr *ctrlr, const uint32_t new_tail,
		     struct nvmf_vfio_user_sq *sq)
{
	struct spdk_nvme_cmd *queue;
	struct nvmf_vfio_user_cq *cq = ctrlr->cqs[sq->cqid];
	int count = 0;
	uint32_t free_cq_slots;

	free_cq_slots = cq_free_slots(cq);
	queue = q_addr(&sq->mapping);
	while (*sq_headp(sq) != new_tail) {
		int err;
		struct spdk_nvme_cmd *cmd;

		if ((free_cq_slots-- <= cq->nr_outstanding)) {
			cq->last_head = *cq_dbl_headp(cq);
			free_cq_slots = cq_free_slots(cq);
			if (free_cq_slots > cq->nr_outstanding) {
				continue;
			}
			if (in_interrupt_mode(ctrlr->transport)) {
				sq_to_poll_group(sq)->need_kick = true;
			}
			break;
		}

		cmd = &queue[*sq_headp(sq)];
		count++;
		cq->nr_outstanding++;
		sq_head_advance(sq);

		err = consume_cmd(ctrlr, sq, cmd);
		if (spdk_unlikely(err != 0)) {
			return err;
		}
	}

	return count;
}

CQ flow control is handled on submission rather than by discovering at completion time that the CQ is full. That design choice avoids creating a completed internal request that has nowhere safe to publish its CQE. The cost is that the SQ may stop draining even though there are still SQEs available.

consume_cmd() routes admin commands to local admin handling and I/O commands to handle_cmd_req(). That function wraps the SQE as an SPDK NVMf request, maps data buffers, and hands off to common NVMf execution:

static int
handle_cmd_req(struct nvmf_vfio_user_ctrlr *ctrlr, struct spdk_nvme_cmd *cmd,
	       struct nvmf_vfio_user_sq *sq)
{
	int err;
	struct nvmf_vfio_user_req *vu_req;
	struct spdk_nvmf_request *req;

	vu_req = get_nvmf_vfio_user_req(sq);
	if (spdk_unlikely(vu_req == NULL)) {
		return post_completion(ctrlr, ctrlr->cqs[sq->cqid], 0, 0, cmd->cid,
				       SPDK_NVME_SC_INTERNAL_DEVICE_ERROR, SPDK_NVME_SCT_GENERIC);
	}
	req = &vu_req->req;

	vu_req->cb_fn = handle_cmd_rsp;
	vu_req->cb_arg = SPDK_CONTAINEROF(req->qpair, struct nvmf_vfio_user_sq, qpair);
	req->cmd->nvme_cmd = *cmd;

	if (nvmf_qpair_is_admin_queue(req->qpair)) {
		err = map_admin_cmd_req(ctrlr, req);
	} else {
		err = map_io_cmd_req(ctrlr, req);
	}

	if (spdk_unlikely(err < 0)) {
		req->rsp->nvme_cpl.status.sct = SPDK_NVME_SCT_GENERIC;
		req->rsp->nvme_cpl.status.sc = err == -ENOTSUP ?
					       SPDK_NVME_SC_INVALID_OPCODE :
					       SPDK_NVME_SC_INTERNAL_DEVICE_ERROR;
		err = handle_cmd_rsp(vu_req, vu_req->cb_arg);
		_nvmf_vfio_user_req_free(sq, vu_req);
		return err;
	}

	vu_req->state = VFIO_USER_REQUEST_STATE_EXECUTING;
	spdk_nvmf_request_exec(req);

	return 0;
}

This is the handoff point. Before this line, you are debugging vfio-user, PCI/NVMe queue state, mappings, and doorbells. After spdk_nvmf_request_exec(req), you are mostly debugging the common NVMf and bdev path.

Data Buffer Mapping For Commands

Queue memory and data buffer memory are related but separate. The SQ itself was mapped from the queue creation command's PRP. A read or write command can also point to data buffers. SPDK maps those buffers into req->iov before executing the request.

static int
map_io_cmd_req(struct nvmf_vfio_user_ctrlr *ctrlr, struct spdk_nvmf_request *req)
{
	int len, iovcnt;
	struct spdk_nvme_cmd *cmd;

	cmd = &req->cmd->nvme_cmd;
	req->xfer = spdk_nvme_opc_get_data_transfer(cmd->opc);

	if (spdk_unlikely(req->xfer == SPDK_NVME_DATA_NONE)) {
		return 0;
	}

	len = get_nvmf_io_req_length(req);
	if (len < 0) {
		return -EINVAL;
	}
	req->length = len;

	iovcnt = vfio_user_map_cmd(ctrlr, req, req->iov, req->length);
	if (iovcnt < 0) {
		SPDK_ERRLOG("%s: failed to map IO OPC %u\n", ctrlr_id(ctrlr), cmd->opc);
		return -EFAULT;
	}
	req->iovcnt = iovcnt;

	return 0;
}

This explains a second common split in failures:

• If SQ polling never reaches handle_cmd_req(), debug doorbells and queue mapping.
• If handle_cmd_req() fails during map_io_cmd_req(), debug PRP/SGL translation and client DMA maps.
• If spdk_nvmf_request_exec() runs and the request stalls, debug NVMf command execution and the bdev graph.

Completion Queue Flow And Interrupts

A bdev completion is not the same thing as a guest-visible NVMe completion. The vfio-user transport still has to write an NVMe CQE into the guest's CQ memory, update the controller-owned CQ tail, and notify the guest if interrupts are enabled and not suppressed.

The request callback first unmaps any mapped data buffers and then calls post_completion():

handle_cmd_rsp(struct nvmf_vfio_user_req *vu_req, void *cb_arg)
{
	struct nvmf_vfio_user_sq *sq = cb_arg;
	struct nvmf_vfio_user_ctrlr *vu_ctrlr = sq->ctrlr;
	uint16_t sqid, cqid;

	if (spdk_likely(vu_req->iovcnt)) {
		vfu_sgl_put(vu_ctrlr->endpoint->vfu_ctx,
			    index_to_sg_t(vu_req->sg, 0),
			    vu_req->iov, vu_req->iovcnt);
	}
	sqid = sq->qid;
	cqid = sq->cqid;

	return post_completion(vu_ctrlr, vu_ctrlr->cqs[cqid],
			       vu_req->req.rsp->nvme_cpl.cdw0,
			       sqid,
			       vu_req->req.cmd->nvme_cmd.cid,
			       vu_req->req.rsp->nvme_cpl.status.sc,
			       vu_req->req.rsp->nvme_cpl.status.sct);
}

post_completion() applies CQ flow control, writes the CQE fields, sets the phase bit, uses a write barrier to make the CQE visible before moving the tail, and triggers an interrupt when the queue and controller state allow it.

static int
post_completion(struct nvmf_vfio_user_ctrlr *ctrlr, struct nvmf_vfio_user_cq *cq,
		uint32_t cdw0, uint16_t sqid, uint16_t cid, uint16_t sc, uint16_t sct)
{
	struct spdk_nvme_status cpl_status = { 0 };
	struct spdk_nvme_cpl *cpl;
	int err;

	if (spdk_unlikely(cq == NULL || q_addr(&cq->mapping) == NULL)) {
		return 0;
	}

	if (cq_is_full(cq)) {
		SPDK_ERRLOG("%s: cqid:%d full (tail=%d, head=%d)\n",
			    ctrlr_id(ctrlr), cq->qid, *cq_tailp(cq),
			    *cq_dbl_headp(cq));
		return -1;
	}

	cpl = ((struct spdk_nvme_cpl *)q_addr(&cq->mapping)) + *cq_tailp(cq);
	cpl->sqhd = *sq_headp(ctrlr->sqs[sqid]);
	cpl->sqid = sqid;
	cpl->cid = cid;
	cpl->cdw0 = cdw0;

	cpl_status.sct = sct;
	cpl_status.sc = sc;
	cpl_status.p = cq->phase;
	cpl->status = cpl_status;

	cq->nr_outstanding--;
	spdk_wmb();
	cq_tail_advance(cq);

	if ((cq->qid == 0 || !ctrlr->adaptive_irqs_enabled) &&
	    cq->ien && ctrlr_interrupt_enabled(ctrlr)) {
		err = vfu_irq_trigger(ctrlr->endpoint->vfu_ctx, cq->iv);
		if (err != 0) {
			SPDK_ERRLOG("%s: failed to trigger interrupt: %m\n",
				    ctrlr_id(ctrlr));
			return err;
		}
	}

	return 0;
}

If the guest is waiting, ask whether the CQE was written, whether the phase bit changed as expected, whether the guest has advanced the CQ head, and whether an interrupt should have been delivered. A request callback firing only proves that SPDK finished internal work. It does not prove that the guest has observed the CQE.

SPDK's interrupt mode documentation adds one more relevant operational rule: in interrupt mode, an fd callback must either drain all pending work or reassert readiness so epoll wakes it again. The vfio-user transport handles this by rearming and kicking poll groups.

static void
ctrlr_kick(struct nvmf_vfio_user_ctrlr *vu_ctrlr)
{
	struct nvmf_vfio_user_poll_group *vu_ctrlr_group;

	SPDK_DEBUGLOG(vfio_user_db, "%s: kicked\n", ctrlr_id(vu_ctrlr));

	vu_ctrlr_group = ctrlr_to_poll_group(vu_ctrlr);
	vu_ctrlr_group->stats.ctrlr_kicks++;

	spdk_thread_send_msg(poll_group_to_thread(vu_ctrlr_group),
			     vfio_user_ctrlr_intr_msg, vu_ctrlr);
}

static void
poll_group_kick(struct nvmf_vfio_user_poll_group *vu_group)
{
	if (spdk_likely(!in_interrupt_mode(vu_group->group.transport))) {
		return;
	}

	vu_group->stats.pg_kicks++;
	eventfd_write(vu_group->intr_fd, 1);
}

The distinction is useful in debugging. ctrlr_kick() moves work to the controller's poll group thread. poll_group_kick() wakes a poll group through its eventfd when interrupt mode would otherwise sleep. A missing kick can look exactly like a lost doorbell.

The vfio-user Protocol Side

SPDK also contains a vfio-user host/client path used by code that connects to vfio-user PCI devices. It is not the target-side NVMf transport, but it is useful for understanding the protocol messages that the target must respond to.

The protocol header and command list are intentionally VFIO-like:

enum vfio_user_command {
	VFIO_USER_VERSION			= 1,
	VFIO_USER_DMA_MAP			= 2,
	VFIO_USER_DMA_UNMAP			= 3,
	VFIO_USER_DEVICE_GET_INFO		= 4,
	VFIO_USER_DEVICE_GET_REGION_INFO	= 5,
	VFIO_USER_DEVICE_GET_REGION_IO_FDS	= 6,
	VFIO_USER_DEVICE_GET_IRQ_INFO		= 7,
	VFIO_USER_DEVICE_SET_IRQS		= 8,
	VFIO_USER_REGION_READ			= 9,
	VFIO_USER_REGION_WRITE			= 10,
	VFIO_USER_DMA_READ			= 11,
	VFIO_USER_DMA_WRITE			= 12,
	VFIO_USER_DEVICE_RESET			= 13,
	VFIO_USER_DIRTY_PAGES			= 14,
	VFIO_USER_MAX,
};

struct vfio_user_header {
	uint16_t	msg_id;
	uint16_t	cmd;
	uint32_t	msg_size;
	struct {
		uint32_t	type     : 4;
#define VFIO_USER_F_TYPE_COMMAND	0
#define VFIO_USER_F_TYPE_REPLY		1
		uint32_t	no_reply : 1;
		uint32_t	error    : 1;
		uint32_t	resvd    : 26;
	} flags;
	uint32_t	error_no;
} __attribute__((packed));

DMA map messages carry the address, size, offset, flags, and often a file descriptor. Region access messages are how non-mappable BAR/config accesses are represented.

struct vfio_user_dma_map {
	uint32_t	argsz;
#define VFIO_USER_F_DMA_REGION_READ	(1 << 0)
#define VFIO_USER_F_DMA_REGION_WRITE	(1 << 1)
	uint32_t	flags;
	uint64_t	offset;
	uint64_t	addr;
	uint64_t	size;
} __attribute__((packed));

struct vfio_user_region_access {
	uint64_t	offset;
	uint32_t	region;
	uint32_t	count;
	uint8_t		data[];
} __attribute__((packed));

The client helper shows that DMA map/unmap can pass fds, while region reads/writes are payload messages on the socket:

int
vfio_user_dev_dma_map_unmap(struct vfio_device *dev, struct vfio_memory_region *mr, bool map)
{
	struct vfio_user_dma_map dma_map = { 0 };
	struct vfio_user_dma_unmap dma_unmap = { 0 };

	if (map) {
		dma_map.argsz = sizeof(struct vfio_user_dma_map);
		dma_map.addr = mr->iova;
		dma_map.size = mr->size;
		dma_map.offset = mr->offset;
		dma_map.flags = VFIO_USER_F_DMA_REGION_READ | VFIO_USER_F_DMA_REGION_WRITE;

		return vfio_user_dev_send_request(dev, VFIO_USER_DMA_MAP,
						  &dma_map, sizeof(dma_map), sizeof(dma_map), &mr->fd, 1);
	} else {
		dma_unmap.argsz = sizeof(struct vfio_user_dma_unmap);
		dma_unmap.addr = mr->iova;
		dma_unmap.size = mr->size;
		return vfio_user_dev_send_request(dev, VFIO_USER_DMA_UNMAP,
						  &dma_unmap, sizeof(dma_unmap), sizeof(dma_unmap), &mr->fd, 1);
	}
}

int
vfio_user_dev_mmio_access(struct vfio_device *dev, uint32_t index, uint64_t offset,
			  size_t len, void *buf, bool is_write)
{
	struct vfio_user_region_access *access;
	size_t arg_len;
	int ret;

	arg_len = sizeof(*access) + len;
	access = calloc(1, arg_len);
	if (!access) {
		return -ENOMEM;
	}

	access->offset = offset;
	access->region = index;
	access->count = len;
	if (is_write) {
		memcpy(access->data, buf, len);
		ret = vfio_user_dev_send_request(dev, VFIO_USER_REGION_WRITE,
						 access, arg_len, arg_len, NULL, 0);
	} else {
		ret = vfio_user_dev_send_request(dev, VFIO_USER_REGION_READ,
						 access, sizeof(*access), arg_len, NULL, 0);
	}

	if (ret) {
		free(access);
		return ret;
	}
}

This protocol split is why "it uses a Unix socket" is not enough to predict performance or behavior. BAR/config accesses may be socket messages, direct BAR subranges may be mmaped, and guest memory may be shared by fd-backed mappings. The exact path depends on region capabilities and the client's memory-sharing mode.

SPDK's vfio-user NVMe Client Path

The local SPDK NVMe driver also has a vfio-user custom transport. This is the initiator/client side for connecting to a vfio-user NVMe target. It helps confirm the same mental model from the other direction: connect to a cntrl socket, set up BAR0, then use the normal PCIe NVMe controller fields for doorbells.

static struct spdk_nvme_ctrlr *
nvme_vfio_ctrlr_construct(const struct spdk_nvme_transport_id *trid,
			  const struct spdk_nvme_ctrlr_opts *opts,
			  void *devhandle)
{
	struct nvme_vfio_ctrlr *vctrlr;
	struct nvme_pcie_ctrlr *pctrlr;
	char ctrlr_path[PATH_MAX];

	snprintf(ctrlr_path, sizeof(ctrlr_path), "%s/cntrl", trid->traddr);
	...
	vctrlr->dev = spdk_vfio_user_setup(ctrlr_path);
	if (!vctrlr->dev) {
		SPDK_ERRLOG("Error to setup vfio device\n");
		free(vctrlr);
		return NULL;
	}

	ret = nvme_vfio_setup_bar0(vctrlr);
	...
	pctrlr = &vctrlr->pctrlr;
	pctrlr->doorbell_base = vctrlr->doorbell_base;
	pctrlr->ctrlr.is_removed = false;
	...
}

From the initiator's perspective, once setup succeeds, the vfio-user controller is wired into the same nvme_pcie_ctrlr structure shape used by PCIe-style NVMe paths. That is another hint that vfio-user NVMe should be debugged as an NVMe queue machine, not as a custom block RPC protocol.

Request Flow Diagram

The diagram separates control setup from data movement. vfio-user messages establish what memory and regions exist. NVMe I/O then runs through shared memory queues, doorbells, and SPDK's common NVMf request path.

Control Plane

SPDK has two related vfio-user control planes that are easy to mix up.

Generic vfu target endpoints use the vfu target RPC base path:

• lib/vfu_tgt/tgt_rpc.c: rpc_vfu_tgt_set_base_path
• lib/vfu_tgt/tgt_rpc.c: SPDK_RPC_REGISTER("vfu_tgt_set_base_path", ...)

That path belongs to the generic lib/vfu_tgt endpoint framework used by RPCs such as vfu_virtio_create_blk_endpoint.

The NVMe-oF VFIOUSER transport in this chapter uses the NVMf control plane instead: create the NVMf transport, create a subsystem and namespace, then add a listener with trtype=VFIOUSER. In lib/nvmf/vfio_user.c, the listener's traddr is the endpoint directory. The transport opens <traddr>/bar0 and creates the libvfio-user controller socket at <traddr>/cntrl. Setting vfu_tgt_set_base_path does not choose the endpoint path for this NVMf VFIOUSER listener.

The conceptual target objects are still subsystem, listener, namespace, qpair, and request. The transport type determines that the listener creates a socket-backed PCI endpoint rather than a TCP or RDMA network listener.

The SPDK NVMe-oF target docs are still useful for this chapter because they explain subsystem/listener/namespace configuration and note that interrupt mode applies to the vfio-user, TCP, and RDMA transports. The unique part of vfio-user is not the NVMf subsystem model; it is how a local PCI/NVMe device surface delivers requests into that model.

Edge Cases And Failure Modes

Doorbell lost or not observed:

The guest can write valid SQEs and still see no progress if SPDK does not observe the SQ tail update. Check whether BAR0 is mappable, whether shadow doorbells are active, whether sq->dbl_tailp points where you think it does, and whether nvmf_vfio_user_sq_poll() is running on the right poll group. In interrupt mode, also check ctrlr_kick(), poll_group_kick(), and poll-group eventfd readiness.

CQ full:

The implementation applies CQ flow control while draining SQEs. If the guest does not advance the CQ head, handle_sq_tdbl_write() can stop taking new commands even though the SQ tail moved. This looks like "submissions stuck" but the root cause is "no completion space." Inspect cq->tail, cq->last_head, *cq_dbl_headp(cq), and cq->nr_outstanding.

Wrong memory mapping:

If guest memory is not mappable or a PRP points outside a mapped DMA range, vfu_addr_to_sgl()/vfu_sgl_get() cannot build a local iovec. Queue mapping failures prevent SQ/CQ access. Data buffer mapping failures show up later in map_io_cmd_req(). The target-side callback explicitly ignores non-mappable DMA regions, so a client that only offers message-based DMA is not enough for this path.

Interrupt versus poll mode:

A CQE can be written while the guest still waits because notification did not happen or because the guest expects to poll and is not polling. Check cq->ien, MSI-X enable state, ctrlr_interrupt_enabled(), adaptive_irqs_enabled, last_irq_failed, and whether vfu_irq_trigger() ran. SPDK interrupt mode adds eventfd rearm/kick responsibilities that do not exist in pure poll mode.

Queue deletion while I/O is outstanding:

SQ and CQ lifetime is separate from individual request lifetime. A queue teardown path must account for outstanding requests, mapped iovecs, and references from CQs shared by multiple SQs. If the VM resets or disconnects while requests are executing, look at delete and close paths before blaming the bdev.

Migration and shadow doorbells:

Shadow doorbells move doorbell state into guest-provided memory. During migration, reset, or doorbell buffer reconfiguration, SPDK may copy doorbells between BAR0 and shadow buffers and switch queue pointers. Debugging only BAR0 is incomplete once ctrlr->sdbl is non-NULL.

Invalid doorbell writes:

nvmf_vfio_user_sq_poll() masks the read tail to 16 bits and reports an NVMe async event when the new tail is outside the queue size. This is a guest-visible controller error, not a normal transient.

Misconceptions To Kill

"vfio-user is just faster vhost."

No. vhost-blk exposes virtio-blk. vfio-user NVMe exposes NVMe controller semantics through a userspace PCI device model.

"Because it uses a Unix socket, I/O data is copied through the socket."

Not usually in the target path described here. The socket carries protocol/control messages and can pass file descriptors. Queue and data access use mapped client memory when the client supports it.

"NVMe-oF means network."

In SPDK, the NVMf target layer has a vfio-user transport. The common NVMf controller/request model is reused, but the transport is local and PCI-like rather than TCP/RDMA network-like.

"A completion callback means the guest has seen the completion."

No. It means SPDK completed internal work. The transport still must publish a CQE, advance CQ state, and notify or rely on guest polling.

"Doorbell value equals queue entry count."

No. Doorbells are ring indices. Wraparound and phase bits matter. Compare head, tail, size, and phase together.

Lab: Queue State Reading

Open lib/nvmf/vfio_user.c and find:

1. struct nvmf_vfio_user_sq
2. struct nvmf_vfio_user_cq
3. sq_head_advance
4. cq_tail_advance
5. vfio_user_ctrlr_switch_doorbells
6. nvmf_vfio_user_sq_poll
7. post_completion

Write a short note answering:

• Where does the backend-owned SQ head live?
• Where does the guest-owned SQ tail live?
• Where does the backend-owned CQ tail live?
• Where does the guest-owned CQ head live?
• How does that answer change when shadow doorbells are active?

Then compare this to the NVMe queue model from the hardware chapters.

Operational Debug Exercise

Symptom: VM sees vfio-user NVMe device but I/O hangs.

Check in this order:

1. Did the endpoint socket accept a connection?
2. Did vfu_realize_ctx() and vfu_attach_ctx() succeed?
3. Did the guest create admin and I/O queues?
4. Are the SQ and CQ mappings non-NULL?
5. Are SQ doorbells changing in the active doorbell location?
6. Is nvmf_vfio_user_sq_poll() running?
7. Are commands reaching handle_cmd_req()?
8. Are data buffers mapping in map_io_cmd_req()?
9. Are requests reaching spdk_nvmf_request_exec()?
10. Are bdev completions returning into handle_cmd_rsp()?
11. Are CQEs written by post_completion()?
12. Is the guest notified through MSI-X/eventfd, or is it expected to poll?

If requests never reach spdk_nvmf_request_exec(), debug vfio-user setup, queue mapping, and doorbells. If requests reach bdev and do not complete, debug the bdev graph. If bdev completes and the guest still hangs, debug CQ visibility and interrupt delivery.

Source Reading Path

Read in this order:

1. include/spdk/vfio_user_spec.h: protocol command IDs, message header, DMA map, region access.
2. lib/nvmf/vfio_user.c: object structs near the top of the file.
3. lib/nvmf/vfio_user.c: nvmf_vfio_user_listen() and vfio_user_dev_info_fill().
4. lib/nvmf/vfio_user.c: map_one(), map_q(), memory_region_add_cb(), memory_region_remove_cb().
5. lib/nvmf/vfio_user.c: queue helpers, vfio_user_ctrlr_switch_doorbells(), handle_doorbell_buffer_config().
6. lib/nvmf/vfio_user.c: nvmf_vfio_user_sq_poll(), handle_sq_tdbl_write(), handle_cmd_req().
7. lib/nvmf/vfio_user.c: handle_cmd_rsp(), post_completion(), ctrlr_kick(), poll_group_kick().
8. lib/nvmf/ctrlr.c and lib/nvmf/ctrlr_bdev.c: common NVMf command execution and bdev handoff.
9. lib/vfio_user/host/vfio_user.c and lib/vfio_user/host/vfio_user_pci.c: SPDK's vfio-user client-side protocol helpers.
10. lib/nvme/nvme_vfio_user.c: SPDK NVMe initiator custom transport for vfio-user.

Self-Check

1. What does an SQ tail doorbell indicate?
2. Why is guest memory mapping central to vfio-user?
3. How does vfio-user NVMe differ from vhost-blk?
4. Which function hands a built vfio-user NVMf request to common NVMf execution?
5. Why can bdev completion still leave the guest waiting?
6. What state tells SPDK whether to read BAR0 doorbells or shadow doorbells?
7. Why does this transport apply CQ flow control before consuming more SQEs?
8. What can go wrong if interrupt mode sleeps without rearming or kicking the poll group?

References

• Local SPDK: lib/nvmf/vfio_user.c
• Local SPDK: include/spdk/vfio_user_spec.h
• Local SPDK: lib/vfio_user/host/vfio_user.c
• Local SPDK: lib/vfio_user/host/vfio_user_pci.c
• Local SPDK: lib/nvme/nvme_vfio_user.c
• Local SPDK: lib/vfu_tgt/tgt_rpc.c
• SPDK NVMe-oF target docs: https://spdk.io/doc/nvmf.html
• SPDK NVMe queue/doorbell doc: https://spdk.io/doc/nvme_spec.html
• SPDK interrupt mode docs: https://spdk.io/doc/interrupt_mode.html
• QEMU vfio-user protocol documentation: https://www.qemu.org/docs/master/interop/vfio-user.html
• QEMU vfio-user system device documentation: https://www.qemu.org/docs/master/system/devices/vfio-user.html
• libvfio-user README: https://qemu.googlesource.com/libvfio-user/+/a8242d117118d5191dad69a96e28a21d66fe8b50/README.md
• libvfio-user memory mapping notes: https://github.com/nutanix/libvfio-user/blob/master/docs/memory-mapping.md
• NVM Express specifications: https://nvmexpress.org/specifications/