SPDK From First Principles

SPDK deep learning path

Chapter 5: NVMe SSDs As Queue Machines

NVMe is not "a faster disk command set." It is a queue protocol designed for many-core hosts and parallel SSD controllers. The host and controller communicate mostly through...

Source: drafts/hardware/05-nvme-queue-machine.md

Chapter Goal

NVMe is not "a faster disk command set." It is a queue protocol designed for many-core hosts and parallel SSD controllers. The host and controller communicate mostly through shared memory queues and small MMIO doorbell writes. SPDK's NVMe driver is built around that model: allocate queue pairs, submit commands, ring doorbells, poll completions, and invoke callbacks.

By the end of this chapter, you should be able to draw the submission/completion flow, explain SQs and CQs, identify command and completion fields in SPDK source, reason about phase bits and queue wraparound, and understand why a qpair is normally owned by one thread.

Beginner Mental Model

An NVMe queue pair is two circular arrays in host memory:

host memory

submission queue (SQ)                  completion queue (CQ)
tail -> next free slot                 head -> next completion to consume

  +-----+-----+-----+-----+              +-----+-----+-----+-----+
  | cmd | cmd |     |     |              | cpl | cpl |     |     |
  +-----+-----+-----+-----+              +-----+-----+-----+-----+
     ^ device reads commands                ^ host reads completions

MMIO doorbells:
  host writes SQ tail to tell controller new commands exist
  host writes CQ head to tell controller completions were consumed

The host writes command entries into the SQ. The controller DMA-reads them. Later the controller DMA-writes completion entries into the CQ. The host polls the CQ and runs callbacks. Doorbells are small MMIO register writes that synchronize producer/consumer positions.

That last sentence is the center of the chapter. The queue entries live in memory that both sides can reach, but the two sides do not share a C data structure in the ordinary sense. The CPU sees virtual addresses and writes cacheable memory. The controller sees bus addresses and performs DMA. Doorbell registers live in the controller's PCIe BAR and are written through MMIO, not through the same path as normal RAM. A qpair is therefore a small protocol machine with three kinds of state:

  • host-owned producer/consumer indexes, such as sq_tail and cq_head;
  • controller-visible queue memory, addressed through DMA-capable bus addresses;
  • MMIO doorbells that tell the controller when an index changed.

NVMe does not require the controller to process commands in the same order that it completes them. It does require the completion entry to name the command that completed. That is why cid is as important as the queue slot: a completion says "command identifier N is done", and SPDK uses that number to find the request object and callback that were saved at submission time.

flowchart LR app[SPDK caller] --> tracker[Request tracker and CID] tracker --> sq[Submission Queue in host memory] sq --> db[MMIO SQ tail doorbell] db --> ctrlr[NVMe controller] ctrlr --> media[SSD media and FTL] media --> cq[Completion Queue in host memory] cq --> poll[SPDK qpair poller] poll --> cb[User completion callback] poll --> cqdb[MMIO CQ head doorbell]

Why NVMe Exists

Legacy storage protocols were designed around fewer queues, deeper kernel mediation, and mechanical disks. SSDs need:

  • many independent queues,
  • high queue depth,
  • low per-command overhead,
  • no interrupt required for every completion,
  • command formats sized for cache lines,
  • parallelism that maps to CPU cores and controller internals.

NVMe over PCIe uses host memory queues and PCIe DMA. NVMe over Fabrics carries the same command model over transports such as RDMA and TCP. This chapter focuses on local PCIe because it exposes the hardware mechanics most directly.

The official NVMe specification set now describes NVMe as a family of base, command-set, and transport specifications. The NVM Express specifications page says the Base specification defines the protocol for host software to communicate with non-volatile memory subsystems over memory-based and message-based transports. The Base Specification page also describes NVMe as designed for SSDs and as a lower-latency, more scalable interface than SATA. Those claims map directly to the queue-pair design: avoid a single serialized command path, expose enough independent queues for multicore hosts, and make the hot path mostly memory stores plus one small doorbell write.

For an SPDK reader, the practical consequence is that "using NVMe" means driving a protocol state machine, not calling a blocking disk function. The application submits work, keeps ownership of the thread/qpair discipline, and later polls for completions. The driver can be fast because it does not hide those costs behind locks, sleeping waits, or per-I/O kernel crossings.

The Command And Completion Structures

SPDK's local copy of the NVMe command layout is in include/spdk/nvme_spec.h:1452. struct spdk_nvme_cmd is 64 bytes (include/spdk/nvme_spec.h:1504). Key fields:

  • opc: opcode, such as read, write, flush, identify, create queue.
  • fuse: fused operation marker.
  • cid: command identifier, used to match completion to request state.
  • nsid: namespace identifier.
  • mptr: metadata pointer.
  • dptr: PRP or SGL data pointer.
  • cdw10 through cdw15: command-specific dwords.

The completion entry is struct spdk_nvme_cpl at include/spdk/nvme_spec.h:1519, and it is 16 bytes (include/spdk/nvme_spec.h:1537). Key fields:

  • cdw0 and cdw1: command-specific result.
  • sqhd: submission queue head pointer from the controller's point of view.
  • sqid: submission queue identifier.
  • cid: command identifier.
  • status: status code, status code type, retry hints, do-not-retry bit, and phase tag.

Beginner trap: the command carries the request; the completion does not contain the original data buffer. The driver uses cid and its own tracker/request tables to find the callback and buffer state.

Here is the front of the command layout from include/spdk/nvme_spec.h. The fields are arranged as NVMe command dwords, not as a high-level SPDK request object:

struct spdk_nvme_cmd {
	/* dword 0 */
	uint16_t opc	:  8;	/* opcode */
	uint16_t fuse	:  2;	/* fused operation */
	uint16_t rsvd1	:  4;
	uint16_t psdt	:  2;
	uint16_t cid;		/* command identifier */

	/* dword 1 */
	uint32_t nsid;		/* namespace identifier */

	/* dword 2-3 */
	uint32_t rsvd2;
	uint32_t rsvd3;

	/* dword 4-5 */
	uint64_t mptr;		/* metadata pointer */

	/* dword 6-9: data pointer */
	union {
		struct {
			uint64_t prp1;		/* prp entry 1 */
			uint64_t prp2;		/* prp entry 2 */
		} prp;

		struct spdk_nvme_sgl_descriptor sgl1;
	} dptr;

The same structure continues with command-specific dwords. SPDK verifies the hardware contract at compile time:

	/* dword 14-15 */
	uint32_t cdw14;		/* command-specific */
	uint32_t cdw15;		/* command-specific */
};
SPDK_STATIC_ASSERT(sizeof(struct spdk_nvme_cmd) == 64, "Incorrect size");

The command contains just enough transport-visible information for the controller to execute the operation: opcode, namespace, data pointer format, metadata pointer, command-specific dwords, and cid. SPDK's larger struct nvme_request owns the user callback, callback argument, payload description, split-child state, and retry/error bookkeeping. That state is intentionally not DMA-read by the controller.

The completion is much smaller:

struct spdk_nvme_status {
	uint16_t p	:  1;	/* phase tag */
	uint16_t sc	:  8;	/* status code */
	uint16_t sct	:  3;	/* status code type */
	uint16_t crd	:  2;   /* command retry delay */
	uint16_t m	:  1;	/* more */
	uint16_t dnr	:  1;	/* do not retry */
};

struct spdk_nvme_cpl {
	uint32_t		cdw0;	/* command-specific */
	uint32_t		cdw1;	/* command-specific */
	uint16_t		sqhd;	/* submission queue head pointer */
	uint16_t		sqid;	/* submission queue identifier */
	uint16_t		cid;	/* command identifier */
	union {
		uint16_t                status_raw;
		struct spdk_nvme_status	status;
	};
};
SPDK_STATIC_ASSERT(sizeof(struct spdk_nvme_cpl) == 16, "Incorrect size");

Read this as a receipt, not as the original work order. sqid identifies which submission queue the command came from, cid identifies the command within that qpair's tracker table, sqhd tells the host what the controller considers consumed on the SQ, and status tells the host whether normal completion, retry, or error handling should follow. The phase bit is not an error bit; it is a freshness bit for a reused CQ slot.

Admin Queues And I/O Queues

Every NVMe controller has an admin queue pair. Admin commands create and delete I/O queues, identify controllers and namespaces, get logs, set features, abort commands, and manage asynchronous events. I/O queue pairs carry reads, writes, flushes, write zeroes, dataset management, and command-set-specific I/O operations.

SPDK's controller register structure in include/spdk/nvme_spec.h:550 includes admin queue attributes and base addresses:

  • aqa: admin queue attributes.
  • asq: admin submission queue base address.
  • acq: admin completion queue base address.

The same register structure exposes doorbells at include/spdk/nvme_spec.h:611: each queue has a submission queue tail doorbell and completion queue head doorbell.

The local register model makes the split explicit. These exact excerpts show the admin queue base registers and the per-qpair doorbell entries:

/** admin queue attributes */
union spdk_nvme_aqa_register	aqa;

uint64_t			asq; /* admin submission queue base addr */
uint64_t			acq; /* admin completion queue base addr */
struct {
	uint32_t	sq_tdbl;	/* submission queue tail doorbell */
	uint32_t	cq_hdbl;	/* completion queue head doorbell */
} doorbell[1];

Admin queue setup starts with registers because the controller needs an initial command path before I/O queues exist. I/O queues are then created with admin commands. In lib/nvme/nvme_pcie_common.c, SPDK builds CREATE_IO_CQ with cmd->dptr.prp.prp1 = pqpair->cpl_bus_addr and CREATE_IO_SQ with cmd->dptr.prp.prp1 = pqpair->cmd_bus_addr. That is the controller-visible side of queue construction: the device receives DMA addresses for the CQ and SQ memory it will use.

The Queue Pair In SPDK

For PCIe, SPDK's struct nvme_pcie_qpair is in lib/nvme/nvme_pcie_internal.h:140. The hot fields are the queue indices and flags:

  • num_entries
  • last_sq_tail
  • sq_tail
  • cq_head
  • sq_head
  • flags.phase
  • sq_vaddr
  • cq_vaddr
  • cmd_bus_addr
  • cpl_bus_addr

The names tell the story. sq_vaddr and cq_vaddr are CPU virtual addresses for the host-side arrays. cmd_bus_addr and cpl_bus_addr are bus/IOVA addresses the device can DMA to or from. The host updates sq_tail; the controller updates completions; the host advances cq_head.

The qpair's hot fields are deliberately placed before the embedded base qpair:

/* Array of trackers indexed by command ID. */
struct nvme_tracker *tr;

uint16_t num_entries;

uint8_t pcie_state;

uint8_t retry_count;

uint16_t max_completions_cap;

uint16_t last_sq_tail;
uint16_t sq_tail;
uint16_t cq_head;
uint16_t sq_head;

struct {
	uint8_t phase			: 1;
	uint8_t delay_cmd_submit	: 1;
	uint8_t has_shadow_doorbell	: 1;
	uint8_t has_pending_vtophys_failures : 1;
	uint8_t defer_destruction	: 1;

	/* Disable merging of physically contiguous SGL entries */
	uint8_t disable_pcie_sgl_merge	: 1;
} flags;

The same structure keeps the queue DMA addresses and host virtual addresses together below the hotter fields:

uint64_t cmd_bus_addr;
uint64_t cpl_bus_addr;

struct spdk_nvme_cmd *sq_vaddr;
struct spdk_nvme_cpl *cq_vaddr;

This excerpt explains several SPDK design choices at once. tr is an array because the completion's cid must become a quick index lookup, not a list search. sq_tail, cq_head, sq_head, and phase are hot because submission and polling touch them constantly. cmd_bus_addr and cpl_bus_addr are colder after setup because the controller already knows where the queues are; the hot path mostly uses cmd/cpl pointers and doorbells.

SPDK's own documentation says qpair scaling is lock-free but thread-constrained. doc/nvme.md:153 through doc/nvme.md:160 explains that queue pairs contain no locks or atomics and a given qpair may only be used by a single thread at a time. Violating this is undefined behavior.

Misconception to kill: "NVMe has many queues so any thread can submit to any queue." The scalable model is many queues with clear ownership, not one shared queue with hidden locks.

Submission Flow

The simplified SPDK PCIe submission path:

application / bdev_nvme
  builds an nvme_request
  calls nvme_qpair_submit_request()

common qpair layer
  queues request if the qpair is backed up
  calls transport submit

PCIe transport
  copies command to SQ slot
  associates CID with tracker
  advances sq_tail
  rings SQ doorbell

controller
  sees new tail
  DMA-reads SQ entries
  executes commands

The common submit wrapper is in lib/nvme/nvme_qpair.c:1171. It handles queued requests and the -EAGAIN case by inserting requests into qpair->queued_req rather than failing the user operation immediately.

At the PCIe transport layer, submission first needs a free tracker. A tracker is SPDK's owner record for one outstanding command. It stores the request pointer and callback state, and its cid becomes the command identifier written into the SQE:

tr = TAILQ_FIRST(&pqpair->free_tr);

if (tr == NULL) {
	pqpair->stat->queued_requests++;
	/* Inform the upper layer to try again later. */
	rc = -EAGAIN;
	goto exit;
}

pqpair->stat->submitted_requests++;
TAILQ_REMOVE(&pqpair->free_tr, tr, tq_list); /* remove tr from free_tr */
TAILQ_INSERT_TAIL(&pqpair->outstanding_tr, tr, tq_list);
pqpair->qpair.queue_depth++;
tr->req = req;
tr->cb_fn = req->cb_fn;
tr->cb_arg = req->cb_arg;
req->cmd.cid = tr->cid;
/* Use PRP by default. This bit will be overridden below if needed. */
req->cmd.psdt = SPDK_NVME_PSDT_PRP;

This is why the completion can be small. The CQE does not need to carry a callback pointer or buffer pointer. It only needs to carry a cid that lets SPDK find pqpair->tr[cid], which still owns the full request state.

After SPDK builds the data pointer or SGL/PRP state, the tracker submission function copies the 64-byte command into the current SQ slot and advances the ring:

/* Copy the command from the tracker to the submission queue. */
nvme_pcie_copy_command(&pqpair->cmd[pqpair->sq_tail], &req->cmd);

if (spdk_unlikely(++pqpair->sq_tail == pqpair->num_entries)) {
	pqpair->sq_tail = 0;
}

if (spdk_unlikely(pqpair->sq_tail == pqpair->sq_head)) {
	NVME_QPAIR_ERRLOG(qpair, "sq_tail is passing sq_head!\n");
}

if (!pqpair->flags.delay_cmd_submit) {
	nvme_pcie_qpair_ring_sq_doorbell(qpair);
}

The ownership sequence matters. The request starts as host-only state. Once copied into pqpair->cmd[pqpair->sq_tail], the controller can safely DMA-read it after the doorbell. The tracker remains host-owned and outstanding until a completion maps back to its cid.

The PCIe doorbell function is nvme_pcie_qpair_ring_sq_doorbell() in lib/nvme/nvme_pcie_internal.h:248. It writes the new SQ tail with spdk_mmio_write_4() at lib/nvme/nvme_pcie_internal.h:272. There is a memory barrier before the MMIO write at lib/nvme/nvme_pcie_internal.h:269; the host must make sure command memory is visible before telling the device to fetch it.

The doorbell code shows both the fused-command exception and the memory-ordering rule:

if (qpair->last_fuse == SPDK_NVME_IO_FLAGS_FUSE_FIRST) {
	/* This is first cmd of two fused commands - don't ring doorbell */
	return;
}

if (spdk_likely(need_mmio)) {
	spdk_wmb();
	pqpair->stat->sq_mmio_doorbell_updates++;
	g_thread_mmio_ctrlr = pctrlr;
	spdk_mmio_write_4(pqpair->sq_tdbl, pqpair->sq_tail);
	g_thread_mmio_ctrlr = NULL;
}

The write barrier is not decoration. If the MMIO doorbell became visible to the device before the SQE stores were visible, the controller could fetch an incomplete or stale command. The barrier makes the intended producer order explicit: fill SQE first, then publish the new tail.

Completion Flow

The simplified completion path:

controller
  DMA-writes CQE
  toggles / sets phase tag as appropriate

host poller
  reads CQE at cq_head
  checks phase bit
  looks up tracker by cid
  completes request callback
  advances cq_head
  rings CQ doorbell

The CQ doorbell function is nvme_pcie_qpair_ring_cq_doorbell() in lib/nvme/nvme_pcie_internal.h:277. It writes the consumed CQ head to the controller at lib/nvme/nvme_pcie_internal.h:295.

SPDK rings the CQ doorbell after it has consumed one or more completions:

if (num_completions > 0) {
	pqpair->stat->completions += num_completions;
	nvme_pcie_qpair_ring_cq_doorbell(qpair);
} else {
	pqpair->stat->idle_polls++;
}

That batching is legal because the CQ head doorbell communicates how much space has been freed, not which callback ran. The controller already wrote the CQEs. The host reads and processes them, then publishes the new head so the controller can reuse those CQ slots.

The phase bit solves a ring-buffer ambiguity. When the CQ wraps, slot 0 is reused. Without an extra bit, the host could not reliably tell whether a slot contains an old completion from the previous lap or a new completion from this lap. SPDK stores the expected phase in flags.phase at lib/nvme/nvme_pcie_internal.h:159.

SPDK initializes the expected phase during qpair reset in lib/nvme/nvme_pcie_common.c:

/* all head/tail vals are set to 0 */
pqpair->last_sq_tail = pqpair->sq_tail = pqpair->sq_head = pqpair->cq_head = 0;

/*
 * First time through the completion queue, HW will set phase
 *  bit on completions to 1.  So set this to 1 here, indicating
 *  we're looking for a 1 to know which entries have completed.
 *  we'll toggle the bit each time when the completion queue
 *  rolls over.
 */
pqpair->flags.phase = 1;
for (i = 0; i < pqpair->num_entries; i++) {
	pqpair->cpl[i].status.p = 0;
}

The actual polling loop reads the CQ slot at cq_head, compares its phase bit against the expected phase, prefetches the next tracker when possible, advances cq_head, toggles phase on wrap, and maps cid back to the tracker:

while (1) {
	cpl = &pqpair->cpl[pqpair->cq_head];

	if (!next_is_valid && cpl->status.p != pqpair->flags.phase) {
		break;
	}

	if (spdk_likely(pqpair->cq_head + 1 != pqpair->num_entries)) {
		next_cq_head = pqpair->cq_head + 1;
		next_phase = pqpair->flags.phase;
	} else {
		next_cq_head = 0;
		next_phase = !pqpair->flags.phase;
	}
	next_cpl = &pqpair->cpl[next_cq_head];
	next_is_valid = (next_cpl->status.p == next_phase);
	if (next_is_valid) {
		__builtin_prefetch(&pqpair->tr[next_cpl->cid]);
	}
if (spdk_unlikely(++pqpair->cq_head == pqpair->num_entries)) {
	pqpair->cq_head = 0;
	pqpair->flags.phase = !pqpair->flags.phase;
}

tr = &pqpair->tr[cpl->cid];
pqpair->sq_head = cpl->sqhd;

if (tr->req) {
	__builtin_prefetch(&tr->req->stailq);
	nvme_pcie_qpair_complete_tracker(qpair, tr, cpl, true);
}

nvme_pcie_qpair_complete_tracker() is where the transport hands the completed request back into the common request completion machinery. That path eventually invokes the user's spdk_nvme_cmd_cb unless the request is being retried, failed internally, or completed through a reset/error path. The important queue-machine point is that callback execution happens on the thread that calls spdk_nvme_qpair_process_completions(), not on a hidden interrupt thread in the normal polling model.

Diagram in prose:

CQ has 4 slots. Expected phase = 1.

lap 1:
  slot 0 phase 1 -> new
  slot 1 phase 1 -> new
  slot 2 phase 1 -> new
  slot 3 phase 1 -> new
  wrap; expected phase becomes 0

lap 2:
  slot 0 phase 0 -> new
  old phase 1 entries are ignored after expected phase flips

Doorbells And MMIO

Doorbells are not normal memory writes. They are MMIO writes to registers mapped from the PCIe device's BAR. MMIO writes can be expensive compared with ordinary cached stores. That is why batching and shadow doorbells exist.

SPDK models controller registers in include/spdk/nvme_spec.h:540 through include/spdk/nvme_spec.h:615. The doorbell array begins at include/spdk/nvme_spec.h:611. In the PCIe qpair code, SPDK optionally updates shadow doorbells and only performs MMIO when required (lib/nvme/nvme_pcie_internal.h:260 through lib/nvme/nvme_pcie_internal.h:274).

The NVMe spec treats invalid doorbell values as protocol errors, not harmless hints. A bad SQ tail can mean software tried to add entries to a full SQ; a bad CQ head can mean software tried to consume entries that were not there. In real driver code, the best defense is boring: keep the qpair single-owned, advance indexes only in the submission/completion paths, and do not let unrelated code write doorbells directly.

Misconception to kill: "Ringing the doorbell moves the command." It does not copy the command. The command is already in host memory. The doorbell tells the controller that the producer index changed.

Namespaces And Controllers

An NVMe controller is the command-processing entity. A namespace is a block address space exposed through that controller. A physical SSD may expose one namespace or many. Multipath and NVMe-oF can make this more complex, but the beginner model is:

controller
  admin qpair
  io qpair 1
  io qpair 2
  namespace 1: logical blocks
  namespace 2: logical blocks

An I/O command usually names the namespace in cmd.nsid and the LBA/range in command-specific dwords. The qpair is the transport path; the namespace is the storage object.

Queue Full, Backpressure, And -EAGAIN

Queues are finite. Trackers are finite. Requests can be temporarily impossible to submit even though the device is healthy. SPDK's common qpair layer turns some -EAGAIN returns into internal queueing (lib/nvme/nvme_qpair.c:1192 through lib/nvme/nvme_qpair.c:1197).

The wrapper is short, but it is a major semantic boundary:

rc = _nvme_qpair_submit_request(qpair, req);
if (rc == -EAGAIN) {
	STAILQ_INSERT_TAIL(&qpair->queued_req, req, stailq);
	req->queued = true;
	rc = 0;
}

return rc;

-EAGAIN from the PCIe transport usually means "no tracker was available right now." The common layer converts that into software queueing, so the caller sees success for "SPDK accepted ownership of this request." It does not mean the command was already copied into the hardware SQ.

Completion processing feeds that queue back into hardware. After the transport returns a positive completion count, the common layer attempts to resubmit the same number of queued requests:

/*
 * At this point, ret must represent the number of completions we reaped.
 * submit as many queued requests as we completed.
 */
if (ret > 0) {
	nvme_qpair_resubmit_requests(qpair, ret);
} else {
	_nvme_qpair_complete_abort_queued_reqs(qpair);
}

This matters operationally. A user-level submit API returning success may mean "accepted by SPDK for eventual transport submission," not necessarily "already placed into the hardware SQ." Completion callback is still the truth for command completion.

Timeouts, Aborts, And Resets

NVMe has explicit error paths:

  • A command can complete with an error status.
  • A qpair can fail or disconnect.
  • A command can time out in the host.
  • The host can issue an abort.
  • The controller or queue can be reset.
  • The bdev layer can reset above the NVMe layer.

The completion status includes dnr ("do not retry") and status code/type fields in include/spdk/nvme_spec.h:1506 through include/spdk/nvme_spec.h:1513. SPDK also has queued-request abort paths around lib/nvme/nvme_qpair.c:1222.

The public completion API also has explicit failed-qpair behavior:

if (spdk_unlikely(qpair->ctrlr->is_failed &&
		  nvme_qpair_get_state(qpair) != NVME_QPAIR_DISCONNECTING)) {
	if (qpair->ctrlr->is_removed) {
		nvme_qpair_set_state(qpair, NVME_QPAIR_DESTROYING);
		nvme_qpair_abort_all_queued_reqs(qpair);
		nvme_transport_qpair_abort_reqs(qpair);
	}
	return -ENXIO;
}

This is why recovery code must distinguish "no completions ready" from "qpair cannot make progress." A return value of 0 from spdk_nvme_qpair_process_completions() can be normal idleness. -ENXIO means the qpair or controller state has moved into a failure/disconnect path and callers need to stop treating the queue as a live submission path.

Beginner trap: an abort is itself a command and may race with normal completion. A command can complete just as the host decides it timed out. Correct code must tolerate late completions, failed aborts, and reset-driven cleanup.

CQ Full And Completion Flow Control

A controller cannot write infinite completions. If the host stops polling, finite CQ space and finite software request resources eventually prevent forward progress. That can stall progress even while earlier SQ entries were submitted correctly.

The source-grounded lesson is enough for this chapter: an SPDK application must drain completions so callbacks run, trackers are freed, queued requests can be resubmitted, and CQ head doorbells advance. lib/nvme/nvme_pcie_common.c:921 through lib/nvme/nvme_pcie_common.c:965 shows the PCIe completion loop checking the CQ phase bit, advancing cq_head, finding the tracker by cpl->cid, and completing the request. lib/nvme/nvme_qpair.c:901 through lib/nvme/nvme_qpair.c:907 then resubmits queued requests after completions are reaped.

SPDK's model is polling-first. doc/nvme.md:111 through doc/nvme.md:116 states that the application submits I/O and must poll each queue pair with outstanding I/O by calling spdk_nvme_qpair_process_completions().

The public header says the same API is non-blocking and calls the request callback for each completed command:

/**
 * Process any outstanding completions for I/O submitted on a queue pair.
 *
 * This call is non-blocking, i.e. it only processes completions that are ready
 * at the time of this function call. It does not wait for outstanding commands
 * to finish.
 *
 * For each completed command, the request's callback function will be called if
 * specified as non-NULL when the request was submitted.
 *
 * The caller must ensure that each queue pair is only used from one thread at a
 * time.
 */

Misconception to kill: "Polling is just busy waiting." In SPDK, polling is the completion engine. If you do not poll, callbacks do not run, buffers are not released, and higher layers may stop making progress.

Multipath And ANA Preview

Asymmetric Namespace Access (ANA) and multipath are advanced topics for later chapters, but they start from this chapter's model. If there are multiple controllers or paths to a namespace, each path has its own queues and state. A path can become optimized, non-optimized, inaccessible, or lost. The host must decide where to submit I/O and how to recover when a path's queues fail.

The key mental model: multipath is not one magic queue. It is multiple queue machines coordinated by policy.

Source Reading Exercise

Read:

  1. include/spdk/nvme_spec.h:1452 through include/spdk/nvme_spec.h:1537.
  2. include/spdk/nvme_spec.h:550 through include/spdk/nvme_spec.h:615.
  3. lib/nvme/nvme_pcie_internal.h:140 through lib/nvme/nvme_pcie_internal.h:203.
  4. lib/nvme/nvme_pcie_internal.h:248 through lib/nvme/nvme_pcie_internal.h:298.
  5. lib/nvme/nvme_pcie_common.c:45 through lib/nvme/nvme_pcie_common.c:58.
  6. lib/nvme/nvme_pcie_common.c:659 through lib/nvme/nvme_pcie_common.c:702.
  7. lib/nvme/nvme_pcie_common.c:872 through lib/nvme/nvme_pcie_common.c:996.
  8. lib/nvme/nvme_pcie_common.c:1672 through lib/nvme/nvme_pcie_common.c:1754.
  9. lib/nvme/nvme_qpair.c:834 through lib/nvme/nvme_qpair.c:910.
  10. lib/nvme/nvme_qpair.c:1171 through lib/nvme/nvme_qpair.c:1200.
  11. include/spdk/nvme.h:2155 through include/spdk/nvme.h:2181.
  12. doc/nvme.md:111 through doc/nvme.md:160.

Answer:

  • Which structure is 64 bytes and which is 16 bytes?
  • Which field matches a completion to a submitted command?
  • Why does SPDK need both virtual addresses and bus addresses for queues?
  • What memory barrier appears before ringing the SQ doorbell?
  • What does the common qpair layer do with -EAGAIN?
  • Which thread runs the completion callback in the polling model?
  • What can happen if the host stops advancing the CQ head?

Operational Lab

Build a simplified paper model with queue depth 4. This is a teaching model, not the complete NVMe full/empty rule. For the submission side, allow at most 4 outstanding commands by keeping an explicit software count. For the completion side, use the NVMe-specific phase-bit idea: when the CQ wraps, the expected phase changes, and an entry is new only when its phase matches what the host expects.

Submit commands with CIDs 10, 11, 12, 13, then complete them out of order as 11, 10, 13, 12. Keep a separate table that maps each CID to an outstanding request or callback. That table stands in for SPDK's tracker/request accounting, such as the tracker array indexed by command ID in lib/nvme/nvme_pcie_internal.h:138 through lib/nvme/nvme_pcie_internal.h:142.

Tasks:

  1. Track sq_tail after each submission.
  2. Track cq_head after each completion is consumed.
  3. Explain why out-of-order completion is fine.
  4. Explain why cid is necessary.
  5. Wrap the completion queue once and show when the phase bit changes.
  6. Name the parts that are NVMe-specific rather than generic ring-buffer conventions: SQ tail doorbell, CQ head doorbell, CQ phase bit, command ID, and SPDK request/tracker accounting.

Self-Check

  1. What is the difference between an SQ and a CQ?
  2. What does a doorbell write communicate?
  3. Why is a qpair usually single-thread owned in SPDK?
  4. Why does a CQE contain cid?
  5. What problem does the phase bit solve?
  6. Why must an SPDK application poll completions?
  7. What is the difference between controller and namespace?

References

  • Local source: include/spdk/nvme_spec.h.
  • Local source: lib/nvme/nvme_pcie_internal.h.
  • Local source: lib/nvme/nvme_pcie_common.c.
  • Local source: lib/nvme/nvme_qpair.c.
  • Local source: include/spdk/nvme.h.
  • Local SPDK documentation: doc/nvme.md.
  • Official SPDK NVMe Driver documentation: https://spdk.io/doc/nvme.html
  • Official SPDK NVMe I/O submission overview: https://spdk.io/doc/nvme_spec.html
  • NVM Express specifications landing page: https://nvmexpress.org/specifications/
  • NVM Express Base Specification page: https://nvmexpress.org/specification/nvm-express-base-specification/
  • Public NVM Express 1.0 PDF used for queue/doorbell/phase wording: https://nvmexpress.org/wp-content/uploads/NVM-Express-1_0-Gold.pdf