SPDK From First Principles

SPDK deep learning path

Chapter 17: NVMe Initiator Library

By the end of this chapter you should be able to explain SPDK's NVMe initiator as a queue-machine library. You should be able to find the source paths for probe, connect,...

Source: drafts/bdev-nvme/17-nvme-initiator-library.md

Reader Promise

By the end of this chapter you should be able to explain SPDK's NVMe initiator as a queue-machine library. You should be able to find the source paths for probe, connect, controller initialization, admin queue progress, I/O qpair allocation, qpair completion polling, poll groups, namespaces, detach, reset, and hotremove callbacks.

This chapter is about lib/nvme, not the bdev module. The bdev module uses this library, but the initiator library can also be used directly by applications.

Mental Model

NVMe is not "a disk API." It is a controller and queue protocol. The host allocates submission queues and completion queues, submits commands, rings doorbells or otherwise notifies the transport, and polls completions. SPDK's NVMe initiator library owns the userspace implementation of that model across PCIe and NVMe-oF transports.

The official SPDK NVMe driver guide describes the library as passive: it does not spawn worker threads on its own, and it only advances work when the application calls into it. That one fact explains much of the code in this chapter. A controller may be "connecting," a qpair may have outstanding commands, or a reset may be waiting for admin completions, but none of those states move unless an SPDK poller or application loop invokes the relevant progress function.

The NVM Express specifications explain why the library is organized around queues instead of around files or disks. The base NVMe model has an admin submission/completion queue pair for controller management and separate I/O queue pairs for data commands. NVMe-oF keeps the command model but changes how queues are created: fabrics queues are established with the Fabrics Connect command rather than PCIe MMIO queue registers and Create I/O Queue admin commands. SPDK hides those transport differences behind struct spdk_nvme_transport, while leaving the queue-machine shape visible in the public API.

The major objects are:

  • struct spdk_nvme_transport_id: where and how to connect.
  • struct spdk_nvme_probe_ctx: async probe/connect state.
  • struct spdk_nvme_ctrlr: one attached NVMe controller.
  • struct spdk_nvme_ns: one namespace under a controller.
  • struct spdk_nvme_qpair: one admin or I/O queue pair.
  • struct spdk_nvme_poll_group: a group of I/O qpairs polled together.
  • struct spdk_nvme_transport: transport-specific operations for PCIe, TCP, RDMA, etc.

Source anchors:

  • include/spdk/nvme.h:struct spdk_nvme_transport_id.
  • include/spdk/nvme.h:struct spdk_nvme_probe_ctx.
  • include/spdk/nvme.h:struct spdk_nvme_ctrlr.
  • include/spdk/nvme.h:struct spdk_nvme_qpair.
  • include/spdk/nvme.h:struct spdk_nvme_ns.
  • include/spdk/nvme.h:struct spdk_nvme_poll_group.
  • lib/nvme/nvme_internal.h:struct spdk_nvme_qpair.
  • lib/nvme/nvme_internal.h:struct spdk_nvme_ctrlr.
  • lib/nvme/nvme_internal.h:struct spdk_nvme_ns.
  • lib/nvme/nvme_internal.h:struct spdk_nvme_probe_ctx.

Why This Matters For diskengine/excloud

diskengine asks SPDK to attach local PCIe NVMe drives on storage nodes and remote NVMe-oF namespaces on baremetal nodes. In both cases, the bdev module eventually uses the NVMe initiator library to create controllers, namespaces, qpairs, and poll groups.

When something fails, the symptom may appear as "bdev missing" or "volume I/O hung," but the actual cause may be lower:

  • PCIe device not bound to VFIO.
  • NVMe-oF connect failed.
  • Admin queue never reached ready.
  • I/O qpair failed.
  • Controller reset is in progress.
  • Namespace changed or disappeared.
  • Poll group is not making completions.
  • Reconnect timeout policy expired.

The NVMe library is where those states are represented.

Probe And Connect

There are two common entry styles:

  • Probe: enumerate matching controllers and use callbacks to decide which to attach.
  • Connect: directly connect to one transport ID.

Public source anchors:

  • include/spdk/nvme.h:spdk_nvme_probe().
  • include/spdk/nvme.h:spdk_nvme_probe_ext().
  • include/spdk/nvme.h:spdk_nvme_connect().
  • include/spdk/nvme.h:spdk_nvme_probe_async_ext().
  • include/spdk/nvme.h:spdk_nvme_connect_async().
  • include/spdk/nvme.h:spdk_nvme_probe_poll_async().

Implementation source anchors:

  • lib/nvme/nvme.c:spdk_nvme_probe().
  • lib/nvme/nvme.c:spdk_nvme_probe_ext().
  • lib/nvme/nvme.c:spdk_nvme_connect().
  • lib/nvme/nvme.c:spdk_nvme_probe_async_ext().
  • lib/nvme/nvme.c:spdk_nvme_connect_async().
  • lib/nvme/nvme.c:spdk_nvme_probe_poll_async().
  • lib/nvme/nvme.c:nvme_probe_ctx_init().
  • lib/nvme/nvme.c:nvme_probe_internal().
  • lib/nvme/nvme.c:nvme_init_controllers().

Synchronous Probe

spdk_nvme_probe() calls spdk_nvme_probe_ext(). If no transport ID is provided, spdk_nvme_probe_ext() creates a PCIe wildcard transport ID. It then creates an async probe context with spdk_nvme_probe_async_ext() and drives it to completion with nvme_init_controllers().

Callbacks:

  • probe_cb: decide whether to attach a discovered controller and optionally modify controller options.
  • attach_cb: receive a ready controller.
  • attach_fail_cb: optional failure callback.
  • remove_cb: optional callback for controllers no longer present.

Direct Connect

spdk_nvme_connect() requires a transport ID. It initializes driver state, copies controller options safely, creates an async connect context with spdk_nvme_connect_async(), drives initialization with nvme_init_controllers(), then finds the attached controller by transport ID and host NQN.

Direct connect is common for NVMe-oF. Probe is common for PCIe discovery and for cases where multiple controllers may match.

Misconception to kill: "connect" still uses the probe machinery internally. It is direct-connect flavored probe, not an entirely separate stack.

The synchronous public functions are small wrappers over the async path. spdk_nvme_probe_ext() supplies a default PCIe wildcard when the caller does not pass a transport ID, creates an async probe context, then drives it until all controllers leave the initializing list. spdk_nvme_connect() is stricter because a direct connect without a transport ID has no target.

int
spdk_nvme_probe_ext(const struct spdk_nvme_transport_id *trid, void *cb_ctx,
		    spdk_nvme_probe_cb probe_cb, spdk_nvme_attach_cb attach_cb,
		    spdk_nvme_attach_fail_cb attach_fail_cb, spdk_nvme_remove_cb remove_cb)
{
	struct spdk_nvme_transport_id trid_pcie;
	struct spdk_nvme_probe_ctx *probe_ctx;

	if (trid == NULL) {
		memset(&trid_pcie, 0, sizeof(trid_pcie));
		spdk_nvme_trid_populate_transport(&trid_pcie, SPDK_NVME_TRANSPORT_PCIE);
		trid = &trid_pcie;
	}

	probe_ctx = spdk_nvme_probe_async_ext(trid, cb_ctx, probe_cb,
					      attach_cb, attach_fail_cb, remove_cb);
	if (!probe_ctx) {
		SPDK_ERRLOG("Create probe context failed\n");
		return -1;
	}

	return nvme_init_controllers(probe_ctx);
}

Source excerpt: lib/nvme/nvme.c:spdk_nvme_probe_ext().

The probe context owns the callbacks and the temporary controller lists during attach. The caller owns the final controller only after attach_cb fires or, for direct connect, after spdk_nvme_connect() returns a non-NULL controller.

struct spdk_nvme_probe_ctx {
	struct spdk_nvme_transport_id		trid;
	const struct spdk_nvme_ctrlr_opts	*opts;
	void					*cb_ctx;
	spdk_nvme_probe_cb			probe_cb;
	spdk_nvme_attach_cb			attach_cb;
	spdk_nvme_attach_fail_cb		attach_fail_cb;
	spdk_nvme_remove_cb			remove_cb;
	TAILQ_HEAD(, spdk_nvme_ctrlr)		init_ctrlrs;
	struct spdk_nvme_detach_ctx		failed_ctxs;
};

Source excerpt: lib/nvme/nvme_internal.h:struct spdk_nvme_probe_ctx.

Direct connect copies versioned controller options through a probe callback when options are supplied. That is why direct connect can reuse the same attach pipeline: nvme_probe_internal() still discovers or creates a controller, and the direct-connect probe callback simply says "yes" while applying the requested options.

static bool
nvme_connect_probe_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
		      struct spdk_nvme_ctrlr_opts *opts)
{
	struct spdk_nvme_ctrlr_opts *requested_opts = cb_ctx;

	assert(requested_opts);
	memcpy(opts, requested_opts, sizeof(*opts));

	return true;
}

Source excerpt: lib/nvme/nvme.c:nvme_connect_probe_cb().

Async Probe Context

The async path exists because controller initialization can take time and must be polled without blocking.

Source anchors:

  • lib/nvme/nvme.c:spdk_nvme_probe_async_ext().
  • lib/nvme/nvme.c:spdk_nvme_connect_async().
  • lib/nvme/nvme.c:spdk_nvme_probe_poll_async().

spdk_nvme_probe_poll_async():

  • Polls every controller in probe_ctx->init_ctrlrs.
  • Polls destruction of failed controllers.
  • Marks the global driver initialized when all init and failed lists are empty.
  • Frees the probe context and returns 0 when done.
  • Returns -EAGAIN while work remains.

This pattern is visible in the NVMe bdev module:

Source anchor: module/bdev/nvme/bdev_nvme.c:bdev_nvme_async_poll().

That poller calls spdk_nvme_probe_poll_async() until the attach work finishes.

The async poll function is the clearest example of SPDK's passive model. It does not sleep waiting for a controller. It walks the initializing controllers, polls each controller state machine, polls destruction for failed controllers, and returns -EAGAIN if the application must call it again.

int
spdk_nvme_probe_poll_async(struct spdk_nvme_probe_ctx *probe_ctx)
{
	struct spdk_nvme_ctrlr *ctrlr, *ctrlr_tmp;
	struct nvme_ctrlr_detach_ctx *detach_ctx, *detach_ctx_tmp;
	int rc;

	if (!spdk_process_is_primary() && probe_ctx->trid.trtype == SPDK_NVME_TRANSPORT_PCIE) {
		free(probe_ctx);
		return 0;
	}

	TAILQ_FOREACH_SAFE(ctrlr, &probe_ctx->init_ctrlrs, tailq, ctrlr_tmp) {
		nvme_ctrlr_poll_internal(ctrlr, probe_ctx);
	}

	TAILQ_FOREACH_SAFE(detach_ctx, &probe_ctx->failed_ctxs.head, link, detach_ctx_tmp) {
		rc = nvme_ctrlr_destruct_poll_async(detach_ctx->ctrlr, detach_ctx);
		if (rc == -EAGAIN) {
			continue;
		}
		TAILQ_REMOVE(&probe_ctx->failed_ctxs.head, detach_ctx, link);
		free(detach_ctx);
	}

	if (TAILQ_EMPTY(&probe_ctx->init_ctrlrs) && TAILQ_EMPTY(&probe_ctx->failed_ctxs.head)) {
		nvme_robust_mutex_lock(&g_spdk_nvme_driver->lock);
		g_spdk_nvme_driver->initialized = true;
		nvme_robust_mutex_unlock(&g_spdk_nvme_driver->lock);
		free(probe_ctx);
		return 0;
	}

	return -EAGAIN;
}

Source excerpt: lib/nvme/nvme.c:spdk_nvme_probe_poll_async().

Two practical details matter when debugging attach. First, PCIe probing in a non-primary process returns complete immediately because only the primary process monitors and attaches PCIe devices. Second, failed controller objects are not simply leaked or dropped; they are put through async destruction, and that destruction must also finish before the probe context is freed.

Controller State Machine

The controller object represents an attached NVMe controller and its admin queue. Initialization is a state machine.

Source anchors:

  • lib/nvme/nvme.c:nvme_ctrlr_poll_internal().
  • lib/nvme/nvme_ctrlr.c:nvme_ctrlr_process_init().
  • lib/nvme/nvme_ctrlr.c:NVME_CTRLR_STATE_CONNECT_ADMINQ.
  • lib/nvme/nvme_ctrlr.c:NVME_CTRLR_STATE_WAIT_FOR_CONNECT_ADMINQ.

One key point in the source: when the controller state is NVME_CTRLR_STATE_CONNECT_ADMINQ, SPDK asks the transport to connect the admin qpair. In NVME_CTRLR_STATE_WAIT_FOR_CONNECT_ADMINQ, it calls spdk_nvme_qpair_process_completions(ctrlr->adminq, 0) and watches the qpair state transition to connected/enabled.

Beginner mental model: the controller is not "ready" when memory for the object is allocated. It becomes ready after a series of admin queue and identify/configuration steps complete.

The controller state machine starts by connecting the admin qpair. The transition is transport-neutral: PCIe, RDMA, TCP, and other transports implement nvme_transport_ctrlr_connect_qpair(), while the controller logic waits for the qpair to reach a connected/enabled state.

case NVME_CTRLR_STATE_CONNECT_ADMINQ:
	rc = nvme_transport_ctrlr_connect_qpair(ctrlr, ctrlr->adminq);
	if (rc == 0) {
		nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_WAIT_FOR_CONNECT_ADMINQ,
				     NVME_TIMEOUT_INFINITE);
	} else {
		nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_ERROR, NVME_TIMEOUT_INFINITE);
	}
	break;

case NVME_CTRLR_STATE_WAIT_FOR_CONNECT_ADMINQ:
	spdk_nvme_qpair_process_completions(ctrlr->adminq, 0);

	switch (nvme_qpair_get_state(ctrlr->adminq)) {
	case NVME_QPAIR_CONNECTING:
		if (ctrlr->is_failed) {
			nvme_transport_ctrlr_disconnect_qpair(ctrlr, ctrlr->adminq);
			break;
		}
		break;
	case NVME_QPAIR_CONNECTED:
		nvme_qpair_set_state(ctrlr->adminq, NVME_QPAIR_ENABLED);
	case NVME_QPAIR_ENABLED:
		nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_READ_VS,
				     NVME_TIMEOUT_INFINITE);
		nvme_qpair_abort_queued_reqs(ctrlr->adminq);
		break;
	default:
		nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_ERROR, NVME_TIMEOUT_INFINITE);
		break;
	}
	break;

Source excerpt: lib/nvme/nvme_ctrlr.c:nvme_ctrlr_process_init().

After adminq connect, the same state machine reads controller registers or properties, disables and enables the controller when needed, identifies the controller and namespaces, configures asynchronous event requests, sets host features, and finally calls the transport ready hook. The shape mirrors the NVMe specifications: the controller is only ready for normal command submission after the host has completed queue setup, controller enable/ready handling, and identify/configuration work.

case NVME_CTRLR_STATE_TRANSPORT_READY:
	rc = nvme_transport_ctrlr_ready(ctrlr);
	if (rc) {
		NVME_CTRLR_ERRLOG(ctrlr, "Transport controller ready step failed: rc %d\n", rc);
		nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_ERROR, NVME_TIMEOUT_INFINITE);
	} else {
		nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_READY, NVME_TIMEOUT_INFINITE);
	}
	break;

case NVME_CTRLR_STATE_READY:
	NVME_CTRLR_DEBUGLOG(ctrlr, "Ctrlr already in ready state\n");
	return 0;

case NVME_CTRLR_STATE_ERROR:
	NVME_CTRLR_ERRLOG(ctrlr, "Ctrlr is in error state\n");
	return -1;

Source excerpt: lib/nvme/nvme_ctrlr.c:nvme_ctrlr_process_init().

Admin Queue

The admin queue is used for controller management: identify, get log page, set features, namespace management, async events, and fabrics connect.

Source anchors:

  • lib/nvme/nvme_ctrlr.c:nvme_ctrlr_process_init().
  • lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_process_admin_completions().
  • lib/nvme/nvme_qpair.c:spdk_nvme_qpair_process_completions().

Admin completions often drive state transitions. If admin queue progress stops, controller initialization, reset, namespace changes, and asynchronous event handling can stall.

A subtle point: many init states are "wait" states that advance only when admin completions are processed. The code avoids recursion if it is already running inside the admin completion path.

case NVME_CTRLR_STATE_READ_VS_WAIT_FOR_VS:
case NVME_CTRLR_STATE_READ_CAP_WAIT_FOR_CAP:
case NVME_CTRLR_STATE_CHECK_EN_WAIT_FOR_CC:
case NVME_CTRLR_STATE_WAIT_FOR_IDENTIFY:
case NVME_CTRLR_STATE_WAIT_FOR_CONFIGURE_AER:
case NVME_CTRLR_STATE_WAIT_FOR_SET_NUM_QUEUES:
case NVME_CTRLR_STATE_WAIT_FOR_IDENTIFY_NS:
case NVME_CTRLR_STATE_WAIT_FOR_HOST_ID:
	/*
	 * nvme_ctrlr_process_init() may be called from the completion context
	 * for the admin qpair. Avoid recursive calls for this case.
	 */
	if (!ctrlr->adminq->in_completion_context) {
		spdk_nvme_qpair_process_completions(ctrlr->adminq, 0);
	}
	break;

Source excerpt: lib/nvme/nvme_ctrlr.c:nvme_ctrlr_process_init().

I/O Qpairs

Applications submit namespace I/O on I/O qpairs, not on the admin queue.

Source anchors:

  • include/spdk/nvme.h:spdk_nvme_ctrlr_alloc_io_qpair().
  • lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_alloc_io_qpair().
  • lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_connect_io_qpair().
  • lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_disconnect_io_qpair().

spdk_nvme_ctrlr_alloc_io_qpair():

  • Locks the controller.
  • Requires controller state NVME_CTRLR_STATE_READY.
  • Copies default qpair options and user overrides.
  • Validates caller-provided SQ/CQ buffers if used.
  • Rejects incompatible interrupt and delayed-submit options.
  • Creates an I/O qpair.
  • Connects it unless create_only was requested.

Misconception to kill: allocating an I/O qpair can fail because the controller is resetting or initializing. It is not just memory allocation.

The allocation path proves that an I/O qpair is a controller resource, not just a heap object. SPDK takes the controller lock, rejects allocation unless the controller is ready, copies versioned user options, validates caller-owned SQ/CQ memory when supplied, and rejects option combinations that cannot work together.

struct spdk_nvme_qpair *
spdk_nvme_ctrlr_alloc_io_qpair(struct spdk_nvme_ctrlr *ctrlr,
			       const struct spdk_nvme_io_qpair_opts *user_opts,
			       size_t opts_size)
{
	struct spdk_nvme_qpair		*qpair = NULL;
	struct spdk_nvme_io_qpair_opts	opts;
	int				rc;

	nvme_ctrlr_lock(ctrlr);

	if (spdk_unlikely(ctrlr->state != NVME_CTRLR_STATE_READY)) {
		/* When controller is resetting or initializing, free_io_qids is deleted or not created yet.
		 * We can't create IO qpair in that case */
		goto unlock;
	}

	spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts));
	if (user_opts) {
		nvme_ctrlr_io_qpair_opts_copy(&opts, user_opts, spdk_min(opts.opts_size, opts_size));
	}

	if (ctrlr->opts.enable_interrupts && opts.delay_cmd_submit) {
		NVME_CTRLR_ERRLOG(ctrlr, "delay command submit cannot work with interrupts\n");
		goto unlock;
	}

Source excerpt: lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_alloc_io_qpair().

The create_only option is important for poll groups. Official SPDK API docs describe it as a way to allocate the qpair without performing the connect portion, so the application can add it to a poll group first and then connect it later. That is why spdk_nvme_poll_group_add() requires a disconnected qpair.

	qpair = nvme_ctrlr_create_io_qpair(ctrlr, &opts);

	if (qpair == NULL || opts.create_only == true) {
		goto unlock;
	}

	rc = spdk_nvme_ctrlr_connect_io_qpair(ctrlr, qpair);
	if (rc != 0) {
		NVME_CTRLR_ERRLOG(ctrlr, "nvme_transport_ctrlr_connect_io_qpair() failed\n");
		nvme_ctrlr_proc_remove_io_qpair(qpair);
		TAILQ_REMOVE(&ctrlr->active_io_qpairs, qpair, tailq);
		spdk_bit_array_set(ctrlr->free_io_qids, qpair->id);
		nvme_transport_ctrlr_delete_io_qpair(ctrlr, qpair);
		qpair = NULL;
		goto unlock;
	}

unlock:
	nvme_ctrlr_unlock(ctrlr);

	return qpair;
}

Source excerpt: lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_alloc_io_qpair().

Namespace Lifecycle

Namespaces are the NVMe units that look like block devices. The initiator library exposes namespace objects, while the bdev module turns namespaces into bdevs.

Source anchors:

  • include/spdk/nvme.h:spdk_nvme_ctrlr_get_ns().
  • include/spdk/nvme.h:spdk_nvme_ns_get_ctrlr().
  • lib/nvme/nvme_internal.h:struct spdk_nvme_ns.

The initiator library can also notice changed namespace lists through admin events and log pages. Higher layers must decide how to present additions/removals. The NVMe bdev module handles this in its namespace populate/depopulate path.

Command Submission And Completion

The namespace command APIs build NVMe commands and submit them on qpairs.

Source anchors:

  • include/spdk/nvme.h:spdk_nvme_ns_cmd_readv().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_readv_with_md().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_readv_ext().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_writev().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_writev_with_md().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_writev_ext().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_flush().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_write_zeroes().
  • lib/nvme/nvme_ns_cmd.c:nvme_ns_cmd_rw_ext().
  • lib/nvme/nvme_ns_cmd.c:nvme_ns_cmd_rwv_ext().
  • lib/nvme/nvme_ns_cmd.c:spdk_nvme_ns_cmd_flush().

Completion progress is explicit:

Source anchor: lib/nvme/nvme_qpair.c:spdk_nvme_qpair_process_completions().

This function:

  • Processes register operations and transport events for admin queues.
  • Detects failed or removed controllers.
  • Rejects work when qpair is not enabled except for connecting/disconnecting states.
  • Handles error injection queues.
  • Calls transport-specific completion processing.
  • Resubmits queued requests when possible.

Misconception to kill: if nobody polls completions, I/O will not finish. SPDK is poll-driven unless using interrupt integrations that still ultimately schedule completion processing.

The completion function is also where controller failure becomes visible to I/O callers. If the controller is failed or removed, qpair polling returns -ENXIO; if the admin queue sees a transport completion error, the controller is failed so other qpairs can observe the failure on their own polling path.

int32_t
spdk_nvme_qpair_process_completions(struct spdk_nvme_qpair *qpair, uint32_t max_completions)
{
	int32_t ret;
	struct nvme_request *req, *tmp;

	if (nvme_qpair_is_admin_queue(qpair)) {
		nvme_complete_register_operations(qpair);
		nvme_transport_ctrlr_process_transport_events(qpair->ctrlr);
	}

	if (spdk_unlikely(qpair->ctrlr->is_failed &&
			  nvme_qpair_get_state(qpair) != NVME_QPAIR_DISCONNECTING)) {
		if (qpair->ctrlr->is_removed) {
			nvme_qpair_set_state(qpair, NVME_QPAIR_DESTROYING);
			nvme_qpair_abort_all_queued_reqs(qpair);
			nvme_transport_qpair_abort_reqs(qpair);
		}
		return -ENXIO;
	}

	if (spdk_unlikely(!nvme_qpair_check_enabled(qpair) &&
			  !(nvme_qpair_get_state(qpair) == NVME_QPAIR_CONNECTING ||
			    nvme_qpair_get_state(qpair) == NVME_QPAIR_DISCONNECTING))) {
		return -ENXIO;
	}

Source excerpt: lib/nvme/nvme_qpair.c:spdk_nvme_qpair_process_completions().

The transport owns the mechanics of reading a CQ, receiving TCP PDUs, polling RDMA completions, or checking PCIe completion queues. The common layer wraps that transport result with completion-context bookkeeping and queued-request resubmission.

	qpair->in_completion_context = 1;
	ret = nvme_transport_qpair_process_completions(qpair, max_completions);
	if (ret < 0) {
		if (ret == -ENXIO && nvme_qpair_get_state(qpair) == NVME_QPAIR_DISCONNECTING) {
			ret = 0;
		} else {
			NVME_QPAIR_ERRLOG(qpair, "CQ transport error %d (%s)\n", ret, spdk_strerror(-ret));
			if (nvme_qpair_is_admin_queue(qpair)) {
				nvme_ctrlr_fail(qpair->ctrlr, false);
			}
		}
	}
	qpair->in_completion_context = 0;

	if (ret > 0) {
		nvme_qpair_resubmit_requests(qpair, ret);
	} else {
		_nvme_qpair_complete_abort_queued_reqs(qpair);
	}

	return ret;
}

Source excerpt: lib/nvme/nvme_qpair.c:spdk_nvme_qpair_process_completions().

Poll Groups

Polling each qpair individually is possible, but poll groups let SPDK group qpairs by transport and process completions together.

Source anchors:

  • include/spdk/nvme.h:spdk_nvme_poll_group_create().
  • include/spdk/nvme.h:spdk_nvme_poll_group_add().
  • include/spdk/nvme.h:spdk_nvme_poll_group_process_completions().
  • lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_create().
  • lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_add().
  • lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_process_completions().

spdk_nvme_poll_group_create() allocates the group, copies optional accel callbacks, creates an fd group for interrupt mode when supported, and initializes transport-group state.

spdk_nvme_poll_group_add() requires the qpair to be disconnected, validates interrupt compatibility, creates a transport poll group if needed, and delegates to the transport.

spdk_nvme_poll_group_process_completions() prevents reentrant polling, loops all transport poll groups, accumulates completions, and returns a negative error if any transport reports one.

The NVMe bdev module builds on this:

Source anchors:

  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_create_poll_group_cb().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_poll().

A poll group is a cross-transport container. The public spdk_nvme_poll_group has one transport poll group per transport represented by its qpairs. That lets the application call one function per SPDK thread or channel while each transport keeps its own batching and event logic.

int
spdk_nvme_poll_group_add(struct spdk_nvme_poll_group *group, struct spdk_nvme_qpair *qpair)
{
	struct spdk_nvme_transport_poll_group *tgroup;
	const struct spdk_nvme_transport *transport;
	int rc;

	if (nvme_qpair_get_state(qpair) != NVME_QPAIR_DISCONNECTED) {
		return -EINVAL;
	}

	if (!group->enable_interrupts_is_valid) {
		group->enable_interrupts_is_valid = true;
		group->enable_interrupts = qpair->ctrlr->opts.enable_interrupts;
		if (group->enable_interrupts) {
			rc = nvme_poll_group_add_disconnect_qpair_fd(group);
			if (rc != 0) {
				return rc;
			}
		}
	} else if (qpair->ctrlr->opts.enable_interrupts != group->enable_interrupts) {
		NVME_QPAIR_ERRLOG(qpair, "Queue pair %s interrupts cannot be added to poll group\n",
				  qpair->ctrlr->opts.enable_interrupts ? "without" : "with");
		return -EINVAL;
	}

Source excerpt: lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_add().

The first qpair added fixes whether the poll group is interrupt-enabled. Later qpairs must match because a single poll group cannot safely mix event-driven and pure polling assumptions. In poll mode, the same grouping idea still helps because the transport can batch completions across its qpairs.

	STAILQ_FOREACH(tgroup, &group->tgroups, link) {
		if (tgroup->transport == qpair->transport) {
			break;
		}
	}

	if (!tgroup) {
		transport = nvme_get_first_transport();
		while (transport != NULL) {
			if (transport == qpair->transport) {
				tgroup = nvme_transport_poll_group_create(transport);
				if (tgroup == NULL) {
					return -ENOMEM;
				}
				tgroup->group = group;
				STAILQ_INSERT_TAIL(&group->tgroups, tgroup, link);
				break;
			}
			transport = nvme_get_next_transport(transport);
		}
	}

	return tgroup ? nvme_transport_poll_group_add(tgroup, qpair) : -ENODEV;
}

Source excerpt: lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_add().

Processing completions on the group is deliberately non-reentrant. This matters because completion callbacks can submit more work, disconnect qpairs, or trigger higher-level error handling. The group accumulates successful completions but preserves the first transport error as the return value.

int64_t
spdk_nvme_poll_group_process_completions(struct spdk_nvme_poll_group *group,
		uint32_t completions_per_qpair, spdk_nvme_disconnected_qpair_cb disconnected_qpair_cb)
{
	struct spdk_nvme_transport_poll_group *tgroup;
	int64_t error_reason = 0, num_completions = 0;

	if (spdk_unlikely(disconnected_qpair_cb == NULL)) {
		return -EINVAL;
	}

	if (spdk_unlikely(group->in_process_completions)) {
		return 0;
	}
	group->in_process_completions = true;

	STAILQ_FOREACH(tgroup, &group->tgroups, link) {
		int64_t local_completions;

		local_completions = nvme_transport_poll_group_process_completions(tgroup, completions_per_qpair,
				    disconnected_qpair_cb);
		if (spdk_unlikely(local_completions < 0)) {
			if (!error_reason) {
				error_reason = local_completions;
			}
		} else {
			num_completions += local_completions;
		}
	}
	group->in_process_completions = false;

	return error_reason ? error_reason : num_completions;
}

Source excerpt: lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_process_completions().

Reset, Detach, And Hotremove

Detach source anchors:

  • include/spdk/nvme.h:spdk_nvme_detach().
  • include/spdk/nvme.h:spdk_nvme_detach_async().
  • include/spdk/nvme.h:spdk_nvme_detach_poll_async().
  • lib/nvme/nvme.c:spdk_nvme_detach().
  • lib/nvme/nvme.c:spdk_nvme_detach_async().
  • lib/nvme/nvme.c:spdk_nvme_detach_poll_async().

Controller shutdown and detach may require polling the admin queue and checking controller status.

Source anchor: lib/nvme/nvme_ctrlr.c:nvme_ctrlr_shutdown_poll_async().

Hotremove can surface as failed qpair processing, remove callbacks from probe, or bdev module hotplug logic. For PCIe, physical presence and VFIO ownership matter. For fabrics, disconnect/reconnect policy matters.

Reset is another synchronous wrapper over asynchronous pieces. It first disconnects the controller under the controller lock, marks I/O qpairs failed, drains admin completions until the old admin path reports -ENXIO, starts reconnect, and polls reconnect until it stops returning -EAGAIN.

int
spdk_nvme_ctrlr_reset(struct spdk_nvme_ctrlr *ctrlr)
{
	int rc;

	nvme_ctrlr_lock(ctrlr);

	rc = nvme_ctrlr_disconnect(ctrlr);
	if (rc == 0) {
		nvme_ctrlr_fail_io_qpairs(ctrlr);
	}

	nvme_ctrlr_unlock(ctrlr);

	if (rc != 0) {
		if (rc == -EBUSY) {
			rc = 0;
		}
		return rc;
	}

	while (1) {
		rc = spdk_nvme_ctrlr_process_admin_completions(ctrlr);
		if (rc == -ENXIO) {
			break;
		}
	}

	spdk_nvme_ctrlr_reconnect_async(ctrlr);

	while (true) {
		rc = spdk_nvme_ctrlr_reconnect_poll_async(ctrlr);
		if (rc != -EAGAIN) {
			break;
		}
	}

	return rc;
}

Source excerpt: lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_reset().

Hotremove starts as a controller failure, but SPDK separates "mark failed" from "complete every outstanding request." nvme_ctrlr_fail() sets controller flags; later qpair polling notices those flags and aborts queued/transport requests. This division is why a hotremoved device may not be fully cleaned up until pollers run.

void
nvme_ctrlr_fail(struct spdk_nvme_ctrlr *ctrlr, bool hot_remove)
{
	/*
	 * Set the flag here and leave the work failure of qpairs to
	 * spdk_nvme_qpair_process_completions().
	 */
	if (hot_remove) {
		ctrlr->is_removed = true;
	}

	if (ctrlr->is_failed) {
		NVME_CTRLR_NOTICELOG(ctrlr, "already in failed state\n");
		return;
	}

Source excerpt: lib/nvme/nvme_ctrlr.c:nvme_ctrlr_fail().

The PCIe hotplug monitor handles both explicit PCI events and devices that are already physically removed. It marks the controller failed under the driver/controller locks, then invokes the user's remove callback outside the driver lock so the application can stop I/O and detach without deadlocking the global driver state.

static int
_nvme_pcie_hotplug_monitor(struct spdk_nvme_probe_ctx *probe_ctx)
{
	struct spdk_nvme_ctrlr *ctrlr, *tmp;
	struct spdk_pci_event event;
	int rc = 0;

	if (g_spdk_nvme_driver->hotplug_fd >= 0) {
		while (spdk_pci_get_event(g_spdk_nvme_driver->hotplug_fd, &event) > 0) {
			_nvme_pcie_event_process(&event, probe_ctx->cb_ctx);
		}
	}

	TAILQ_FOREACH_SAFE(ctrlr, &g_spdk_nvme_driver->shared_attached_ctrlrs, tailq, tmp) {
		bool do_remove = false;
		struct nvme_pcie_ctrlr *pctrlr;

		if (ctrlr->trid.trtype != SPDK_NVME_TRANSPORT_PCIE) {
			continue;
		}

		pctrlr = nvme_pcie_ctrlr(ctrlr);
		if (spdk_pci_device_is_removed(pctrlr->devhandle)) {
			do_remove = true;
			rc = 1;
		}

		if (do_remove) {
			nvme_ctrlr_lock(ctrlr);
			nvme_ctrlr_fail(ctrlr, true);
			nvme_ctrlr_unlock(ctrlr);
			if (ctrlr->remove_cb) {
				nvme_robust_mutex_unlock(&g_spdk_nvme_driver->lock);
				ctrlr->remove_cb(ctrlr->cb_ctx, ctrlr);
				nvme_robust_mutex_lock(&g_spdk_nvme_driver->lock);
			}
		}
	}
	return rc;
}

Source excerpt: lib/nvme/nvme_pcie.c:_nvme_pcie_hotplug_monitor().

For fabrics, the failure often appears as transport completion errors, keep-alive failures, or disconnected qpairs rather than a PCIe removal event. The common response is the same from the upper layer's perspective: stop submitting on failed qpairs, poll or wait for disconnected qpair handling, and let the controller reset/reconnect policy decide whether the controller becomes ready again or is removed.

Prose Diagram

Imagine a controller as a box with one admin qpair at the top and multiple I/O qpairs below it. To the left is spdk_nvme_transport_id, which says PCIe address or fabrics address. To the right is a poll loop.

Probe/connect creates a probe context. The probe context creates controller objects. Controller initialization drives the admin qpair until the controller is ready. The application allocates I/O qpairs. Namespace commands go down I/O qpairs. The poll loop calls either spdk_nvme_qpair_process_completions() or spdk_nvme_poll_group_process_completions() and completions invoke user callbacks.

Edge Cases And Failure Modes

  • Probe callback rejects a controller: no attach callback for it.
  • Attach callback receives a controller that is ready, but later namespace changes still need handling.
  • Direct connect with bad transport ID: probe/connect context may fail or attach nothing.
  • Controller already attached: shared controller logic and transport ID comparison matter.
  • Controller not ready: spdk_nvme_ctrlr_alloc_io_qpair() returns NULL.
  • Interrupt mode mismatch: poll group rejects qpairs with incompatible interrupt settings.
  • Qpair failed or removed: completion polling can return -ENXIO.
  • Admin queue stuck: controller initialization, reset, or detach may not progress.
  • No polling: commands remain outstanding.
  • PCIe secondary process: probe polling has special behavior for non-primary processes.
  • User detaches while other threads use controller: API docs warn the application must ensure no other users remain.

Misconceptions To Kill

  • "SPDK NVMe is synchronous because spdk_nvme_connect() returns a controller." Internally it drives an async initialization process.
  • "A namespace is a bdev." No. Namespace is an NVMe library object. NVMe bdev maps namespace to SPDK bdev.
  • "I/O qpairs are global." They are resources tied to a controller and commonly allocated per thread or channel.
  • "Poll group is optional decoration." For high-scale initiator use, poll groups are central to efficient completion handling.
  • "Admin queue is only used at startup." It is also used for health, async events, log pages, reset, detach, and passthrough admin commands.

Source Reading Exercise

Read:

  1. lib/nvme/nvme.c:spdk_nvme_connect().
  2. lib/nvme/nvme.c:spdk_nvme_connect_async().
  3. lib/nvme/nvme.c:spdk_nvme_probe_poll_async().
  4. lib/nvme/nvme_ctrlr.c:nvme_ctrlr_process_init().
  5. lib/nvme/nvme_ctrlr.c:spdk_nvme_ctrlr_alloc_io_qpair().
  6. lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_process_completions().
  7. lib/nvme/nvme_qpair.c:spdk_nvme_qpair_process_completions().

Questions:

  • How does spdk_nvme_connect() reuse probe machinery?
  • What return value means async probe is still in progress?
  • Why can I/O qpair allocation fail when memory is available?
  • Which function must run to make completions happen?
  • Where is interrupt-mode compatibility checked for poll groups?

Operational Lab

Build a debug checklist for "NVMe-oF attach hangs":

  1. Was the transport ID parsed correctly?
  2. Did spdk_nvme_connect_async() return a probe context?
  3. Is spdk_nvme_probe_poll_async() being called repeatedly?
  4. Did an attach callback run?
  5. Did controller initialization reach ready?
  6. Is admin queue completion processing happening?
  7. Were I/O qpairs allocated and connected?
  8. Is a poll group processing completions?
  9. Did qpair failure reason become non-none?

For each item, write the source function you would instrument first.

Self-Check

  1. What is the difference between probe and connect?
  2. Why does async probe return -EAGAIN?
  3. What does the admin qpair do during controller initialization?
  4. Why are I/O qpairs separate from the admin qpair?
  5. What does a poll group contain?
  6. Why can completion polling return a negative value?
  7. What object represents a namespace in lib/nvme?
  8. Why is direct use of the NVMe library lower level than bdev use?

References

  • Local source: include/spdk/nvme.h.
  • Local source: lib/nvme/nvme.c.
  • Local source: lib/nvme/nvme_ctrlr.c.
  • Local source: lib/nvme/nvme_qpair.c.
  • Local source: lib/nvme/nvme_poll_group.c.
  • Local source: lib/nvme/nvme_ns_cmd.c.
  • SPDK NVMe Driver documentation: https://spdk.io/doc/nvme.html
  • SPDK nvme.h API reference: https://spdk.io/doc/nvme_8h.html
  • SPDK spdk_nvme_io_qpair_opts reference: https://spdk.io/doc/structspdk__nvme__io__qpair__opts.html
  • SPDK Interrupt Mode documentation: https://spdk.io/doc/interrupt_mode.html
  • NVM Express specifications landing page: https://nvmexpress.org/specifications/
  • NVM Express Base Specification, Revision 2.3: https://nvmexpress.org/wp-content/uploads/NVM-Express-Base-Specification-Revision-2.3-2025.08.01-Ratified.pdf
  • NVM Express over Fabrics 1.1a: https://nvmexpress.org/wp-content/uploads/NVMe-over-Fabrics-1.1a-2021.07.12-Ratified.pdf