SPDK From First Principles

SPDK deep learning path

Chapter 18: NVMe bdev Module

By the end of this chapter you should be able to trace `bdev_nvme_attach_controller` from JSON-RPC to NVMe connect to namespace bdev registration, then trace a read or write...

Source: drafts/bdev-nvme/18-nvme-bdev-module.md

Reader Promise

By the end of this chapter you should be able to trace bdev_nvme_attach_controller from JSON-RPC to NVMe connect to namespace bdev registration, then trace a read or write from bdev I/O to spdk_nvme_ns_cmd_* and completion polling. You should also understand the per-thread qpair model, poll groups, multipath/failover basics, reset, namespace changes, health/stat surfaces, and common failure modes.

This chapter connects Part 4's bdev model to Chapter 17's NVMe initiator library.

Mental Model

The NVMe bdev module adapts NVMe controllers and namespaces into SPDK bdevs.

At the bottom:

  • lib/nvme owns controllers, namespaces, qpairs, commands, and completions.

In the middle:

  • module/bdev/nvme owns controller groups, namespace-to-bdev mapping, per-thread bdev channels, NVMe qpair channels, poll groups, multipath, reconnect, and bdev-specific options.

At the top:

  • Applications see ordinary bdevs named from the attach controller base name and namespace ID.

Key source anchors:

  • module/bdev/nvme/bdev_nvme.c:nvme_if.
  • module/bdev/nvme/bdev_nvme.c:struct nvme_bdev_io.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_bdev_ctrlr.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_bdev.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_qpair.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_ctrlr_channel.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_io_path.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_bdev_channel.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_poll_group.

Why This Matters For diskengine/excloud

diskengine uses NVMe bdevs in two major ways:

  • Storage node: attach local PCIe NVMe SSDs and build higher-level storage on top.
  • Baremetal or consumer node: attach remote NVMe-oF namespaces, often with multipath/failover policy.

The NVMe bdev module is where control-plane terms such as "controller name," "transport ID," "host NQN," "subsystem NQN," "reconnect delay," "controller loss timeout," "multipath," and "namespace bdev name" become source-level state.

When a diskengine operation says "attach this remote disk," this module is usually the bridge from JSON-RPC intent to real queues.

Module Registration And Options

The module is registered as nvme.

Source anchors:

  • module/bdev/nvme/bdev_nvme.c:nvme_if.
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_get_ctx_size().
  • module/bdev/nvme/bdev_nvme.c:SPDK_BDEV_MODULE_REGISTER(nvme, &nvme_if).

The module sets:

  • module_init = bdev_nvme_init.
  • module_fini = bdev_nvme_fini.
  • async_fini = true.
  • config_json = bdev_nvme_config_json.
  • get_ctx_size = bdev_nvme_get_ctx_size.

Per-I/O context is struct nvme_bdev_io.

Source anchor: module/bdev/nvme/bdev_nvme.c:struct nvme_bdev_io.

The registration is deliberately small. The bdev core does not need to know about NVMe transports, namespaces, ANA, or reconnect. It only needs a module name, lifecycle hooks, a config dump hook, and the size of the per-I/O private context that the bdev core will reserve for this module.

static int
bdev_nvme_get_ctx_size(void)
{
	return sizeof(struct nvme_bdev_io);
}

static struct spdk_bdev_module nvme_if = {
	.name = "nvme",
	.async_fini = true,
	.module_init = bdev_nvme_init,
	.module_fini = bdev_nvme_fini,
	.config_json = bdev_nvme_config_json,
	.get_ctx_size = bdev_nvme_get_ctx_size,

};
SPDK_BDEV_MODULE_REGISTER(nvme, &nvme_if)

The private I/O context is where the module remembers the NVMe-specific state that does not belong in the generic struct spdk_bdev_io. The most important field for the main read/write path is io_path: after path selection it points to the namespace and qpair that will carry this command. The same context also stores SGL iteration state, NVMe completion status, retry state, zone-report state, and optional extended command options.

struct nvme_bdev_io {
	struct iovec *iovs;
	int iovcnt;
	int iovpos;
	uint32_t iov_offset;

	/* I/O path the current I/O or admin passthrough is submitted on, or the I/O path
	 *  being reset in a reset I/O.
	 */
	struct nvme_io_path *io_path;

	/** Saved status for admin passthru completion event, PI error verification, or intermediate compare-and-write status */
	struct spdk_nvme_cpl cpl;

	/** Extended IO opts passed by the user to bdev layer and mapped to NVME format */
	struct spdk_nvme_ns_cmd_ext_io_opts ext_opts;

	/* How many times the current I/O was retried. */
	int32_t retry_count;

	/** Expiration value in ticks to retry the current I/O. */
	uint64_t retry_ticks;

	/* Current tsc at submit time. */
	uint64_t submit_tsc;
};

That context stores:

  • SGL iteration state.
  • Current I/O path.
  • NVMe completion status.
  • Extended I/O options.
  • fused-command state.
  • retry count and retry time.
  • zone-report state.
  • submit timestamp.

RPC Attach Controller

The main user-facing RPC is bdev_nvme_attach_controller.

Source anchors:

  • module/bdev/nvme/bdev_nvme_rpc.c:struct rpc_bdev_nvme_attach_controller.
  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_attach_controller_decoders.
  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_attach_controller().
  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_attach_controller_done().
  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_attach_controller_examined().
  • module/bdev/nvme/bdev_nvme_rpc.c:SPDK_RPC_REGISTER(\"bdev_nvme_attach_controller\", ...).

The RPC:

  1. Allocates request context.
  2. Gets default NVMe controller options with spdk_nvme_ctrlr_get_default_ctrlr_opts().
  3. Gets default bdev NVMe controller options with spdk_bdev_nvme_get_default_ctrlr_opts().
  4. Decodes JSON.
  5. Parses trtype, traddr, optional adrfam, trsvcid, subnqn, host fields, digest/auth fields, timeout fields, and multipath mode.
  6. Checks duplicate controller/path cases.
  7. Validates num_io_queues.
  8. Calls spdk_bdev_nvme_create().
  9. Waits for bdev examine before returning the created bdev names.

Misconception to kill: bdev_nvme_attach_controller does not itself create queues and bdevs inline. It validates RPC input and hands off to the async attach path.

The front half of the RPC is mostly translation and guardrails. JSON gives the module strings such as trtype, traddr, trsvcid, subnqn, and host fields. The NVMe library wants a struct spdk_nvme_transport_id plus controller option structures. That conversion happens before any connect attempt, so bad input is rejected while the RPC is still on the caller-facing path.

spdk_nvme_ctrlr_get_default_ctrlr_opts(&ctx->req.drv_opts, sizeof(ctx->req.drv_opts));
spdk_bdev_nvme_get_default_ctrlr_opts(&ctx->req.bdev_opts);
ctx->req.multipath = BDEV_NVME_MP_MODE_MULTIPATH;
ctx->req.max_bdevs = DEFAULT_MAX_BDEVS_PER_RPC;

if (spdk_json_decode_object(params, rpc_bdev_nvme_attach_controller_decoders,
			    SPDK_COUNTOF(rpc_bdev_nvme_attach_controller_decoders),
			    &ctx->req)) {
	SPDK_ERRLOG("spdk_json_decode_object failed\n");
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR,
					 "spdk_json_decode_object failed");
	goto cleanup;
}

/* Parse trstring */
rc = spdk_nvme_transport_id_populate_trstring(&trid, ctx->req.trtype);
if (rc < 0) {
	SPDK_ERRLOG("Failed to parse trtype: %s\n", ctx->req.trtype);
	spdk_jsonrpc_send_error_response_fmt(request, -EINVAL, "Failed to parse trtype: %s",
					     ctx->req.trtype);
	goto cleanup;
}

The duplicate-path checks are not cosmetic. One logical NVMe bdev controller name may be reused to add failover or multipath paths, but the reused name must still refer to the same subsystem and host identity. Otherwise one SPDK bdev name would silently mix unrelated storage targets.

ctrlr = nvme_ctrlr_get_by_name(ctx->req.name);

if (ctrlr) {
	if (ctx->req.multipath == BDEV_NVME_MP_MODE_DISABLE) {
		spdk_jsonrpc_send_error_response_fmt(request, -EALREADY,
						     "A controller named %s already exists and multipath is disabled",
						     ctx->req.name);
		goto cleanup;
	}

	drv_opts = spdk_nvme_ctrlr_get_opts(ctrlr->ctrlr);
	ctrlr_trid = spdk_nvme_ctrlr_get_transport_id(ctrlr->ctrlr);

	if (strncmp(trid.subnqn, ctrlr_trid->subnqn, SPDK_NVMF_NQN_MAX_LEN) != 0) {
		spdk_jsonrpc_send_error_response_fmt(request, -EINVAL,
						     "A controller named %s already exists, but uses a different subnqn (%s)",
						     ctx->req.name, ctrlr_trid->subnqn);
		goto cleanup;
	}

	if (strncmp(ctx->req.drv_opts.hostnqn, drv_opts->hostnqn, SPDK_NVMF_NQN_MAX_LEN) != 0) {
		spdk_jsonrpc_send_error_response_fmt(request, -EINVAL,
						     "A controller named %s already exists, but uses a different hostnqn (%s)",
						     ctx->req.name, drv_opts->hostnqn);
		goto cleanup;
	}
}

After validation, the RPC stops being the owner of the attach mechanics. It passes the parsed transport ID, options, name array, and callback into the bdev module API. The callback waits for bdev examine before writing the RPC result, which is why the RPC can return names only after the bdev layer has seen the new devices.

rc = spdk_bdev_nvme_create(&trid, ctx->req.name, ctx->names, ctx->req.max_bdevs,
			   rpc_bdev_nvme_attach_controller_done, ctx, &ctx->req.drv_opts,
			   &ctx->req.bdev_opts);
if (rc) {
	spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc));
	goto cleanup;
}

return;

Programmatic Create Path

The bdev module public create API is:

Source anchors:

  • include/spdk/module/bdev/nvme.h:spdk_bdev_nvme_create().
  • module/bdev/nvme/bdev_nvme.c:spdk_bdev_nvme_create().

spdk_bdev_nvme_create():

  • Rejects duplicate transport ID/host NQN.
  • Validates controller name length.
  • Validates controller loss/reconnect/fast-fail parameters.
  • Allocates nvme_async_probe_ctx.
  • Copies base name, output names array, callback, transport ID, bdev options, and driver options.
  • Applies module-wide options such as transport retry count, keep-alive timeout, admin read-ANA behavior, TOS, and interrupt mode.
  • Resolves PSK or DH-HMAC-CHAP keys if configured.
  • Starts an async NVMe probe/connect and registers a poller to finish it.

Connect callbacks:

  • module/bdev/nvme/bdev_nvme.c:connect_attach_cb().
  • module/bdev/nvme/bdev_nvme.c:connect_set_failover_cb().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_async_poll().

bdev_nvme_async_poll() calls spdk_nvme_probe_poll_async() until the NVMe library attach process is done.

The public create path is the point where the bdev module becomes an NVMe initiator client. It validates the logical controller name, rejects an exact duplicate controller/host pair, copies options into an async probe context, and then calls spdk_nvme_connect_async(). The connect is asynchronous because NVMe fabrics connection setup, authentication, controller identify, and namespace discovery are not instantaneous, and SPDK does not block a reactor thread while that work proceeds.

if (nvme_ctrlr_get(trid, drv_opts->hostnqn) != NULL) {
	SPDK_ERRLOG("A controller with the provided trid (traddr: %s, hostnqn: %s) "
		    "already exists.\n", trid->traddr, drv_opts->hostnqn);
	return -EEXIST;
}

len = strnlen(base_name, SPDK_CONTROLLER_NAME_MAX);

if (len == 0 || len == SPDK_CONTROLLER_NAME_MAX) {
	SPDK_ERRLOG("controller name must be between 1 and %d characters\n", SPDK_CONTROLLER_NAME_MAX - 1);
	return -EINVAL;
}

if (bdev_opts != NULL &&
    !bdev_nvme_check_io_error_resiliency_params(bdev_opts->ctrlr_loss_timeout_sec,
		    bdev_opts->reconnect_delay_sec,
		    bdev_opts->fast_io_fail_timeout_sec)) {
	return -EINVAL;
}

The create function also decides whether a successful low-level connection should become a real attached controller path or just a failover transport ID for an existing controller. In multipath mode, every path stays connected and has qpairs. In failover mode, a secondary path can be validated and stored, then disconnected until it is needed.

if (nvme_bdev_ctrlr_get_by_name(base_name) == NULL || ctx->bdev_opts.multipath) {
	attach_cb = connect_attach_cb;
} else {
	attach_cb = connect_set_failover_cb;
}

nvme_ctrlr = nvme_ctrlr_get_by_name(ctx->base_name);
if (nvme_ctrlr  && nvme_ctrlr->opts.multipath != ctx->bdev_opts.multipath) {
	free_nvme_async_probe_ctx(ctx);
	return -EINVAL;
}

ctx->probe_ctx = spdk_nvme_connect_async(trid, &ctx->drv_opts, attach_cb);
if (ctx->probe_ctx == NULL) {
	SPDK_ERRLOG("No controller was found with provided trid (traddr: %s)\n", trid->traddr);
	free_nvme_async_probe_ctx(ctx);
	return -ENODEV;
}
ctx->poller = SPDK_POLLER_REGISTER(bdev_nvme_async_poll, ctx, 1000);

The async poller is intentionally small. The NVMe library owns progress for the probe/connect state machine, and the bdev module watches for the terminal state. If no attach callback fired, the module reports an attach failure. If namespace population already completed, the context can be freed.

static int
bdev_nvme_async_poll(void *arg)
{
	struct nvme_async_probe_ctx	*ctx = arg;
	int				rc;

	rc = spdk_nvme_probe_poll_async(ctx->probe_ctx);
	if (spdk_unlikely(rc != -EAGAIN)) {
		ctx->probe_done = true;
		spdk_poller_unregister(&ctx->poller);
		if (!ctx->ctrlr_attached) {
			ctx->reported_bdevs = 0;
			populate_namespaces_cb(ctx, -EIO);
		} else if (ctx->namespaces_populated) {
			free_nvme_async_probe_ctx(ctx);
		}
	}

	return SPDK_POLLER_BUSY;
}

Controller Grouping

The NVMe bdev module groups one or more NVMe controllers under a bdev controller name.

Source anchors:

  • module/bdev/nvme/bdev_nvme.h:struct nvme_bdev_ctrlr.
  • module/bdev/nvme/bdev_nvme.c:g_nvme_bdev_ctrlrs.
  • module/bdev/nvme/bdev_nvme.c:nvme_bdev_ctrlr_get_by_name().
  • module/bdev/nvme/bdev_nvme.c:nvme_bdev_ctrlr_create().

struct nvme_bdev_ctrlr contains:

  • name: logical controller name from RPC.
  • ctrlrs: one or more struct nvme_ctrlr paths.
  • bdevs: namespace bdevs created from those controllers.

This structure is what lets SPDK represent multipath or failover for one logical controller name.

There are three different "controller" ideas that beginners often collapse into one:

controller.

path. It holds bdev-specific options, path IDs, ANA state, admin polling, reset/reconnect state, and namespace objects.

RPC. It aggregates one or more struct nvme_ctrlr paths and the namespace bdevs created from them.

  • struct spdk_nvme_ctrlr is the NVMe library object for one connected NVMe
  • struct nvme_ctrlr is the bdev module's wrapper around one such controller
  • struct nvme_bdev_ctrlr is the logical bdev controller group named by the
struct nvme_bdev_ctrlr {
	char				*name;
	TAILQ_HEAD(, nvme_ctrlr)	ctrlrs;
	TAILQ_HEAD(, nvme_bdev)		bdevs;
	TAILQ_ENTRY(nvme_bdev_ctrlr)	tailq;
};

struct nvme_bdev {
	struct spdk_bdev			disk;
	uint32_t				nsid;
	struct nvme_bdev_ctrlr			*nbdev_ctrlr;
	pthread_mutex_t				mutex;
	int					ref;
	enum spdk_bdev_nvme_multipath_policy	mp_policy;
	enum spdk_bdev_nvme_multipath_selector	mp_selector;
	uint32_t				rr_min_io;
	TAILQ_HEAD(, nvme_ns)			nvme_ns_list;
	struct nvme_error_stat			*err_stat;
};

When a newly connected controller is attached to a logical controller group, the module either finds the existing group or creates one. A second controller path is allowed only if the controllers advertise multipath support and the controller IDs are not duplicated. The later namespace matching step decides whether each namespace becomes a new bdev or an alternate path to an existing bdev.

static bool
bdev_nvme_check_multipath(struct nvme_bdev_ctrlr *nbdev_ctrlr, struct spdk_nvme_ctrlr *ctrlr)
{
	struct nvme_ctrlr *tmp;
	const struct spdk_nvme_ctrlr_data *cdata, *tmp_cdata;

	cdata = spdk_nvme_ctrlr_get_data(ctrlr);

	if (!cdata->cmic.mctrs) {
		SPDK_ERRLOG("Ctrlr%u does not support multipath.\n", cdata->cntlid);
		return false;
	}

	TAILQ_FOREACH(tmp, &nbdev_ctrlr->ctrlrs, tailq) {
		tmp_cdata = spdk_nvme_ctrlr_get_data(tmp->ctrlr);
		if (!tmp_cdata->cmic.mctrs || cdata->cntlid == tmp_cdata->cntlid) {
			return false;
		}
	}

	return true;
}

Edge case: when adding another path to an existing controller name, the module checks that multipath/failover configuration, host NQN, subnqn, and path details are compatible.

Source anchors:

  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_attach_controller().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_check_multipath().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_add_secondary_trid().

Namespace To bdev Mapping

Once a controller is attached, namespaces are populated and mapped to bdevs.

Source anchors:

  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespaces().
  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespace().
  • module/bdev/nvme/bdev_nvme.c:nvme_bdev_create().
  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespace_done().
  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespaces_done().

nvme_bdev_create():

  • Allocates struct nvme_bdev.
  • Builds the embedded struct spdk_bdev with nbdev_create().
  • Registers an io_device for the NVMe bdev.
  • Links the namespace to the bdev.
  • Links the bdev to the controller group.
  • Calls spdk_bdev_register().

If a bdev for the namespace already exists, nvme_ctrlr_populate_namespace() adds the namespace path to the existing bdev instead of creating a duplicate.

Namespace population runs on the app thread because it mutates global bdev registration state. For each namespace, the module first applies ANA information if it has an ANA log page, then looks up an existing bdev with the same NSID in the logical controller group. A miss means "new bdev." A hit means "additional namespace path for an existing bdev."

static void
nvme_ctrlr_populate_namespace(struct nvme_ctrlr *nvme_ctrlr, struct nvme_ns *nvme_ns)
{
	struct nvme_bdev	*bdev;
	int			rc = 0;

	if (nvme_ctrlr->ana_log_page != NULL) {
		bdev_nvme_parse_ana_log_page(nvme_ctrlr, nvme_ns_set_ana_state, nvme_ns);
	}

	bdev = nvme_bdev_ctrlr_get_bdev(nvme_ctrlr->nbdev_ctrlr, nvme_ns->id);
	if (bdev == NULL) {
		rc = nvme_bdev_create(nvme_ctrlr, nvme_ns);
	} else {
		rc = nvme_bdev_add_ns(bdev, nvme_ns);
		if (rc == 0) {
			return;
		}
	}

	nvme_ctrlr_populate_namespace_done(nvme_ns, rc);
}

Creating the bdev is where the namespace ID becomes an SPDK block-device identity. The embedded spdk_bdev gets initialized, the nvme_bdev is registered as an io_device so it can have per-thread channels, and finally the generic bdev layer is told about it with spdk_bdev_register().

static int
nvme_bdev_create(struct nvme_ctrlr *nvme_ctrlr, struct nvme_ns *nvme_ns)
{
	struct nvme_bdev *nbdev;
	struct nvme_bdev_ctrlr *nbdev_ctrlr = nvme_ctrlr->nbdev_ctrlr;
	int rc;

	assert(spdk_thread_is_app_thread(NULL));

	nbdev = nvme_bdev_alloc();
	if (nbdev == NULL) {
		return -ENOMEM;
	}

	rc = nbdev_create(&nbdev->disk, nbdev_ctrlr->name, nvme_ctrlr->ctrlr,
			  nvme_ns->ns, &nvme_ctrlr->opts, nbdev);
	if (rc != 0) {
		nvme_bdev_free(nbdev);
		return rc;
	}

	spdk_io_device_register(nbdev, bdev_nvme_create_bdev_channel_cb,
				bdev_nvme_destroy_bdev_channel_cb,
				sizeof(struct nvme_bdev_channel), nbdev->disk.name);

	nvme_ns->bdev = nbdev;
	nbdev->nsid = nvme_ns->id;
	TAILQ_INSERT_TAIL(&nbdev->nvme_ns_list, nvme_ns, tailq);

	nbdev->nbdev_ctrlr = nbdev_ctrlr;
	TAILQ_INSERT_TAIL(&nbdev_ctrlr->bdevs, nbdev, tailq);

	rc = spdk_bdev_register(&nbdev->disk);
	/* Error cleanup and success return follow in the full source. */

The RPC response is an array because namespace population may produce more than one bdev. The module fills the caller-provided name array only after namespace population has finished, and it fails if the caller's max_bdevs cap is too small to report all created names.

/*
 * Report the new bdevs that were created in this call.
 * There can be more than one bdev per NVMe controller.
 */
j = 0;

RB_FOREACH(nvme_ns, nvme_ns_tree, &nvme_ctrlr->namespaces) {
	nvme_bdev = nvme_ns->bdev;
	if (j < ctx->max_bdevs) {
		ctx->names[j] = nvme_bdev->disk.name;
		j++;
	} else {
		NVME_CTRLR_ERRLOG(nvme_ctrlr,
				  "Maximum number of namespaces supported per NVMe controller is %du. "
				  "Unable to return all names of created bdevs\n",
				  ctx->max_bdevs);
		ctx->reported_bdevs = 0;
		populate_namespaces_cb(ctx, -ERANGE);
		return;
	}
}

Namespace removal source anchors:

  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_depopulate_namespace().
  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_depopulate_namespace_done().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_delete_io_path_done().

If the last namespace path for a bdev disappears, the module unregisters the bdev.

Misconception to kill: one NVMe controller attach can create multiple bdevs, one per namespace. The RPC returns an array of bdev names for that reason.

Per-Thread qpair And Channel Model

The bdev layer asks the module for an I/O channel:

Source anchor: module/bdev/nvme/bdev_nvme.c:bdev_nvme_get_io_channel().

The NVMe bdev channel is:

Source anchor: module/bdev/nvme/bdev_nvme.h:struct nvme_bdev_channel.

It stores:

  • Current I/O path.
  • Multipath policy and selector.
  • Round-robin state.
  • List of available I/O paths.
  • Retry I/O list and retry poller.
  • Resetting flag.

Poll group state is:

Source anchor: module/bdev/nvme/bdev_nvme.h:struct nvme_poll_group.

It stores:

  • struct spdk_nvme_poll_group *group.
  • accel channel.
  • poller.
  • qpair list.
  • interrupt state.
  • spin-time stats.

Poll group creation:

Source anchor: module/bdev/nvme/bdev_nvme.c:bdev_nvme_create_poll_group_cb().

It calls spdk_nvme_poll_group_create(), registers bdev_nvme_poll() as the poller, and sets up interrupt integration when enabled.

Completion polling:

Source anchor: module/bdev/nvme/bdev_nvme.c:bdev_nvme_poll().

It calls spdk_nvme_poll_group_process_completions() and checks disconnected qpairs.

Beginner mental model: an NVMe bdev channel is the bdev-facing per-thread object. Under it are one or more NVMe I/O paths, each pointing at a namespace and a qpair. The poll group is the per-thread completion engine for those qpairs.

The important ownership split is:

  • The bdev channel belongs to one nvme_bdev on one SPDK thread.
  • The controller channel belongs to one nvme_ctrlr on one SPDK thread.
  • The qpair hangs off the controller channel.
  • The I/O path connects the bdev channel to a namespace and that qpair.
struct nvme_qpair {
	struct nvme_ctrlr		*ctrlr;
	struct spdk_nvme_qpair		*qpair;
	struct nvme_poll_group		*group;
	struct nvme_ctrlr_channel	*ctrlr_ch;
	TAILQ_HEAD(, nvme_io_path)	io_path_list;
	TAILQ_ENTRY(nvme_qpair)		tailq;
};

struct nvme_io_path {
	struct nvme_ns			*nvme_ns;
	struct nvme_qpair		*qpair;
	STAILQ_ENTRY(nvme_io_path)	stailq;
	struct nvme_bdev_channel	*nbdev_ch;
	TAILQ_ENTRY(nvme_io_path)	tailq;
	struct spdk_bdev_io_stat	*stat;
};

struct nvme_bdev_channel {
	struct nvme_io_path			*current_io_path;
	enum spdk_bdev_nvme_multipath_policy	mp_policy;
	enum spdk_bdev_nvme_multipath_selector	mp_selector;
	uint32_t				rr_min_io;
	uint32_t				rr_counter;
	STAILQ_HEAD(, nvme_io_path)		io_path_list;
	TAILQ_HEAD(retry_io_head, nvme_bdev_io)	retry_io_list;
	struct spdk_poller			*retry_io_poller;
	bool					resetting;
};

When a bdev channel is created, the module walks the bdev's namespace-path list and adds an I/O path for each namespace. The call to spdk_get_io_channel() on nvme_ns->ctrlr is what creates or retrieves the per-thread controller channel, and from there the module obtains the qpair.

static int
_bdev_nvme_add_io_path(struct nvme_bdev_channel *nbdev_ch, struct nvme_ns *nvme_ns)
{
	struct nvme_io_path *io_path;
	struct spdk_io_channel *ch;
	struct nvme_ctrlr_channel *ctrlr_ch;
	struct nvme_qpair *nvme_qpair;

	io_path = nvme_io_path_alloc();
	if (io_path == NULL) {
		return -ENOMEM;
	}

	io_path->nvme_ns = nvme_ns;

	ch = spdk_get_io_channel(nvme_ns->ctrlr);
	if (ch == NULL) {
		nvme_io_path_free(io_path);
		return -ENOMEM;
	}

	ctrlr_ch = spdk_io_channel_get_ctx(ch);
	nvme_qpair = ctrlr_ch->qpair;
	io_path->qpair = nvme_qpair;
	TAILQ_INSERT_TAIL(&nvme_qpair->io_path_list, io_path, tailq);

	io_path->nbdev_ch = nbdev_ch;
	STAILQ_INSERT_TAIL(&nbdev_ch->io_path_list, io_path, stailq);
	bdev_nvme_clear_current_io_path(nbdev_ch);

	return 0;
}

The qpair itself is created from the controller channel. It also gets a poll group channel for the thread-wide NVMe completion engine. This is why one SPDK thread can submit without locks: its bdev channels, controller channels, qpairs, and poll group are all thread-local dynamic contexts.

static int
nvme_qpair_create(struct nvme_ctrlr *nvme_ctrlr, struct nvme_ctrlr_channel *ctrlr_ch)
{
	struct nvme_qpair *nvme_qpair;
	struct spdk_io_channel *pg_ch;
	int rc;

	nvme_qpair = calloc(1, sizeof(*nvme_qpair));
	if (!nvme_qpair) {
		return -1;
	}

	TAILQ_INIT(&nvme_qpair->io_path_list);
	nvme_qpair->ctrlr = nvme_ctrlr;
	nvme_qpair->ctrlr_ch = ctrlr_ch;

	pg_ch = spdk_get_io_channel(&g_nvme_bdev_ctrlrs);
	if (!pg_ch) {
		free(nvme_qpair);
		return -1;
	}

	nvme_qpair->group = spdk_io_channel_get_ctx(pg_ch);
	rc = bdev_nvme_create_qpair(nvme_qpair);
	/* Retry handling, group insertion, and ref accounting follow. */

Poll group creation registers the poller that will drive completions. In poll mode the poller runs periodically according to nvme_ioq_poll_period_us; in interrupt mode it also integrates with an fd group.

static int
bdev_nvme_create_poll_group_cb(void *io_device, void *ctx_buf)
{
	struct nvme_poll_group *group = ctx_buf;
	uint64_t period;
	int rc;

	TAILQ_INIT(&group->qpair_list);

	group->group = spdk_nvme_poll_group_create(group, &g_bdev_nvme_accel_fn_table);
	if (group->group == NULL) {
		return -1;
	}

	period = spdk_interrupt_mode_is_enabled() ? 0 : g_opts.nvme_ioq_poll_period_us;
	group->poller = SPDK_POLLER_REGISTER(bdev_nvme_poll, group, period);
	/* Interrupt integration and return handling follow. */

I/O Submission

The bdev function table points to:

Source anchors:

  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_submit_request_initial().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_submit_request().
  • module/bdev/nvme/bdev_nvme.c:_bdev_nvme_submit_request().

bdev_nvme_submit_request_initial() initializes retry tracking, then calls bdev_nvme_submit_request().

bdev_nvme_submit_request():

  • Stores submit timestamp.
  • Records trace.
  • Chooses an I/O path with bdev_nvme_find_io_path().
  • Fails non-admin I/O with -ENXIO if no path exists.
  • Calls _bdev_nvme_submit_request().

The first submission wrapper initializes retry accounting only once. A retried I/O re-enters bdev_nvme_submit_request() later with updated timing, but it does not need to be treated as a brand-new bdev I/O.

static void
bdev_nvme_submit_request_initial(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)
{
	struct nvme_bdev_io *nbdev_io = (struct nvme_bdev_io *)bdev_io->driver_ctx;

	/* Initialize our values of submit tsc and retry count here
	 * so that it doesn't interfere with the retry process
	 */
	nbdev_io->submit_tsc = 0;
	nbdev_io->retry_count = 0;

	bdev_nvme_submit_request(ch, bdev_io);
}

The next wrapper is where an ordinary bdev I/O becomes tied to a concrete NVMe path. If no path is available, regular reads and writes fail with -ENXIO. Admin commands are allowed to continue into admin-specific handling because they do not use the same optimal I/O path selection.

static void
bdev_nvme_submit_request(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)
{
	struct nvme_bdev_channel *nbdev_ch = spdk_io_channel_get_ctx(ch);
	struct nvme_bdev_io *nbdev_io = (struct nvme_bdev_io *)bdev_io->driver_ctx;

	if (spdk_likely(nbdev_io->submit_tsc == 0)) {
		nbdev_io->submit_tsc = spdk_bdev_io_get_submit_tsc(bdev_io);
	} else {
		nbdev_io->submit_tsc = spdk_get_ticks();
	}

	spdk_trace_record(TRACE_BDEV_NVME_IO_START, 0, 0, (uintptr_t)nbdev_io, (uintptr_t)bdev_io);
	nbdev_io->io_path = bdev_nvme_find_io_path(nbdev_ch);
	if (spdk_unlikely(!nbdev_io->io_path)) {
		if (!bdev_nvme_io_type_is_admin(bdev_io->type)) {
			bdev_nvme_io_complete(nbdev_io, -ENXIO);
			return;
		}
	}

	_bdev_nvme_submit_request(nbdev_ch, bdev_io);
}

_bdev_nvme_submit_request() switches on bdev_io->type and calls NVMe-specific helpers.

Relevant I/O helper source anchors:

  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_readv().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_writev().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_unmap().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_flush().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_get_buf_cb().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_admin_passthru().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_abort().

NVMe library command source anchors used by the module:

  • include/spdk/nvme.h:spdk_nvme_ns_cmd_readv_with_md().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_readv_ext().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_writev_with_md().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_writev_ext().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_flush().
  • include/spdk/nvme.h:spdk_nvme_ns_cmd_write_zeroes().
  • include/spdk/nvme.h:spdk_nvme_ctrlr_cmd_admin_raw().

Completion source anchors:

  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_io_complete_nvme_status().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_io_complete().

The NVMe completion status is preserved so upper layers can inspect NVMe status code type and status code through bdev error helpers.

The switch statement is the protocol adapter in miniature. It accepts generic bdev operation types and calls helpers that know how to build the corresponding NVMe command. Reset, admin passthrough, abort, and NSSR are special because they do not follow the normal read/write/unmap path.

switch (bdev_io->type) {
case SPDK_BDEV_IO_TYPE_READ:
	if (bdev_io->u.bdev.iovs && bdev_io->u.bdev.iovs[0].iov_base) {
		rc = bdev_nvme_readv(nbdev_io,
				     bdev_io->u.bdev.iovs,
				     bdev_io->u.bdev.iovcnt,
				     bdev_io->u.bdev.md_buf,
				     bdev_io->u.bdev.num_blocks,
				     bdev_io->u.bdev.offset_blocks,
				     bdev_io->u.bdev.dif_check_flags,
				     bdev_io->u.bdev.memory_domain,
				     bdev_io->u.bdev.memory_domain_ctx,
				     bdev_io->u.bdev.accel_sequence);
	} else {
		spdk_bdev_io_get_buf(bdev_io, bdev_nvme_get_buf_cb,
				     bdev_io->u.bdev.num_blocks * bdev->blocklen);
		rc = 0;
	}
	break;
case SPDK_BDEV_IO_TYPE_WRITE:
	rc = bdev_nvme_writev(nbdev_io,
			      bdev_io->u.bdev.iovs,
			      bdev_io->u.bdev.iovcnt,
			      bdev_io->u.bdev.md_buf,
			      bdev_io->u.bdev.num_blocks,
			      bdev_io->u.bdev.offset_blocks,
			      bdev_io->u.bdev.dif_check_flags,
			      bdev_io->u.bdev.memory_domain,
			      bdev_io->u.bdev.memory_domain_ctx,
			      bdev_io->u.bdev.accel_sequence,
			      bdev_io->u.bdev.nvme_cdw12,
			      bdev_io->u.bdev.nvme_cdw13);
	break;
case SPDK_BDEV_IO_TYPE_RESET:
	nbdev_io->io_path = NULL;
	bdev_nvme_reset_io(bdev->ctxt, nbdev_io);
	return;
}

That bdev->blocklen multiplication is in module/core code. A public caller looking through a descriptor may see a metadata-adjusted block size when the descriptor hides metadata; module internals still use the raw bdev geometry.

The read helper shows the final bridge into the NVMe library. By this point the module has a selected struct nvme_io_path; that gives it the NVMe namespace and qpair needed by spdk_nvme_ns_cmd_read*(). Single-iovec commands can use the simpler buffer form, while multi-iovec commands provide SGL callbacks. The extended single-iovec case uses spdk_nvme_ns_cmd_read_ext(), not the readv helper, because it does not need SGL reset/next callbacks.

static int
bdev_nvme_readv(struct nvme_bdev_io *bio, struct iovec *iov, int iovcnt,
		void *md, uint64_t lba_count, uint64_t lba, uint32_t flags,
		struct spdk_memory_domain *domain, void *domain_ctx,
		struct spdk_accel_sequence *seq)
{
	struct spdk_nvme_ns *ns = bio->io_path->nvme_ns->ns;
	struct spdk_nvme_qpair *qpair = bio->io_path->qpair->qpair;
	int rc;

	bio->iovs = iov;
	bio->iovcnt = iovcnt;
	bio->iovpos = 0;
	bio->iov_offset = 0;

	if (domain != NULL || seq != NULL) {
		bio->ext_opts.size = SPDK_SIZEOF(&bio->ext_opts, accel_sequence);
		bio->ext_opts.memory_domain = domain;
		bio->ext_opts.memory_domain_ctx = domain_ctx;
		bio->ext_opts.io_flags = flags;
		bio->ext_opts.metadata = md;
		bio->ext_opts.accel_sequence = seq;

		if (iovcnt == 1) {
			rc = spdk_nvme_ns_cmd_read_ext(ns, qpair, iov[0].iov_base,
						       lba, lba_count, bdev_nvme_readv_done,
						       bio, &bio->ext_opts);
		} else {
			rc = spdk_nvme_ns_cmd_readv_ext(ns, qpair, lba, lba_count,
							bdev_nvme_readv_done, bio,
							bdev_nvme_queued_reset_sgl,
							bdev_nvme_queued_next_sge,
							&bio->ext_opts);
		}
	} else if (iovcnt == 1) {
		rc = spdk_nvme_ns_cmd_read_with_md(ns, qpair, iov[0].iov_base,
						   md, lba, lba_count, bdev_nvme_readv_done,
						   bio, flags, 0, 0);
	} else {
		rc = spdk_nvme_ns_cmd_readv_with_md(ns, qpair, lba, lba_count,
						    bdev_nvme_readv_done, bio, flags,
						    bdev_nvme_queued_reset_sgl,
						    bdev_nvme_queued_next_sge, md, 0, 0);
	}
	/* Error logging and return follow. */

Completions are not interrupts by default. The bdev module polls the NVMe poll group, lets the NVMe library process completions for every qpair in that group, and clears cached paths if qpair processing reports failure. That is the loop that eventually calls the command completion callbacks such as bdev_nvme_readv_done().

static int
bdev_nvme_poll(void *arg)
{
	struct nvme_poll_group *group = arg;
	int64_t num_completions;

	num_completions = spdk_nvme_poll_group_process_completions(group->group, 0,
			  bdev_nvme_disconnected_qpair_cb);

	if (spdk_unlikely(num_completions < 0)) {
		bdev_nvme_check_io_qpairs(group);
	}

	return num_completions > 0 ? SPDK_POLLER_BUSY : SPDK_POLLER_IDLE;
}

Multipath, Failover, ANA, And I/O Paths

The bdev module's I/O path object is:

Source anchor: module/bdev/nvme/bdev_nvme.h:struct nvme_io_path.

It ties:

  • nvme_ns: namespace path.
  • qpair: qpair for that path.
  • nbdev_ch: bdev channel cache.
  • optional per-path stats.

Multipath selection is stored on struct nvme_bdev and copied/cache-managed on channels.

Source anchors:

  • module/bdev/nvme/bdev_nvme.h:struct nvme_bdev.
  • module/bdev/nvme/bdev_nvme.h:struct nvme_bdev_channel.
  • include/spdk/module/bdev/nvme.h:spdk_bdev_nvme_set_multipath_policy().

ANA state can influence path selection for NVMe-oF multipath. Namespace populate can parse ANA log information before adding namespace paths.

Source anchors:

  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespace().
  • module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_read_ana_log_page().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_parse_ana_log_page().

Misconception to kill: multipath does not mean every I/O is blindly sprayed across every controller. The module tracks path state, policy, selector, retry, ANA information, and qpair health.

The path selector first filters out paths that cannot carry I/O. A path is not available if its qpair is missing, failed, or resetting, or if the namespace is not accessible. Only after that health check does the module look at ANA state and multipath policy. In this source, "active" and "accessible" are distinct: nvme_ns_is_active() rejects a namespace whose ANA state is updating or whose NVMe namespace pointer is missing; nvme_ns_is_accessible() additionally requires ANA optimized or non-optimized state.

static inline bool
nvme_qpair_is_connected(struct nvme_qpair *nvme_qpair)
{
	if (spdk_unlikely(nvme_qpair->qpair == NULL)) {
		return false;
	}

	if (spdk_unlikely(spdk_nvme_qpair_get_failure_reason(nvme_qpair->qpair) !=
			  SPDK_NVME_QPAIR_FAILURE_NONE)) {
		return false;
	}

	if (spdk_unlikely(nvme_qpair->ctrlr_ch->reset_iter != NULL)) {
		return false;
	}

	return true;
}

static inline bool
nvme_io_path_is_available(struct nvme_io_path *io_path)
{
	if (spdk_unlikely(!nvme_qpair_is_connected(io_path->qpair))) {
		return false;
	}

	if (spdk_unlikely(!nvme_ns_is_accessible(io_path->nvme_ns))) {
		return false;
	}

	return true;
}

For active-passive or round-robin selection, _bdev_nvme_find_io_path() walks the per-channel path list in a circular fashion. It prefers ANA optimized paths, remembers a non-optimized path only as a fallback, and caches the selected path on the channel until policy or path health changes clear that cache.

static struct nvme_io_path *
_bdev_nvme_find_io_path(struct nvme_bdev_channel *nbdev_ch)
{
	struct nvme_io_path *io_path, *start, *non_optimized = NULL;

	start = nvme_io_path_get_next(nbdev_ch, nbdev_ch->current_io_path);

	io_path = start;
	do {
		if (spdk_likely(nvme_io_path_is_available(io_path))) {
			switch (io_path->nvme_ns->ana_state) {
			case SPDK_NVME_ANA_OPTIMIZED_STATE:
				nbdev_ch->current_io_path = io_path;
				return io_path;
			case SPDK_NVME_ANA_NON_OPTIMIZED_STATE:
				if (non_optimized == NULL) {
					non_optimized = io_path;
				}
				break;
			default:
				assert(false);
				break;
			}
		}
		io_path = nvme_io_path_get_next(nbdev_ch, io_path);
	} while (io_path != start);

	nbdev_ch->current_io_path = non_optimized;
	return non_optimized;
}

The queue-depth selector is a separate code path. It does not use the cached current path and does not just rotate through the list. It scans connected paths, skips namespaces that are not active, checks each qpair's outstanding request count, and chooses the lowest-depth optimized path if one exists. A non-optimized path is only the fallback.

static struct nvme_io_path *
_bdev_nvme_find_io_path_min_qd(struct nvme_bdev_channel *nbdev_ch)
{
	struct nvme_io_path *io_path;
	struct nvme_io_path *optimized = NULL, *non_optimized = NULL;
	uint32_t opt_min_qd = UINT32_MAX, non_opt_min_qd = UINT32_MAX;
	uint32_t num_outstanding_reqs;

	STAILQ_FOREACH(io_path, &nbdev_ch->io_path_list, stailq) {
		if (spdk_unlikely(!nvme_qpair_is_connected(io_path->qpair))) {
			continue;
		}

		if (spdk_unlikely(!nvme_ns_is_active(io_path->nvme_ns))) {
			continue;
		}

		num_outstanding_reqs =
			spdk_nvme_qpair_get_num_outstanding_reqs(io_path->qpair->qpair);
		switch (io_path->nvme_ns->ana_state) {
		case SPDK_NVME_ANA_OPTIMIZED_STATE:
			if (num_outstanding_reqs < opt_min_qd) {
				opt_min_qd = num_outstanding_reqs;
				optimized = io_path;
			}
			break;
		case SPDK_NVME_ANA_NON_OPTIMIZED_STATE:
			if (num_outstanding_reqs < non_opt_min_qd) {
				non_opt_min_qd = num_outstanding_reqs;
				non_optimized = io_path;
			}
			break;
		default:
			break;
		}
	}

	if (optimized != NULL) {
		return optimized;
	}

	return non_optimized;
}

bdev_nvme_find_io_path() dispatches between those selectors. Active-passive and round-robin use _bdev_nvme_find_io_path(). Queue-depth policy uses _bdev_nvme_find_io_path_min_qd().

if (nbdev_ch->mp_policy == BDEV_NVME_MP_POLICY_ACTIVE_PASSIVE ||
    nbdev_ch->mp_selector == BDEV_NVME_MP_SELECTOR_ROUND_ROBIN) {
	return _bdev_nvme_find_io_path(nbdev_ch);
} else {
	return _bdev_nvme_find_io_path_min_qd(nbdev_ch);
}

The policy setter updates the persistent nvme_bdev policy under the bdev mutex, then walks existing bdev channels and refreshes their cached policy. That walk matters because channels are per-thread objects, not a single shared structure.

void
spdk_bdev_nvme_set_multipath_policy(const char *name, enum spdk_bdev_nvme_multipath_policy policy,
				    enum spdk_bdev_nvme_multipath_selector selector, uint32_t rr_min_io,
				    spdk_bdev_nvme_set_multipath_policy_cb cb_fn, void *cb_arg)
{
	struct spdk_bdev *bdev;
	struct nvme_bdev *nbdev;
	int rc;

	switch (policy) {
	case BDEV_NVME_MP_POLICY_ACTIVE_PASSIVE:
		break;
	case BDEV_NVME_MP_POLICY_ACTIVE_ACTIVE:
		switch (selector) {
		case BDEV_NVME_MP_SELECTOR_ROUND_ROBIN:
			if (rr_min_io == UINT32_MAX) {
				rr_min_io = 1;
			} else if (rr_min_io == 0) {
				rc = -EINVAL;
				goto exit;
			}
			break;
		case BDEV_NVME_MP_SELECTOR_QUEUE_DEPTH:
			break;
		default:
			rc = -EINVAL;
			goto exit;
		}
		break;
	default:
		rc = -EINVAL;
		goto exit;
	}
	/* The bdev is opened, policy is stored, and channels are updated below. */

ANA is read from an NVMe log page. The bdev module allocates a DMA-capable log page buffer, accounts for descriptor alignment, and submits a Get Log Page command for SPDK_NVME_LOG_ASYMMETRIC_NAMESPACE_ACCESS. Later namespace population parses that page and tags each nvme_ns with its ANA state.

static int
nvme_ctrlr_init_ana_log_page(struct nvme_ctrlr *nvme_ctrlr,
			     struct nvme_async_probe_ctx *ctx)
{
	struct spdk_nvme_ctrlr *ctrlr = nvme_ctrlr->ctrlr;
	const struct spdk_nvme_ctrlr_data *cdata;
	uint32_t ana_log_page_size;

	cdata = spdk_nvme_ctrlr_get_data(ctrlr);

	ana_log_page_size = sizeof(struct spdk_nvme_ana_page) + cdata->nanagrpid *
			    sizeof(struct spdk_nvme_ana_group_descriptor) + cdata->mnan *
			    sizeof(uint32_t);

	nvme_ctrlr->ana_log_page = spdk_zmalloc(ana_log_page_size, 64, NULL,
						SPDK_ENV_NUMA_ID_ANY, SPDK_MALLOC_DMA);
	if (nvme_ctrlr->ana_log_page == NULL) {
		return -ENXIO;
	}

	nvme_ctrlr->max_ana_log_page_size = ana_log_page_size;
	nvme_ctrlr->probe_ctx = ctx;
	ana_log_page_size = nvme_ctrlr_get_ana_log_page_size(nvme_ctrlr);

	return spdk_nvme_ctrlr_cmd_get_log_page(ctrlr,
						SPDK_NVME_LOG_ASYMMETRIC_NAMESPACE_ACCESS,
						SPDK_NVME_GLOBAL_NS_TAG,
						nvme_ctrlr->ana_log_page,
						ana_log_page_size, 0,
						nvme_ctrlr_init_ana_log_page_done,
						nvme_ctrlr);
}

Reset And Reconnect

There are two layers of reset:

  • bdev core reset: freezes all bdev channels and submits an I/O of type SPDK_BDEV_IO_TYPE_RESET.
  • NVMe bdev reset: resets one or more NVMe controller paths and rebuilds qpairs.

Source anchors:

  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_io().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_freeze_bdev_channel().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_freeze_bdev_channel_done().
  • module/bdev/nvme/bdev_nvme.c:_bdev_nvme_reset_io().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_io_continue().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_io_complete().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_ctrlr().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_ctrlr_unsafe().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_create_qpair().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_destroy_qpairs().

The reset I/O freezes NVMe bdev channels, resets controller paths sequentially, and then unfreezes channels and aborts retry I/O. When a controller is already resetting, reset I/O can be queued to avoid fighting the app framework's reset strategy.

The reset entry point does not reset only the channel that submitted the reset I/O. It iterates all channels for the bdev and marks them resetting. That prevents new work from racing through stale path state while controller qpairs are being torn down and recreated.

static void
bdev_nvme_freeze_bdev_channel(struct nvme_bdev_channel_iter *i,
			      struct nvme_bdev *nbdev,
			      struct nvme_bdev_channel *nbdev_ch, void *ctx)
{
	nbdev_ch->resetting = true;

	nvme_bdev_for_each_channel_continue(i, 0);
}

static void
bdev_nvme_reset_io(struct nvme_bdev *nbdev, struct nvme_bdev_io *bio)
{
	NVME_BDEV_INFOLOG(nbdev, null_ctrlr, "reset_io %p started.\n", bio);
	nvme_bdev_for_each_channel(nbdev,
				   bdev_nvme_freeze_bdev_channel,
				   bio,
				   bdev_nvme_freeze_bdev_channel_done);
}

After channels are frozen, the reset path starts with the first I/O path on the submitting channel and resets controller paths one at a time. In multipath, at least one successful controller reset is enough to make the reset I/O succeed; if every path fails, the reset remains failed.

static void
bdev_nvme_freeze_bdev_channel_done(struct nvme_bdev *nbdev, void *ctx, int status)
{
	struct nvme_bdev_io *bio = ctx;
	struct spdk_bdev_io *bdev_io = spdk_bdev_io_from_ctx(bio);
	struct nvme_bdev_channel *nbdev_ch;
	struct nvme_io_path *io_path;
	int rc;

	nbdev_ch = spdk_io_channel_get_ctx(spdk_bdev_io_get_io_channel(bdev_io));

	/* Initialize with failed status. With multipath it is enough to have at least one successful
	 * nvme_ctrlr reset. If there is none, reset status will remain failed.
	 */
	bio->cpl.cdw0 = 1;

	/* Reset all nvme_ctrlrs of a bdev controller sequentially. */
	io_path = STAILQ_FIRST(&nbdev_ch->io_path_list);
	assert(io_path != NULL);

	rc = _bdev_nvme_reset_io(io_path, bio);
	if (rc != 0) {
		rc = (rc == -EALREADY) ? 0 : rc;
		bdev_nvme_reset_io_continue(bio, rc);
	}
}

The lower reset helper is careful around concurrency. If a controller reset is already in progress, a reset I/O from the app framework can be queued on that controller instead of starting a second overlapping reset. Otherwise the helper records a completion callback and sends the actual controller reset work to the app thread.

static int
_bdev_nvme_reset_io(struct nvme_io_path *io_path, struct nvme_bdev_io *bio)
{
	struct spdk_bdev_io *bdev_io = spdk_bdev_io_from_ctx(bio);
	struct nvme_bdev *nbdev = (struct nvme_bdev *)bdev_io->bdev->ctxt;
	struct nvme_ctrlr *nvme_ctrlr = io_path->qpair->ctrlr;
	spdk_msg_fn msg_fn;
	int rc;

	assert(bio->io_path == NULL);
	bio->io_path = io_path;

	pthread_mutex_lock(&nvme_ctrlr->mutex);
	rc = bdev_nvme_reset_ctrlr_unsafe(nvme_ctrlr, &msg_fn);
	if (rc == -EBUSY) {
		TAILQ_INSERT_TAIL(&nvme_ctrlr->pending_resets, bio, retry_link);
	}
	pthread_mutex_unlock(&nvme_ctrlr->mutex);

	if (rc == 0) {
		nvme_ctrlr->ctrlr_op_cb_fn = bdev_nvme_reset_io_continue;
		nvme_ctrlr->ctrlr_op_cb_arg = bio;
		spdk_thread_send_msg(spdk_thread_get_app_thread(), msg_fn, nvme_ctrlr);
		NVME_BDEV_INFOLOG(nbdev, nvme_ctrlr, "reset_io %p started resetting ctrlr.\n", bio);
	} else if (rc == -EBUSY) {
		rc = 0;
		NVME_BDEV_INFOLOG(nbdev, nvme_ctrlr, "reset_io %p was queued to ctrlr.\n", bio);
	}

	return rc;
}

Reconnect and failover use controller loss parameters from the attach options:

  • ctrlr_loss_timeout_sec.
  • reconnect_delay_sec.
  • fast_io_fail_timeout_sec.

Validation source anchor: module/bdev/nvme/bdev_nvme.c:bdev_nvme_check_io_error_resiliency_params().

Edge cases:

  • ctrlr_loss_timeout_sec == 0 means no reconnect delay/fast-fail timeout should be set.
  • ctrlr_loss_timeout_sec == -1 means keep trying indefinitely, but reconnect delay must be nonzero.
  • fast_io_fail_timeout_sec must not be less than reconnect delay.
  • finite controller loss timeout must not be less than reconnect delay or fast-fail timeout.

The resiliency validation encodes those rules before a connect attempt begins. That matters operationally: a bad timeout combination would otherwise create a controller whose failure behavior cannot be reasoned about.

static bool
bdev_nvme_check_io_error_resiliency_params(int32_t ctrlr_loss_timeout_sec,
		uint32_t reconnect_delay_sec,
		uint32_t fast_io_fail_timeout_sec)
{
	if (ctrlr_loss_timeout_sec < -1) {
		SPDK_ERRLOG("ctrlr_loss_timeout_sec can't be less than -1.\n");
		return false;
	} else if (ctrlr_loss_timeout_sec == -1) {
		if (reconnect_delay_sec == 0) {
			SPDK_ERRLOG("reconnect_delay_sec can't be 0 if ctrlr_loss_timeout_sec is not 0.\n");
			return false;
		} else if (fast_io_fail_timeout_sec != 0 &&
			   fast_io_fail_timeout_sec < reconnect_delay_sec) {
			SPDK_ERRLOG("reconnect_delay_sec can't be more than fast_io-fail_timeout_sec.\n");
			return false;
		}
	} else if (ctrlr_loss_timeout_sec != 0) {
		if (reconnect_delay_sec == 0) {
			SPDK_ERRLOG("reconnect_delay_sec can't be 0 if ctrlr_loss_timeout_sec is not 0.\n");
			return false;
		} else if (reconnect_delay_sec > (uint32_t)ctrlr_loss_timeout_sec) {
			SPDK_ERRLOG("reconnect_delay_sec can't be more than ctrlr_loss_timeout_sec.\n");
			return false;
		} else if (fast_io_fail_timeout_sec != 0) {
			if (fast_io_fail_timeout_sec < reconnect_delay_sec) {
				SPDK_ERRLOG("reconnect_delay_sec can't be more than fast_io_fail_timeout_sec.\n");
				return false;
			} else if (fast_io_fail_timeout_sec > (uint32_t)ctrlr_loss_timeout_sec) {
				SPDK_ERRLOG("fast_io_fail_timeout_sec can't be more than ctrlr_loss_timeout_sec.\n");
				return false;
			}
		}
	} else if (reconnect_delay_sec != 0 || fast_io_fail_timeout_sec != 0) {
		SPDK_ERRLOG("Both reconnect_delay_sec and fast_io_fail_timeout_sec must be 0 if ctrlr_loss_timeout_sec is 0.\n");
		return false;
	}

	return true;
}

Health, Stats, And Config

The NVMe bdev module exposes module options and controller information through RPC/config JSON paths.

Source anchors:

  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_set_options().
  • module/bdev/nvme/bdev_nvme_rpc.c:rpc_dump_nvme_bdev_controller_info().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_config_json().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_device_stat().
  • module/bdev/nvme/bdev_nvme.c:bdev_nvme_format_nvme_status().

Per-bdev error statistics can be enabled through module options. Path statistics can also be enabled. These are useful when a logical bdev has multiple paths and one path is unhealthy.

The controller information RPC surfaces the logical grouping that the attach path built. It writes the logical bdev controller name and then dumps every underlying nvme_ctrlr path in that group. This is the control-plane view that lets operators distinguish "one bdev controller with two paths" from "two unrelated controllers."

static void
rpc_dump_nvme_bdev_controller_info(struct nvme_bdev_ctrlr *nbdev_ctrlr, void *ctx)
{
	struct spdk_json_write_ctx	*w = ctx;
	struct nvme_ctrlr		*nvme_ctrlr;

	spdk_json_write_object_begin(w);
	spdk_json_write_named_string(w, "name", nbdev_ctrlr->name);

	spdk_json_write_named_array_begin(w, "ctrlrs");
	TAILQ_FOREACH(nvme_ctrlr, &nbdev_ctrlr->ctrlrs, tailq) {
		nvme_ctrlr_info_json(w, nvme_ctrlr);
	}
	spdk_json_write_array_end(w);
	spdk_json_write_object_end(w);
}

Config replay is also path-aware. bdev_nvme_config_json() emits module options, then writes one attach-style config entry for the active path and for each alternate path. It also emits non-default multipath policy per namespace bdev. That is why saved config can reconstruct both the controller connection and the bdev-level path selection behavior.

static int
bdev_nvme_config_json(struct spdk_json_write_ctx *w)
{
	struct nvme_bdev_ctrlr	*nbdev_ctrlr;
	struct nvme_ctrlr	*nvme_ctrlr;
	struct spdk_nvme_path_id	*path_id;

	assert(spdk_thread_is_app_thread(NULL));

	bdev_nvme_opts_config_json(w);

	TAILQ_FOREACH(nbdev_ctrlr, &g_nvme_bdev_ctrlrs, tailq) {
		struct nvme_bdev *nbdev;

		TAILQ_FOREACH(nvme_ctrlr, &nbdev_ctrlr->ctrlrs, tailq) {
			path_id = nvme_ctrlr->active_path_id;
			assert(path_id == TAILQ_FIRST(&nvme_ctrlr->trids));
			nvme_ctrlr_config_json(w, nvme_ctrlr, path_id);

			path_id = TAILQ_NEXT(path_id, link);
			while (path_id != NULL) {
				nvme_ctrlr_config_json(w, nvme_ctrlr, path_id);
				path_id = TAILQ_NEXT(path_id, link);
			}
		}

		TAILQ_FOREACH(nbdev, &nbdev_ctrlr->bdevs, tailq) {
			bdev_nvme_multipath_config_json(nbdev, w);
		}
	}
	/* Discovery config and success return follow. */

Admin queue health is separate from I/O qpair health. The admin poller processes admin completions; if that fails, it invokes the controller's disconnected callback or initiates failover. If the admin qpair reports a failure reason, the module clears I/O path caches so channels stop reusing a path whose controller state has changed.

static int
bdev_nvme_poll_adminq(void *arg)
{
	int32_t rc;
	struct nvme_ctrlr *nvme_ctrlr = arg;
	nvme_ctrlr_disconnected_cb disconnected_cb;

	rc = spdk_nvme_ctrlr_process_admin_completions(nvme_ctrlr->ctrlr);
	if (rc < 0) {
		disconnected_cb = nvme_ctrlr->disconnected_cb;
		nvme_ctrlr->disconnected_cb = NULL;

		if (disconnected_cb != NULL) {
			bdev_nvme_change_adminq_poll_period(nvme_ctrlr,
							    g_opts.nvme_adminq_poll_period_us);
			disconnected_cb(nvme_ctrlr);
		} else {
			bdev_nvme_failover_ctrlr(nvme_ctrlr);
		}
	} else if (spdk_nvme_ctrlr_get_admin_qp_failure_reason(nvme_ctrlr->ctrlr) !=
		   SPDK_NVME_QPAIR_FAILURE_NONE) {
		bdev_nvme_clear_io_path_caches(nvme_ctrlr);
	}

	return rc == 0 ? SPDK_POLLER_IDLE : SPDK_POLLER_BUSY;
}

Path statistics are allocated per nvme_io_path only when the module option is enabled. That keeps the default fast path smaller while still giving operators a per-path view when diagnosing an unhealthy fabric path.

static struct nvme_io_path *
nvme_io_path_alloc(void)
{
	struct nvme_io_path *io_path;

	io_path = calloc(1, sizeof(*io_path));
	if (io_path == NULL) {
		SPDK_ERRLOG("Failed to alloc io_path.\n");
		return NULL;
	}

	if (g_opts.io_path_stat) {
		io_path->stat = calloc(1, sizeof(struct spdk_bdev_io_stat));
		if (io_path->stat == NULL) {
			free(io_path);
			SPDK_ERRLOG("Failed to alloc io_path stat.\n");
			return NULL;
		}
		spdk_bdev_reset_io_stat(io_path->stat, SPDK_BDEV_RESET_STAT_MAXMIN);
	}

	return io_path;
}

Prose Diagram

Imagine three layers.

Top layer: bdev API. A caller opens Nvme0n1, gets a bdev channel, submits a read.

Middle layer: NVMe bdev module. The bdev channel points to struct nvme_bdev_channel, which has a list of nvme_io_path objects. Path selection picks one path. The selected path points to a namespace and qpair.

Bottom layer: NVMe library. The namespace command API builds an NVMe read command and submits it on the qpair. The qpair belongs to a poll group. bdev_nvme_poll() polls the poll group. The NVMe completion callback completes the bdev I/O.

Side boxes: controller group, namespace tree, ANA state, retry queue, reset state, reconnect timers, and JSON-RPC configuration.

Edge Cases And Failure Modes

  • Attach creates zero bdevs: controller may attach but namespaces are unsupported, inactive, filtered, or populate failed.
  • max_bdevs too small: attach can create more bdevs than names returned to RPC; the module logs when it cannot return all names.
  • Duplicate controller name with same path: RPC rejects it.
  • Duplicate controller name with different subnqn or hostnqn: RPC rejects it.
  • Multipath disabled: adding a second path to same controller name is rejected.
  • Same namespace reached through multiple controllers: module adds namespace path to existing bdev.
  • Last path removed: namespace bdev unregisters.
  • No I/O path: non-admin I/O completes -ENXIO.
  • Qpair failure: poll path clears I/O path caches.
  • Admin queue failure: admin poller triggers disconnected/failover handling.
  • Reset while another reset runs: reset I/O can queue.
  • Reconnect settings invalid: create rejects them before connect.
  • Interrupt mode with non-PCIe: create rejects it.
  • Base bdev semantics not relevant: NVMe bdev is physical in bdev terms, but remote fabrics may disappear like a network resource.

Misconceptions To Kill

  • "NVMe bdev is just a thin wrapper around spdk_nvme_connect()." No. It adds bdev registration, namespace mapping, channels, qpairs, poll groups, multipath, retry, reset, stats, JSON config, and RPC validation.
  • "One controller attach means one bdev." No. One controller can expose multiple namespaces.
  • "One bdev means one controller." Not with multipath. One namespace bdev can have multiple namespace paths.
  • "Reset is handled entirely by bdev core." No. bdev core coordinates reset I/O, then NVMe bdev resets controllers and qpairs.
  • "A path failure immediately means the bdev is gone." Not necessarily. Multipath or reconnect may keep the bdev visible.
  • "Admin commands use the current I/O path." Admin passthrough has its own handling and can proceed even when no regular I/O path is selected.

Source Reading Exercise

Trace attach:

  1. module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_attach_controller().
  2. include/spdk/module/bdev/nvme.h:spdk_bdev_nvme_create().
  3. module/bdev/nvme/bdev_nvme.c:spdk_bdev_nvme_create().
  4. module/bdev/nvme/bdev_nvme.c:connect_attach_cb().
  5. module/bdev/nvme/bdev_nvme.c:nvme_bdev_ctrlr_create().
  6. module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespaces().
  7. module/bdev/nvme/bdev_nvme.c:nvme_ctrlr_populate_namespace().
  8. module/bdev/nvme/bdev_nvme.c:nvme_bdev_create().
  9. lib/bdev/bdev.c:spdk_bdev_register().

Trace one read:

  1. lib/bdev/bdev.c:spdk_bdev_readv_blocks().
  2. lib/bdev/bdev.c:bdev_io_submit().
  3. module/bdev/nvme/bdev_nvme.c:bdev_nvme_submit_request_initial().
  4. module/bdev/nvme/bdev_nvme.c:bdev_nvme_submit_request().
  5. module/bdev/nvme/bdev_nvme.c:bdev_nvme_readv().
  6. include/spdk/nvme.h:spdk_nvme_ns_cmd_readv_with_md() or include/spdk/nvme.h:spdk_nvme_ns_cmd_readv_ext().
  7. module/bdev/nvme/bdev_nvme.c:bdev_nvme_poll().
  8. lib/nvme/nvme_poll_group.c:spdk_nvme_poll_group_process_completions().
  9. module/bdev/nvme/bdev_nvme.c:bdev_nvme_io_complete_nvme_status().

Questions:

  • Where is JSON converted to struct spdk_nvme_transport_id?
  • Where does the module decide whether this is a second path?
  • Where does namespace ID become bdev identity?
  • Where is the per-thread qpair reached from a bdev channel?
  • Where does no-path turn into an I/O failure?

Operational Lab

Debug "remote NVMe bdev exists but I/O hangs" on paper.

Checklist:

  1. Is bdev_nvme_poll() registered on the thread that owns the qpair?
  2. Does spdk_nvme_poll_group_process_completions() return completions, zero, or negative?
  3. Does qpair failure reason indicate a disconnected qpair?
  4. Did bdev_nvme_check_io_qpairs() clear path caches?
  5. Does bdev_nvme_find_io_path() return NULL?
  6. Are I/O sitting on retry_io_list?
  7. Is nbdev_ch->resetting true?
  8. Is admin queue poller seeing failure and triggering failover?
  9. Are reconnect and fast-fail timeouts configured coherently?

For each "yes/no," write the source function where you would add a log line.

Self-Check

  1. Why does bdev_nvme_attach_controller return an array?
  2. What is the difference between struct nvme_bdev_ctrlr and struct nvme_ctrlr?
  3. What does struct nvme_io_path connect together?
  4. Why does the NVMe bdev module need a poll group?
  5. What happens when no I/O path is available for a normal read?
  6. Why are reconnect timeout combinations validated before attach?
  7. How does a namespace removal become a bdev unregister?
  8. Why is multipath more than a list of transport IDs?

References

  • Local source: include/spdk/module/bdev/nvme.h.
  • Local source: module/bdev/nvme/bdev_nvme.c.
  • Local source: module/bdev/nvme/bdev_nvme.h.
  • Local source: module/bdev/nvme/bdev_nvme_rpc.c.
  • Local source: include/spdk/nvme.h.
  • Local source: lib/nvme/nvme.c.
  • Local source: lib/nvme/nvme_poll_group.c.
  • NVM Express specifications: https://nvmexpress.org/specifications/
  • SPDK Block Device User Guide, NVMe bdev: https://spdk.io/doc/bdev.html#bdev_config_nvme
  • SPDK JSON-RPC, bdev_nvme_attach_controller and NVMe bdev RPCs: https://spdk.io/doc/jsonrpc.html
  • SPDK NVMe Driver: https://spdk.io/doc/nvme.html
  • SPDK NVMe Multipath: https://spdk.io/doc/nvme_multipath.html
  • SPDK NVMe-oF Multipath HOWTO: https://spdk.io/doc/nvmf_multipath_howto.html