SPDK From First Principles

SPDK deep learning path

Chapter 23: NVMe-oF Initiator Through `bdev_nvme`

This chapter explains how SPDK acts as an NVMe initiator and then exposes connected namespaces as SPDK bdevs. For diskengine, this is the core of baremetal mode: compute-side...

Source: drafts/transport-diskengine/23-nvme-of-initiator-bdev-nvme.md

Chapter Goal

This chapter explains how SPDK acts as an NVMe initiator and then exposes connected namespaces as SPDK bdevs. For diskengine, this is the core of baremetal mode: compute-side SPDK connects to storage-node NVMe-oF subsystems and turns each remote lvol namespace into a local bdev that can be assembled into RAID and exported to QEMU.

Beginner Mental Model

The NVMe-oF target chapter described a remote subsystem that exports a namespace. The initiator side is the mirror image. It asks:

Given an NQN and a transport address, can I create a controller connection and produce one or more local bdev names?

SPDK's bdev_nvme module wraps the lower-level NVMe library. The NVMe library knows how to connect, create admin and I/O qpairs, submit commands, poll completions, reconnect, and reset. The bdev module translates generic bdev I/O into NVMe namespace commands.

To a user of the bdev layer, the result is just another bdev. It can be a base for RAID, lvol, vhost, NBD, or another virtual bdev. The fact that it is remote over RDMA is hidden behind the bdev module.

The useful beginner distinction is:

  • A transport address says where to dial: RDMA IP plus service ID, TCP IP plus service ID, or PCIe BDF.
  • A subsystem NQN says which NVMe subsystem at that address the host wants.
  • A controller is the live association SPDK creates after the fabrics connect.
  • A namespace is the logical block device exposed by that controller.
  • A bdev is SPDK's generic block-device wrapper around that namespace.

For direct known-subsystem attach, diskengine already knows the storage node RDMA address and the lvol subsystem NQN. It therefore asks SPDK to connect directly. SPDK also supports discovery-service attach, where the host first connects to the discovery NQN and receives discovery log entries that point at NVM subsystems. The NVMe-oF specification defines discovery so a host can learn which subsystems it may access; a discovery controller is not the data controller for guest I/O.

NQN naming is not decorative. The NVM Express specifications use an NVMe Qualified Name to identify hosts and NVM subsystems for identification and authentication, and SPDK compares target NQNs byte-for-byte. If diskengine changes an NQN string, or changes how it derives controller names from it, the local bdev graph changes even if the same storage is behind it.

Why This Matters For diskengine/excloud

diskengine baremetal mode does not write to storage-node lvols by calling storage-node APIs per I/O. It connects once through SPDK, assembles local bdev graph objects, and then data I/O stays in SPDK. The Go process reconciles attachment state; it is not in the write fast path.

The key diskengine source anchors are:

  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go: startNvmeAttachLoop
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go: reconcileNVMeConnections
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go: attachNvmeSoft
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go: attachNvme
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go: controllerNameForNQN
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/utils.go: baseBdevNameFromNQN
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: BdevNvmeAttachController
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: BdevNvmeGetControllers
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: BdevNvmeGetIoPaths
  • /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go: BdevNvmeSetMultipathPolicy

The baremetal attach loop is deliberately reconciler-shaped. It wakes periodically, asks the database what NVMe-oF paths this host should have, checks SPDK's current controllers, and attaches missing paths. This is control-plane work. Once the bdev exists, reads and writes do not call this loop.

From /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go:

multipath := "multipath"
reconnectDelaySec := 1
ctrlrLossTimeoutSec := 10
fastIoFailTimeoutSec := 0
name := controllerNameForNQN(conn.NQN)

params := spdkclient.BdevNvmeAttachControllerParams{
	Name:                 name,
	Subnqn:               &conn.NQN,
	Trtype:               "rdma",
	Traddr:               conn.RDMAIP.String(),
	Trsvcid:              &svc,
	Adrfam:               &adrfam,
	Multipath:            &multipath,
	ReconnectDelaySec:    &reconnectDelaySec,
	CtrlrLossTimeoutSec:  &ctrlrLossTimeoutSec,
	FastIoFailTimeoutSec: &fastIoFailTimeoutSec,
}
if hostaddr != nil {
	params.Hostaddr = hostaddr
}

This tells you most of the diskengine policy in one place. The target is RDMA, the subsystem is selected by Subnqn, the local bdev controller name is deterministic, and SPDK is asked to keep multipath semantics active. The timeout values are short because the host should retry transient loss but should not keep a dead controller forever.

SPDK treats hostaddr as an optional attach field, but current diskengine's IPv4 RDMA path is stricter. pickHostAddr() reads config.Value.RDMA_IPS, chooses a local IPv4 source address, and ensureHostAddrInterfaceUp(hostaddr) runs before the RPC. If RDMA_IPS is missing, has no usable IPv4 address, names an address not present on a local interface, or names an interface that is down, diskengine fails the attach before SPDK is called.

RPC To bdev Creation

The RPC entry point is:

  • module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_attach_controller
  • module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_attach_controller_decoders
  • module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_attach_controller_done
  • module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_attach_controller_examined

That handler decodes fields such as name, trtype, traddr, trsvcid, adrfam, subnqn, hostnqn, hostaddr, multipath, ctrlr_loss_timeout_sec, reconnect_delay_sec, fast_io_fail_timeout_sec, psk, dhchap_key, and dhchap_ctrlr_key. It then calls:

  • module/bdev/nvme/bdev_nvme.c: spdk_bdev_nvme_create

The decoder table is the contract between JSON-RPC and the local struct rpc_bdev_nvme_attach_controller. Required fields are omitted from the table's optional flag; optional fields have the final true. The excerpt below is trimmed to the fields that matter most in this chapter, but the local SPDK table also includes PI, TCP digest, queue-count, security, max_bdevs, and CSI options.

From module/bdev/nvme/bdev_nvme_rpc.c:

static const struct spdk_json_object_decoder rpc_bdev_nvme_attach_controller_decoders[] = {
	{"name", offsetof(struct rpc_bdev_nvme_attach_controller, name), spdk_json_decode_string},
	{"trtype", offsetof(struct rpc_bdev_nvme_attach_controller, trtype), spdk_json_decode_string},
	{"traddr", offsetof(struct rpc_bdev_nvme_attach_controller, traddr), spdk_json_decode_string},

	{"adrfam", offsetof(struct rpc_bdev_nvme_attach_controller, adrfam), spdk_json_decode_string, true},
	{"trsvcid", offsetof(struct rpc_bdev_nvme_attach_controller, trsvcid), spdk_json_decode_string, true},
	{"priority", offsetof(struct rpc_bdev_nvme_attach_controller, priority), spdk_json_decode_string, true},
	{"subnqn", offsetof(struct rpc_bdev_nvme_attach_controller, subnqn), spdk_json_decode_string, true},
	{"hostnqn", offsetof(struct rpc_bdev_nvme_attach_controller, hostnqn), spdk_json_decode_string, true},
	{"hostaddr", offsetof(struct rpc_bdev_nvme_attach_controller, hostaddr), spdk_json_decode_string, true},
	{"hostsvcid", offsetof(struct rpc_bdev_nvme_attach_controller, hostsvcid), spdk_json_decode_string, true},

	{"multipath", offsetof(struct rpc_bdev_nvme_attach_controller, multipath), bdev_nvme_decode_multipath, true},
	{"ctrlr_loss_timeout_sec", offsetof(struct rpc_bdev_nvme_attach_controller, bdev_opts.ctrlr_loss_timeout_sec), spdk_json_decode_int32, true},
	{"reconnect_delay_sec", offsetof(struct rpc_bdev_nvme_attach_controller, bdev_opts.reconnect_delay_sec), spdk_json_decode_uint32, true},
	{"fast_io_fail_timeout_sec", offsetof(struct rpc_bdev_nvme_attach_controller, bdev_opts.fast_io_fail_timeout_sec), spdk_json_decode_uint32, true},
	{"psk", offsetof(struct rpc_bdev_nvme_attach_controller, psk), spdk_json_decode_string, true},
	{"max_bdevs", offsetof(struct rpc_bdev_nvme_attach_controller, max_bdevs), spdk_json_decode_uint32, true},
	{"dhchap_key", offsetof(struct rpc_bdev_nvme_attach_controller, dhchap_key), spdk_json_decode_string, true},
	{"dhchap_ctrlr_key", offsetof(struct rpc_bdev_nvme_attach_controller, dhchap_ctrlr_key), spdk_json_decode_string, true},
};

After decoding, the handler copies JSON strings into the NVMe transport ID and controller options. traddr, trsvcid, and adrfam describe the network endpoint. subnqn selects the remote subsystem. hostnqn, hostaddr, and hostsvcid are host-side identity and source-address controls. psk configures TLS pre-shared-key use, while dhchap_key and dhchap_ctrlr_key configure DH-HMAC-CHAP host and controller authentication keys. This is also where SPDK protects multipath identity: adding another path to an existing controller name is allowed only when the new path still names the same subnqn and hostnqn.

Current diskengine has Go struct fields for Psk, DhchapKey, and DhchapCtrlrKey in internal/spdkclient/types.go, so the JSON-RPC boundary can carry those values. The inspected baremetal RDMA attach loop does not set them. In current deployments, initiator security therefore depends on the isolated fabric and the target-side access policy unless diskengine grows explicit host/key management.

From module/bdev/nvme/bdev_nvme_rpc.c:

if (ctrlr) {
	if (ctx->req.multipath == BDEV_NVME_MP_MODE_DISABLE) {
		spdk_jsonrpc_send_error_response_fmt(request, -EALREADY,
				"A controller named %s already exists and multipath is disabled",
				ctx->req.name);
		goto cleanup;
	}

	if (strncmp(trid.subnqn,
		    ctrlr_trid->subnqn,
		    SPDK_NVMF_NQN_MAX_LEN) != 0) {
		spdk_jsonrpc_send_error_response_fmt(request, -EINVAL,
				"A controller named %s already exists, but uses a different subnqn (%s)",
				ctx->req.name, ctrlr_trid->subnqn);
		goto cleanup;
	}

	if (strncmp(ctx->req.drv_opts.hostnqn, drv_opts->hostnqn, SPDK_NVMF_NQN_MAX_LEN) != 0) {
		spdk_jsonrpc_send_error_response_fmt(request, -EINVAL,
				"A controller named %s already exists, but uses a different hostnqn (%s)",
				ctx->req.name, drv_opts->hostnqn);
		goto cleanup;
	}
}

The call into the bdev module is asynchronous. The RPC allocates a names array, passes its callback, and does not send the JSON-RPC result until spdk_bdev_wait_for_examine() says bdev examine work has caught up. That wait matters because the user expects the returned names to be visible in the bdev layer, not merely scheduled for later creation.

From module/bdev/nvme/bdev_nvme_rpc.c:

static void
rpc_bdev_nvme_attach_controller_done(void *cb_ctx, size_t bdev_count, int rc)
{
	struct rpc_bdev_nvme_attach_controller_ctx *ctx = cb_ctx;
	struct spdk_jsonrpc_request *request = ctx->request;

	if (rc < 0) {
		spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc));
		free_rpc_bdev_nvme_attach_controller_ctx(ctx);
		return;
	}

	ctx->bdev_count = bdev_count;
	spdk_bdev_wait_for_examine(rpc_bdev_nvme_attach_controller_examined, ctx);
}

rc = spdk_bdev_nvme_create(&trid, ctx->req.name, ctx->names, ctx->req.max_bdevs,
			   rpc_bdev_nvme_attach_controller_done, ctx, &ctx->req.drv_opts,
			   &ctx->req.bdev_opts);

The public header for the module is:

  • include/spdk/module/bdev/nvme.h: spdk_bdev_nvme_create
  • include/spdk/module/bdev/nvme.h: spdk_bdev_nvme_delete
  • include/spdk/module/bdev/nvme.h: spdk_bdev_nvme_set_multipath_policy
  • include/spdk/module/bdev/nvme.h: struct spdk_bdev_nvme_ctrlr_opts

Controller and bdev object creation flows through:

  • module/bdev/nvme/bdev_nvme.c: nvme_bdev_ctrlr_create
  • module/bdev/nvme/bdev_nvme.c: nvme_bdev_create
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_create_ctrlr_channel_cb
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_create_bdev_channel_cb
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_get_io_channel

The module registers a bdev function table with:

  • module/bdev/nvme/bdev_nvme.c: nvmelib_fn_table

The function table is where the bdev layer learns how to submit I/O to this module.

spdk_bdev_nvme_create() is the first bdev-module-owned function after the RPC bridge. It rejects duplicate controller transport identities, checks the base name length, validates reconnect/loss/fast-fail relationships, and stores the callback context that will be completed after probe and namespace population.

From module/bdev/nvme/bdev_nvme.c:

int
spdk_bdev_nvme_create(struct spdk_nvme_transport_id *trid,
		      const char *base_name,
		      const char **names,
		      uint32_t count,
		      spdk_bdev_nvme_create_cb cb_fn,
		      void *cb_ctx,
		      struct spdk_nvme_ctrlr_opts *drv_opts,
		      struct spdk_bdev_nvme_ctrlr_opts *bdev_opts)
{
	struct nvme_async_probe_ctx *ctx;
	int len;

	if (nvme_ctrlr_get(trid, drv_opts->hostnqn) != NULL) {
		SPDK_ERRLOG("A controller with the provided trid (traddr: %s, hostnqn: %s) "
			    "already exists.\n", trid->traddr, drv_opts->hostnqn);
		return -EEXIST;
	}

	len = strnlen(base_name, SPDK_CONTROLLER_NAME_MAX);
	if (len == 0 || len == SPDK_CONTROLLER_NAME_MAX) {
		return -EINVAL;
	}

	if (bdev_opts != NULL &&
	    !bdev_nvme_check_io_error_resiliency_params(bdev_opts->ctrlr_loss_timeout_sec,
			    bdev_opts->reconnect_delay_sec,
			    bdev_opts->fast_io_fail_timeout_sec)) {
		return -EINVAL;
	}

	ctx = calloc(1, sizeof(*ctx));
	if (!ctx) {
		return -ENOMEM;
	}
	ctx->base_name = strdup(base_name);
	ctx->names = names;
	ctx->max_bdevs = count;
	ctx->cb_fn = cb_fn;
	ctx->cb_ctx = cb_ctx;
	ctx->trid = *trid;

The excerpt is intentionally cut before the lower-level probe mechanics. The important ownership point is that the RPC-owned request has been converted into an nvme_async_probe_ctx. From here on, the attach is driven by the NVMe library and bdev module callbacks, not by diskengine.

Discovery Versus Direct Connect

The SPDK NVMe documentation says the transport ID may name either a discovery service or a single NVM subsystem. If it names a discovery service, SPDK reads discovery log entries and probes each discovered subsystem. If it names a subsystem directly, SPDK probes only that subsystem.

The lower-level NVMe fabrics code makes that branch explicit. The discovery NQN is special. Anything else is treated as a direct subsystem connection.

From lib/nvme/nvme_fabric.c:

if (strcmp(probe_ctx->trid.subnqn, SPDK_NVMF_DISCOVERY_NQN) != 0) {
	/* It is not a discovery_ctrlr info and try to directly connect it */
	rc = nvme_ctrlr_probe(&probe_ctx->trid, probe_ctx, NULL);
	return rc;
}

spdk_nvme_ctrlr_get_default_ctrlr_opts(&discovery_opts, sizeof(discovery_opts));
if (direct_connect && probe_ctx->probe_cb) {
	probe_ctx->probe_cb(probe_ctx->cb_ctx, &probe_ctx->trid, &discovery_opts);
}

discovery_ctrlr = nvme_transport_ctrlr_construct(&probe_ctx->trid, &discovery_opts, NULL);

SPDK's bdev module also has a discovery-service mode through bdev_nvme_start_discovery. That path reads the discovery log, builds transport IDs from log entries, and eventually calls the same spdk_bdev_nvme_create() used by direct attach. The following excerpt is abridged around duplicate-detection and error handling so the main path is visible.

From module/bdev/nvme/bdev_nvme.c:

for (i = 0; i < numrec; i++) {
	found = false;
	new_entry = &log_page->entries[i];
	if (new_entry->subtype == SPDK_NVMF_SUBTYPE_DISCOVERY_CURRENT ||
	    new_entry->subtype == SPDK_NVMF_SUBTYPE_DISCOVERY) {
		struct spdk_nvme_transport_id trid = {};

		build_trid_from_log_page_entry(&trid, new_entry);
		new_ctx = create_discovery_entry_ctx(ctx, &trid);
		TAILQ_INSERT_TAIL(&ctx->discovery_entry_ctxs, new_ctx, tailq);
		continue;
	}

	if (!found) {
		struct discovery_entry_ctx *subnqn_ctx = NULL, *new_ctx;

		new_ctx = calloc(1, sizeof(*new_ctx));
		memcpy(&new_ctx->entry, new_entry, sizeof(*new_entry));
		build_trid_from_log_page_entry(&new_ctx->trid, new_entry);
		if (subnqn_ctx) {
			snprintf(new_ctx->name, sizeof(new_ctx->name), "%s", subnqn_ctx->name);
		} else {
			snprintf(new_ctx->name, sizeof(new_ctx->name), "%s%d", ctx->name, ctx->index++);
		}
		rc = spdk_bdev_nvme_create(&new_ctx->trid, new_ctx->name, NULL, 0,
					   discovery_attach_controller_done, new_ctx,
					   &new_ctx->drv_opts, &ctx->bdev_opts);
	}
}

diskengine's baremetal path is direct connect, not discovery-service connect. That is a system design choice: diskengine's database already maps a volume replica to an NQN, RDMA IP, and port, so discovery would add another moving part without adding information to the attach loop.

Namespace To bdev Registration

After the NVMe library has a controller and namespace objects, bdev_nvme decides whether the namespace becomes a new bdev or an additional path for an existing bdev. A new namespace ID under a controller name normally produces a bdev name like <base_name>n<nsid>, so diskengine's baseBdevNameFromNQN() assumes namespace 1 and appends n1.

The bdev registration code runs on the app thread. It allocates an nvme_bdev, initializes the SPDK bdev fields through nbdev_create(), registers an I/O device so each thread can have a channel, links the namespace to the bdev, and finally calls spdk_bdev_register().

From module/bdev/nvme/bdev_nvme.c:

static int
nvme_bdev_create(struct nvme_ctrlr *nvme_ctrlr, struct nvme_ns *nvme_ns)
{
	struct nvme_bdev *nbdev;
	struct nvme_bdev_ctrlr *nbdev_ctrlr = nvme_ctrlr->nbdev_ctrlr;
	int rc;

	assert(spdk_thread_is_app_thread(NULL));

	nbdev = nvme_bdev_alloc();
	rc = nbdev_create(&nbdev->disk, nbdev_ctrlr->name, nvme_ctrlr->ctrlr,
			  nvme_ns->ns, &nvme_ctrlr->opts, nbdev);

	spdk_io_device_register(nbdev,
				bdev_nvme_create_bdev_channel_cb,
				bdev_nvme_destroy_bdev_channel_cb,
				sizeof(struct nvme_bdev_channel),
				nbdev->disk.name);

	nvme_ns->bdev = nbdev;
	nbdev->nsid = nvme_ns->id;
	TAILQ_INSERT_TAIL(&nbdev->nvme_ns_list, nvme_ns, tailq);

	rc = spdk_bdev_register(&nbdev->disk);

This is the point where a remote namespace becomes a generic SPDK bdev. The app that later opens the bdev does not need to know whether nbdev->disk represents a PCIe controller or an NVMe-oF controller.

bdev I/O To NVMe Command

Once the controller and namespace bdev exist, normal bdev I/O enters:

  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_submit_request_initial
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_submit_request
  • module/bdev/nvme/bdev_nvme.c: _bdev_nvme_submit_request

Reads and writes reach:

  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_readv
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_readv_done
  • module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev_done

The actual NVMe namespace commands are lower-level library calls:

  • lib/nvme/nvme_ns_cmd.c: spdk_nvme_ns_cmd_readv
  • lib/nvme/nvme_ns_cmd.c: spdk_nvme_ns_cmd_writev
  • lib/nvme/nvme_ns_cmd.c: spdk_nvme_ns_cmd_read_ext
  • lib/nvme/nvme_ns_cmd.c: spdk_nvme_ns_cmd_write_ext

For RDMA transport mechanics, start at:

  • lib/nvme/nvme_rdma.c
  • lib/nvme/nvme_fabric.c
  • lib/nvme/nvme_qpair.c
  • lib/nvme/nvme_poll_group.c

The bdev layer calls the function table installed by bdev_nvme. The NVMe bdev submit function gets the per-thread bdev channel, chooses an I/O path, and then routes the generic bdev operation into an NVMe operation. This is where multipath becomes part of the fast path: bdev_nvme_find_io_path() chooses the path according to current path state and policy before the read or write helper submits to the NVMe library.

From module/bdev/nvme/bdev_nvme.c:

static void
bdev_nvme_submit_request(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)
{
	struct nvme_bdev_channel *nbdev_ch = spdk_io_channel_get_ctx(ch);
	struct nvme_bdev_io *nbdev_io = (struct nvme_bdev_io *)bdev_io->driver_ctx;

	spdk_trace_record(TRACE_BDEV_NVME_IO_START, 0, 0, (uintptr_t)nbdev_io, (uintptr_t)bdev_io);
	nbdev_io->io_path = bdev_nvme_find_io_path(nbdev_ch);
	if (spdk_unlikely(!nbdev_io->io_path)) {
		if (!bdev_nvme_io_type_is_admin(bdev_io->type)) {
			bdev_nvme_io_complete(nbdev_io, -ENXIO);
			return;
		}
	}

	_bdev_nvme_submit_request(nbdev_ch, bdev_io);
}

The inner dispatcher is intentionally boring: a bdev read calls a read helper, a bdev write calls a write helper, flush calls flush, and so on. That is the point of bdev_nvme: it hides controller, qpair, and transport details behind the bdev function table.

From module/bdev/nvme/bdev_nvme.c:

switch (bdev_io->type) {
case SPDK_BDEV_IO_TYPE_READ:
	rc = bdev_nvme_readv(nbdev_io,
			     bdev_io->u.bdev.iovs,
			     bdev_io->u.bdev.iovcnt,
			     bdev_io->u.bdev.md_buf,
			     bdev_io->u.bdev.num_blocks,
			     bdev_io->u.bdev.offset_blocks,
			     bdev_io->u.bdev.dif_check_flags,
			     bdev_io->u.bdev.memory_domain,
			     bdev_io->u.bdev.memory_domain_ctx,
			     bdev_io->u.bdev.accel_sequence);
	break;
case SPDK_BDEV_IO_TYPE_WRITE:
	rc = bdev_nvme_writev(nbdev_io,
			      bdev_io->u.bdev.iovs,
			      bdev_io->u.bdev.iovcnt,
			      bdev_io->u.bdev.md_buf,
			      bdev_io->u.bdev.num_blocks,
			      bdev_io->u.bdev.offset_blocks,
			      bdev_io->u.bdev.dif_check_flags,
			      bdev_io->u.bdev.memory_domain,
			      bdev_io->u.bdev.memory_domain_ctx,
			      bdev_io->u.bdev.accel_sequence,
			      bdev_io->u.bdev.nvme_cdw12,
			      bdev_io->u.bdev.nvme_cdw13);
	break;
}

On an RDMA attach, the selected io_path ultimately contains an NVMe qpair whose transport implementation is RDMA. The qpair is polled by SPDK threads; guest I/O completion returns through NVMe completion polling into bdev_nvme_io_complete(), then back to the generic bdev completion path.

Prose Diagram: Initiator Object Stack

Draw a top-down stack:

  1. diskengine baremetal loop.
  2. SPDK JSON-RPC bdev_nvme_attach_controller.
  3. bdev_nvme controller object.
  4. NVMe controller connection and admin qpair.
  5. Per-thread bdev channel.
  6. NVMe I/O qpair.
  7. Remote NVMe-oF subsystem namespace.
  8. Storage-node lvol bdev.

Next to the stack, draw a separate horizontal data path:

bdev write to NvmeRemoteNqn1 -> bdev_nvme_submit_request -> spdk_nvme_ns_cmd_write* -> RDMA/TCP qpair -> storage-node NVMf target -> storage-node bdev.

The diagram should show diskengine only above the stack, not in the data path.

flowchart TB de[diskengine baremetal reconciler] --> rpc[SPDK JSON-RPC bdev_nvme_attach_controller] rpc --> nbdev_ctrlr[bdev_nvme controller object] nbdev_ctrlr --> nvme_ctrlr[NVMe-oF controller association and admin qpair] nvme_ctrlr --> ns[Remote namespace] ns --> bdev[Local SPDK bdev name] bdev --> raid[RAID or other local bdev graph] raid --> qemu[QEMU-facing export] write[Guest or upper bdev write] --> submit[bdev_nvme_submit_request] submit --> path[bdev_nvme_find_io_path] path --> cmd[spdk_nvme_ns_cmd_write*] cmd --> fabric[RDMA/TCP qpair] fabric --> target[storage-node NVMf target] target --> lvol[storage-node lvol bdev]

The left side is setup and object ownership. The right side is data I/O. diskengine appears above the attach path and does not sit between the guest write and the storage-node lvol.

Multipath In The Beginner Model

Multipath means one logical NVMe bdev may have more than one path to storage. A path may be another controller connection or route to the same namespace. The policy chooses which path receives I/O, and failover behavior controls what happens when a path degrades or disappears.

SPDK's multipath documentation distinguishes failover mode from multipath mode. Failover keeps one active connection and connects alternates during switch-over. Multipath maintains active connections for every path and chooses among them with active-passive or active-active policy. For diskengine, that means two storage paths can appear as one logical base bdev, and RAID can consume that bdev without knowing every controller path behind it.

SPDK source anchors:

  • module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_get_io_paths
  • module/bdev/nvme/bdev_nvme_rpc.c: _rpc_bdev_nvme_get_io_paths
  • module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_set_multipath_policy
  • module/bdev/nvme/bdev_nvme.c: nvme_bdev_channel
  • module/bdev/nvme/bdev_nvme.h: enum spdk_bdev_nvme_multipath_policy

diskengine baremetal attaches with deterministic controller names derived from NQN, refreshes I/O path information, and calls bdev_nvme_set_multipath_policy for managed bdevs. This matters because storage nodes and networks fail independently. A single logical volume may have primary and secondary lvol placements.

The policy values are small, but they are a useful map for reading the code and RPCs:

From include/spdk/module/bdev/nvme.h:

enum spdk_bdev_nvme_multipath_policy {
	BDEV_NVME_MP_POLICY_ACTIVE_PASSIVE,
	BDEV_NVME_MP_POLICY_ACTIVE_ACTIVE,
};

enum spdk_bdev_nvme_multipath_selector {
	BDEV_NVME_MP_SELECTOR_ROUND_ROBIN = 1,
	BDEV_NVME_MP_SELECTOR_QUEUE_DEPTH,
};

bdev_nvme_get_io_paths walks SPDK I/O channels, not just controller objects. This is why diskengine uses it as a better signal for multipath policy refresh: it reports the paths attached to poll groups and bdevs, which is closer to data path reality than just seeing that a controller exists.

From module/bdev/nvme/bdev_nvme_rpc.c:

static void
_rpc_bdev_nvme_get_io_paths(struct spdk_io_channel_iter *i)
{
	struct spdk_io_channel *_ch = spdk_io_channel_iter_get_channel(i);
	struct nvme_poll_group *group = spdk_io_channel_get_ctx(_ch);
	struct rpc_get_io_paths_ctx *ctx = spdk_io_channel_iter_get_ctx(i);
	struct nvme_qpair *qpair;
	struct nvme_io_path *io_path;
	struct nvme_bdev *nbdev;

	spdk_json_write_named_string(ctx->w, "thread", spdk_thread_get_name(spdk_get_thread()));
	spdk_json_write_named_array_begin(ctx->w, "io_paths");

	TAILQ_FOREACH(qpair, &group->qpair_list, tailq) {
		TAILQ_FOREACH(io_path, &qpair->io_path_list, tailq) {
			nbdev = io_path->nvme_ns->bdev;
			nvme_io_path_info_json(ctx->w, io_path);
		}
	}
}

diskengine's policy refresh is intentionally scoped. It asks for I/O paths, collects bdev names whose prefixes match controller names derived from expected NQNs, and sets only those bdevs to active-active round-robin. That avoids changing unrelated NVMe bdevs the same SPDK process may own.

From /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go:

expectedPrefixes := make([]string, 0, len(expectedNQNs))
for nqn := range expectedNQNs {
	expectedPrefixes = append(expectedPrefixes, controllerNameForNQN(nqn))
}

seen := make(map[string]struct{})
var names []string
for _, pg := range res.PollGroups {
	for _, p := range pg.IoPaths {
		if p.BdevName == "" {
			continue
		}
		if _, ok := seen[p.BdevName]; ok {
			continue
		}
		seen[p.BdevName] = struct{}{}
		if !hasMatchingPrefix(p.BdevName, expectedPrefixes) {
			continue
		}
		names = append(names, p.BdevName)
	}
}

Reconnect, Loss Timeout, And Fast Fail

Three knobs are easy to confuse:

  • reconnect_delay_sec: how long to wait between reconnect attempts.
  • ctrlr_loss_timeout_sec: how long a controller may be lost before SPDK gives up.
  • fast_io_fail_timeout_sec: how soon queued I/O may fail while the controller is unavailable.

diskengine uses short reconnect-oriented values in baremetal attach references from /home/lolwierd/Projects/excloud/diskengine/diskengine/docs/baremetal.md, and the code constructs those params in internal/baremetal/nvme_attach.go: attachNvmeSoft and internal/baremetal/nvme_attach.go: attachNvme.

The operational tradeoff is simple: a long loss timeout hides transient network loss but can make guest I/O appear hung. A short fast-fail timeout reveals problems quickly but may surface transient blips to the VM.

SPDK validates that these knobs form a coherent state machine. If ctrlr_loss_timeout_sec is zero, reconnect and fast fail must also be zero. If controller loss timeout is nonzero or infinite, reconnect delay must be nonzero. Fast fail, when set, must not be shorter than the reconnect delay, and for finite controller loss it must not exceed the controller loss timeout.

From module/bdev/nvme/bdev_nvme.c:

if (ctrlr_loss_timeout_sec == -1) {
	if (reconnect_delay_sec == 0) {
		SPDK_ERRLOG("reconnect_delay_sec can't be 0 if ctrlr_loss_timeout_sec is not 0.\n");
		return false;
	} else if (fast_io_fail_timeout_sec != 0 &&
		   fast_io_fail_timeout_sec < reconnect_delay_sec) {
		SPDK_ERRLOG("reconnect_delay_sec can't be more than fast_io-fail_timeout_sec.\n");
		return false;
	}
} else if (ctrlr_loss_timeout_sec != 0) {
	if (reconnect_delay_sec == 0) {
		SPDK_ERRLOG("reconnect_delay_sec can't be 0 if ctrlr_loss_timeout_sec is not 0.\n");
		return false;
	} else if (reconnect_delay_sec > (uint32_t)ctrlr_loss_timeout_sec) {
		SPDK_ERRLOG("reconnect_delay_sec can't be more than ctrlr_loss_timeout_sec.\n");
		return false;
	}
}

The reset path later uses those values to decide whether to delay reconnect or destruct the controller. In diskengine's current attach policy, reconnect_delay_sec=1, ctrlr_loss_timeout_sec=10, and fast_io_fail_timeout_sec=0 means SPDK can retry quickly for up to about ten seconds before giving up on the lost controller, while I/O fast-fail remains disabled.

From module/bdev/nvme/bdev_nvme.c:

static bool
bdev_nvme_check_ctrlr_loss_timeout(struct nvme_ctrlr *nvme_ctrlr)
{
	uint32_t elapsed;

	if (nvme_ctrlr->opts.ctrlr_loss_timeout_sec == 0 ||
	    nvme_ctrlr->opts.ctrlr_loss_timeout_sec == -1) {
		return false;
	}

	elapsed = (spdk_get_ticks() - nvme_ctrlr->reset_start_tsc) / spdk_get_ticks_hz();
	return elapsed >= (uint32_t)nvme_ctrlr->opts.ctrlr_loss_timeout_sec;
}

static void
bdev_nvme_start_reconnect_delay_timer(struct nvme_ctrlr *nvme_ctrlr)
{
	spdk_poller_pause(nvme_ctrlr->adminq_timer_poller);
	nvme_ctrlr->reconnect_is_delayed = true;
	nvme_ctrlr->reconnect_delay_timer = SPDK_POLLER_REGISTER(bdev_nvme_reconnect_delay_timer_expired,
					    nvme_ctrlr,
					    nvme_ctrlr->opts.reconnect_delay_sec * SPDK_SEC_TO_USEC);
}

If a guest sees a stall during storage-node or network failure, these settings are part of the answer. They decide how long SPDK keeps trying to preserve the controller before the bdev path visibly fails.

NQN Naming And diskengine bdev Names

SPDK's NVMe-oF docs show the formal shape of NQNs and note that comparisons are byte-for-byte. That is why the chapter keeps saying subnqn rather than "name string." It is a protocol identifier. In diskengine, the NQN also becomes the seed for local SPDK object names.

From /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go:

func controllerNameForNQN(nqn string) string {
	if nqn == "" {
		return "nvme_subsys_unknown"
	}
	replacer := strings.NewReplacer(":", "_", ".", "_", "/", "_", " ", "_")
	name := "nvme_subsys_" + replacer.Replace(nqn)
	if len(name) > 100 {
		name = name[:100]
	}
	return name
}

From /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/utils.go:

// baseBdevNameFromNQN derives the NVMe bdev name for a given subsystem NQN.
// We assume a single namespace (n1) per subsystem.
func baseBdevNameFromNQN(nqn string) (string, error) {
	nqn = strings.TrimSpace(nqn)
	if nqn == "" {
		return "", fmt.Errorf("empty NQN")
	}
	return controllerNameForNQN(nqn) + "n1", nil
}

The assumption is practical but narrow: one exported namespace per subsystem. If storage-node code starts exporting multiple namespaces under one subsystem, baseBdevNameFromNQN() will keep pointing at n1, and RAID assembly may miss the intended namespace. If controller names are truncated into collisions, two distinct NQNs could also map to one local prefix. The current naming works because diskengine controls NQN construction and expects one lvol namespace per subsystem.

Edge Cases And Failure Modes

Attach succeeds but no bdev appears:

The controller may connect while namespace discovery or bdev examine lags. Check the RPC return from bdev_nvme_attach_controller, bdev_get_bdevs, and bdev_nvme_get_controllers.

Controller name collision:

SPDK controller names are not just labels. diskengine derives names from NQN using controllerNameForNQN; if naming changes, cleanup and base-bdev derivation can break.

NQN maps to expected bdev name incorrectly:

diskengine assumes one namespace per subsystem in baseBdevNameFromNQN, producing a controller-derived name plus n1. If a subsystem exports multiple namespaces, that assumption fails.

Local RDMA source address missing:

Baremetal attach may fail before SPDK connect if diskengine cannot find a usable local RDMA interface. See internal/baremetal/nvme_attach.go: localRDMAHostAddr.

Concurrent attach/detach:

The diskengine client notes in internal/spdkclient/coord.go: LockController explain why overlapping controller operations are risky. Even when SPDK is asynchronous, object state transitions can be externally visible through RPCs.

Reset or reconnect during bdev graph inspection:

diskengine avoids some bdev_get_bdevs paths during reset because production crashes were observed. See internal/baremetal/utils.go: areBaseBdevsReady and internal/baremetal/utils.go: areBaseBdevsPresentViaIoPaths.

Misconceptions To Kill

"The remote lvol is copied to baremetal when attached."

No. Attach creates a controller connection and local bdev representation. Data is read and written remotely as I/O occurs.

"A controller is the same thing as a bdev."

No. A controller can expose one or more namespaces. Each namespace can become a bdev. diskengine mostly assumes one namespace, but the concepts remain separate.

"Multipath means RAID."

No. Multipath is multiple transport paths to the same logical namespace. RAID combines multiple bdevs into another bdev.

"If bdev_nvme_get_controllers shows enabled, all data paths are healthy."

Not always. Use bdev_nvme_get_io_paths, bdev I/O stats, and RAID state to understand actual path use.

Lab: Attach Sequence Walkthrough

Given:

  • NQN: nqn.2024-01.io.excloud:storage.nodeA.disk123.lvol456
  • Target RDMA IP: 10.10.1.20
  • Target RDMA port: 4420
  • Local host RDMA IP: 10.10.2.30

Write the intended bdev_nvme_attach_controller params:

  • name: deterministic name from NQN.
  • trtype: RDMA.
  • adrfam: IPv4 or IPv6.
  • traddr: target IP.
  • trsvcid: target port.
  • subnqn: NQN.
  • hostaddr: local RDMA IP. SPDK can omit it, but current diskengine requires RDMA_IPS to select one for IPv4 RDMA attach.
  • reconnect/loss/fast-fail values.
  • multipath: enabled where diskengine expects it.
  • security fields: psk, dhchap_key, and dhchap_ctrlr_key if the fabric requires TLS PSK or DH-HMAC-CHAP. Current diskengine leaves these unset.

Then inspect module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_attach_controller_decoders and verify every field exists.

Source Reading Exercise

Start at module/bdev/nvme/bdev_nvme_rpc.c: rpc_bdev_nvme_attach_controller. Follow the path until bdev registration:

  1. Where is JSON decoded?
  2. Where is spdk_bdev_nvme_create called?
  3. Where does a namespace become an SPDK bdev?
  4. Where is the bdev function table installed?

Then start at module/bdev/nvme/bdev_nvme.c: bdev_nvme_writev and identify where the completion callback returns to the bdev layer.

Operational Debug Exercise

Symptom: RAID stays configuring on baremetal.

Check in this order:

  1. Does bdev_nvme_get_controllers show enabled controllers for all required NQNs?
  2. Does bdev_nvme_get_io_paths show bdev names matching diskengine's baseBdevNameFromNQN output?
  3. Does bdev_raid_get_bdevs category=all show the RAID with missing bases?
  4. Did diskengine skip bdev_get_bdevs and rely on controller/path checks during reset?
  5. Are storage-node exports still present in nvmf_get_subsystems?

Self-Check

  1. What is the difference between subnqn and traddr?
  2. Why does diskengine need deterministic controller names?
  3. Which SPDK function is the RPC bridge into NVMe bdev creation?
  4. Why is bdev_nvme still a bdev module even when the target is remote?
  5. What is the risk of assuming every subsystem has exactly one namespace?

References

  • Local SPDK: module/bdev/nvme/bdev_nvme_rpc.c
  • Local SPDK: module/bdev/nvme/bdev_nvme.c
  • Local SPDK: include/spdk/module/bdev/nvme.h
  • Local SPDK: lib/nvme/nvme_fabric.c
  • Local SPDK: lib/nvme/nvme_rdma.c
  • Local SPDK docs: doc/nvme.md
  • Local SPDK docs: doc/bdev.md
  • Local SPDK docs: doc/nvme_multipath.md
  • Local SPDK docs: doc/nvmf.md
  • Local SPDK docs: doc/nvmf_multipath_howto.md
  • Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/nvme_attach.go
  • Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/baremetal/utils.go
  • Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/types.go
  • Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go
  • Local diskengine docs: /home/lolwierd/Projects/excloud/diskengine/diskengine/docs/baremetal.md
  • SPDK NVMe driver documentation: https://spdk.io/doc/nvme.html
  • SPDK bdev documentation: https://spdk.io/doc/bdev.html
  • SPDK NVMe multipath documentation: https://spdk.io/doc/nvme_multipath.html
  • SPDK NVMe-oF target and NQN documentation: https://spdk.io/doc/nvmf.html
  • NVM Express Base Specification, Revision 2.3: https://nvmexpress.org/wp-content/uploads/NVM-Express-Base-Specification-Revision-2.3-2025.08.01-Ratified.pdf
  • NVM Express over Fabrics Revision 1.0: https://nvmexpress.org/wp-content/uploads/NVMe_over_Fabrics_1_0_Gold_20160605-1.pdf