Chapter 22: NVMe-oF Target

Chapter Goal

After this chapter, the reader should be able to explain how SPDK exposes a local bdev as a remote NVMe namespace. They should know the difference between a target, transport, subsystem, listener, namespace, controller, qpair, poll group, and request. They should also be able to trace a remote write from an RDMA/TCP/vfio-user transport callback into spdk_nvmf_request_exec() and then into bdev I/O.

The official SPDK NVMe-oF target guide describes nvmf_tgt as a user-space target application that presents block devices over fabrics such as Ethernet, InfiniBand, or Fibre Channel. It also uses the NVMe specification's terms: the software exporting subsystems is a target, and the connecting peer is a host. The NVM Express specification set is broader than one document: the base specification defines host-to-subsystem protocol, command-set documents define commands and status values, and transport specifications define how that protocol is bound to PCIe, RDMA, TCP, and other transports.

For this chapter, the practical interpretation is simple: SPDK is not inventing a storage protocol above NVMe. It is implementing an NVMe subsystem in software and letting a host reach that subsystem through a fabric transport.

Beginner Mental Model

NVMe-oF is NVMe queue semantics carried over a fabric. A remote host still believes it is submitting NVMe commands to a controller. The controller is not a physical PCIe device on that host; it is represented by SPDK inside nvmf_tgt.

Think of the target as a building:

• The target is the whole building.
• A transport is a road type into the building: RDMA, TCP, FC, or vfio-user.
• A listener is a doorway on a road: an address and port/socket.
• A subsystem is a named storage tenant, identified by NQN.
• A namespace is one block device inside that subsystem.
• A controller is one host's live association with a subsystem.
• A qpair is one active queue connection from a host.
• A request is one NVMe command flowing through that qpair.
• A poll group is the CPU-local worker that polls transport events and bdev completions.

NVMe-oF is not a network filesystem. It does not understand files, directories, extents, or VM images. It exports block namespaces. The guest or host above it owns the filesystem or partition table.

The word "namespace" is easy to underweight. In NVMe, the namespace is the block-addressable object. SPDK backs that namespace with a bdev descriptor and bdev I/O channel. That backing bdev might be a physical NVMe namespace, an lvol, a RAID bdev, a malloc bdev, or another virtual bdev. The host sees an NVMe namespace either way.

Why This Matters For diskengine/excloud

In diskengine storage-node mode, each lvol becomes reachable from compute/baremetal nodes by being added to an NVMe-oF subsystem as a namespace. The storage node loop creates or verifies:

• an RDMA transport,
• a subsystem NQN,
• one or more listeners using storage-node RDMA IPs and port,
• a namespace pointing at the lvol bdev.

The local diskengine reconciler makes that mapping explicit. This is not SPDK source, but it is the application-level control plane that Chapter 22 is meant to explain:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go
// nvmeofExportLoop is a non-destructive reconciler that ensures NQN, RDMA transport,
// listener, and namespace exports exist for all UP/RESIZING lvols. It does not delete anything.
func nvmeofExportLoop(ctx context.Context) {
	logger.Info.Println("nvmeofExportReconcileLoop: starting")
	spdkClient, err := spdkclient.CreateClientWithJsonCodec("unix", config.Value.SPDK_RPC_SOCK)
	if err != nil {
		logger.Error.Printf("nvmeofExportReconcileLoop: failed to create spdk client: %v", err)
		return
	}
	defer spdkClient.Close()

Later in the same reconciler, diskengine checks for the RDMA transport, creates a subsystem if the NQN is missing, adds listeners for the desired RDMA addresses, and then attaches the lvol UUID as the namespace bdev. The important design point is that diskengine is not pushing data during this step. It is programming SPDK control-plane state so later host I/O has a route to the bdev.

In diskengine baremetal mode, the other side of the same relationship appears as bdev_nvme_attach_controller. The baremetal node connects to the storage node's NQN and sees a local SPDK bdev, usually named from the controller name plus namespace suffix.

Object Lifecycle

The event subsystem initializes the target before user workloads can use it. The source tour starts at:

• module/event/subsystems/nvmf/nvmf_tgt.c: nvmf_subsystem_init
• module/event/subsystems/nvmf/nvmf_tgt.c: nvmf_tgt_advance_state
• module/event/subsystems/nvmf/nvmf_tgt.c: nvmf_tgt_create_target
• module/event/subsystems/nvmf/nvmf_tgt.c: nvmf_tgt_create_poll_groups
• module/event/subsystems/nvmf/nvmf_tgt.c: nvmf_subsystem_write_config_json

The target object itself is created by the library:

• lib/nvmf/nvmf.c: spdk_nvmf_tgt_create
• lib/nvmf/nvmf.c: spdk_nvmf_tgt_destroy
• lib/nvmf/nvmf.c: spdk_nvmf_poll_group_create
• lib/nvmf/nvmf.c: spdk_nvmf_tgt_new_qpair

Target creation is deliberately small at the event layer. nvmf_tgt asks the library to create the target object, then creates the discovery subsystem. Discovery is special because hosts use it to find subsystems before connecting to a normal NVM subsystem.

/* module/event/subsystems/nvmf/nvmf_tgt.c */
static int
nvmf_tgt_create_target(void)
{
	g_spdk_nvmf_tgt = spdk_nvmf_tgt_create(&g_spdk_nvmf_tgt_conf.opts);
	if (!g_spdk_nvmf_tgt) {
		SPDK_ERRLOG("spdk_nvmf_tgt_create() failed\n");
		return -1;
	}

	if (nvmf_add_discovery_subsystem() != 0) {
		SPDK_ERRLOG("nvmf_add_discovery_subsystem failed\n");
		return -1;
	}

	return 0;
}

The library target object owns global target state: the target name, transport list, subsystem tree, poll-group list, discovery generation counter, and locks around shared target state. The excerpt below is worth reading for what it initializes, not for every option field.

/* lib/nvmf/nvmf.c */
tgt = calloc(1, sizeof(*tgt));
if (!tgt) {
	return NULL;
}

snprintf(tgt->name, NVMF_TGT_NAME_MAX_LENGTH, "%s", opts.name);

if (!opts.max_subsystems) {
	tgt->max_subsystems = SPDK_NVMF_DEFAULT_MAX_SUBSYSTEMS;
} else {
	tgt->max_subsystems = opts.max_subsystems;
}

TAILQ_INIT(&tgt->transports);
TAILQ_INIT(&tgt->poll_groups);
TAILQ_INIT(&tgt->referrals);
tgt->num_poll_groups = 0;

tgt->subsystem_ids = spdk_bit_array_create(tgt->max_subsystems);
if (tgt->subsystem_ids == NULL) {
	free(tgt);
	return NULL;
}

RB_INIT(&tgt->subsystems);
pthread_mutex_init(&tgt->mutex, NULL);

The event subsystem then creates poll-group threads. This is where the target becomes an SPDK threaded service rather than one blocking network loop. Each created SPDK thread receives a message to create its poll group.

/* module/event/subsystems/nvmf/nvmf_tgt.c */
static void
nvmf_tgt_create_poll_groups(void)
{
	uint32_t cpu, count = 0;
	char thread_name[32];
	struct spdk_thread *thread;

	g_tgt_init_thread = spdk_get_thread();
	assert(g_tgt_init_thread != NULL);

	SPDK_ENV_FOREACH_CORE(cpu) {
		if (g_poll_groups_mask && !spdk_cpuset_get_cpu(g_poll_groups_mask, cpu)) {
			continue;
		}
		snprintf(thread_name, sizeof(thread_name), "nvmf_tgt_poll_group_%03u", count++);

		thread = spdk_thread_create(thread_name, g_poll_groups_mask);
		assert(thread != NULL);

		spdk_thread_send_msg(thread, nvmf_tgt_create_poll_group, NULL);
	}
}

A poll group is the unit that ties an SPDK thread to transport pollers, qpairs, and subsystem namespace channels. When a new qpair arrives from a transport, SPDK chooses an optimal poll group if the transport can provide one; otherwise it round-robins through target poll groups. The qpair is added by message to the poll-group thread, preserving SPDK's thread-affinity rule.

/* lib/nvmf/nvmf.c */
void
spdk_nvmf_tgt_new_qpair(struct spdk_nvmf_tgt *tgt, struct spdk_nvmf_qpair *qpair)
{
	struct spdk_nvmf_poll_group *group;
	struct nvmf_new_qpair_ctx *ctx;

	group = spdk_nvmf_get_optimal_poll_group(qpair);
	if (group == NULL) {
		if (tgt->next_poll_group == NULL) {
			tgt->next_poll_group = TAILQ_FIRST(&tgt->poll_groups);
			if (tgt->next_poll_group == NULL) {
				SPDK_ERRLOG("No poll groups exist.\n");
				spdk_nvmf_qpair_disconnect(qpair);
				return;
			}
		}
		group = tgt->next_poll_group;
		tgt->next_poll_group = TAILQ_NEXT(group, link);
	}

	ctx = calloc(1, sizeof(*ctx));
	if (!ctx) {
		SPDK_ERRLOG("Unable to send message to poll group.\n");
		spdk_nvmf_qpair_disconnect(qpair);
		return;
	}

	ctx->qpair = qpair;
	ctx->group = group;
	spdk_thread_send_msg(group->thread, _nvmf_poll_group_add, ctx);
}

Transports And Listeners

Transport implementations register themselves behind a common interface:

• lib/nvmf/transport.c: spdk_nvmf_transport_register
• lib/nvmf/transport.c: spdk_nvmf_transport_create_async
• lib/nvmf/transport.c: spdk_nvmf_transport_listen
• lib/nvmf/transport.c: nvmf_transport_poll_group_create
• include/spdk/nvmf_transport.h: struct spdk_nvmf_transport_ops

A transport is not a subsystem and it is not a namespace. It is the implementation of a fabric binding: TCP sockets, RDMA verbs, Fibre Channel, or vfio-user. The common target does not need to know the verbs or socket details once the transport has converted a received command into an spdk_nvmf_request.

/* include/spdk/nvmf_transport.h */
struct spdk_nvmf_transport_ops {
	char name[SPDK_NVMF_TRSTRING_MAX_LEN];
	enum spdk_nvme_transport_type type;

	void (*opts_init)(struct spdk_nvmf_transport_opts *opts);
	struct spdk_nvmf_transport *(*create)(struct spdk_nvmf_transport_opts *opts);
	int (*create_async)(struct spdk_nvmf_transport_opts *opts,
			    spdk_nvmf_transport_create_done_cb cb_fn,
			    void *cb_arg);

	int (*listen)(struct spdk_nvmf_transport *transport,
		      const struct spdk_nvme_transport_id *trid,
		      struct spdk_nvmf_listen_opts *opts);
	void (*listen_dump_opts)(struct spdk_nvmf_transport *transport,
				 const struct spdk_nvme_transport_id *trid,
				 struct spdk_json_write_ctx *w);

nvmf_create_transport creates the transport object. It does not make every subsystem reachable. Listener creation is separate because one transport can listen at several addresses, and each subsystem decides which of those addresses it allows. The common spdk_nvmf_transport_listen() path keeps a reference-counted listener record and calls the transport-specific listen operation only for a new address.

/* lib/nvmf/transport.c */
listener = nvmf_transport_find_listener(transport, trid);
if (!listener) {
	listener = calloc(1, sizeof(*listener));
	if (!listener) {
		return -ENOMEM;
	}

	listener->ref = 1;
	listener->trid = *trid;
	listener->sock_impl = opts->sock_impl;
	TAILQ_INSERT_TAIL(&transport->listeners, listener, link);
	pthread_mutex_lock(&transport->mutex);
	rc = transport->ops->listen(transport, &listener->trid, opts);
	pthread_mutex_unlock(&transport->mutex);
	if (rc != 0) {
		TAILQ_REMOVE(&transport->listeners, listener, link);
		free(listener);
	}
	return rc;
}

++listener->ref;
return 0;

The transport also creates a transport poll group under each NVMf poll group. This explains why a poll group has both common state and transport-specific state. The common NVMf layer owns the qpair list and subsystem namespace channels; each transport owns its polling mechanism and buffer handling.

/* lib/nvmf/transport.c */
static struct spdk_nvmf_transport_poll_group *
nvmf_transport_poll_group_create(struct spdk_nvmf_transport *transport,
				 struct spdk_nvmf_poll_group *group)
{
	struct spdk_nvmf_transport_poll_group *tgroup;
	struct spdk_iobuf_opts opts_iobuf = {};

	pthread_mutex_lock(&transport->mutex);
	tgroup = transport->ops->poll_group_create(transport, group);
	pthread_mutex_unlock(&transport->mutex);
	if (!tgroup) {
		return NULL;
	}
	tgroup->transport = transport;
	nvmf_transport_poll_group_create_poller(tgroup);

	STAILQ_INIT(&tgroup->pending_buf_queue);

Subsystems, Listeners, Namespaces, Controllers

Subsystems and namespaces are managed here:

• lib/nvmf/subsystem.c: spdk_nvmf_subsystem_create
• lib/nvmf/subsystem.c: spdk_nvmf_subsystem_start
• lib/nvmf/subsystem.c: spdk_nvmf_subsystem_stop
• lib/nvmf/subsystem.c: spdk_nvmf_subsystem_add_listener_ext
• lib/nvmf/subsystem.c: spdk_nvmf_subsystem_add_ns_ext
• include/spdk/nvmf.h: spdk_nvmf_subsystem_add_ns_ext
• include/spdk/nvmf.h: spdk_nvmf_subsystem_add_listener_ext

A subsystem owns the NQN. A listener owns addressability. A namespace owns the mapping from an NSID to a bdev. A controller represents a connected host's live association with a subsystem. The relationship is easier to see in the internal structs:

/* lib/nvmf/nvmf_internal.h */
struct spdk_nvmf_ns {
	uint32_t nsid;
	uint32_t anagrpid;
	struct spdk_nvmf_subsystem *subsystem;
	struct spdk_bdev *bdev;
	struct spdk_bdev_desc *desc;
	struct spdk_nvmf_ns_opts opts;
	bool zcopy;
	enum spdk_nvme_csi csi;
	TAILQ_HEAD(, spdk_nvmf_host) hosts;
	bool always_visible;
	uint32_t passthru_nsid;
};

struct spdk_nvmf_ctrlr {
	uint16_t cntlid;
	char hostnqn[SPDK_NVMF_NQN_MAX_LEN + 1];
	struct spdk_nvmf_subsystem *subsys;
	struct spdk_bit_array *visible_ns;
	struct spdk_nvmf_qpair *admin_qpair;
	struct spdk_thread *thread;
	const struct spdk_nvmf_subsystem_listener *listener;
};

The same file shows that the subsystem contains the namespace array and controller list:

/* lib/nvmf/nvmf_internal.h */
struct spdk_nvmf_subsystem {
	struct spdk_thread *thread;
	uint32_t id;
	enum spdk_nvmf_subsystem_state state;
	enum spdk_nvmf_subtype subtype;
	bool allow_any_host;
	struct spdk_nvmf_tgt *tgt;

	/* Array of pointers to namespaces of size max_nsid indexed by nsid - 1 */
	struct spdk_nvmf_ns **ns;
	uint32_t max_nsid;

	TAILQ_HEAD(, spdk_nvmf_ctrlr) ctrlrs;
	TAILQ_HEAD(, spdk_nvmf_subsystem_listener) listeners;
};

Adding a listener to a subsystem is intentionally not just appending an address string. The RPC path first ensures the target transport is listening, then attaches the listener to the subsystem. This is why "transport exists" and "subsystem listens on RDMA IP:port" are different debug questions.

/* lib/nvmf/nvmf_rpc.c */
if (nvmf_subsystem_find_listener(subsystem, &ctx->trid)) {
	SPDK_ERRLOG("Listener already exists\n");
	spdk_jsonrpc_send_error_response(ctx->request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
					 "Invalid parameters");
	ctx->response_sent = true;
	break;
}

rc = spdk_nvmf_tgt_listen_ext(ctx->tgt, &ctx->trid, &ctx->opts);
if (rc) {
	spdk_jsonrpc_send_error_response(ctx->request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
					 "Invalid parameters");
	ctx->response_sent = true;
	break;
}

spdk_nvmf_subsystem_add_listener_ext(ctx->subsystem, &ctx->trid,
				     nvmf_rpc_subsystem_listen, ctx,
				     &ctx->listener_opts);

Adding a namespace is where the NQN-to-bdev bridge is built. The code only allows the operation while the subsystem is inactive or paused. That state rule prevents a simple array mutation from racing live I/O. The function selects an NSID if the caller did not provide one, opens the named bdev, stores its descriptor and bdev pointer, and claims the bdev for the NVMf namespace module.

/* lib/nvmf/subsystem.c */
if (!(subsystem->state == SPDK_NVMF_SUBSYSTEM_INACTIVE ||
      subsystem->state == SPDK_NVMF_SUBSYSTEM_PAUSED)) {
	return 0;
}

spdk_nvmf_ns_opts_get_defaults(&opts, sizeof(opts));
if (user_opts) {
	nvmf_ns_opts_copy(&opts, user_opts, opts_size);
}

if (opts.nsid == 0) {
	for (opts.nsid = 1; opts.nsid <= subsystem->max_nsid; opts.nsid++) {
		if (_nvmf_subsystem_get_ns(subsystem, opts.nsid) == NULL) {
			break;
		}
	}
	if (opts.nsid > subsystem->max_nsid) {
		SPDK_ERRLOG("No free namespace slot available in the subsystem\n");
		return 0;
	}
}

/* lib/nvmf/subsystem.c */
rc = spdk_bdev_open_ext_v2(bdev_name, true, nvmf_ns_event, ns,
			   &open_opts, &ns->desc);
if (rc != 0) {
	SPDK_ERRLOG("Subsystem %s: bdev %s cannot be opened, error=%d\n",
		    subsystem->subnqn, bdev_name, rc);
	free(ns);
	return 0;
}

ns->bdev = spdk_bdev_desc_get_bdev(ns->desc);

rc = spdk_bdev_module_claim_bdev(ns->bdev, ns->desc, &ns_bdev_module);
if (rc != 0) {
	spdk_bdev_close(ns->desc);
	free(ns);
	return 0;
}

spdk_nvmf_subsystem_add_ns_ext() returns the assigned namespace ID, not a Unix-style success code. In this path 0 means no NSID was assigned, so namespace add failed. The JSON-RPC wrapper treats a returned 0 as invalid parameters and sends an RPC error instead of reporting success.

Control-plane RPCs for this chapter are registered in:

• lib/nvmf/nvmf_rpc.c: rpc_nvmf_create_transport
• lib/nvmf/nvmf_rpc.c: rpc_nvmf_create_subsystem
• lib/nvmf/nvmf_rpc.c: rpc_nvmf_subsystem_add_listener
• lib/nvmf/nvmf_rpc.c: rpc_nvmf_subsystem_add_ns
• lib/nvmf/nvmf_rpc.c: rpc_nvmf_get_subsystems

The official SPDK target guide shows the same conceptual RPC sequence: create a transport, create a bdev, create a subsystem, add the bdev as a namespace, and add a listener. diskengine follows that pattern with lvol bdevs and RDMA listeners.

Qpairs And Requests

The transport-facing qpair and request structs live in include/spdk/nvmf_transport.h. A qpair is not just a socket or RDMA QP handle; it is the common NVMf representation of a queue connection after the transport has accepted it. It points to its transport, controller, poll group, and outstanding request list.

/* include/spdk/nvmf_transport.h */
struct spdk_nvmf_qpair {
	uint8_t state;
	uint16_t qid;
	uint16_t sq_head;
	uint16_t sq_head_max;

	struct spdk_nvmf_transport *transport;
	struct spdk_nvmf_ctrlr *ctrlr;
	struct spdk_nvmf_poll_group *group;

	TAILQ_HEAD(, spdk_nvmf_request) outstanding;
	TAILQ_ENTRY(spdk_nvmf_qpair) link;

	bool connect_received;
	bool disconnect_started;
	uint16_t queue_depth;
};

A request is the common object for one command. The transport fills in command/response pointers, data direction, payload length, iovecs, memory-domain information, and any transport buffer state. The bdev command path later uses the same object as callback context.

/* include/spdk/nvmf_transport.h */
struct spdk_nvmf_request {
	struct spdk_nvmf_qpair *qpair;
	uint32_t length;
	uint8_t xfer;
	uint8_t iovcnt;
	union nvmf_h2c_msg *cmd;
	union nvmf_c2h_msg *rsp;
	TAILQ_ENTRY(spdk_nvmf_request) link;

	struct spdk_memory_domain *memory_domain;
	void *memory_domain_ctx;
	struct spdk_accel_sequence *accel_sequence;

	struct iovec iov[NVMF_REQ_MAX_BUFFERS];
	struct spdk_bdev_io_wait_entry bdev_io_wait;
	spdk_nvmf_nvme_passthru_cmd_cb cmd_cb_fn;
	struct spdk_nvmf_request *first_fused_req;
};

This object model is why the common NVMf layer can be transport-neutral. RDMA, TCP, and vfio-user differ in how they receive a command, move data, and send a completion. Once an spdk_nvmf_request is ready to execute, the common controller path classifies it the same way.

The I/O Path

A host sends a command over RDMA, TCP, FC, or vfio-user. The transport decodes enough of the wire/device protocol to create an spdk_nvmf_request. It then calls the common execution path:

• lib/nvmf/ctrlr.c: spdk_nvmf_request_exec

That function classifies the command. Fabrics commands go to:

• lib/nvmf/ctrlr.c: nvmf_ctrlr_process_fabrics_cmd

Admin commands go to:

• lib/nvmf/ctrlr.c: nvmf_ctrlr_process_admin_cmd

I/O commands go to:

• lib/nvmf/ctrlr.c: nvmf_ctrlr_process_io_cmd

The execution entry point first checks subsystem and qpair state. If both are usable, it places the request on the qpair's outstanding list and dispatches by command type. Only commands that return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE finish immediately. Bdev-backed reads and writes normally return asynchronous status.

/* lib/nvmf/ctrlr.c */
void
spdk_nvmf_request_exec(struct spdk_nvmf_request *req)
{
	struct spdk_nvmf_qpair *qpair = req->qpair;
	enum spdk_nvmf_request_exec_status status;

	if (spdk_unlikely(!nvmf_check_subsystem_active(req))) {
		return;
	}
	if (spdk_unlikely(!nvmf_check_qpair_active(req))) {
		return;
	}

	TAILQ_INSERT_TAIL(&qpair->outstanding, req, link);

	if (spdk_unlikely(req->cmd->nvmf_cmd.opcode == SPDK_NVME_OPC_FABRIC)) {
		status = nvmf_ctrlr_process_fabrics_cmd(req);
	} else if (spdk_unlikely(nvmf_qpair_is_admin_queue(qpair))) {
		status = nvmf_ctrlr_process_admin_cmd(req);
	} else {
		status = nvmf_ctrlr_process_io_cmd(req);
	}

	if (status == SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE) {
		_nvmf_request_complete(req);
	}
}

For namespace I/O, the controller code resolves nsid to an NVMf namespace, validates controller state, and checks ANA path state before submitting bdev I/O. This is one place where "listener exists but namespace missing" becomes a real NVMe status rather than a generic network error.

/* lib/nvmf/ctrlr.c */
static int
nvmf_ctrlr_process_io_cmd(struct spdk_nvmf_request *req)
{
	uint32_t nsid;
	struct spdk_nvmf_ns *ns;
	struct spdk_nvmf_qpair *qpair = req->qpair;
	struct spdk_nvmf_ctrlr *ctrlr = qpair->ctrlr;
	struct spdk_nvme_cmd *cmd = &req->cmd->nvme_cmd;
	struct spdk_nvme_cpl *response = &req->rsp->nvme_cpl;

	response->status.sc = SPDK_NVME_SC_SUCCESS;
	nsid = cmd->nsid;

	assert(ctrlr != NULL);
	if (spdk_unlikely(ctrlr->vcprop.cc.bits.en != 1)) {
		response->status.sct = SPDK_NVME_SCT_GENERIC;
		response->status.sc = SPDK_NVME_SC_COMMAND_SEQUENCE_ERROR;
		return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
	}

	ns = nvmf_ctrlr_get_ns(ctrlr, nsid);
	if (spdk_unlikely(ns == NULL || ns->bdev == NULL)) {
		response->status.sc = SPDK_NVME_SC_INVALID_NAMESPACE_OR_FORMAT;
		response->status.dnr = 1;
		return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
	}

The bdev descriptor and I/O channel are fetched from the namespace and the poll group's per-subsystem namespace state. That is the thread-affine handoff: the namespace owns the bdev descriptor, but the poll group owns the channel used on this SPDK thread.

/* lib/nvmf/ctrlr.c */
int
spdk_nvmf_request_get_bdev(uint32_t nsid, struct spdk_nvmf_request *req,
			   struct spdk_bdev **bdev, struct spdk_bdev_desc **desc,
			   struct spdk_io_channel **ch)
{
	struct spdk_nvmf_ctrlr *ctrlr = req->qpair->ctrlr;
	struct spdk_nvmf_ns *ns;
	struct spdk_nvmf_poll_group *group = req->qpair->group;
	struct spdk_nvmf_subsystem_pg_ns_info *ns_info;

	*bdev = NULL;
	*desc = NULL;
	*ch = NULL;

	ns = nvmf_ctrlr_get_ns(ctrlr, nsid);
	if (ns == NULL || ns->bdev == NULL) {
		return -EINVAL;
	}

	assert(group != NULL && group->sgroups != NULL);
	ns_info = &group->sgroups[ctrlr->subsys->id].ns_info[nsid - 1];
	*bdev = ns->bdev;
	*desc = ns->desc;
	*ch = ns_info->channel;

	return 0;
}

The bdev-backed command helpers live in:

• lib/nvmf/ctrlr_bdev.c: nvmf_ctrlr_process_io_cmd_resubmit
• lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrl_queue_io
• lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_read_cmd
• lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_write_cmd
• lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_flush_cmd
• lib/nvmf/ctrlr_bdev.c: nvmf_bdev_ctrlr_unmap

The read and write helpers translate the NVMe command fields into LBA and block counts, validate request size, and submit asynchronous bdev I/O. A successful submission returns SPDK_NVMF_REQUEST_EXEC_STATUS_ASYNCHRONOUS, so spdk_nvmf_request_exec() does not complete the request inline.

/* lib/nvmf/ctrlr_bdev.c */
rc = spdk_bdev_readv_blocks_ext(desc, ch, req->iov, req->iovcnt,
				start_lba, num_blocks,
				nvmf_bdev_ctrlr_complete_cmd, req, &opts);
if (spdk_unlikely(rc)) {
	if (rc == -ENOMEM) {
		nvmf_bdev_ctrl_queue_io(req, bdev, ch,
					nvmf_ctrlr_process_io_cmd_resubmit, req);
		return SPDK_NVMF_REQUEST_EXEC_STATUS_ASYNCHRONOUS;
	}
	rsp->status.sct = SPDK_NVME_SCT_GENERIC;
	rsp->status.sc = SPDK_NVME_SC_INTERNAL_DEVICE_ERROR;
	return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
}

return SPDK_NVMF_REQUEST_EXEC_STATUS_ASYNCHRONOUS;

/* lib/nvmf/ctrlr_bdev.c */
rc = spdk_bdev_writev_blocks_ext(desc, ch, req->iov, req->iovcnt,
				 start_lba, num_blocks,
				 nvmf_bdev_ctrlr_complete_cmd, req, &opts);
if (spdk_unlikely(rc)) {
	if (rc == -ENOMEM) {
		nvmf_bdev_ctrl_queue_io(req, bdev, ch,
					nvmf_ctrlr_process_io_cmd_resubmit, req);
		return SPDK_NVMF_REQUEST_EXEC_STATUS_ASYNCHRONOUS;
	}
	rsp->status.sct = SPDK_NVME_SCT_GENERIC;
	rsp->status.sc = SPDK_NVME_SC_INTERNAL_DEVICE_ERROR;
	return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
}

Completion happens when the bdev module calls back. The callback copies NVMe status out of the bdev I/O, completes the NVMf request, and frees the bdev I/O object.

/* lib/nvmf/ctrlr_bdev.c */
static void
nvmf_bdev_ctrlr_complete_cmd(struct spdk_bdev_io *bdev_io, bool success,
			     void *cb_arg)
{
	struct spdk_nvmf_request *req = cb_arg;
	struct spdk_nvme_cpl *response = &req->rsp->nvme_cpl;
	int sc = 0, sct = 0;
	uint32_t cdw0 = 0;

	spdk_bdev_io_get_nvme_status(bdev_io, &cdw0, &sct, &sc);

	response->cdw0 = cdw0;
	response->status.sc = sc;
	response->status.sct = sct;

	spdk_nvmf_request_complete(req);
	spdk_bdev_free_io(bdev_io);
}

spdk_nvmf_request_complete() moves completion onto the qpair's poll-group thread before the internal completion path returns the request to the transport. That final transport callback is where RDMA, TCP, vfio-user, or FC sends the right completion format back to the host.

/* lib/nvmf/ctrlr.c */
int
spdk_nvmf_request_complete(struct spdk_nvmf_request *req)
{
	struct spdk_nvmf_qpair *qpair = req->qpair;

	spdk_thread_exec_msg(qpair->group->thread, _nvmf_request_complete, req);

	return 0;
}

Prose Diagram: Target Request Flow

Picture a left-to-right diagram with five vertical lanes:

1. Remote host NVMe driver.
2. SPDK transport.
3. SPDK NVMf controller.
4. SPDK bdev layer.
5. Physical or virtual backing bdev.

The arrows are:

Host submits SQE -> transport receives capsule/request -> spdk_nvmf_request_exec -> nvmf_ctrlr_process_io_cmd -> spdk_nvmf_request_get_bdev -> spdk_bdev_writev_blocks_ext or spdk_bdev_readv_blocks_ext -> backing bdev finishes -> bdev callback -> spdk_nvmf_request_complete -> transport sends CQE -> host observes completion.

The important visual detail is that completion is a separate arrow coming back later. Nothing should be drawn as a blocking function call waiting for the SSD.

RDMA, TCP, And vfio-user Distinctions

The common NVMf layer does not care whether a write arrived from RDMA or TCP once it has an spdk_nvmf_request. The transport matters before and after that point.

RDMA:

• Uses RDMA queue pairs and memory registration.
• Sensitive to RNIC, RDMA CM, MTU, PFC/ECN, and hostaddr binding.
• Common in diskengine's storage-node to baremetal path.
• Source anchors: lib/nvmf/rdma.c: spdk_nvmf_request_exec call sites, lib/nvmf/rdma.c: spdk_nvmf_request_complete call sites.

TCP:

• Uses sockets instead of RDMA verbs.
• Easier to bring up, often lower operational barrier, usually higher CPU cost.
• Source anchors: lib/nvmf/tcp.c: spdk_nvmf_request_exec call sites.

vfio-user:

• Looks like a local PCIe device to a VM or client process over a Unix socket.
• Uses guest memory mapping and doorbell handling rather than network packets.
• Source anchors: lib/nvmf/vfio_user.c: nvmf_vfio_user_poll_group, lib/nvmf/vfio_user.c: vfio_user_ctrlr_switch_doorbells, lib/nvmf/vfio_user.c: spdk_nvmf_request_exec call sites.

The official SPDK guide calls out operational differences that matter during debugging: RDMA needs RDMA-capable NICs and verbs support, TCP is built into nvmf_tgt by default, and interrupt mode is supported for vfio-user, TCP, and RDMA. It also calls out RDMA memory-region pressure as a real failure mode when too many small hugepage regions must be registered.

Edge Cases And Failure Modes

Listener exists but subsystem has no namespace:

The host may discover or connect to a subsystem but see no usable capacity. Check nvmf_get_subsystems and verify namespace entries. In diskengine, this usually points at /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go: provisionLvol, /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go: reconcileExports, or a missing lvol bdev.

Namespace bdev disappeared:

If the bdev backing a namespace is removed, outstanding I/O may fail and reconnect behavior depends on the initiator. Read lib/nvmf/subsystem.c namespace removal paths and lib/nvmf/ctrlr.c: spdk_nvmf_request_get_bdev. The namespace stores a bdev pointer and descriptor, so bdev lifetime events have to be handled deliberately.

Host NQN mismatch:

If allow_any_host is false and the host was not added, connect fails even though the network path is fine. Source anchors: include/spdk/nvmf.h: spdk_nvmf_subsystem_add_host_ext, include/spdk/nvmf.h: spdk_nvmf_subsystem_set_allow_any_host. The SPDK target guide's NQN section is worth reading because SPDK compares NQNs byte-for-byte, including UUID letter case.

Transport exists but does not listen:

nvmf_create_transport creates the transport object. It does not by itself create every subsystem listener. Listener creation is separate through nvmf_subsystem_add_listener, whose RPC path first calls spdk_nvmf_tgt_listen_ext() and then associates the address with the subsystem.

Buffer pressure:

Transports may need iobufs for request data. Source anchors: lib/nvmf/transport.c: spdk_nvmf_request_get_buffers, lib/nvmf/transport.c: nvmf_request_iobuf_get_cb, and lib/nvmf/transport.c: nvmf_transport_poll_group_create. RDMA also has NIC memory-registration limits; the official SPDK guide recommends 1 GB hugepages or pre-reserving memory for some cases where too many 2 MB hugepages produce too many memory regions.

Subsystem state transitions:

Some changes require pause/stop/resume semantics. A beginner mistake is to think a namespace list is just an array that can be mutated freely while I/O is running. spdk_nvmf_subsystem_add_ns_ext() checks for inactive or paused state before changing the namespace table; if it is active, the call returns failure.

Bdev resource exhaustion:

Read and write helpers queue I/O on -ENOMEM by using spdk_bdev_queue_io_wait(). That means a transient bdev resource shortage does not necessarily fail the host command immediately. Other submission errors become NVMe internal device errors.

ANA/path state:

The I/O path checks ANA state before bdev submission. In multipath or multi-listener setups, a namespace can exist and the controller can be enabled, but a path state can still reject I/O with a path-related status.

Misconceptions To Kill

"NVMe-oF exports disks."

More precisely, it exports namespaces backed by SPDK bdevs. The bdev may be an NVMe namespace, an lvol, a RAID bdev, a malloc bdev, or another virtual bdev.

"An NQN is an IP address."

An NQN is a name. A listener provides addressability. diskengine stores both because compute nodes need the NQN and the RDMA endpoint.

"Creating a subsystem moves data."

Creating a subsystem changes the target's namespace/control-plane state. Data moves only when a host sends I/O.

"The target thread blocks on remote writes."

The target submits asynchronous bdev I/O and returns later through completion callbacks. Blocking a reactor would harm every qpair on that thread.

"A qpair owns the namespace."

The qpair owns queue state and outstanding requests. The subsystem owns namespaces. The controller links a connected host to one subsystem, and request execution uses that controller plus the request NSID to find the namespace.

Lab: Build A Minimal Mental Config

Without running SPDK, write the minimal JSON-RPC sequence for a storage node exporting one lvol named abcd-uuid through RDMA:

1. nvmf_create_transport with trtype=RDMA.
2. nvmf_create_subsystem with an NQN such as nqn.2024-01.io.excloud:storage.node.disk.lvol.
3. nvmf_subsystem_add_listener with traddr, trsvcid, adrfam, and trtype.
4. nvmf_subsystem_add_ns with namespace bdev abcd-uuid.

Then inspect lib/nvmf/nvmf_rpc.c and identify which C RPC handler decodes each step. Compare your sequence with the official SPDK guide's malloc-bdev example, then replace the malloc bdev name with the lvol UUID that diskengine stores.

Source Reading Exercise

Start at lib/nvmf/ctrlr.c: spdk_nvmf_request_exec. Follow only the I/O command path. Write down:

1. Where the opcode is classified.
2. Where nsid becomes an spdk_nvmf_ns.
3. Where the namespace becomes a bdev descriptor and I/O channel.
4. Where the bdev call is submitted.
5. Where spdk_nvmf_request_complete is called after bdev completion.

Do not read the whole file linearly. Use symbol search and call references.

Suggested reading path:

1. include/spdk/nvmf_transport.h for spdk_nvmf_qpair, spdk_nvmf_request, and spdk_nvmf_transport_ops.
2. lib/nvmf/nvmf_internal.h for spdk_nvmf_subsystem, spdk_nvmf_ns, and spdk_nvmf_ctrlr.
3. module/event/subsystems/nvmf/nvmf_tgt.c for app initialization and poll-group thread creation.
4. lib/nvmf/transport.c for transport creation, listen, poll-group creation, and completion handoff.
5. lib/nvmf/subsystem.c for subsystem state, listener association, and namespace add/remove.
6. lib/nvmf/ctrlr.c for request execution, controller checks, namespace lookup, and completion.
7. lib/nvmf/ctrlr_bdev.c for NVMe read/write/flush/unmap translation into bdev I/O.
8. /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go and provisionlvol.go for diskengine's control-plane use of the SPDK RPCs.

Operational Debug Exercise

Symptom: baremetal node cannot attach a volume over RDMA.

Classify it:

1. Does storage node show the RDMA transport in nvmf_get_transports?
2. Does nvmf_get_subsystems show the target NQN?
3. Does that subsystem have a listener with the expected RDMA IP and port?
4. Does it have a namespace whose bdev_name is the lvol UUID?
5. On the baremetal node, does bdev_nvme_attach_controller fail during connect or succeed but no bdev appears?

Only after answering these should you suspect the bdev layer or lvol metadata. If connect fails before a controller appears, focus on network/RDMA, listener, host NQN, authentication, and transport options. If connect succeeds but no usable namespace appears, focus on namespace visibility, bdev presence, NSID, ANA/path state, and bdev I/O errors.

Self-Check

1. What object owns the NQN?
2. What object owns the RDMA IP and port?
3. Why can a subsystem exist without being useful to a host?
4. Why is spdk_nvmf_request_complete not called immediately after spdk_bdev_writev_blocks_ext?
5. Which diskengine storage-node functions create or verify NVMe-oF exports?
6. Why does spdk_nvmf_request_get_bdev() need both the namespace and the poll group?
7. What is different about the work a transport does before spdk_nvmf_request_exec() and after spdk_nvmf_request_complete()?

References

• Local SPDK: include/spdk/nvmf.h
• Local SPDK: include/spdk/nvmf_transport.h
• Local SPDK: lib/nvmf/nvmf_internal.h
• Local SPDK: module/event/subsystems/nvmf/nvmf_tgt.c
• Local SPDK: lib/nvmf/nvmf.c
• Local SPDK: lib/nvmf/transport.c
• Local SPDK: lib/nvmf/nvmf_rpc.c
• Local SPDK: lib/nvmf/subsystem.c
• Local SPDK: lib/nvmf/ctrlr.c
• Local SPDK: lib/nvmf/ctrlr_bdev.c
• Local SPDK: lib/nvmf/rdma.c
• Local SPDK: lib/nvmf/tcp.c
• Local SPDK: lib/nvmf/vfio_user.c
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go
• Local diskengine: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/utils.go
• SPDK NVMe-oF target documentation: https://spdk.io/doc/nvmf.html
• NVM Express specifications: https://nvmexpress.org/specifications/