Chapter 13: bdev Object Model

Reader Promise

By the end of this chapter you should be able to look at a block device in SPDK and answer five practical questions:

1. What object represents the device?
2. Who registered it?
3. Who is allowed to open it?
4. What per-thread resources are used to submit I/O?
5. Which function is called when an I/O reaches the module?

The short version is this: a bdev is not a disk. A bdev is SPDK's common contract for anything that behaves like a block device. It might be a physical NVMe namespace, a file, a malloc-backed fake disk, a logical volume, a RAID volume, or a virtual bdev stacked on another bdev. The bdev layer gives all of these things one uniform API and one uniform I/O object.

The official SPDK bdev docs describe the same split from two directions. The user guide frames bdev as the block layer equivalent that sits above device drivers and provides pluggable modules, JSON-RPC configuration, stacking, queueing, timeout, reset, and lockless per-thread queues. The programming guide names the basic objects a caller sees: spdk_bdev, spdk_bdev_desc, spdk_bdev_io, and per-thread I/O channels. This chapter stays closer to the source and explains how those objects are wired together.

Why This Matters For diskengine/excloud

diskengine treats SPDK as an external storage engine. It asks SPDK to create, discover, stack, export, resize, and delete storage objects through JSON-RPC. Almost every diskengine operation eventually names a bdev: an NVMe namespace bdev, an lvol bdev, a RAID bdev, or a bdev exported through NVMe-oF, vhost, or vfio-user.

If a volume is missing, stuck, busy, or returning I/O errors, the first debugging step is usually not "look at NVMe." It is "understand the bdev object graph":

• Is the bdev registered?
• Is it still examining?
• Is it open by another module?
• Is it claimed by a virtual bdev module?
• Does the descriptor have write permission?
• Does the caller have a channel on the right SPDK thread?
• Did the module say the I/O type is supported?

Those questions are answered by the object model.

The Core Objects

struct spdk_bdev

The bdev object is the central descriptor for a block-device surface. It contains geometry, capabilities, ownership, module callbacks, and internal lifecycle state.

Source anchor: include/spdk/bdev_module.h:struct spdk_bdev.

Important public-facing fields:

• name: unique bdev name.
• aliases: alternate names.
• product_name: human-readable device class.
• blocklen: logical block size.
• phys_blocklen: physical block size.
• blockcnt: number of logical blocks.
• md_len: metadata bytes per block, when metadata exists.
• dif_type, dif_pi_format, dif_check_flags: protection information details.
• required_alignment: buffer alignment requirement. The bdev layer may allocate bounce buffers if a request violates this.
• split_on_write_unit, split_on_optimal_io_boundary, write_unit_size, and optimal_io_boundary: fields that decide whether core splitting is advisory or mandatory for some I/O shapes.
• preferred_write_alignment, preferred_write_granularity, optimal_write_size, preferred_unmap_alignment, and preferred_unmap_granularity: backend preference fields surfaced by modules such as NVMe.
• max_segment_size, max_num_segments, max_rw_size: constraints that can force request splitting.
• max_unmap, max_unmap_segments, max_write_zeroes, max_copy: operation-specific limits.
• zoned, zone_size, max_zone_append_size, max_open_zones, max_active_zones, and optimal_open_zones: zoned namespace capability fields.
• ctratt and nsid: NVMe-specific attributes exposed through the bdev surface when the module has that information.
• reset_io_drain_timeout: reset behavior control.
• module: the module that registered the bdev.
• fn_table: the module operations called by the bdev layer.

Important internal fields:

• internal.status: normal, unregistering, removing, etc.
• internal.open_descs: descriptors currently open on this bdev.
• internal.examine_in_progress: protects descriptor-backed claims while examine_disk is iterating them.
• internal.claim_type and internal.claim: ownership by virtual modules. The union has a legacy v1 single-module claim and a v2 list of descriptor-backed claims.
• internal.qos: rate-limit state.
• internal.reset_in_progress and internal.queued_resets: reset serialization.
• internal.locked_ranges and internal.pending_locked_ranges: quiesce and range-lock state.
• internal.stat: accumulated statistics from destroyed channels.

Beginner mental model: struct spdk_bdev is a device record. It does not itself do I/O. It points at the module function table that does I/O.

The public part of the structure is intentionally boring: name, geometry, capabilities, and pointers back to the owning module. The important design choice is that the bdev core puts common policy around these fields. For example, alignment and splitting limits are not just documentation; they drive generic bdev behavior before a module sees the request.

struct spdk_bdev {
	/** User context passed in by the backend */
	void *ctxt;

	/** Unique name for this block device. */
	char *name;

	/** Unique product name for this kind of block device. */
	char *product_name;

	/** Size in bytes of a logical block for the backend */
	uint32_t blocklen;

	/** Size in bytes of a physical block for the backend */
	uint32_t phys_blocklen;

	/** Bitmap of supported io types */
	uint32_t io_type_supported;

	/** Number of blocks */
	uint64_t blockcnt;

The source later switches from module-owned description to bdev-core-owned lifecycle state. This boundary matters when reading module code: a module fills the public fields and then calls spdk_bdev_register(). It must not patch the internal descriptor lists, claim state, QoS objects, or unregister callbacks directly.

	/**
	 * Pointer to the bdev module that registered this bdev.
	 */
	struct spdk_bdev_module *module;

	/** function table for all LUN ops */
	const struct spdk_bdev_fn_table *fn_table;

	/** Fields that are used internally by the bdev subsystem.  Bdev modules
	 *  must not read or write to these fields.
	 */
	struct __bdev_internal_fields {
		/** Quality of service parameters */
		struct spdk_bdev_qos *qos;

		/** The bdev status */
		enum spdk_bdev_status status;

		/**
		 * The claim type: used in conjunction with claim. Must hold spinlock on all
		 * updates.
		 */
		enum spdk_bdev_claim_type claim_type;

That is the first ownership rule: the module owns the backend context (ctxt) and the static description of the exported device. The bdev core owns the registry, open descriptor list, claim bookkeeping, reset/QoS state, and notification path.

struct spdk_bdev_module

A bdev module is a producer of one or more bdevs. The module might be physical, like NVMe, or virtual, like passthru, lvol, RAID, crypto, or delay.

Source anchor: include/spdk/bdev_module.h:struct spdk_bdev_module.

Important callbacks and fields:

• module_init: called during bdev subsystem startup.
• module_fini: called during shutdown.
• fini_start: optional early shutdown hook.
• config_json: emits module-level JSON config.
• name: module name.
• get_ctx_size: tells the bdev layer how much per-I/O driver_ctx memory to append to every struct spdk_bdev_io.
• examine_config: first examine pass for virtual modules. It must complete synchronously.
• examine_disk: second examine pass for virtual modules. It may do I/O and complete asynchronously.
• async_init, async_fini, async_fini_start: tell the bdev subsystem that callbacks finish later.

The registration macro is:

Source anchor: include/spdk/bdev_module.h:SPDK_BDEV_MODULE_REGISTER().

Example source anchors:

• module/bdev/null/bdev_null.c:null_if.
• module/bdev/null/bdev_null.c:SPDK_BDEV_MODULE_REGISTER(null, &null_if).
• module/bdev/passthru/vbdev_passthru.c:passthru_if.
• module/bdev/passthru/vbdev_passthru.c:SPDK_BDEV_MODULE_REGISTER(passthru, &passthru_if).
• module/bdev/nvme/bdev_nvme.c:nvme_if.
• module/bdev/nvme/bdev_nvme.c:SPDK_BDEV_MODULE_REGISTER(nvme, &nvme_if).

Misconception to kill: registering a module is not the same as registering a bdev. A module becomes known at process startup. A bdev becomes visible only when the module allocates a struct spdk_bdev, fills it in, and calls spdk_bdev_register().

The module structure is closer to a driver vtable than a disk object. It says how to initialize the module as a subsystem participant, how much per-I/O private space the module needs, and whether the module wants to examine bdevs created by other modules.

struct spdk_bdev_module {
	/**
	 * Initialization function for the module. Called by the bdev library
	 * during startup.
	 */
	int (*module_init)(void);

	/**
	 * Finish function for the module. Called by the bdev library
	 * after all bdevs for all modules have been unregistered.
	 */
	void (*module_fini)(void);

	/** Name for the modules being defined. */
	const char *name;

	/**
	 * Returns the allocation size required for the backend for uses such as local
	 * command structs, local SGL, iovecs, or other user context.
	 */
	int (*get_ctx_size)(void);

The examine callbacks are why virtual modules can appear automatically. A physical module such as NVMe can register an Nvme0n1 bdev, and then virtual modules get a chance to inspect it. examine_config is the synchronous, no-I/O pass for config-driven claims. examine_disk is the second pass where a virtual module may perform I/O and complete asynchronously.

	/**
	 * First notification that a bdev should be examined by a virtual bdev module.
	 * Virtual bdev modules may use this to examine newly-added bdevs and automatically
	 * create their own vbdevs, but no I/O to device can be send to bdev at this point.
	 */
	void (*examine_config)(struct spdk_bdev *bdev);

	/**
	 * Second notification that a bdev should be examined by a virtual bdev module.
	 * Virtual bdev modules may use this to examine newly-added bdevs and automatically
	 * create their own vbdevs. This callback may use I/O operations and finish asynchronously.
	 * Once complete spdk_bdev_module_examine_done() must be called.
	 */
	void (*examine_disk)(struct spdk_bdev *bdev);

The null module shows the difference between module registration and bdev registration in a tiny example. This constructor attribute puts null_if on the bdev module list at process startup. It does not create Null0; it only makes the null module available.

static int
bdev_null_get_ctx_size(void)
{
	return sizeof(struct null_bdev_io);
}

static struct spdk_bdev_module null_if = {
	.name = "null",
	.module_init = bdev_null_initialize,
	.module_fini = bdev_null_finish,
	.async_fini = true,
	.get_ctx_size = bdev_null_get_ctx_size,
};

SPDK_BDEV_MODULE_REGISTER(null, &null_if)

The get_ctx_size callback feeds bdev-core allocation. During bdev subsystem initialization, core asks every registered module for its private I/O size and creates a global pool large enough for the largest module.

static int
bdev_module_get_max_ctx_size(void)
{
	struct spdk_bdev_module *bdev_module;
	int max_bdev_module_size = 0;

	TAILQ_FOREACH(bdev_module, &g_bdev_mgr.bdev_modules, internal.tailq) {
		if (bdev_module->get_ctx_size && bdev_module->get_ctx_size() > max_bdev_module_size) {
			max_bdev_module_size = bdev_module->get_ctx_size();
		}
	}

	return max_bdev_module_size;
}

The practical result is that a module gets bdev_io->driver_ctx memory without allocating it on every I/O. That is a performance and simplicity choice: the generic pool pays the size cost up front, and the hot path avoids a backend-specific malloc.

struct spdk_bdev_fn_table

The function table is the module's implementation of the bdev contract.

Source anchor: include/spdk/bdev_module.h:struct spdk_bdev_fn_table.

Important entries:

• destruct(void *ctx): destroy the backend object. May return 1 for asynchronous destruct and later call spdk_bdev_destruct_done().
• submit_request(struct spdk_io_channel ch, struct spdk_bdev_io bdev_io): handle one bdev I/O.
• io_type_supported(void *ctx, enum spdk_bdev_io_type type): advertise which I/O operations this bdev supports.
• get_io_channel(void *ctx): return a module channel for the current SPDK thread.
• dump_info_json, write_config_json: optional JSON output.
• get_memory_domains: optional memory-domain support.
• reset_device_stat, dump_device_stat_json: optional module-specific statistics.

Example source anchors:

• module/bdev/null/bdev_null.c:null_fn_table.
• module/bdev/passthru/vbdev_passthru.c:vbdev_passthru_fn_table.

Beginner mental model: bdev core code owns the generic policy and lifecycle; the module function table owns the backend-specific work.

The function table is the narrow bridge between common bdev policy and backend-specific implementation. The bdev core can validate ranges, queue for QoS, split a request, retry NOMEM, and track statistics without knowing whether the backend is NVMe, malloc, null, or another bdev. When it is finally time to do backend work, it calls submit_request.

struct spdk_bdev_fn_table {
	/** Destroy the backend block device object. */
	int (*destruct)(void *ctx);

	/** Process the IO. */
	void (*submit_request)(struct spdk_io_channel *ch, struct spdk_bdev_io *);

	/** Check if the block device supports a specific I/O type. */
	bool (*io_type_supported)(void *ctx, enum spdk_bdev_io_type);

	/** Get an I/O channel for the specific bdev for the calling thread. */
	struct spdk_io_channel *(*get_io_channel)(void *ctx);

The null module's table is the minimal shape most new module readers should start with. It names destruction, submission, capability checking, per-thread channel lookup, and JSON config output.

static const struct spdk_bdev_fn_table null_fn_table = {
	.destruct		= bdev_null_destruct,
	.submit_request		= bdev_null_submit_request,
	.io_type_supported	= bdev_null_io_type_supported,
	.get_io_channel		= bdev_null_get_io_channel,
	.write_config_json	= bdev_null_write_config_json,
};

Core dispatch is deliberately direct once generic policy has finished. bdev_submit_request() receives the module channel stored in the bdev channel and calls the module function table. Completion flows back through spdk_bdev_io_complete(), not by returning a status from submit_request().

static inline void
bdev_submit_request(struct spdk_bdev *bdev, struct spdk_io_channel *ioch,
		    struct spdk_bdev_io *bdev_io)
{
	/* The generic bdev layer should not pass an I/O with a dif_check_flags set that
	 * the underlying bdev does not support. Add an assert to check this.
	 */
	assert((bdev_io->type != SPDK_BDEV_IO_TYPE_WRITE &&
		bdev_io->type != SPDK_BDEV_IO_TYPE_READ) ||
	       ((bdev_io->u.bdev.dif_check_flags & bdev->dif_check_flags) ==
		bdev_io->u.bdev.dif_check_flags));

	bdev->fn_table->submit_request(ioch, bdev_io);
}

struct spdk_bdev_desc

A descriptor is an open handle. Applications and modules do not normally submit I/O by holding only a struct spdk_bdev *; they open it and get a descriptor.

Source anchors:

• lib/bdev/bdev.c:spdk_bdev_open_ext().
• lib/bdev/bdev.c:spdk_bdev_open_ext_v2().
• lib/bdev/bdev.c:bdev_open().
• lib/bdev/bdev.c:spdk_bdev_close().

Descriptor facts:

• It is bound to the SPDK thread that opened it.
• It records whether the opener requested write access.
• It stores the event callback used for remove and media-management events.
• It participates in the open_descs list on the bdev.
• It can own claims through newer claim APIs.

Why write permission matters: bdev_open() rejects a write descriptor if the bdev is already claimed by a module in a way that prevents additional writers. You can have many readers, but write access is deliberately constrained because virtual modules need exclusive control when they stack on a base bdev.

The public open API is name-based because applications usually discover or receive bdev names through JSON-RPC configuration. Core looks up the name, allocates a descriptor, records the callback and options, then inserts the descriptor into the bdev's internal open list.

static int
bdev_open(struct spdk_bdev *bdev, bool write, struct spdk_bdev_desc *desc)
{
	struct spdk_thread *thread;
	int rc = 0;

	thread = spdk_get_thread();
	if (!thread) {
		SPDK_ERRLOG("Cannot open bdev from non-SPDK thread.\n");
		return -ENOTSUP;
	}

	desc->bdev = bdev;
	desc->thread = thread;
	desc->write = write;

	spdk_spin_lock(&bdev->internal.spinlock);
	if (bdev->internal.status == SPDK_BDEV_STATUS_UNREGISTERING ||
	    bdev->internal.status == SPDK_BDEV_STATUS_REMOVING) {
		spdk_spin_unlock(&bdev->internal.spinlock);
		return -ENODEV;
	}

The write check happens before the descriptor is published on open_descs. This is why "open for read works, open for write fails" often points to a claim, not to missing hardware.

	if (write && bdev->internal.claim_type != SPDK_BDEV_CLAIM_NONE) {
		LOG_ALREADY_CLAIMED_ERROR("already claimed", bdev);
		spdk_spin_unlock(&bdev->internal.spinlock);
		return -EPERM;
	}

	rc = bdev_start_qos(bdev);
	if (rc != 0) {
		SPDK_ERRLOG("Failed to start QoS on bdev %s\n", bdev->name);
		spdk_spin_unlock(&bdev->internal.spinlock);
		return rc;
	}

	TAILQ_INSERT_TAIL(&bdev->internal.open_descs, desc, link);

Closing has the opposite thread rule. The descriptor records the opening SPDK thread, and close asserts that the same thread is closing it. Virtual modules that open a base bdev and later destruct on a different thread must send a message back to the original thread before closing.

void
spdk_bdev_close(struct spdk_bdev_desc *desc)
{
	struct spdk_bdev *bdev = spdk_bdev_desc_get_bdev(desc);

	assert(desc->thread == spdk_get_thread());

	spdk_poller_unregister(&desc->io_timeout_poller);

	spdk_spin_lock(&g_bdev_mgr.spinlock);
	bdev_close(bdev, desc);
	spdk_spin_unlock(&g_bdev_mgr.spinlock);
}

struct spdk_io_channel And struct spdk_bdev_channel

The public channel type is struct spdk_io_channel. The bdev layer stores bdev-specific state in a struct spdk_bdev_channel as the channel context.

Source anchors:

• lib/bdev/bdev.c:spdk_bdev_get_io_channel().
• lib/bdev/bdev.c:bdev_channel_create().
• lib/bdev/bdev.c:bdev_channel_destroy().

bdev_channel_create() does several important things:

• Calls the module get_io_channel() callback.
• Gets an accel channel.
• Gets a bdev management channel.
• Creates or reuses a shared resource for NOMEM retry state.
• Initializes submitted, locked, QoS, accel, and memory-domain queues.
• Allocates per-channel statistics.
• Copies existing locked ranges into the new channel.
• Enables QoS on the channel if the bdev already has QoS.

Misconception to kill: a channel is not a queue pair by definition. For NVMe bdevs, a channel will lead to an NVMe qpair. For a null bdev, it leads to a simple poller queue. For virtual bdevs, it often contains a base bdev channel. "Channel" means per-thread module state, not a specific hardware object.

The bdev channel is created by the generic SPDK io_device framework. Core first asks the module for its channel, then layers bdev-owned resources around it: accel channel, bdev management channel, NOMEM retry sharing, queues, statistics, locked ranges, and QoS state.

static int
bdev_channel_create(void *io_device, void *ctx_buf)
{
	struct spdk_bdev		*bdev = __bdev_from_io_dev(io_device);
	struct spdk_bdev_channel	*ch = ctx_buf;
	struct spdk_io_channel		*mgmt_io_ch;

	ch->bdev = bdev;
	ch->channel = bdev->fn_table->get_io_channel(bdev->ctxt);
	if (!ch->channel) {
		return -1;
	}

	ch->accel_channel = spdk_accel_get_io_channel();
	if (!ch->accel_channel) {
		spdk_put_io_channel(ch->channel);
		return -1;
	}

	mgmt_io_ch = spdk_get_io_channel(&g_bdev_mgr);
	if (!mgmt_io_ch) {
		spdk_put_io_channel(ch->channel);
		spdk_put_io_channel(ch->accel_channel);
		return -1;
	}

Channel destruction explains why statistics and queued I/O belong in the object model. A channel is per-thread, so its counters would disappear when the thread releases the channel unless core rolls them into bdev-wide accumulated stats. Queued I/O must also be failed or aborted because the per-thread queue owner is going away.

static void
bdev_channel_destroy(void *io_device, void *ctx_buf)
{
	struct spdk_bdev_channel *ch = ctx_buf;

	/* This channel is going away, so add its statistics into the bdev so that they don't get lost. */
	spdk_spin_lock(&ch->bdev->internal.spinlock);
	spdk_bdev_add_io_stat(ch->bdev->internal.stat, ch->stat);
	spdk_spin_unlock(&ch->bdev->internal.spinlock);

	bdev_channel_abort_queued_ios(ch);

	if (ch->histogram) {
		spdk_histogram_data_free(ch->histogram);
	}

	bdev_channel_destroy_resource(ch);
}

struct spdk_bdev_io

Every I/O submitted through the bdev layer becomes a struct spdk_bdev_io.

Source anchor: include/spdk/bdev_module.h:struct spdk_bdev_io.

Public-ish fields:

• bdev: target bdev.
• type: operation type.
• u.bdev: block I/O parameters for read, write, unmap, flush, write zeroes, copy, and zcopy.
• u.reset: reset parameters.
• u.abort: abort parameters.
• u.nvme_passthru: NVMe passthrough command parameters.
• driver_ctx: per-I/O memory reserved for the module using spdk_bdev_module.get_ctx_size.

Internal fields:

• internal.ch: bdev channel.
• internal.desc: descriptor used to submit.
• internal.cb and internal.caller_ctx: user completion callback.
• internal.status: pending, success, failed, NOMEM, NVMe error, etc.
• internal.submit_tsc: timestamp for latency accounting.
• internal.split: parent/child split tracking.
• internal.buf and internal.bounce_buf: iobuf and alignment handling.
• internal.link: queue link reused for NOMEM, QoS, memory-domain, accel, and reset queues.

Misconception to kill: driver_ctx is not a malloc you do yourself per I/O. The bdev layer sizes struct spdk_bdev_io to include module-private memory. The module advertises the size through get_ctx_size().

The I/O object is the handoff record between the caller, bdev core, and the module. The top-level fields identify the target and operation. The union holds operation-specific parameters. The internal section remembers where the I/O came from and how core should complete it. The flexible driver_ctx tail is module-private.

struct spdk_bdev_io {
	/** The block device that this I/O belongs to. */
	struct spdk_bdev *bdev;

	/** Enumerated value representing the I/O type. */
	uint8_t type;

	/** Parameters filled in by the user */
	union {
		struct spdk_bdev_io_block_params bdev;
		struct spdk_bdev_io_reset_params reset;
		struct spdk_bdev_io_abort_params abort;
		struct spdk_bdev_io_nvme_passthru_params nvme_passthru;
		struct spdk_bdev_io_zone_mgmt_params zone_mgmt;
	} u;

	/**
	 *  Fields that are used internally by the bdev subsystem.  Bdev modules
	 *  must not read or write to these fields.
	 */
	struct spdk_bdev_io_internal_fields internal;

	/**
	 * Per I/O context for use by the bdev module.
	 */
	uint8_t driver_ctx[0];

The internal fields are where core stores the bdev channel, descriptor, user completion callback, latency timestamp, status, split state, bounce buffer state, and queue linkage. When debugging completion bugs, this is the map of what core needs to route the I/O back to the original caller.

struct spdk_bdev_io_internal_fields {
	/** The bdev I/O channel that this was handled on. */
	struct spdk_bdev_channel *ch;

	/** Status for the IO */
	int8_t status;

	/** Retry state (resubmit, re-pull, re-push, etc.) */
	uint8_t retry_state;

	/** The bdev descriptor that was used when submitting this I/O. */
	struct spdk_bdev_desc *desc;

	/** User function that will be called when this completes */
	spdk_bdev_io_completion_cb cb;

	/** Context that will be passed to the completion callback */
	void *caller_ctx;

	/** Current tsc at submit time. Used to calculate latency at completion. */
	uint64_t submit_tsc;

The global bdev I/O pool includes the largest advertised module context. That is the other half of the driver_ctx story.

g_bdev_mgr.bdev_io_pool = spdk_mempool_create(mempool_name,
			  g_bdev_opts.bdev_io_pool_size,
			  sizeof(struct spdk_bdev_io) +
			  bdev_module_get_max_ctx_size(),
			  0,
			  SPDK_ENV_NUMA_ID_ANY);

When a channel needs a new I/O, it first tries a per-thread cache and then the global pool. If other callers are already waiting for I/O objects, it does not jump the line.

struct spdk_bdev_io *
bdev_channel_get_io(struct spdk_bdev_channel *channel)
{
	struct spdk_bdev_mgmt_channel *ch = channel->shared_resource->mgmt_ch;
	struct spdk_bdev_io *bdev_io;

	if (ch->per_thread_cache_count > 0) {
		bdev_io = STAILQ_FIRST(&ch->per_thread_cache);
		STAILQ_REMOVE_HEAD(&ch->per_thread_cache, internal.buf_link);
		ch->per_thread_cache_count--;
	} else if (spdk_unlikely(!TAILQ_EMPTY(&ch->io_wait_queue))) {
		bdev_io = NULL;
	} else {
		bdev_io = spdk_mempool_get(g_bdev_mgr.bdev_io_pool);
	}

	return bdev_io;
}

Registration Lifecycle

The simple registration flow is:

1. Module startup registers or prepares module-global state.
2. A concrete bdev object is allocated.
3. The module fills struct spdk_bdev.
4. The module sets bdev->ctxt, bdev->fn_table, and bdev->module.
5. The module calls spdk_bdev_register().
6. The bdev layer inserts the name, creates internal state, opens a temporary descriptor, and runs examine callbacks.
7. When examine is complete, the bdev becomes generally usable.

Source anchors:

• lib/bdev/bdev.c:spdk_bdev_register().
• lib/bdev/bdev.c:bdev_register().
• lib/bdev/bdev.c:bdev_examine().
• lib/bdev/bdev.c:spdk_bdev_wait_for_examine().
• module/bdev/null/bdev_null.c:bdev_null_create().
• module/bdev/null/bdev_null.c:bdev_null_initialize().

The important detail in spdk_bdev_register() is thread ownership. It checks spdk_thread_is_app_thread(NULL) and rejects registration from the wrong thread. This is why modules often bounce lifecycle work back to the app thread.

Here is the core wrapper. It enforces app-thread registration, performs generic registration, opens a temporary descriptor to keep the bdev alive during examine, runs examine, and waits for examine completion before sending the register notification.

int
spdk_bdev_register(struct spdk_bdev *bdev)
{
	struct spdk_bdev_desc *desc;
	struct spdk_thread *thread = spdk_get_thread();
	int rc;

	if (spdk_unlikely(!spdk_thread_is_app_thread(NULL))) {
		SPDK_ERRLOG("Cannot register bdev %s on thread %p (%s)\n", bdev->name, thread,
			    thread ? spdk_thread_get_name(thread) : "null");
		return -EINVAL;
	}

	rc = bdev_register(bdev);
	if (rc != 0) {
		return rc;
	}

	/* A descriptor is opened to prevent bdev deletion during examination */
	rc = bdev_desc_alloc(bdev, _tmp_bdev_event_cb, NULL, NULL, &desc);

The rest of the wrapper shows the temporary descriptor's purpose. Examine can cause virtual modules to inspect or stack on the new bdev. The temporary descriptor keeps the object from being destroyed while that process is still active.

	rc = bdev_open(bdev, false, desc);
	if (rc != 0) {
		bdev_desc_free(desc);
		spdk_bdev_unregister(bdev, NULL, NULL);
		return rc;
	}

	/* Examine configuration before initializing I/O */
	bdev_examine(bdev);

	rc = spdk_bdev_wait_for_examine(bdev_register_finished, desc);
	if (rc != 0) {
		bdev_close(bdev, desc);
		spdk_bdev_unregister(bdev, NULL, NULL);
	}

	return rc;
}

Inside bdev_register(), core initializes the internal state before publishing the name. The order is important: once the name is inserted into the global tree, other threads may find the bdev and create channels.

bdev->internal.status = SPDK_BDEV_STATUS_READY;
bdev->internal.measured_queue_depth = UINT64_MAX;
bdev->internal.claim_type = SPDK_BDEV_CLAIM_NONE;
memset(&bdev->internal.claim, 0, sizeof(bdev->internal.claim));
bdev->internal.qd_poller = NULL;
bdev->internal.qos = NULL;

TAILQ_INIT(&bdev->internal.open_descs);
TAILQ_INIT(&bdev->internal.locked_ranges);
TAILQ_INIT(&bdev->internal.pending_locked_ranges);
TAILQ_INIT(&bdev->internal.queued_resets);
TAILQ_INIT(&bdev->aliases);

The name tree is the global namespace for both primary names and aliases. Duplicate names fail here, before the bdev is inserted into the global bdev list.

/*
 * Register bdev name only after the bdev object is ready.
 * After bdev_name_add returns, it is possible for other threads to start using the bdev,
 * create IO channels...
 */
ret = bdev_name_add(&bdev->internal.bdev_name, bdev, bdev->name);
if (ret != 0) {
	spdk_io_device_unregister(__bdev_to_io_dev(bdev), NULL);
	if (strcmp(bdev->name, uuid) != 0) {
		spdk_bdev_alias_del(bdev, uuid);
	}
	bdev_free_io_stat(bdev->internal.stat);
	spdk_spin_destroy(&bdev->internal.spinlock);
	free(bdev_name);
	return ret;
}

A module-created bdev follows the same pattern regardless of backend. Null bdev creation is a compact example: validate constructor options, allocate the backend object, fill public bdev fields, set the context/function-table/module pointers, then register.

null_disk->bdev.product_name = "Null disk";

null_disk->bdev.write_cache = 0;
null_disk->bdev.blocklen = block_size;
null_disk->bdev.phys_blocklen = opts->physical_block_size;
null_disk->bdev.blockcnt = opts->num_blocks;
null_disk->bdev.md_len = opts->md_size;
null_disk->bdev.md_interleave = true;
null_disk->bdev.dif_type = opts->dif_type;
null_disk->bdev.dif_is_head_of_md = opts->dif_is_head_of_md;

null_disk->bdev.ctxt = null_disk;
null_disk->bdev.fn_table = &null_fn_table;
null_disk->bdev.module = &null_if;

rc = spdk_bdev_register(&null_disk->bdev);
if (rc) {
	free(null_disk->bdev.name);
	free(null_disk);
	return rc;
}

NVMe bdev creation fills the same fields from namespace/controller data instead of RPC options. The object model is the same; only the source of geometry and limits changes.

disk->name = spdk_sprintf_alloc("%sn%d", base_name, spdk_nvme_ns_get_id(ns));
if (!disk->name) {
	return -ENOMEM;
}

disk->write_cache = 0;
if (cdata->vwc.present) {
	/* Enable if the Volatile Write Cache exists */
	disk->write_cache = 1;
}
disk->blocklen = spdk_nvme_ns_get_extended_sector_size(ns);
disk->blockcnt = spdk_nvme_ns_get_num_sectors(ns);
disk->max_segment_size = spdk_nvme_ctrlr_get_max_xfer_size(ctrlr);
disk->nsid = spdk_nvme_ns_get_id(ns);

disk->ctxt = ctx;
disk->fn_table = &nvmelib_fn_table;
disk->module = &nvme_if;

disk->numa.id_valid = 1;
disk->numa.id = spdk_nvme_ctrlr_get_numa_id(ctrlr);

Examine is the bridge from one module's bdev to another module's virtual bdev. Core first runs examine_config for all interested modules. Then it uses the claim state to decide who receives the I/O-capable examine_disk pass.

TAILQ_FOREACH(module, &g_bdev_mgr.bdev_modules, internal.tailq) {
	if (module->examine_config) {
		spdk_spin_lock(&module->internal.spinlock);
		action = module->internal.action_in_progress;
		module->internal.action_in_progress++;
		spdk_spin_unlock(&module->internal.spinlock);
		module->examine_config(bdev);
		if (action != module->internal.action_in_progress) {
			SPDK_ERRLOG("examine_config for module %s did not call "
				    "spdk_bdev_module_examine_done()\n", module->name);
		}
	}
}

switch (bdev->internal.claim_type) {
case SPDK_BDEV_CLAIM_NONE:
	/* Examine by all bdev modules */
	TAILQ_FOREACH(module, &g_bdev_mgr.bdev_modules, internal.tailq) {
		if (module->examine_disk) {
			spdk_spin_lock(&module->internal.spinlock);
			module->internal.action_in_progress++;
			spdk_spin_unlock(&module->internal.spinlock);
			spdk_spin_unlock(&bdev->internal.spinlock);
			module->examine_disk(bdev);
			spdk_spin_lock(&bdev->internal.spinlock);
		}
	}
	break;
case SPDK_BDEV_CLAIM_EXCL_WRITE:
	/* Examine by the one bdev module with a v1 claim */
	module = bdev->internal.claim.v1.module;
	break;
default:
	/* Examine by all bdev modules with a v2 claim */
	assert(claim_type_is_v2(bdev->internal.claim_type));
	bdev->internal.examine_in_progress++;

	TAILQ_FOREACH(claim, &bdev->internal.claim.v2.claims, link) {
		module = claim->module;
		if (module == NULL || module->examine_disk == NULL) {
			continue;
		}
		module->examine_disk(bdev);
	}
}

That default branch is the current descriptor-backed claim path. Core may call more than one claiming module's examine_disk() when the claim type permits a set of compatible v2 claims. Claims released while that iteration is in progress are left as vestigial nodes until the iteration finishes, then removed.

Open, Claim, And Stack

Virtual bdevs sit on base bdevs. To do this safely, they usually:

1. Open the base bdev with spdk_bdev_open_ext().
2. Store the base descriptor.
3. Claim the base bdev with a claim type that matches the virtual module's write semantics.
4. Create and register a new virtual bdev.
5. On destruct, release the claim if it is legacy-owned, then close the base descriptor. Descriptor-backed claims are released when their descriptor is closed.

Source anchors:

• include/spdk/bdev_module.h:enum spdk_bdev_claim_type.
• include/spdk/bdev_module.h:spdk_bdev_module_claim_bdev().
• include/spdk/bdev_module.h:spdk_bdev_module_claim_bdev_desc().
• include/spdk/bdev_module.h:spdk_bdev_module_release_bdev().
• module/bdev/passthru/vbdev_passthru.c:vbdev_passthru_register().
• module/bdev/passthru/vbdev_passthru.c:vbdev_passthru_destruct().

The passthru module is intentionally simple and therefore valuable. In vbdev_passthru_register(), it opens the base bdev, copies geometry to the virtual bdev, registers an io_device for per-thread virtual-bdev state, claims the base bdev, and then registers the virtual bdev. In vbdev_passthru_destruct(), it removes itself from the global list, releases the base claim, closes the base descriptor on the original thread, and unregisters its io_device.

Misconception to kill: a claim is not the same as an open descriptor. A descriptor says "I have a handle." A claim says "my module owns a stacking relationship that constrains other writers."

For new virtual bdev consumers, the preferred claim API is spdk_bdev_module_claim_bdev_desc(). It attaches the claim to the descriptor that was used to open the base bdev. The claim is then released automatically when that descriptor is closed, which makes lifetime easier to audit.

The current claim types are deliberately more precise than "claimed or not":

• SPDK_BDEV_CLAIM_READ_MANY_WRITE_ONE: many readers are allowed, and this descriptor is the only writer.
• SPDK_BDEV_CLAIM_READ_MANY_WRITE_NONE: readers are allowed, but writers are blocked.
• SPDK_BDEV_CLAIM_EXCL_WRITE: the legacy exclusive-writer shape used by spdk_bdev_module_claim_bdev().
• SPDK_BDEV_CLAIM_READ_MANY_WRITE_SHARED: cooperating writers are allowed when they use the same shared claim key.

A module can open the base read-only and then request READ_MANY_WRITE_ONE or compatible shared write access. If the claim is granted, the descriptor can be promoted to write access. This is why modern code should choose a claim type that describes the real stacking contract instead of reaching for a blanket exclusive claim by habit.

The older claim helper is still in the tree and is short enough to read directly. Passthru uses it, so it remains a useful example of the legacy v1 claim path. It refuses to claim an already-claimed bdev, upgrades the descriptor to write if one was supplied, and records the claiming module in internal state.

int
spdk_bdev_module_claim_bdev(struct spdk_bdev *bdev, struct spdk_bdev_desc *desc,
			    struct spdk_bdev_module *module)
{
	spdk_spin_lock(&bdev->internal.spinlock);

	if (bdev->internal.claim_type != SPDK_BDEV_CLAIM_NONE) {
		LOG_ALREADY_CLAIMED_ERROR("already claimed", bdev);
		spdk_spin_unlock(&bdev->internal.spinlock);
		return -EPERM;
	}

	if (desc && !desc->write) {
		desc->write = true;
	}

	bdev->internal.claim_type = SPDK_BDEV_CLAIM_EXCL_WRITE;
	bdev->internal.claim.v1.module = module;

Passthru shows the full stacking relationship in ordinary module code. It opens the base bdev for write, stores the descriptor and base pointer, copies the base bdev properties that define the virtual surface, sets the virtual bdev's callback pointers, registers a per-thread io_device, then claims the base bdev before registering the virtual bdev.

/* The base bdev that we're attaching to. */
rc = spdk_bdev_open_ext(bdev_name, true, vbdev_passthru_base_bdev_event_cb,
			NULL, &pt_node->base_desc);
if (rc) {
	if (rc != -ENODEV) {
		SPDK_ERRLOG("could not open bdev %s\n", bdev_name);
	}
	free(pt_node->pt_bdev.name);
	free(pt_node);
	break;
}

bdev = spdk_bdev_desc_get_bdev(pt_node->base_desc);
pt_node->base_bdev = bdev;

/* Copy some properties from the underlying base bdev. */
pt_node->pt_bdev.write_cache = bdev->write_cache;
pt_node->pt_bdev.required_alignment = bdev->required_alignment;
pt_node->pt_bdev.optimal_io_boundary = bdev->optimal_io_boundary;
pt_node->pt_bdev.blocklen = bdev->blocklen;
pt_node->pt_bdev.blockcnt = bdev->blockcnt;

pt_node->pt_bdev.md_interleave = bdev->md_interleave;
pt_node->pt_bdev.md_len = bdev->md_len;
pt_node->pt_bdev.dif_type = bdev->dif_type;
pt_node->pt_bdev.dif_is_head_of_md = bdev->dif_is_head_of_md;
pt_node->pt_bdev.dif_check_flags = bdev->dif_check_flags;
pt_node->pt_bdev.dif_pi_format = bdev->dif_pi_format;

pt_node->pt_bdev.ctxt = pt_node;
pt_node->pt_bdev.fn_table = &vbdev_passthru_fn_table;
pt_node->pt_bdev.module = &passthru_if;
TAILQ_INSERT_TAIL(&g_pt_nodes, pt_node, link);

spdk_io_device_register(pt_node, pt_bdev_ch_create_cb, pt_bdev_ch_destroy_cb,
			sizeof(struct pt_io_channel),
			name->vbdev_name);

pt_node->thread = spdk_get_thread();

rc = spdk_bdev_module_claim_bdev(bdev, pt_node->base_desc, pt_node->pt_bdev.module);

Destruct reverses that ownership. The virtual module removes its global node, releases the base claim, and closes the base descriptor on the thread where it was opened. That last detail follows directly from spdk_bdev_close() asserting descriptor thread ownership.

static int
vbdev_passthru_destruct(void *ctx)
{
	struct vbdev_passthru *pt_node = (struct vbdev_passthru *)ctx;

	TAILQ_REMOVE(&g_pt_nodes, pt_node, link);

	/* Unclaim the underlying bdev. */
	spdk_bdev_module_release_bdev(pt_node->base_bdev);

	/* Close the underlying bdev on its same opened thread. */
	if (pt_node->thread && pt_node->thread != spdk_get_thread()) {
		spdk_thread_send_msg(pt_node->thread, _vbdev_passthru_destruct, pt_node->base_desc);
	} else {
		spdk_bdev_close(pt_node->base_desc);
	}

Prose Diagram

Imagine a vertical diagram with five boxes:

Top box: "Application or upper SPDK layer." It holds a spdk_bdev_desc and a spdk_io_channel.

Second box: "bdev core." It validates block ranges, allocates spdk_bdev_io, applies splitting, QoS, reset, NOMEM retry, statistics, and completion routing.

Third box: "struct spdk_bdev." This is the named object with geometry and a pointer to fn_table.

Fourth box: "module function table." submit_request, get_io_channel, io_type_supported, destruct.

Bottom box: "backend." This may be an NVMe namespace, a file, malloc memory, another bdev, or a network connection.

Arrows go down for submission. Arrows go up for completion. Side arrows from the bdev box point to descriptors, claims, QoS, reset state, and locked ranges.

Edge Cases And Failure Modes

Duplicate names fail at registration time because the name tree is global across bdev names and aliases. This is why a UUID alias can collide with some other bdev name, and why a module must treat -EEXIST from spdk_bdev_register() as a real publication failure rather than as a harmless warning.

Thread placement errors usually show up as registration or close failures. Registration must happen on the SPDK app thread, and descriptor close asserts that the current thread is the descriptor's opening thread. A virtual module that opens a base bdev on one thread and destructs on another must bounce the close back to the original thread, as passthru does.

Open failures are often permission or lifecycle failures, not device-discovery failures. spdk_bdev_open_ext_v2() rejects a missing event callback because open descriptors must receive remove/media-management events. bdev_open() rejects write opens when the bdev is already claimed, and it rejects opens while the bdev is unregistering or removing. When debugging this class of issue, inspect the bdev's claim state and internal status before chasing the physical backend.

Unregister is a start signal, not always a completed destruction. Core notifies every open descriptor of hotremove and may defer final destruction until descriptors close. If a module's destruct() cannot finish immediately, it can return 1 and later call spdk_bdev_destruct_done(). This is why shutdown bugs often look like a stuck object graph: some descriptor, channel, claim, or async destruct path is still holding part of the lifecycle open.

Channel teardown has data-path consequences. When a channel is destroyed, core rolls the channel's stats into the bdev-wide accumulator and aborts queued I/O that was waiting on NOMEM or iobuf state. If a caller releases channels while work is still expected to complete, the failure will surface as aborted or failed I/O, not as a backend-specific error.

Virtual bdev geometry bugs are easy to create because a virtual bdev must present a complete and consistent surface. Copying only blocklen and blockcnt is not enough if metadata, DIF, alignment, write unit, memory domains, or operation limits differ from the base. Hidden metadata adds another wrinkle: descriptor options can change the block size observed by a caller, so bdev->blocklen is not always the byte count a descriptor-facing path should assume.

Claims must be released as part of virtual bdev teardown. If a module keeps a base bdev claimed after its virtual bdev has disappeared, later write opens can fail and shutdown can remain blocked. The claim is the ownership relationship; the descriptor is only the handle used to access the base.

Misconceptions To Kill

• "A bdev is always hardware." No. It is an abstraction.
• "The module owns all policy." No. The bdev core owns common policy like splitting, QoS, reset gating, NOMEM retry, and completion routing.
• "The public channel is the module channel." Not exactly. The public channel is a bdev channel whose context contains or points to the module's channel.
• "A descriptor can be closed anywhere." No. It is tied to the opening SPDK thread.
• "A virtual bdev should just keep a pointer to its base bdev." It normally needs a descriptor, a claim, an event callback, and per-thread base channels.
• "Returning success from an RPC means all examine side effects are visible." Not always. Some paths wait for examine; others may require understanding async examine.

Source Reading Exercise

Read these in order:

1. module/bdev/null/bdev_null.c:bdev_null_create().
2. module/bdev/null/bdev_null.c:null_fn_table.
3. lib/bdev/bdev.c:spdk_bdev_register().
4. lib/bdev/bdev.c:spdk_bdev_open_ext().
5. lib/bdev/bdev.c:bdev_channel_create().
6. include/spdk/bdev_module.h:struct spdk_bdev_io.

Questions:

• Where is the null bdev's backend context stored?
• Which function returns the null module's per-thread channel?
• Where does the bdev layer store the user's completion callback?
• Why does spdk_bdev_register() open a temporary descriptor?
• Which parts of struct spdk_bdev should a module fill, and which parts are explicitly internal?

Operational Lab

No live SPDK system is required.

1. Pick a bdev name that appears in an RPC config, for example Nvme0n1.
2. Determine which module owns it by finding the constructor RPC. For NVMe, the constructor is usually bdev_nvme_attach_controller.
3. Find that module's struct spdk_bdev_module.
4. Find the module's struct spdk_bdev_fn_table.
5. Find the function that fills bdev->name, bdev->blocklen, bdev->blockcnt, bdev->ctxt, bdev->fn_table, and bdev->module.
6. Write down how the module would destroy that bdev.

Expected outcome: you should be able to explain the bdev's owner, lifecycle, and I/O dispatch function without running SPDK.

Self-Check

1. What is the difference between struct spdk_bdev, struct spdk_bdev_desc, and struct spdk_io_channel?
2. Why does a module provide get_ctx_size()?
3. What is the role of spdk_bdev_module_claim_bdev() in a virtual bdev?
4. Why can unregister be delayed after spdk_bdev_unregister() is called?
5. Why is submit_request() not a public API for applications?
6. Which source function should you read first when debugging "bdev exists but open fails"?
7. Which source function should you read first when debugging "channel creation fails"?
8. Why is copying a base bdev's block size not enough to implement a correct virtual bdev?

References

• Local source: include/spdk/bdev_module.h.
• Local source: lib/bdev/bdev.c.
• Local source: module/bdev/null/bdev_null.c.
• Local source: module/bdev/passthru/vbdev_passthru.c.
• Local source: module/bdev/nvme/bdev_nvme.c.
• SPDK Block Device User Guide: https://spdk.io/doc/bdev.html
• SPDK Block Device Layer Programming Guide: https://spdk.io/doc/bdev_pg.html
• SPDK Writing a Custom Block Device Module: https://spdk.io/doc/bdev_module.html
• SPDK bdev module header reference: https://spdk.io/doc/bdev__module_8h.html