Chapter 15: Writing A bdev Module | SPDK From First Principles

Reader Promise

By the end of this chapter you should be able to sketch a small bdev module and know where the hard parts are. You will know the difference between a physical bdev module and a virtual bdev module, how to allocate and register a bdev, how to handle I/O, how to forward I/O to a base bdev, how claims work, and how to tear down without leaving dangling descriptors or channels.

This is not a full coding tutorial with a patch to compile. It is a source-guided design draft. The goal is to make the existing SPDK examples readable before you write code.

Primary references for this chapter:

SPDK documentation: Writing a Custom Block Device Module.
SPDK documentation: bdev_module.h File Reference.
SPDK documentation: Block Device Layer Programming Guide.
Local source: include/spdk/bdev_module.h, module/bdev/null/bdev_null.c, and module/bdev/passthru/vbdev_passthru.c.

The official custom-module guide makes two important points that shape the chapter: a module is SPDK's driver-like integration point for the bdev layer, and the module must allocate/fill a struct spdk_bdev before calling spdk_bdev_register(). The generated bdev_module.h docs add the current API details, including asynchronous destruct rules and the app-thread requirement for registration.

The Two Objects You Implement

A bdev module has two related but different objects. The module object describes the driver family: its name, initialization and finish callbacks, how much per-I/O context it needs, and optional examine callbacks used by virtual bdevs. The bdev object describes one exposed block device: its name, size, block length, metadata format, function table, module pointer, and module-owned context.

The important division is that the module object is global to the module, while each bdev object is one exported disk. A single module can own zero, one, or many bdevs. For example, the null module's module object is one static null_if, while each null disk created by RPC gets its own struct null_bdev containing an embedded struct spdk_bdev.

include/spdk/bdev_module.h shows the module callbacks that a module author chooses from. This excerpt is intentionally partial; the full struct includes more optional hooks.

struct spdk_bdev_module {
	/**
	 * Initialization function for the module. Called by the bdev library
	 * during startup.
	 *
	 * Modules are required to define this function.
	 */
	int (*module_init)(void);

	/**
	 * Finish function for the module. Called by the bdev library
	 * after all bdevs for all modules have been unregistered.
	 */
	void (*module_fini)(void);

	/** Name for the modules being defined. */
	const char *name;

	/**
	 * Returns the allocation size required for the backend for uses such as local
	 * command structs, local SGL, iovecs, or other user context.
	 */
	int (*get_ctx_size)(void);

	/**
	 * First notification that a bdev should be examined by a virtual bdev module.
	 */
	void (*examine_config)(struct spdk_bdev *bdev);

get_ctx_size() is easy to underestimate. It is the normal place to reserve per-I/O scratch space. When bdev core allocates a struct spdk_bdev_io for this module, it also makes room for the module's private context, and the module later reaches it through bdev_io->driver_ctx. That avoids a malloc() in the hot submit path for common per-I/O state.

The bdev function table is the per-device operational contract. If a bdev is registered, bdev core can ask whether it supports an I/O type, ask for the current thread's channel, submit an I/O, and eventually call its destructor.

struct spdk_bdev_fn_table {
	/** Destroy the backend block device object. If the destruct process
	 *  for the bdev is asynchronous, return 1 from this function, and
	 *  then call spdk_bdev_destruct_done() once the async work is
	 *  complete.
	 */
	int (*destruct)(void *ctx);

	/** Process the IO. */
	void (*submit_request)(struct spdk_io_channel *ch, struct spdk_bdev_io *);

	/** Check if the block device supports a specific I/O type. */
	bool (*io_type_supported)(void *ctx, enum spdk_bdev_io_type);

	/** Get an I/O channel for the specific bdev for the calling thread. */
	struct spdk_io_channel *(*get_io_channel)(void *ctx);

	int (*dump_info_json)(void *ctx, struct spdk_json_write_ctx *w);
	void (*write_config_json)(struct spdk_bdev *bdev, struct spdk_json_write_ctx *w);

The function table is not only for data movement. It is also where lifecycle and introspection enter. destruct() owns the last chance to release module-private memory. write_config_json() or the module-level config_json() lets SPDK save enough JSON-RPC state to reconstruct a bdev. io_type_supported() is part of correctness because upper layers use it to decide what commands they may submit.

Physical vs Virtual bdev Modules

A physical bdev module presents a device-like backend directly. Examples include NVMe, malloc, null, aio, and uring. A physical module's submit_request() usually talks to hardware, a file descriptor, memory, or a network transport.

A virtual bdev module presents a new bdev stacked on one or more base bdevs. Examples include passthru, lvol, RAID, crypto, delay, and split. A virtual module's submit_request() usually transforms or forwards I/O to base bdevs. The block-device programming guide calls these "virtual bdevs" and describes layering as the way SPDK implements RAID, caching, logical volumes, and similar constructs.

The difference matters because a virtual bdev module has extra responsibilities:

Open each base bdev with spdk_bdev_open_ext().
Register an event callback for base bdev removal.
Claim the base bdev when exclusive stacking is required.
Create per-thread base channels under the virtual bdev's own channel.
Submit new base I/O through public bdev APIs.
Complete the original virtual I/O after the base I/O completes.
Release claims and close descriptors during destruct.

A physical bdev can still have complicated thread, DMA, and reset logic. The difference is ownership: a physical module owns its backend directly, while a virtual module is a consumer of one or more bdevs and a provider of a new bdev at the same time.

Minimal Physical Module: null

The null bdev is a compact physical example. It does not store data; it accepts reads and writes, completes them later from a poller, and optionally generates or verifies DIF. It is intentionally simple, but it still demonstrates the same object shapes used by real modules.

The module has three local state shapes:

struct null_bdev_io is per-I/O context.
struct null_bdev is one exported disk and embeds struct spdk_bdev.
struct null_io_channel is per-thread state and owns a poller plus a queue of pending null I/O.

struct null_bdev_io {
	TAILQ_ENTRY(null_bdev_io) link;
};

struct null_bdev {
	struct spdk_bdev	bdev;
	TAILQ_ENTRY(null_bdev)	tailq;
};

struct null_io_channel {
	struct spdk_poller		*poller;
	TAILQ_HEAD(, null_bdev_io)	io;
};

static int
bdev_null_get_ctx_size(void)
{
	return sizeof(struct null_bdev_io);
}

This is the smallest useful ownership model. The struct spdk_bdev is embedded in the module's object so that the destructor can recover the whole struct null_bdev from bdev.ctxt. The per-I/O context is only a queue link because null does not need an operation descriptor, DMA handle, or backend request. A real physical module often puts hardware command handles, SGL state, or retry state there.

Null's module object and registration macro connect that local implementation to the bdev subsystem:

static struct spdk_bdev_module null_if = {
	.name = "null",
	.module_init = bdev_null_initialize,
	.module_fini = bdev_null_finish,
	.async_fini = true,
	.get_ctx_size = bdev_null_get_ctx_size,
};

SPDK_BDEV_MODULE_REGISTER(null, &null_if)

SPDK_BDEV_MODULE_REGISTER() puts the module on SPDK's module list through a constructor. That does not create a disk. It only makes the module known so bdev initialization can call module_init() and so RPC code can later create bdev instances.

Registering A null bdev

bdev_null_create() validates input, allocates the backend object, fills the embedded spdk_bdev, and calls spdk_bdev_register(). The exact geometry fields vary by module, but the core pattern is stable: name, product, block geometry, metadata, context, function table, and module pointer.

null_disk = calloc(1, sizeof(*null_disk));
if (!null_disk) {
	SPDK_ERRLOG("could not allocate null_bdev\n");
	return -ENOMEM;
}

null_disk->bdev.name = strdup(opts->name);
if (!null_disk->bdev.name) {
	free(null_disk);
	return -ENOMEM;
}
null_disk->bdev.product_name = "Null disk";

null_disk->bdev.write_cache = 0;
null_disk->bdev.blocklen = block_size;
null_disk->bdev.phys_blocklen = opts->physical_block_size;
null_disk->bdev.blockcnt = opts->num_blocks;
null_disk->bdev.md_len = opts->md_size;
null_disk->bdev.md_interleave = true;

Later in the same function, null binds the bdev to its callbacks and registers it.

null_disk->bdev.ctxt = null_disk;
null_disk->bdev.fn_table = &null_fn_table;
null_disk->bdev.module = &null_if;

rc = spdk_bdev_register(&null_disk->bdev);
if (rc) {
	free(null_disk->bdev.name);
	free(null_disk);
	return rc;
}

*bdev = &(null_disk->bdev);

TAILQ_INSERT_TAIL(&g_null_bdev_head, null_disk, tailq);

The order matters. Until spdk_bdev_register() succeeds, this object is private allocation owned by the create function. After it succeeds, bdev core can expose it to callers. If registration fails, null frees the name and object immediately. That reverse-order cleanup is the pattern to copy into your own create path.

The function table is deliberately boring:

static const struct spdk_bdev_fn_table null_fn_table = {
	.destruct		= bdev_null_destruct,
	.submit_request		= bdev_null_submit_request,
	.io_type_supported	= bdev_null_io_type_supported,
	.get_io_channel		= bdev_null_get_io_channel,
	.write_config_json	= bdev_null_write_config_json,
};

The useful lesson is that "minimal" does not mean "only submit_request." A bdev without an honest capability callback, channel callback, and destruct path is not a complete bdev.

Channels And Poller Completion

Null registers an io_device during module initialization. An io_device is the key SPDK's channel layer uses to allocate one channel context per SPDK thread. Null uses the address of its global tailq as the key because it needs one module-wide io_device rather than one per disk.

bdev_null_initialize(void)
{
	g_null_read_buf = spdk_zmalloc(SPDK_BDEV_LARGE_BUF_MAX_SIZE, 0, NULL,
				       SPDK_ENV_NUMA_ID_ANY, SPDK_MALLOC_DMA);
	if (g_null_read_buf == NULL) {
		return -1;
	}

	spdk_io_device_register(&g_null_bdev_head, null_bdev_create_cb, null_bdev_destroy_cb,
				sizeof(struct null_io_channel), "null_bdev");

	return 0;
}

static struct spdk_io_channel *
bdev_null_get_io_channel(void *ctx)
{
	return spdk_get_io_channel(&g_null_bdev_head);
}

The channel create callback initializes the per-thread queue and registers a poller. The destroy callback unregisters the poller. This is why submit_request() can be lightweight: it only appends work to the current thread's queue.

static int
null_bdev_create_cb(void *io_device, void *ctx_buf)
{
	struct null_io_channel *ch = ctx_buf;

	TAILQ_INIT(&ch->io);
	ch->poller = SPDK_POLLER_REGISTER(null_io_poll, ch, 0);

	return 0;
}

static void
null_bdev_destroy_cb(void *io_device, void *ctx_buf)
{
	struct null_io_channel *ch = ctx_buf;

	spdk_poller_unregister(&ch->poller);
}

Even a "do nothing" bdev should avoid teaching itself bad habits by completing every request inline. Null queues supported I/O and completes it from the poller, which more closely resembles a real asynchronous backend.

switch (bdev_io->type) {
case SPDK_BDEV_IO_TYPE_READ:
	/* read buffer and DIF handling omitted */
	TAILQ_INSERT_TAIL(&ch->io, null_io, link);
	break;
case SPDK_BDEV_IO_TYPE_WRITE:
	/* DIF verification omitted */
	TAILQ_INSERT_TAIL(&ch->io, null_io, link);
	break;
case SPDK_BDEV_IO_TYPE_WRITE_ZEROES:
case SPDK_BDEV_IO_TYPE_RESET:
	TAILQ_INSERT_TAIL(&ch->io, null_io, link);
	break;
case SPDK_BDEV_IO_TYPE_ABORT:
	if (bdev_null_abort_io(ch, bdev_io->u.abort.bio_to_abort)) {
		spdk_bdev_io_complete(bdev_io, SPDK_BDEV_IO_STATUS_SUCCESS);
	} else {
		spdk_bdev_io_complete(bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
	}
	break;
case SPDK_BDEV_IO_TYPE_FLUSH:
case SPDK_BDEV_IO_TYPE_UNMAP:
default:
	spdk_bdev_io_complete(bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
	break;
}

Notice the exactly-once rule in the switch. A queued read/write/reset/write-zeroes is completed later. Abort, unsupported flush, unsupported unmap, and unknown types are completed before returning. There is no path where a received I/O is ignored.

The poller drains the queue by swapping it into a local list first. That keeps the channel queue available for new submissions while completions are being issued.

null_io_poll(void *arg)
{
	struct null_io_channel		*ch = arg;
	TAILQ_HEAD(, null_bdev_io)	io;
	struct null_bdev_io		*null_io;

	TAILQ_INIT(&io);
	TAILQ_SWAP(&ch->io, &io, null_bdev_io, link);

	if (TAILQ_EMPTY(&io)) {
		return SPDK_POLLER_IDLE;
	}

	while (!TAILQ_EMPTY(&io)) {
		null_io = TAILQ_FIRST(&io);
		TAILQ_REMOVE(&io, null_io, link);
		spdk_bdev_io_complete(spdk_bdev_io_from_ctx(null_io), SPDK_BDEV_IO_STATUS_SUCCESS);
	}

	return SPDK_POLLER_BUSY;
}

The destructor is synchronous because no backend close, hardware reset, or async unregister remains. It removes the disk from the module list, frees the bdev name, frees the containing object, and returns zero.

bdev_null_destruct(void *ctx)
{
	struct null_bdev *bdev = ctx;

	TAILQ_REMOVE(&g_null_bdev_head, bdev, tailq);
	free(bdev->bdev.name);
	free(bdev);

	return 0;
}

Null's module_fini() is separate from per-bdev destruct. Destruct frees one disk. Module fini releases module-wide resources, here the shared read buffer and io_device. Because null_if.async_fini = true, bdev_null_finish() must eventually call spdk_bdev_module_fini_done(), either directly or from its unregister callback.

Minimal Virtual Module: passthru

Passthru is the canonical first virtual bdev because it mostly forwards I/O unchanged. That makes it useful for learning object ownership, even though real virtual bdevs often modify offsets, encrypt data, split I/O, store metadata, or coordinate several base devices.

The object layout shows the central difference from null. Passthru owns a virtual pt_bdev, but it also stores a pointer and descriptor for the base bdev it consumes.

struct vbdev_passthru {
	struct spdk_bdev		*base_bdev; /* the thing we're attaching to */
	struct spdk_bdev_desc		*base_desc; /* its descriptor we get from open */
	struct spdk_bdev		pt_bdev;    /* the PT virtual bdev */
	TAILQ_ENTRY(vbdev_passthru)	link;
	struct spdk_thread		*thread;    /* thread where base device is opened */
};

struct pt_io_channel {
	struct spdk_io_channel	*base_ch; /* IO channel of base device */
};

struct passthru_bdev_io {
	uint8_t test;
	struct spdk_io_channel *ch;
	struct spdk_bdev_io_wait_entry bdev_io_wait;
};

The descriptor is the open handle. The base channel is per-thread and is obtained from that descriptor. The stored thread exists because spdk_bdev_close() must happen on the same thread that opened the descriptor. This thread ownership is not decoration; getting it wrong can leave descriptors open or close them from the wrong execution context.

Passthru's module object uses examine_config, which is the callback virtual modules use to inspect newly appearing bdevs and decide whether to stack on them.

static struct spdk_bdev_module passthru_if = {
	.name = "passthru",
	.module_init = vbdev_passthru_init,
	.get_ctx_size = vbdev_passthru_get_ctx_size,
	.examine_config = vbdev_passthru_examine,
	.module_fini = vbdev_passthru_finish,
	.config_json = vbdev_passthru_config_json
};

SPDK_BDEV_MODULE_REGISTER(passthru, &passthru_if)

For virtual modules, examine is a discovery bridge. An RPC may store "when base X appears, create virtual Y." Later, when bdev core notifies the module about a candidate base bdev, examine_config() can create the virtual bdev if the name matches.

static void
vbdev_passthru_examine(struct spdk_bdev *bdev)
{
	vbdev_passthru_register(bdev->name);

	spdk_bdev_module_examine_done(&passthru_if);
}

The call to spdk_bdev_module_examine_done() is mandatory. bdev_module.h documents that examine_config must call it before returning, while examine_disk may do asynchronous work and call it later. Forgetting this call can stall bdev subsystem progress because SPDK waits for modules to finish examining a bdev.

Notice the return type: both examine_config and examine_disk are void callbacks. Completion is reported by calling spdk_bdev_module_examine_done(), not by returning a status code.

Opening, Claiming, And Registering

Passthru creation starts by opening the base bdev. The event callback is not optional for a stacked bdev, because the base may disappear while the virtual bdev still exists.

rc = spdk_bdev_open_ext(bdev_name, true, vbdev_passthru_base_bdev_event_cb,
			NULL, &pt_node->base_desc);
if (rc) {
	if (rc != -ENODEV) {
		SPDK_ERRLOG("could not open bdev %s\n", bdev_name);
	}
	free(pt_node->pt_bdev.name);
	free(pt_node);
	break;
}

bdev = spdk_bdev_desc_get_bdev(pt_node->base_desc);
pt_node->base_bdev = bdev;

The true argument requests write access to the base. A read-only virtual module may use different permissions, but a forwarding write path needs a writable descriptor. The block-device programming guide notes that opening with write permission may fail if another virtual bdev has claimed the base.

After opening the base, passthru copies the properties that make the virtual bdev look like the base. The virtual bdev does not magically inherit these fields. The module author decides what to expose.

pt_node->pt_bdev.write_cache = bdev->write_cache;
pt_node->pt_bdev.required_alignment = bdev->required_alignment;
pt_node->pt_bdev.optimal_io_boundary = bdev->optimal_io_boundary;
pt_node->pt_bdev.blocklen = bdev->blocklen;
pt_node->pt_bdev.blockcnt = bdev->blockcnt;

pt_node->pt_bdev.md_interleave = bdev->md_interleave;
pt_node->pt_bdev.md_len = bdev->md_len;
pt_node->pt_bdev.dif_type = bdev->dif_type;
pt_node->pt_bdev.dif_is_head_of_md = bdev->dif_is_head_of_md;
pt_node->pt_bdev.dif_check_flags = bdev->dif_check_flags;
pt_node->pt_bdev.dif_pi_format = bdev->dif_pi_format;

Then passthru fills the virtual bdev's context, function table, and module pointer, registers an io_device for per-thread virtual channels, remembers the opening thread, claims the base, and registers the virtual bdev.

pt_node->pt_bdev.ctxt = pt_node;
pt_node->pt_bdev.fn_table = &vbdev_passthru_fn_table;
pt_node->pt_bdev.module = &passthru_if;
TAILQ_INSERT_TAIL(&g_pt_nodes, pt_node, link);

spdk_io_device_register(pt_node, pt_bdev_ch_create_cb, pt_bdev_ch_destroy_cb,
			sizeof(struct pt_io_channel),
			name->vbdev_name);

pt_node->thread = spdk_get_thread();

rc = spdk_bdev_module_claim_bdev(bdev, pt_node->base_desc, pt_node->pt_bdev.module);
if (rc) {
	SPDK_ERRLOG("could not claim bdev %s\n", bdev_name);
	spdk_bdev_close(pt_node->base_desc);
	TAILQ_REMOVE(&g_pt_nodes, pt_node, link);
	spdk_io_device_unregister(pt_node, NULL);
	free(pt_node->pt_bdev.name);
	free(pt_node);
	break;
}

The claim protects stacking semantics. It tells the bdev layer that this module owns the base bdev for module-level use, so other writers cannot unexpectedly modify it underneath the virtual layer. Claims are especially important for modules that maintain metadata, reorder writes, or combine devices. Passthru is simple, but it still demonstrates the pattern.

For new virtual bdevs, prefer the descriptor-backed claim helper: spdk_bdev_module_claim_bdev_desc(desc, type, opts, module). The main claim types are SPDK_BDEV_CLAIM_READ_MANY_WRITE_ONE, SPDK_BDEV_CLAIM_READ_MANY_WRITE_NONE, SPDK_BDEV_CLAIM_EXCL_WRITE, and SPDK_BDEV_CLAIM_READ_MANY_WRITE_SHARED. Descriptor-backed claims are released when the descriptor is closed, so the teardown path is usually easier to reason about than the legacy spdk_bdev_module_claim_bdev() plus spdk_bdev_module_release_bdev() pairing. Passthru still uses the older helper, which is why it is a good source example for legacy v1 claims but not the only shape a new module should copy.

Registration happens only after the base has been opened, the virtual io_device exists, and the base has been claimed. Read the local vbdev_passthru_register() error paths as a failure-unwind checklist more than as code to paste blindly. The important idea is reverse-order cleanup: if registration fails after the module has opened, inserted, registered an io_device, and claimed, the error path must undo those effects before freeing the object.

Virtual Channels

The virtual bdev has its own channel because upper layers submit I/O to the virtual bdev, not directly to the base bdev. The virtual channel stores the base channel for the same SPDK thread.

static struct spdk_io_channel *
vbdev_passthru_get_io_channel(void *ctx)
{
	struct vbdev_passthru *pt_node = (struct vbdev_passthru *)ctx;
	struct spdk_io_channel *pt_ch = NULL;

	pt_ch = spdk_get_io_channel(pt_node);

	return pt_ch;
}

static int
pt_bdev_ch_create_cb(void *io_device, void *ctx_buf)
{
	struct pt_io_channel *pt_ch = ctx_buf;
	struct vbdev_passthru *pt_node = io_device;

	pt_ch->base_ch = spdk_bdev_get_io_channel(pt_node->base_desc);

	return 0;
}

The destroy callback must put the base channel. The get/put pairing is per thread and mirrors normal descriptor/channel use by bdev consumers.

static void
pt_bdev_ch_destroy_cb(void *io_device, void *ctx_buf)
{
	struct pt_io_channel *pt_ch = ctx_buf;

	spdk_put_io_channel(pt_ch->base_ch);
}

This is one of the easiest places to leak resources in a virtual module. Every virtual channel created on a thread owns one base channel reference on that thread. If destroy forgets to put it, teardown can hang or the base bdev can remain busy.

Forwarding I/O

The virtual module receives one struct spdk_bdev_io from bdev core. It does not send that same object to the base bdev. Instead, it submits a new I/O to the base using public bdev APIs and passes the original virtual I/O as the callback argument.

static void
_pt_complete_io(struct spdk_bdev_io *bdev_io, bool success, void *cb_arg)
{
	struct spdk_bdev_io *orig_io = cb_arg;
	struct passthru_bdev_io *io_ctx = (struct passthru_bdev_io *)orig_io->driver_ctx;

	if (io_ctx->test != 0x5a) {
		SPDK_ERRLOG("Error, original IO device_ctx is wrong! 0x%x\n",
			    io_ctx->test);
	}

	spdk_bdev_io_complete_base_io_status(orig_io, bdev_io);

	spdk_bdev_free_io(bdev_io);
}

spdk_bdev_io_complete_base_io_status() preserves the base I/O's completion status when completing the original virtual I/O. That is better than collapsing every base failure to a generic failed status because upper layers may care about detailed NVMe/SCSI style status information.

The submit path shows the forwarding pattern. Reads call spdk_bdev_io_get_buf() first, because the original read may not have a data buffer assigned yet. Writes and other commands submit directly to the base descriptor and base channel.

switch (bdev_io->type) {
case SPDK_BDEV_IO_TYPE_READ:
	spdk_bdev_io_get_buf(bdev_io, pt_read_get_buf_cb,
			     bdev_io->u.bdev.num_blocks * bdev_io->bdev->blocklen);
	break;
case SPDK_BDEV_IO_TYPE_WRITE:
	pt_init_ext_io_opts(bdev_io, &io_opts);
	rc = spdk_bdev_writev_blocks_ext(pt_node->base_desc, pt_ch->base_ch, bdev_io->u.bdev.iovs,
					 bdev_io->u.bdev.iovcnt, bdev_io->u.bdev.offset_blocks,
					 bdev_io->u.bdev.num_blocks, _pt_complete_io,
					 bdev_io, &io_opts);
	break;
case SPDK_BDEV_IO_TYPE_WRITE_ZEROES:
	rc = spdk_bdev_write_zeroes_blocks(pt_node->base_desc, pt_ch->base_ch,
					   bdev_io->u.bdev.offset_blocks,
					   bdev_io->u.bdev.num_blocks,
					   _pt_complete_io, bdev_io);
	break;
case SPDK_BDEV_IO_TYPE_UNMAP:
	rc = spdk_bdev_unmap_blocks(pt_node->base_desc, pt_ch->base_ch,
				    bdev_io->u.bdev.offset_blocks,
				    bdev_io->u.bdev.num_blocks,
				    _pt_complete_io, bdev_io);
	break;

The switch continues for flush, reset, zcopy, abort, and copy. The pattern is the same: translate the original request into a public base-bdev API call, then complete the original I/O from the callback.

Submission failure is different from I/O failure. If a public bdev API returns nonzero, then the base I/O was not successfully submitted. There may never be a completion callback. The virtual module must either queue for retry on -ENOMEM or complete the original I/O itself.

if (rc != 0) {
	if (rc == -ENOMEM) {
		SPDK_ERRLOG("No memory, start to queue io for passthru.\n");
		io_ctx->ch = ch;
		vbdev_passthru_queue_io(bdev_io);
	} else {
		SPDK_ERRLOG("ERROR on bdev_io submission!\n");
		spdk_bdev_io_complete(bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
	}
}

Passthru queues memory-pressure retries using spdk_bdev_queue_io_wait(). The wait entry points back at the original virtual I/O. When resources are available, the bdev layer calls the wait callback and passthru resubmits.

static void
vbdev_passthru_resubmit_io(void *arg)
{
	struct spdk_bdev_io *bdev_io = (struct spdk_bdev_io *)arg;
	struct passthru_bdev_io *io_ctx = (struct passthru_bdev_io *)bdev_io->driver_ctx;

	vbdev_passthru_submit_request(io_ctx->ch, bdev_io);
}

static void
vbdev_passthru_queue_io(struct spdk_bdev_io *bdev_io)
{
	struct passthru_bdev_io *io_ctx = (struct passthru_bdev_io *)bdev_io->driver_ctx;
	struct pt_io_channel *pt_ch = spdk_io_channel_get_ctx(io_ctx->ch);
	int rc;

	io_ctx->bdev_io_wait.bdev = bdev_io->bdev;
	io_ctx->bdev_io_wait.cb_fn = vbdev_passthru_resubmit_io;
	io_ctx->bdev_io_wait.cb_arg = bdev_io;

	rc = spdk_bdev_queue_io_wait(bdev_io->bdev, pt_ch->base_ch, &io_ctx->bdev_io_wait);

This is why the per-I/O context stores the virtual channel. A later wait callback needs enough state to retry the same logical I/O on the same virtual path.

Passthru advertises exactly what its base bdev advertises:

static bool
vbdev_passthru_io_type_supported(void *ctx, enum spdk_bdev_io_type io_type)
{
	struct vbdev_passthru *pt_node = (struct vbdev_passthru *)ctx;

	return spdk_bdev_io_type_supported(pt_node->base_bdev, io_type);
}

That is correct for a transparent forwarding layer. It would not be correct for a virtual module that changes capabilities. For example, a module may decide not to expose raw NVMe passthrough even if the base supports it, or it may emulate write zeroes even if the base does not.

Hotremove And Destruct

The base descriptor's event callback handles asynchronous base events. Passthru only cares about removal and unregisters virtual bdevs stacked on the removed base.

static void
vbdev_passthru_base_bdev_hotremove_cb(struct spdk_bdev *bdev_find)
{
	struct vbdev_passthru *pt_node, *tmp;

	TAILQ_FOREACH_SAFE(pt_node, &g_pt_nodes, link, tmp) {
		if (bdev_find == pt_node->base_bdev) {
			spdk_bdev_unregister(&pt_node->pt_bdev, NULL, NULL);
		}
	}
}

static void
vbdev_passthru_base_bdev_event_cb(enum spdk_bdev_event_type type, struct spdk_bdev *bdev,
				  void *event_ctx)
{
	switch (type) {
	case SPDK_BDEV_EVENT_REMOVE:
		vbdev_passthru_base_bdev_hotremove_cb(bdev);
		break;
	default:
		SPDK_NOTICELOG("Unsupported bdev event: type %d\n", type);
		break;
	}
}

The event callback is not where the virtual object is freed. It starts unregistration. Bdev core later calls the virtual bdev's destruct() when it is safe to tear down that bdev object.

Passthru destruct shows the reverse ownership order: remove from the module list, release the claim, close the base descriptor on the opening thread, unregister the virtual io_device, and return.

static int
vbdev_passthru_destruct(void *ctx)
{
	struct vbdev_passthru *pt_node = (struct vbdev_passthru *)ctx;

	TAILQ_REMOVE(&g_pt_nodes, pt_node, link);

	spdk_bdev_module_release_bdev(pt_node->base_bdev);

	if (pt_node->thread && pt_node->thread != spdk_get_thread()) {
		spdk_thread_send_msg(pt_node->thread, _vbdev_passthru_destruct, pt_node->base_desc);
	} else {
		spdk_bdev_close(pt_node->base_desc);
	}

	spdk_io_device_unregister(pt_node, _device_unregister_cb);

	return 0;
}

The actual object free happens in the io_device unregister callback in passthru. That callback is outside the excerpt, but the design is visible here: unregistering the io_device is the last step because channel contexts may still exist until SPDK finishes channel cleanup.

The API supports asynchronous destruct when synchronous return is not enough:

/**
 * Notify the bdev layer that an asynchronous destruct operation is complete.
 *
 * A Bdev with an asynchronous destruct path should return 1 from its
 * destruct function and call this function at the conclusion of that path.
 * Bdevs with synchronous destruct paths should return 0 from their destruct
 * path.
 */
void spdk_bdev_destruct_done(struct spdk_bdev *bdev, int bdeverrno);

Use asynchronous destruct when a module has to wait for backend work, remote teardown, hardware state, or a message callback before the bdev is truly gone. Do not return zero and then finish cleanup later unless the object is already safe for bdev core to forget.

Contrasting Virtual Module Patterns

Passthru is useful because it is small, but it is not the only virtual bdev shape in the tree.

The delay module looks passthru-like at construction time: it opens a base bdev, copies geometry, creates a virtual io_device, claims the base, and forwards I/O. The important difference is completion ownership. Delay records completed base I/O in per-channel queues such as avg_read_io, p99_read_io, avg_write_io, and p99_write_io, then a poller completes the original virtual I/O after the configured latency. Its reset path has to abort delayed completions on every channel before forwarding reset to the base. That is the lesson: once a virtual module queues completed work internally, reset and abort handling are part of the design, not an afterthought.

The error module shows a different construction model. Instead of manually mirroring every base-bdev field in a passthru-style object, it uses the partition framework through spdk_bdev_part_base_construct_ext(). The module then layers error injection behavior on top of a partition-backed virtual bdev. That pattern is useful when the partition helpers already provide the base open, hotremove, channel, and part lifecycle structure your module needs.

A KISS Module Checklist

For a physical module:

Define a backend object containing struct spdk_bdev.
Define per-channel state if the module has queues, pollers, hardware queues, or base resources.
Define per-I/O context through get_ctx_size() instead of allocating in the hot path when possible.
Define struct spdk_bdev_module and register it with SPDK_BDEV_MODULE_REGISTER().
Implement module_init() and register any io_device.
Implement module_fini() and unregister module-wide resources.
Implement struct spdk_bdev_fn_table.
Fill bdev name, geometry, capabilities, module, function table, and context.
Call spdk_bdev_register() only after the object is ready for callers.
Complete every submitted I/O exactly once.
Free per-bdev resources in destruct().

For a virtual module, add:

Store configured base name and virtual name.
Use examine to attach when a configured base appears.
Open the base with an event callback.
Claim the base before exposing the virtual bdev when stacking needs writer or reader exclusion. Prefer descriptor-backed claims for new code.
Create a virtual io_device.
Get and put base channels in virtual channel create/destroy.
Forward I/O through public bdev APIs.
Translate, preserve, or deliberately replace completion status.
On base remove, unregister virtual bdevs depending on that base.
On destruct, release legacy claims if used and close descriptors on the correct thread. Descriptor-backed claims release with descriptor close.

The Hard Rules

Do Not Block

Module callbacks run on SPDK threads. Blocking in submit_request() stalls that thread and all other work scheduled there. Null completes from a poller, and passthru returns after submitting to the base. Real modules should use pollers, asynchronous APIs, and messages for work that cannot finish immediately.

Complete Exactly Once

Every I/O delivered to submit_request() must be completed exactly once. A missing completion hangs upper layers. A double completion can trip bdev core assertions or corrupt request ownership. The safe mental model is to mark each switch case as one of these:

Completes inline before returning.
Queues work and completes later.
Submits to another asynchronous API that will complete later.
Fails submission and completes inline.

Avoid cases that "fall through" into no completion.

Respect Thread Ownership

Descriptors and many lifecycle actions are thread-bound. Passthru records the thread where the base descriptor was opened and sends a message back there before closing when needed. The same concern applies to channel references: get and put them in matched per-thread lifecycle callbacks.

Separate Submission Failure From I/O Failure

If spdk_bdev_writev_blocks_ext() or another public bdev API returns nonzero, the new base I/O was not submitted. No base completion callback is guaranteed. The virtual module must complete or queue the original I/O itself. If the API returns zero and the base later completes with an error, complete the original I/O from the completion callback using the base status.

Do Not Invent Unsupported Capabilities

io_type_supported() must reflect reality. A transparent virtual module can mirror the base. A transforming module should advertise only the operations it can implement correctly. Claiming support for reset, flush, raw NVMe passthrough, zcopy, or copy without preserving the required semantics creates bugs above the bdev layer.

Pair Every Lifecycle Edge

Most bdev module bugs are not in the successful read path. They are in the edges:

calloc() succeeds and strdup() fails.
Base open succeeds and claim fails.
Claim succeeds and virtual registration fails.
Base remove arrives during I/O.
A channel exists on a thread when destruct starts.
-ENOMEM is returned during base submission.
Destruct needs asynchronous work but returns zero.

Write cleanup as reverse ownership. If a create path did A, then B, then C, the failure path after C usually has to undo C, then B, then A.

Testing A New Module

SPDK has several useful patterns to copy before inventing your own test harness.

test/rpc/rpc.sh includes a small RPC integrity check for passthru. It creates a malloc bdev, creates a passthru bdev on top, verifies that the bdev count increased, deletes the passthru bdev, deletes the base, and verifies that the bdev list is empty again.

malloc=$($rpc bdev_malloc_create 8 512)
bdevs=$($rpc bdev_get_bdevs)
[ "$(jq length <<< "$bdevs")" == "1" ]

$rpc bdev_passthru_create -b "$malloc" -p Passthru0
bdevs=$($rpc bdev_get_bdevs)
[ "$(jq length <<< "$bdevs")" == "2" ]

$rpc bdev_passthru_delete Passthru0
$rpc bdev_malloc_delete $malloc

That test does not prove data correctness, but it catches common registration, RPC, delete, and cleanup errors. For a new module, a smoke test like this is the first bar.

test/bdev/bdev_raid.sh uses passthru bdevs as RAID base devices in a larger flow. The important lesson is not RAID itself; it is that a virtual bdev must behave like a normal bdev when another virtual module stacks on it.

$rpc_py bdev_malloc_create 32 $base_blocklen $base_malloc_params -b $bdev_malloc
$rpc_py bdev_passthru_create -b $bdev_malloc -p $bdev_pt -u $bdev_pt_uuid

$rpc_py bdev_raid_create $strip_size_create_arg -r $raid_level \
	-b "'${base_bdevs_pt[*]}'" -n $raid_bdev_name -s
verify_raid_bdev_state $raid_bdev_name "online" $raid_level $strip_size $num_base_bdevs

For a physical bdev, add bdevperf or bdevio coverage for reads, writes, resets, and each I/O type you advertise. For a virtual bdev, test both the virtual bdev itself and another module stacked on top when that is a realistic use case.

test/external_code/README.md points at another practical path: building an external application and custom bdev module against SPDK libraries. That is useful when your module is not ready to live under module/bdev, or when you want a smaller link/build loop while learning the interfaces.

Prose Diagram

Physical module flow:

SPDK_BDEV_MODULE_REGISTER() makes the module discoverable. During bdev subsystem startup, module_init() allocates module-wide resources and registers an io_device. An RPC calls a create function. The create function allocates a backend object containing struct spdk_bdev, fills the bdev fields, and calls spdk_bdev_register(). Later, bdev core calls get_io_channel() on each submitting thread and submit_request() for each I/O. The module completes I/O from a backend callback, poller, or immediate failure path. During teardown, bdev core calls destruct() for each bdev and module_fini() for module-wide cleanup.

Virtual module flow:

An RPC stores "base bdev name -> virtual bdev name." Examine sees the base bdev and calls the virtual module's register path. The module opens the base with an event callback, copies or adapts geometry, registers its own io_device, claims the base, and registers its virtual bdev. Each virtual channel owns a base channel for the same thread. I/O enters the virtual bdev, gets submitted to the base bdev as a separate base I/O, base completion returns, and the virtual module completes the original I/O. If the base is removed, the event callback unregisters the dependent virtual bdev.

Edge Cases And Failure Modes

Create succeeds partly, then register fails. Clean up name, io_device, descriptor, claim, and allocated object in reverse order. The exact set depends on how far create got.

The base bdev is not present yet. Some virtual modules intentionally store configuration and create later during examine. That is different from a create RPC that promises immediate construction; be explicit in the RPC contract.

The base bdev is removed. The event callback must unregister virtual bdevs that depend on it. Ignoring remove events leaves dangling base pointers and descriptors.

The base descriptor was opened on another thread. Close it on the opening thread. Passthru uses spdk_thread_send_msg() for that case.

Claim fails. Another module or writer may already own the base. Do not register the virtual bdev.

Base submission returns -ENOMEM. Queue a wait entry if retry is appropriate. If not, complete the original I/O failed. Do not assume a completion callback will arrive.

Base submission returns another error. Complete the original I/O failed unless your module has a deliberate retry policy for that error.

Base completion carries detailed status. Use helper functions such as spdk_bdev_io_complete_base_io_status() when preserving detail matters.

Reset is advertised but not safely implemented. Either fail reset or forward/coordinate it correctly. Do not complete reset before outstanding I/O has been accounted for.

Asynchronous destruct is needed. Return 1 from destruct() and call spdk_bdev_destruct_done() later. Synchronous destruct should return 0 only when bdev core can safely consider the destruct complete.

Examine does not call spdk_bdev_module_examine_done(). Bdev subsystem progress can hang while waiting for that module.

Misconceptions To Kill

"A virtual bdev can skip claims if it is just forwarding." Not when it needs to protect write ownership and stacking semantics.
"A module can malloc per-I/O context in submit_request()." It can, but the normal hot-path design is get_ctx_size() plus driver_ctx.
"The base bdev event callback is optional." If you stack on a base bdev, ignoring remove events creates dangling state.
"A create RPC should always fail if the base bdev is missing." Some virtual modules intentionally defer creation until examine sees the base.
"The destructor can free everything immediately." Only if no asynchronous close, unregister, channel, backend, or device cleanup remains.
"Forwarding means reusing the same struct spdk_bdev_io." A virtual bdev receives one I/O and usually submits a separate base I/O.
"If base submission fails, the base completion callback will report that failure." A nonzero submission return usually means no base I/O was submitted.

Source Reading Path

Read these in order when implementing your first module:

include/spdk/bdev_module.h: struct spdk_bdev_module, struct spdk_bdev_fn_table, spdk_bdev_destruct_done(), and spdk_bdev_module_examine_done().
module/bdev/null/bdev_null.c: object layout, null_if, bdev_null_create(), null_fn_table, bdev_null_submit_request(), null_io_poll(), bdev_null_destruct(), and bdev_null_finish().
module/bdev/passthru/vbdev_passthru.c: struct vbdev_passthru, vbdev_passthru_register(), pt_bdev_ch_create_cb(), vbdev_passthru_submit_request(), _pt_complete_io(), vbdev_passthru_base_bdev_event_cb(), and vbdev_passthru_destruct().
module/bdev/passthru/vbdev_passthru_rpc.c and module/bdev/null/bdev_null_rpc.c: how JSON-RPC parameters enter create/delete functions.
module/bdev/delay/vbdev_delay.c: delayed completions, reset-time abort of queued completions, and base reset forwarding.
module/bdev/error/vbdev_error.c: partition-framework construction through spdk_bdev_part_base_construct_ext().
test/rpc/rpc.sh: a minimal create/list/delete smoke pattern.
test/external_code/README.md and test/external_code/passthru/: how SPDK demonstrates external module linking.