Chapter 21: RAID And Virtual bdev Stacking | SPDK From First Principles

Beginner Mental Model

RAID in SPDK is a virtual bdev module. It takes several base bdevs and exposes one logical bdev. The bdev user sees one device; the RAID module sees base-device descriptors, per-thread base-device channels, RAID-level mapping rules, metadata, state, and sometimes a background process such as rebuild.

The key idea is layering. The bdev layer does not require every block device to be a hardware namespace. A bdev may be a software object that routes IO to other bdevs. SPDK's bdev programmer guide describes that pattern directly:

From doc/bdev_pg.md:

Bdevs can be layered, such that some bdevs service I/O by routing requests to
other bdevs. This can be used to implement caching, RAID, logical volume
management, and more. Bdevs that route I/O to other bdevs are often referred
to as virtual bdevs, or *vbdevs* for short.

For RAID, that layered graph looks like this:

base bdev 0   base bdev 1   base bdev 2
     \             |             /
      \            |            /
       +------ RAID bdev -------+
                    |
              bdev consumers

The RAID bdev is not just a naming wrapper. It is both a data mapping layer and a lifecycle layer. It has to answer these questions on behalf of upper layers:

Which base bdevs belong to this RAID bdev?
Is the array online, configuring, or offline?
Can IO proceed if one base is missing?
Does on-disk metadata identify this array?
Is a rebuild or other process active?
What happens if a base bdev is removed, fails, or resizes?
Which thread owns each state transition and callback?

The official SPDK bdev user guide summarizes the user-visible RAID feature set:

From doc/bdev.md:

RAID virtual bdev module provides functionality to combine any SPDK bdevs into one
RAID bdev. Currently SPDK supports RAID0, Concat, RAID1 and RAID5F levels. To enable
RAID5F, configure SPDK using the `--with-raid5f` option. For RAID levels with redundancy
(1 and 5F) degraded operation and rebuild are supported. RAID metadata may be stored
on member disks if enabled when creating the RAID bdev, so user does not have to
recreate the RAID volume when restarting application.

That short paragraph explains most of the implementation pressure in this chapter. RAID0 and concat need mapping. RAID1 and RAID5f also need degraded operation and rebuild. Superblock-enabled arrays need metadata reads and writes. Every RAID level still has to fit into the same bdev module model.

Source Map

Core RAID files:

module/bdev/raid/bdev_raid.h
module/bdev/raid/bdev_raid.c
module/bdev/raid/bdev_raid_sb.c
module/bdev/raid/bdev_raid_rpc.c

RAID level modules:

module/bdev/raid/raid0.c
module/bdev/raid/raid1.c
module/bdev/raid/concat.c
module/bdev/raid/raid5f.c

Useful bdev docs and common-layer code:

doc/bdev_pg.md
doc/bdev.md
doc/jsonrpc.md.jinja2
lib/bdev/bdev.c
include/spdk/bdev.h

Tests:

test/bdev/bdev_raid.sh
test/unit/lib/bdev/raid/bdev_raid.c/bdev_raid_ut.c
test/unit/lib/bdev/raid/bdev_raid_sb.c/bdev_raid_sb_ut.c
test/unit/lib/bdev/raid/raid0.c/raid0_ut.c
test/unit/lib/bdev/raid/raid1.c/raid1_ut.c
test/unit/lib/bdev/raid/concat.c/concat_ut.c
test/unit/lib/bdev/raid/raid5f.c/raid5f_ut.c

Core Objects

RAID keeps one object for the exported RAID bdev and one object per base slot. A slot is important: it may name a missing disk, a currently configured disk, a failed disk, or a replacement target. The slot survives longer than a single base bdev descriptor.

The state enum is intentionally small:

/* module/bdev/raid/bdev_raid.h */
enum raid_bdev_state {
	RAID_BDEV_STATE_ONLINE,

	/*
	 * raid bdev is configuring, not all underlying bdevs are present.
	 * And can't be seen by upper layers.
	 */
	RAID_BDEV_STATE_CONFIGURING,

	/*
	 * In offline state, raid bdev layer will complete all incoming commands without
	 * submitting to underlying base nvme bdevs
	 */
	RAID_BDEV_STATE_OFFLINE,

	/* raid bdev state max, new states should be added before this */
	RAID_BDEV_STATE_MAX
};

The difference between "configured in memory" and "online bdev" matters. An array can be known to RAID while still waiting for base bdevs or metadata. It becomes visible to normal bdev consumers only after the RAID module registers the embedded struct spdk_bdev.

The base slot structure is the first place to look when debugging ownership:

/* module/bdev/raid/bdev_raid.h */
struct raid_base_bdev_info {
	/* The raid bdev that this base bdev belongs to */
	struct raid_bdev	*raid_bdev;

	/* name of the bdev */
	char			*name;

	/* uuid of the bdev */
	struct spdk_uuid	uuid;

	/*
	 * Pointer to base bdev descriptor opened by raid bdev. This is NULL when the bdev for
	 * this slot is missing.
	 */
	struct spdk_bdev_desc	*desc;

	/* offset in blocks from the start of the base bdev to the start of the data region */
	uint64_t		data_offset;

	/* size in blocks of the base bdev's data region */
	uint64_t		data_size;
};

The fields that follow in the real structure add lifecycle state: remove_scheduled, app_thread_ch, is_configured, is_process_target, and is_failed. Those flags explain why base removal and rebuild cannot be implemented by simply clearing desc. A slot may be missing but still be part of the array's identity.

The RAID bdev object embeds the public bdev and points at its level module:

/* module/bdev/raid/bdev_raid.h */
struct raid_bdev {
	/* raid bdev device, this will get registered in bdev layer */
	struct spdk_bdev		bdev;

	/* the raid bdev descriptor, opened for internal use */
	struct spdk_bdev_desc		*self_desc;

	/* link of raid bdev to link it to global raid bdev list */
	TAILQ_ENTRY(raid_bdev)		global_link;

	/* array of base bdev info */
	struct raid_base_bdev_info	*base_bdev_info;

	/* strip size of raid bdev in blocks */
	uint32_t			strip_size;

	/* state of raid bdev */
	enum raid_bdev_state		state;

	/* number of base bdevs comprising raid bdev  */
	uint8_t				num_base_bdevs;
};

Later fields track discovered and operational base counts, the minimum number required to operate, the RAID level, superblock state, and the active background process. That split is useful when reading failures: discovered means "present/configured now"; operational means "the array expects this many working members"; minimum operational means "below this, the exported bdev cannot keep its promises."

The per-IO context is separate from the exported spdk_bdev_io. RAID stores it in driver_ctx, then uses it to collect child IO completions:

/* module/bdev/raid/bdev_raid.h */
struct raid_bdev_io {
	/* The raid bdev associated with this IO */
	struct raid_bdev *raid_bdev;

	uint64_t offset_blocks;
	uint64_t num_blocks;
	struct iovec *iovs;
	int iovcnt;
	enum spdk_bdev_io_type type;
	struct spdk_memory_domain *memory_domain;
	void *memory_domain_ctx;
	void *md_buf;

	/* Context of the original channel for this IO */
	struct raid_bdev_io_channel	*raid_ch;

	/* Used for tracking progress on io requests sent to member disks. */
	uint64_t			base_bdev_io_remaining;
	uint8_t				base_bdev_io_submitted;
	enum spdk_bdev_io_status	base_bdev_io_status;
};

The level modules plug into common RAID through struct raid_bdev_module:

/* module/bdev/raid/bdev_raid.h */
struct raid_bdev_module {
	/* RAID level implemented by this module */
	enum raid_level level;

	/* Minimum required number of base bdevs. Must be > 0. */
	uint8_t base_bdevs_min;

	/* Handler for R/W requests */
	void (*submit_rw_request)(struct raid_bdev_io *raid_io);

	/* Handler for requests without payload (flush, unmap). Optional. */
	void (*submit_null_payload_request)(struct raid_bdev_io *raid_io);

	/*
	 * Called when a base_bdev is resized to resize the raid if the condition
	 * is satisfied. Optional.
	 */
	bool (*resize)(struct raid_bdev *raid_bdev);

	/* Handler for raid process requests. Required for raid modules with redundancy. */
	int (*submit_process_request)(struct raid_bdev_process_request *process_req,
				      struct raid_bdev_io_channel *raid_ch);
};

This is the main separation of responsibilities. Common RAID owns bdev registration, base discovery, events, superblocks, process scheduling, and callbacks. The level module owns the mapping from a RAID logical IO to one or more base-device IOs.

RAID States

State anchors:

module/bdev/raid/bdev_raid.h:enum raid_bdev_state
module/bdev/raid/bdev_raid.c:g_raid_state_names
module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_get_bdevs
doc/jsonrpc.md.jinja2:bdev_raid_get_bdevs

The JSON-RPC documentation exposes the same state model that the source uses:

From doc/jsonrpc.md.jinja2:

Category should be one of 'all', 'online', 'configuring' or 'offline'.
'online' is the raid bdev which is registered with bdev layer. 'configuring'
is the raid bdev which does not have full configuration discovered yet.
'offline' is the raid bdev which is not registered with bdev as of now and
it has encountered any error or user has requested to offline the raid bdev.

Beginner misconception to kill: "configured in JSON" and "online bdev exists" are not the same state. A RAID bdev can exist as an in-memory configuration object before it is registered as a bdev. That is how SPDK can remember a planned array while waiting for later base bdev registration or superblock discovery.

RAID Creation

RPC and implementation anchors:

module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_create
module/bdev/raid/bdev_raid.c:raid_bdev_create
module/bdev/raid/bdev_raid.c:_raid_bdev_create
module/bdev/raid/bdev_raid.c:raid_bdev_add_base_bdev
module/bdev/raid/bdev_raid.c:raid_bdev_configure_base_bdev
module/bdev/raid/bdev_raid.c:raid_bdev_configure
module/bdev/raid/bdev_raid.c:raid_bdev_configure_cont

Creation flow:

raid_bdev_create RPC
  -> allocate raid_bdev object
  -> store level, strip size, expected base count, UUID/superblock flags
  -> add configured base names
  -> when enough base bdevs are open and configured:
       initialize exported bdev fields
       optionally write superblocks
       register RAID bdev

The first low-level act of stacking is opening and claiming a base bdev. The bdev programmer guide explains the reason for claiming:

From doc/bdev_pg.md:

Opening a bdev with write permission may fail if a virtual bdev module
has *claimed* the bdev. Virtual bdev modules implement logic like RAID or
logical volume management and forward their I/O to lower level bdevs, so they
mark these lower level bdevs as claimed to prevent outside users from issuing
writes.

The RAID implementation follows that model. It opens the base with write access, verifies identity, claims it for the RAID module, and gets an app-thread IO channel:

/* module/bdev/raid/bdev_raid.c */
rc = spdk_bdev_open_ext(base_info->name, true, raid_bdev_event_base_bdev, NULL, &desc);
if (rc != 0) {
	if (rc != -ENODEV) {
		SPDK_ERRLOG("Unable to create desc on bdev '%s'\n", base_info->name);
	}
	return rc;
}

bdev = spdk_bdev_desc_get_bdev(desc);
bdev_uuid = spdk_bdev_get_uuid(bdev);

rc = spdk_bdev_module_claim_bdev(bdev, NULL, &g_raid_if);
if (rc != 0) {
	SPDK_ERRLOG("Unable to claim this bdev as it is already claimed\n");
	spdk_bdev_close(desc);
	return rc;
}

base_info->app_thread_ch = spdk_bdev_get_io_channel(desc);
base_info->desc = desc;
base_info->blockcnt = bdev->blockcnt;

This excerpt is the source-level answer to "how does RAID sit on top of base bdevs?" It has a descriptor for each base, and it has a channel used for app-thread operations such as reset, metadata writes, and event handling. Per-IO channels for normal IO are handled separately through the RAID bdev's IO device channel.

RAID also requires base devices to agree on block and metadata layout. It either copies the first base's format into the RAID bdev or rejects later bases that do not match:

/* module/bdev/raid/bdev_raid.c */
if (raid_bdev->bdev.blocklen == 0) {
	raid_bdev->bdev.blocklen = bdev->blocklen;
	raid_bdev->bdev.md_len = spdk_bdev_get_md_size(bdev);
	raid_bdev->bdev.md_interleave = spdk_bdev_is_md_interleaved(bdev);
	raid_bdev->bdev.dif_type = spdk_bdev_get_dif_type(bdev);
	raid_bdev->bdev.dif_check_flags = bdev->dif_check_flags;
	raid_bdev->bdev.dif_is_head_of_md = spdk_bdev_is_dif_head_of_md(bdev);
	raid_bdev->bdev.dif_pi_format = bdev->dif_pi_format;
} else {
	if (raid_bdev->bdev.blocklen != bdev->blocklen) {
		SPDK_ERRLOG("Raid bdev '%s' blocklen %u differs from base bdev '%s' blocklen %u\n",
			    raid_bdev->bdev.name, raid_bdev->bdev.blocklen, bdev->name, bdev->blocklen);
		rc = -EINVAL;
		goto out;
	}

	if (raid_bdev->bdev.md_len != spdk_bdev_get_md_size(bdev) ||
	    raid_bdev->bdev.md_interleave != spdk_bdev_is_md_interleaved(bdev) ||
	    raid_bdev->bdev.dif_type != spdk_bdev_get_dif_type(bdev) ||
	    raid_bdev->bdev.dif_check_flags != bdev->dif_check_flags ||
	    raid_bdev->bdev.dif_is_head_of_md != spdk_bdev_is_dif_head_of_md(bdev) ||
	    raid_bdev->bdev.dif_pi_format != bdev->dif_pi_format) {
		SPDK_ERRLOG("Raid bdev '%s' has different metadata format than base bdev '%s'\n",
			    raid_bdev->bdev.name, bdev->name);
		rc = -EINVAL;
		goto out;
	}
}

That check exists because upper layers submit IO to one logical bdev. They cannot safely handle a RAID bdev whose members disagree about data block size, metadata size, DIF type, or metadata placement. RAID needs one logical contract.

RAID bdev IO Path

Common entry:

module/bdev/raid/bdev_raid.c:raid_bdev_submit_request
module/bdev/raid/bdev_raid.c:raid_bdev_io_init
module/bdev/raid/bdev_raid.c:raid_bdev_submit_rw_request
module/bdev/raid/bdev_raid.c:raid_bdev_submit_null_payload_request
module/bdev/raid/bdev_raid.c:raid_bdev_io_complete
module/bdev/raid/bdev_raid.c:raid_bdev_queue_io_wait
module/bdev/raid/bdev_raid.c:raid_bdev_io_split
module/bdev/raid/bdev_raid.c:g_raid_bdev_fn_table

Level-specific entries:

module/bdev/raid/raid0.c:raid0_submit_rw_request
module/bdev/raid/raid1.c:raid1_submit_rw_request
module/bdev/raid/concat.c:concat_submit_rw_request
module/bdev/raid/raid5f.c:raid5f_submit_rw_request

Common IO flow:

bdev user IO to RAID bdev
  -> lib/bdev/bdev.c:bdev_submit_request
  -> module/bdev/raid/bdev_raid.c:raid_bdev_submit_request
  -> initialize raid_bdev_io from parent bdev_io
  -> for reads, obtain/read buffer if needed
  -> dispatch to common reset/null-payload path or level-specific read/write path
  -> submit one or more child IOs to base bdevs
  -> collect child completions
  -> complete original bdev_io

The common submit function is short because it delegates the level-specific mapping:

/* module/bdev/raid/bdev_raid.c */
static void
raid_bdev_submit_request(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)
{
	struct raid_bdev_io *raid_io = (struct raid_bdev_io *)bdev_io->driver_ctx;

	raid_bdev_io_init(raid_io, spdk_io_channel_get_ctx(ch), bdev_io->type,
			  bdev_io->u.bdev.offset_blocks, bdev_io->u.bdev.num_blocks,
			  bdev_io->u.bdev.iovs, bdev_io->u.bdev.iovcnt, bdev_io->u.bdev.md_buf,
			  bdev_io->u.bdev.memory_domain, bdev_io->u.bdev.memory_domain_ctx);

	switch (bdev_io->type) {
	case SPDK_BDEV_IO_TYPE_READ:
		spdk_bdev_io_get_buf(bdev_io, raid_bdev_get_buf_cb,
				     bdev_io->u.bdev.num_blocks * bdev_io->bdev->blocklen);
		break;
	case SPDK_BDEV_IO_TYPE_WRITE:
		raid_bdev_submit_rw_request(raid_io);
		break;

	case SPDK_BDEV_IO_TYPE_RESET:
		raid_bdev_submit_reset_request(raid_io);
		break;

	case SPDK_BDEV_IO_TYPE_FLUSH:
	case SPDK_BDEV_IO_TYPE_UNMAP:
		raid_bdev_submit_null_payload_request(raid_io);
		break;
	}
}

Reads go through spdk_bdev_io_get_buf() because the bdev layer may need to allocate or normalize buffers before the lower module can fill them. Writes already have payload buffers. Flush, unmap, and reset are not normal read/write mapping operations, but level modules may still need to forward them to multiple base bdevs.

RAID0 Mapping

Source anchors:

module/bdev/raid/raid0.c:raid0_submit_rw_request
module/bdev/raid/raid0.c:raid0_submit_null_payload_request
module/bdev/raid/raid0.c:g_raid0_module

RAID0 stripes logical blocks across base devices. Reads and writes go to the base that owns the strip segment. Large IO may split before it reaches raid0_submit_rw_request(), so the level function assumes the request fits in one strip.

For strip size S, bases 0 and 1:

logical blocks:
0..S-1       -> base0 offset 0
S..2S-1      -> base1 offset 0
2S..3S-1     -> base0 offset S
3S..4S-1     -> base1 offset S

The mapping math is the heart of RAID0:

/* module/bdev/raid/raid0.c */
start_strip = raid_io->offset_blocks >> raid_bdev->strip_size_shift;
end_strip = (raid_io->offset_blocks + raid_io->num_blocks - 1) >>
	    raid_bdev->strip_size_shift;
if (start_strip != end_strip && raid_bdev->num_base_bdevs > 1) {
	assert(false);
	SPDK_ERRLOG("I/O spans strip boundary!\n");
	raid_bdev_io_complete(raid_io, SPDK_BDEV_IO_STATUS_FAILED);
	return;
}

pd_strip = start_strip / raid_bdev->num_base_bdevs;
pd_idx = start_strip % raid_bdev->num_base_bdevs;
offset_in_strip = raid_io->offset_blocks & (raid_bdev->strip_size - 1);
pd_lba = (pd_strip << raid_bdev->strip_size_shift) + offset_in_strip;
pd_blocks = raid_io->num_blocks;
base_info = &raid_bdev->base_bdev_info[pd_idx];

pd_idx is the member disk. pd_lba is the offset inside that member's data region. The bit shifts work because RAID validates strip size so it can use strip_size_shift rather than a divide in the hot path.

RAID0 has no redundancy. Losing a base means the logical address space has holes, so the array cannot serve complete data.

RAID1 Mapping

Source anchors:

module/bdev/raid/raid1.c:raid1_submit_rw_request
module/bdev/raid/raid1.c:raid1_submit_read_request
module/bdev/raid/raid1.c:raid1_submit_write_request
module/bdev/raid/raid1.c:raid1_submit_process_request
module/bdev/raid/raid1.c:g_raid1_module

RAID1 mirrors data. A write is submitted to operational bases. A read chooses one operational base, and if that read fails the code can try another mirror. This is why RAID1 can be online while degraded: a missing mirror reduces redundancy, not necessarily readability.

Read selection is simple load balancing by outstanding blocks on the current RAID IO channel:

/* module/bdev/raid/raid1.c */
idx = raid1_channel_next_read_base_bdev(raid_bdev, raid_ch);
if (spdk_unlikely(idx == UINT8_MAX)) {
	raid_bdev_io_complete(raid_io, SPDK_BDEV_IO_STATUS_FAILED);
	return 0;
}

base_info = &raid_bdev->base_bdev_info[idx];
base_ch = raid_bdev_channel_get_base_channel(raid_ch, idx);

raid1_init_ext_io_opts(&io_opts, raid_io);
ret = raid_bdev_readv_blocks_ext(base_info, base_ch, raid_io->iovs, raid_io->iovcnt,
				 raid_io->offset_blocks, raid_io->num_blocks,
				 raid1_read_bdev_io_completion, raid_io, &io_opts);

if (spdk_likely(ret == 0)) {
	raid1_channel_inc_read_counters(raid_ch, idx, raid_io->num_blocks);
	raid_io->base_bdev_io_submitted = idx;
}

Writes walk every base slot. Missing slots complete that child part as failed, but the aggregate RAID IO completion logic decides whether the parent IO should succeed according to remaining completions and default status:

/* module/bdev/raid/raid1.c */
raid1_init_ext_io_opts(&io_opts, raid_io);
for (idx = raid_io->base_bdev_io_submitted; idx < raid_bdev->num_base_bdevs; idx++) {
	base_info = &raid_bdev->base_bdev_info[idx];
	base_ch = raid_bdev_channel_get_base_channel(raid_io->raid_ch, idx);

	if (base_ch == NULL) {
		/* skip a missing base bdev's slot */
		raid_io->base_bdev_io_submitted++;
		raid_bdev_io_complete_part(raid_io, 1, SPDK_BDEV_IO_STATUS_FAILED);
		continue;
	}

	ret = raid_bdev_writev_blocks_ext(base_info, base_ch, raid_io->iovs, raid_io->iovcnt,
					  raid_io->offset_blocks, raid_io->num_blocks,
					  raid1_write_bdev_io_completion, raid_io, &io_opts);
	if (spdk_unlikely(ret != 0)) {
		if (spdk_unlikely(ret == -ENOMEM)) {
			raid_bdev_queue_io_wait(raid_io, spdk_bdev_desc_get_bdev(base_info->desc),
						base_ch, _raid1_submit_rw_request);
			return 0;
		}
	}

	raid_io->base_bdev_io_submitted++;
}

The -ENOMEM path is not a generic failure. It means the child IO could not be submitted due to temporary resource pressure, so RAID queues an IO wait and retries the same logical operation later. That retry behavior appears in several level modules.

Do not read the write path as "every mirror must complete successfully for the parent write to succeed." RAID1 initializes the aggregate write status to failed, then raid_bdev_io_complete_part() changes the parent status when any submitted mirror write reports success. A degraded RAID1 write can therefore complete successfully if at least one mirror write succeeds; failed mirrors are marked through the base failure path and the array remains usable only while it satisfies min_base_bdevs_operational.

Concat Mapping

Source anchors:

module/bdev/raid/concat.c:concat_submit_rw_request
module/bdev/raid/concat.c:concat_submit_null_payload_request
module/bdev/raid/concat.c:g_concat_module

Concat appends base devices end-to-end. It has no striping and no redundancy:

logical 0..end(base0)-1       -> base0
logical end(base0)..next-1    -> base1
logical next..next+base2-1    -> base2

The implementation searches a precomputed base range table and subtracts the selected base's start offset:

/* module/bdev/raid/concat.c */
pd_idx = -1;
for (i = 0; i < raid_bdev->num_base_bdevs; i++) {
	if (block_range[i].start > raid_io->offset_blocks) {
		break;
	}
	pd_idx = i;
}
assert(pd_idx >= 0);
assert(raid_io->offset_blocks >= block_range[pd_idx].start);
pd_lba = raid_io->offset_blocks - block_range[pd_idx].start;
pd_blocks = raid_io->num_blocks;
base_info = &raid_bdev->base_bdev_info[pd_idx];

Concat is useful when learning virtual bdevs because it shows the simplest possible many-to-one mapping: choose one child and adjust the offset.

RAID5f Mapping

Source anchors:

module/bdev/raid/raid5f.c:raid5f_submit_rw_request
module/bdev/raid/raid5f.c:raid5f_submit_read_request
module/bdev/raid/raid5f.c:raid5f_submit_write_request
module/bdev/raid/raid5f.c:raid5f_submit_reconstruct_read
module/bdev/raid/raid5f.c:raid5f_submit_process_request
module/bdev/raid/raid5f.c:g_raid5f_module

RAID5f stores parity and can reconstruct reads when one chunk is unavailable. The implementation works in stripe units. A logical offset is converted into a stripe index, an offset inside the stripe, and a data chunk index. The parity chunk rotates by stripe.

The read path shows the degraded case directly:

/* module/bdev/raid/raid5f.c */
raid5f_init_ext_io_opts(&io_opts, raid_io);
if (base_ch == NULL) {
	return raid5f_submit_reconstruct_read(raid_io, stripe_index, chunk_idx, chunk_offset,
					      raid5f_stripe_request_reconstruct_xor_done);
}

ret = raid_bdev_readv_blocks_ext(base_info, base_ch, raid_io->iovs, raid_io->iovcnt,
				 base_offset_blocks, raid_io->num_blocks,
				 raid5f_chunk_read_complete, raid_io, &io_opts);
if (spdk_unlikely(ret == -ENOMEM)) {
	raid_bdev_queue_io_wait(raid_io, spdk_bdev_desc_get_bdev(base_info->desc),
				base_ch, _raid5f_submit_rw_request);
	return 0;
}

If the target base channel is missing, RAID5f does not immediately fail the read. It creates a reconstruct read over the rest of the stripe and XORs the surviving data/parity into the requested buffer. That is the operational difference between RAID5f and RAID0: missing member data can be rebuilt from parity as long as the redundancy limit is not exceeded.

The top-level read/write dispatcher also encodes RAID5f's write-shape assumptions:

/* module/bdev/raid/raid5f.c */
switch (raid_io->type) {
case SPDK_BDEV_IO_TYPE_READ:
	assert(raid_io->num_blocks <= raid_bdev->strip_size);
	ret = raid5f_submit_read_request(raid_io, stripe_index, stripe_offset);
	break;
case SPDK_BDEV_IO_TYPE_WRITE:
	assert(stripe_offset == 0);
	assert(raid_io->num_blocks == r5f_info->stripe_blocks);
	ret = raid5f_submit_write_request(raid_io, stripe_index);
	break;
default:
	ret = -EINVAL;
	break;
}

In this implementation, RAID5f write handling is full-stripe shaped at the level-module boundary. A write must start at stripe offset 0 and cover exactly r5f_info->stripe_blocks, while a read is constrained to at most one strip here. Partial user writes have to be split or shaped before this function; the assertions document an internal contract, not a supported partial-stripe write path inside raid5f_submit_rw_request().

Beginner path: understand RAID0 offset mapping first, then RAID1 fan-out and degraded reads, then RAID5f's stripe request helpers.

Superblocks

Source anchors:

module/bdev/raid/bdev_raid_sb.c:raid_bdev_alloc_superblock
module/bdev/raid/bdev_raid_sb.c:raid_bdev_init_superblock
module/bdev/raid/bdev_raid_sb.c:raid_bdev_write_superblock
module/bdev/raid/bdev_raid_sb.c:_raid_bdev_write_superblock
module/bdev/raid/bdev_raid_sb.c:raid_bdev_load_base_bdev_superblock
module/bdev/raid/bdev_raid.c:raid_bdev_examine_load_sb
module/bdev/raid/bdev_raid.c:raid_bdev_examine_sb
module/bdev/raid/bdev_raid.c:raid_bdev_create_from_sb

Superblocks let RAID discover arrays from base bdev metadata. Without superblocks, a config-driven RAID must be recreated by config/RPC. With superblocks, examine can read member metadata and reconstruct the RAID object.

Superblock metadata is outside the RAID data area. When a superblock-enabled base bdev is configured, common RAID chooses data_offset from the minimum reserved area, rounds it up to the base bdev's optimal IO boundary when needed, and then uses data_size = blockcnt - data_offset unless existing metadata already supplies a data size. Child IO is shifted by data_offset, so raw base capacity is not the same as usable member capacity. Capacity calculations should use the data_offset and data_size fields reported by bdev_raid_get_bdevs, not only the raw base bdev block count.

Config Save And Replay

Source anchors:

module/bdev/raid/bdev_raid.c:raid_bdev_write_config_json
module/bdev/raid/bdev_raid.c:raid_bdev_opts_config_json
module/bdev/raid/bdev_raid.c:raid_bdev_config_json
lib/bdev/bdev.c:bdev_write_config_json

Save-config behavior depends on whether the array is superblock-backed. raid_bdev_write_config_json() returns without writing a bdev_raid_create entry when raid_bdev->superblock_enabled is true, because the array membership and geometry are stored on member disks. Module-wide RAID options are still saved by raid_bdev_opts_config_json() as bdev_raid_set_options.

For replay, that means two different restore paths:

Config-only RAID needs its saved bdev_raid_create RPC replayed so the in-memory RAID object knows the expected bases.
Superblock-backed RAID needs the lower bdevs recreated first; then examine reads the member superblocks and recreates or updates the RAID object. bdev_wait_for_examine is the synchronization point before upper layers assume the RAID bdev exists.

The parser treats the superblock as a small on-disk contract. It checks signature, size, CRC, version, and base slots before trusting it:

/* module/bdev/raid/bdev_raid_sb.c */
if (memcmp(sb->signature, RAID_BDEV_SB_SIG, sizeof(sb->signature))) {
	SPDK_DEBUGLOG(bdev_raid_sb, "invalid signature\n");
	return -EINVAL;
}

if (spdk_divide_round_up(sb->length, spdk_bdev_get_data_block_size(bdev)) >
    spdk_divide_round_up(ctx->buf_size, bdev->blocklen)) {
	if (sb->length > RAID_BDEV_SB_MAX_LENGTH) {
		SPDK_WARNLOG("Incorrect superblock length on bdev %s\n",
			     spdk_bdev_get_name(bdev));
		return -EINVAL;
	}

	return -EAGAIN;
}

if (!raid_bdev_sb_check_crc(sb)) {
	SPDK_WARNLOG("Incorrect superblock crc on bdev %s\n", spdk_bdev_get_name(bdev));
	return -EINVAL;
}

Superblock writes are fan-out writes to all configured, non-removing base devices. The write path also handles temporary bdev resource pressure with spdk_bdev_queue_io_wait():

/* module/bdev/raid/bdev_raid_sb.c */
for (i = ctx->submitted; i < raid_bdev->num_base_bdevs; i++) {
	base_info = &raid_bdev->base_bdev_info[i];

	if (!base_info->is_configured || base_info->remove_scheduled) {
		assert(ctx->remaining > 1);
		raid_bdev_write_sb_base_bdev_done(0, ctx);
		ctx->submitted++;
		continue;
	}

	rc = spdk_bdev_write(base_info->desc, base_info->app_thread_ch,
			     raid_bdev->sb_io_buf, 0, raid_bdev->sb_io_buf_size,
			     raid_bdev_write_superblock_cb, ctx);
	if (rc != 0) {
		struct spdk_bdev *bdev = spdk_bdev_desc_get_bdev(base_info->desc);

		if (rc == -ENOMEM) {
			ctx->wait_entry.bdev = bdev;
			ctx->wait_entry.cb_fn = _raid_bdev_write_superblock;
			ctx->wait_entry.cb_arg = ctx;
			spdk_bdev_queue_io_wait(bdev, base_info->app_thread_ch, &ctx->wait_entry);
			return;
		}
	}
}

This is why metadata changes are asynchronous. Removing a base, finishing a rebuild, or resizing the array may need to update member superblocks, and those writes can complete later through callbacks.

Edge cases:

Superblock version may be newer than the running code expects.
A base may have stale RAID metadata from a different array.
Some arrays may be config-only without superblock discovery.
Metadata writes can fail and leave future examine/recovery paths to reconcile the state.
Interleaved metadata needs special IO buffers because data and metadata share the physical block format.

Degraded Mode And Base Removal

Source anchors:

module/bdev/raid/bdev_raid.c:raid_bdev_remove_base_bdev
module/bdev/raid/bdev_raid.c:_raid_bdev_remove_base_bdev
module/bdev/raid/bdev_raid.c:raid_bdev_remove_base_bdev_quiesce
module/bdev/raid/bdev_raid.c:raid_bdev_remove_base_bdev_on_quiesced
module/bdev/raid/bdev_raid.c:raid_bdev_deconfigure
module/bdev/raid/bdev_raid.c:raid_bdev_deconfigure_base_bdev

Degraded mode is a lifecycle state, not a different IO API. Upper layers still submit normal IO to the RAID bdev. The RAID module decides whether the array has enough operational members to continue and whether a level module can reconstruct or mirror the requested data.

Base removal can happen because the user removes a member, because the lower bdev reports an event, or because a child IO failure causes RAID to mark a base failed. Event dispatch is intentionally small:

/* module/bdev/raid/bdev_raid.c */
switch (type) {
case SPDK_BDEV_EVENT_REMOVE:
	rc = raid_bdev_remove_base_bdev(bdev, NULL, NULL);
	if (rc != 0) {
		SPDK_ERRLOG("Failed to remove base bdev %s: %s\n",
			    spdk_bdev_get_name(bdev), spdk_strerror(-rc));
	}
	break;
case SPDK_BDEV_EVENT_RESIZE:
	raid_bdev_resize_base_bdev(bdev);
	break;
default:
	SPDK_NOTICELOG("Unsupported bdev event: type %d\n", type);
	break;
}

The removal path distinguishes arrays that can tolerate a missing base from arrays that cannot:

/* module/bdev/raid/bdev_raid.c */
assert(base_info->desc);
base_info->remove_scheduled = true;

if (raid_bdev->state != RAID_BDEV_STATE_ONLINE) {
	raid_bdev_free_base_bdev_resource(base_info);
	base_info->remove_scheduled = false;
	if (raid_bdev->num_base_bdevs_discovered == 0 &&
	    raid_bdev->state == RAID_BDEV_STATE_OFFLINE) {
		raid_bdev_cleanup_and_free(raid_bdev);
	}
	if (cb_fn != NULL) {
		cb_fn(cb_ctx, 0);
	}
} else if (raid_bdev->min_base_bdevs_operational == raid_bdev->num_base_bdevs) {
	/* This raid bdev does not tolerate removing a base bdev. */
	raid_bdev->num_base_bdevs_operational--;
	raid_bdev_deconfigure(raid_bdev, cb_fn, cb_ctx);
} else {
	base_info->remove_cb = cb_fn;
	base_info->remove_cb_ctx = cb_ctx;

	if (raid_bdev->process != NULL) {
		ret = raid_bdev_process_base_bdev_remove(raid_bdev->process, base_info);
	} else {
		ret = raid_bdev_remove_base_bdev_quiesce(base_info);
	}
}

The quiesce step exists to stop normal foreground IO at a point where RAID can safely change channel state, mark metadata, reset the base, and close descriptors. Beginner misconception to kill: base removal is not just clearing a pointer. There may be in-flight IOs, per-thread channels, child IOs waiting on resources, superblock updates, and rebuild state.

Rebuild And Background Processes

Source anchors:

module/bdev/raid/bdev_raid.h:enum raid_process_type
module/bdev/raid/bdev_raid.c:raid_bdev_start_rebuild
module/bdev/raid/bdev_raid.c:raid_bdev_process_alloc
module/bdev/raid/bdev_raid.c:raid_bdev_process_start
module/bdev/raid/bdev_raid.c:raid_bdev_process_thread_init
module/bdev/raid/bdev_raid.c:raid_bdev_process_thread_run
module/bdev/raid/bdev_raid.c:raid_bdev_process_lock_window_range
module/bdev/raid/bdev_raid.c:raid_bdev_submit_process_request
module/bdev/raid/bdev_raid.c:raid_bdev_process_request_complete
module/bdev/raid/bdev_raid.c:raid_bdev_process_finish
module/bdev/raid/raid1.c:raid1_submit_process_request
module/bdev/raid/raid5f.c:raid5f_submit_process_request

Rebuild is implemented as a background process. It has a target base bdev, a process thread, a RAID IO channel, reusable process request objects, a window offset, a maximum window size, optional bandwidth limiting, and finish actions.

The JSON-RPC option docs explain the user-facing meaning of the process window:

From doc/jsonrpc.md.jinja2:

The `process_window_size_kb` parameter defines the size of the "window" (LBA range of the raid bdev)
in which a background process like rebuild performs its work. Any positive value is valid, but the value
actually used by a raid bdev can be adjusted to the size of the raid bdev or the write unit size.

The process loop locks a range of the RAID bdev before submitting process IO. That lock prevents foreground IO from racing with the rebuild update for the same logical range:

/* module/bdev/raid/bdev_raid.c */
rc = spdk_bdev_quiesce_range(&raid_bdev->bdev, &g_raid_if,
			     process->window_offset, process->max_window_size,
			     raid_bdev_process_window_range_locked, process);
if (rc != 0) {
	raid_bdev_process_window_range_locked(process, rc);
}
return true;

A process run advances one window at a time:

/* module/bdev/raid/bdev_raid.c */
static void
raid_bdev_process_thread_run(struct raid_bdev_process *process)
{
	struct raid_bdev *raid_bdev = process->raid_bdev;

	assert(spdk_get_thread() == process->thread);
	assert(process->window_remaining == 0);
	assert(process->window_range_locked == false);

	if (process->state == RAID_PROCESS_STATE_STOPPING) {
		raid_bdev_process_do_finish(process);
		return;
	}

	if (process->window_offset == raid_bdev->bdev.blockcnt) {
		SPDK_DEBUGLOG(bdev_raid, "process completed on %s\n", raid_bdev->bdev.name);
		raid_bdev_process_finish(process, 0);
		return;
	}

	process->max_window_size = spdk_min(raid_bdev->bdev.blockcnt - process->window_offset,
					    process->max_window_size);
	raid_bdev_process_lock_window_range(process);
}

When all process requests in the current window complete, RAID updates every channel's process offset and unlocks the window:

/* module/bdev/raid/bdev_raid.c */
if (status != 0) {
	process->window_status = status;
}

process->window_remaining -= process_req->num_blocks;
if (process->window_remaining == 0) {
	if (process->window_status != 0) {
		raid_bdev_process_finish(process, process->window_status);
		return;
	}

	spdk_for_each_channel(process->raid_bdev, raid_bdev_process_channel_update, process,
			      raid_bdev_process_channels_update_done);
}

RAID1's process request shows the level-specific side. Rebuild reads from an existing mirror through normal RAID1 read logic, then writes the result to the replacement target:

/* module/bdev/raid/raid1.c */
raid_bdev_io_init(raid_io, raid_ch, SPDK_BDEV_IO_TYPE_READ,
		  process_req->offset_blocks, process_req->num_blocks,
		  &process_req->iov, 1, process_req->md_buf, NULL, NULL);
raid_io->completion_cb = raid1_process_read_completed;

ret = raid1_submit_read_request(raid_io);
if (spdk_likely(ret == 0)) {
	return process_req->num_blocks;
} else if (ret < 0) {
	return ret;
} else {
	return -EINVAL;
}

Key subtlety: normal foreground IO may overlap the rebuild boundary. The per-channel process offset tells the submit path which ranges are already rebuilt and which ranges still need old degraded behavior.

Resize

Source anchors:

module/bdev/raid/bdev_raid.c:raid_bdev_resize_base_bdev
module/bdev/raid/bdev_raid.c:raid_bdev_resize_write_sb_cb
module/bdev/raid/bdev_raid.c:raid_bdev_event_base_bdev
module/bdev/raid/raid1.c:raid1_resize

Base resize can affect RAID capacity, but only according to the RAID level's rules. For mirror-like layouts, the smallest member usually controls exposed capacity. For concat, each base contributes a range. For RAID0 and RAID5f, striping/parity constraints matter.

Common resize handling updates the base's block count, asks the level module whether the exported RAID bdev changed size, and writes updated superblocks if needed:

/* module/bdev/raid/bdev_raid.c */
SPDK_NOTICELOG("base_bdev '%s' was resized: old size %" PRIu64 ", new size %" PRIu64 "\n",
	       base_bdev->name, base_info->blockcnt, base_bdev->blockcnt);

base_info->blockcnt = base_bdev->blockcnt;

if (!raid_bdev->module->resize) {
	return;
}

blockcnt_old = raid_bdev->bdev.blockcnt;
if (raid_bdev->module->resize(raid_bdev) == false) {
	return;
}

SPDK_NOTICELOG("raid bdev '%s': block count was changed from %" PRIu64 " to %" PRIu64 "\n",
	       raid_bdev->bdev.name, blockcnt_old, raid_bdev->bdev.blockcnt);

if (raid_bdev->superblock_enabled) {
	struct raid_bdev_superblock *sb = raid_bdev->sb;
	uint8_t i;

	for (i = 0; i < sb->base_bdevs_size; i++) {
		struct raid_bdev_sb_base_bdev *sb_base_bdev = &sb->base_bdevs[i];

		if (sb_base_bdev->slot < raid_bdev->num_base_bdevs) {
			base_info = &raid_bdev->base_bdev_info[sb_base_bdev->slot];
			sb_base_bdev->data_size = base_info->data_size;
		}
	}
	sb->raid_size = raid_bdev->bdev.blockcnt;
	raid_bdev_write_superblock(raid_bdev, raid_bdev_resize_write_sb_cb, NULL);
}

RAID1's resize implementation demonstrates the "smallest usable member" rule:

/* module/bdev/raid/raid1.c */
RAID_FOR_EACH_BASE_BDEV(raid_bdev, base_info) {
	struct spdk_bdev *base_bdev;

	if (base_info->desc == NULL) {
		continue;
	}
	base_bdev = spdk_bdev_desc_get_bdev(base_info->desc);
	min_blockcnt = spdk_min(min_blockcnt, base_bdev->blockcnt - base_info->data_offset);
}

if (min_blockcnt == raid_bdev->bdev.blockcnt) {
	return false;
}

rc = spdk_bdev_notify_blockcnt_change(&raid_bdev->bdev, min_blockcnt);

Resize is not the same as lvolstore grow. RAID may notify that its exported bdev has a new block count, but upper layers that keep their own metadata, such as blobstore/lvolstore, need their own grow path and metadata updates.

Edge cases:

One base grows but others do not; the RAID logical size may not change.
A base shrinks below current mapping requirements; the level resize function may refuse to expose a new size.
Resize during rebuild or removal has to interact with process/channel state.
Upper layers may need explicit grow operations after RAID grows.

JSON-RPC Surface

RPC anchors:

module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_create
module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_delete
module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_get_bdevs
module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_add_base_bdev
module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_remove_base_bdev
module/bdev/raid/bdev_raid_rpc.c:rpc_bdev_raid_set_options

The RPC surface is mostly lifecycle control:

bdev_raid_create constructs the RAID object and records the intended base bdevs.
bdev_raid_get_bdevs reports state, member slots, data offsets, data sizes, and process state.
bdev_raid_add_base_bdev fills an empty slot and may start rebuild for redundant levels.
bdev_raid_remove_base_bdev removes a member and may deconfigure or degrade the array.
bdev_raid_set_options controls future background process window and bandwidth settings.

The add-base docs call out compatibility requirements:

From doc/jsonrpc.md.jinja2:

Add base bdev to existing raid bdev. The raid bdev must have an empty base bdev slot.
The bdev must be large enough and have the same block size and metadata format as the other base bdevs.

This maps directly to the base configuration checks shown earlier.

Stacking With lvol

Two common stacks:

RAID under lvol:
  base0 + base1 -> RAID bdev -> lvolstore -> lvol bdevs

lvol under RAID:
  lvol bdev A + lvol bdev B -> RAID bdev

RAID under lvol is usually easier to reason about for shared durability: one lvolstore sees one reliability layer beneath it. That is an operational inference from the ownership graph, not a hard rule in SPDK. lvol under RAID is possible as a bdev graph, but it couples independent lvolstores to RAID lifecycle and metadata decisions.

The source-level responsibilities remain the same either way:

Each layer claims its immediate base.
Each layer handles remove and resize events from its base.
Each layer exposes a new bdev that may be examined by other modules.
Each layer translates IO offsets, metadata, and completions.

When debugging stacked systems, walk one edge at a time. Ask which module claimed the base, which module owns the event callback, and whether the upper layer has its own metadata that must be grown, repaired, or reloaded.

Misconceptions To Kill

"RAID is special outside bdev." No. It is a bdev module that exports a virtual bdev.
"Online means all bases are present." Not necessarily for redundant RAID levels; degraded online operation may be allowed.
"Configuring means broken." It may mean the module is waiting for more bases or metadata.
"Superblocks are required for all RAID." Config-driven RAID can exist, but superblocks enable disk discovery.
"Rebuild is a single IO." It is a windowed background process with quiesce/unquiesce and per-channel updates.
"Resize of one base automatically grows every upper layer." RAID may resize its bdev, but lvolstore/blobstore grow is a separate operation.
"Metadata only means RAID superblocks." It can also mean bdev data-integrity metadata, and RAID checks that member metadata formats match.

Source Reading Exercise

Read the RAID1 write path:

module/bdev/raid/bdev_raid.c:raid_bdev_submit_request
module/bdev/raid/bdev_raid.c:raid_bdev_submit_rw_request
module/bdev/raid/raid1.c:raid1_submit_rw_request
module/bdev/raid/raid1.c:raid1_submit_write_request
module/bdev/raid/bdev_raid.c:raid_bdev_io_complete_part
module/bdev/raid/bdev_raid.c:raid_bdev_io_complete

Questions:

How many child IOs can one write produce?
Where does the level module decide which base bdevs receive IO?
How are completions collected?
Where would -ENOMEM or IO-wait retry enter the path?
What happens if one base channel is missing?

Then read the degraded/rebuild path:

module/bdev/raid/bdev_raid.c:raid_bdev_event_base_bdev
module/bdev/raid/bdev_raid.c:_raid_bdev_remove_base_bdev
module/bdev/raid/bdev_raid.c:raid_bdev_configure_base_bdev_cont
module/bdev/raid/bdev_raid.c:raid_bdev_start_rebuild
module/bdev/raid/bdev_raid.c:raid_bdev_process_thread_run
module/bdev/raid/raid1.c:raid1_submit_process_request or module/bdev/raid/raid5f.c:raid5f_submit_process_request

Operational Lab

Use test/bdev/bdev_raid.sh as the main lab script.

Suggested manual lab:

1. Create three malloc bdevs.
2. Create RAID1 or RAID0 using two of them.
3. Run bdev_get_bdevs and bdev_raid_get_bdevs with category=all.
4. Create an lvolstore on the RAID bdev.
5. Create an lvol and write data.
6. Remove one RAID base bdev.
7. Observe RAID state and lvol IO behavior.
8. Add a replacement base.
9. Observe rebuild/process state.
10. Grow base bdevs and determine which layers see the new size.

Debug checklist:

Is the RAID bdev online, configuring, or offline?
Are all expected base names present?
Did superblock load succeed?
Is a background process active?
Is a base marked is_process_target?
Did upper layers wait for examine after base creation?
Is the lvolstore on top of RAID using spdk_bs_grow_live() or an equivalent grow path after RAID size changes?
Are base bdev metadata formats identical?
Did a child IO return -ENOMEM, causing an IO-wait retry rather than immediate failure?

Self-Check

What does RAID add beyond a simple bdev wrapper?
Why can a RAID bdev exist but not be registered as an online bdev?
Which functions read and write RAID superblocks?
Why does rebuild quiesce ranges?
How does RAID0 map logical offsets to base offsets?
How does RAID1 differ for read and write?
What happens if a base is removed during a background process?
Why is RAID resize not the same as lvolstore grow?
Why must base bdev block size and metadata format match?
Why does RAID claim base bdevs?

References

Local official bdev programmer guide: doc/bdev_pg.md
Local official bdev user guide RAID section: doc/bdev.md
Local official JSON-RPC RAID docs: doc/jsonrpc.md.jinja2
Local RAID core: module/bdev/raid/bdev_raid.c
Local RAID structures: module/bdev/raid/bdev_raid.h
Local RAID superblocks: module/bdev/raid/bdev_raid_sb.c
Local RAID RPC: module/bdev/raid/bdev_raid_rpc.c
Local RAID levels: module/bdev/raid/raid0.c, module/bdev/raid/raid1.c, module/bdev/raid/concat.c, module/bdev/raid/raid5f.c
Local bdev layer: lib/bdev/bdev.c, include/spdk/bdev.h
Local tests: test/bdev/bdev_raid.sh, test/unit/lib/bdev/raid/bdev_raid.c/bdev_raid_ut.c