Chapter 3: What A Block Device Promises

Chapter Goal

A block device is the lie that makes storage programming possible. It tells software: "Give me a logical block address and a length, and I will read or write that range." It hides heads, cylinders, NAND pages, erase blocks, caches, remaps, retries, and controller firmware. SPDK is built around that same promise: almost every storage object eventually becomes a struct spdk_bdev, even when the backing thing is an NVMe namespace, a malloc buffer, a file, a RAID volume, a logical volume, or a remote export.

By the end of this chapter, you should be able to explain the difference between bytes, sectors, and logical blocks; reason about alignment and atomicity; explain why flush, protocol-specific write-through flags such as NVMe FUA, unmap, write zeroes, and metadata matter; and read the SPDK bdev structure without treating its fields as random driver trivia.

Beginner Mental Model

Imagine a huge numbered shelf of fixed-size boxes:

logical block number:   0      1      2      3      4      5
                      +------+------+------+------+------+------+
contents:             | 4KiB | 4KiB | 4KiB | 4KiB | 4KiB | 4KiB |
                      +------+------+------+------+------+------+

The host does not ask "put this byte at NAND die 3, plane 1, block 22, page 17." It asks "write 8 logical blocks starting at LBA 128." A block device turns byte-oriented user data into block-oriented commands. The simplest block device contract has four pieces:

1. A fixed logical block size, commonly 512 bytes or 4096 bytes.
2. A count of logical blocks.
3. Operations such as read, write, flush, unmap, reset, and write zeroes.
4. Rules about which addresses, lengths, and buffers are legal or fast.

That last line is where most production bugs hide. A block device is not an infinite byte array with magical durability. It is an asynchronous state machine with geometry, caching, media limits, and failure modes.

Bytes, Sectors, And Logical Blocks

A byte is the CPU's smallest addressable unit. A sector was historically the disk drive's native transfer unit, often 512 bytes. A logical block is the unit exposed by a modern storage API. In practice people still say "sector" when they mean "logical block," but the distinction matters when devices expose 512-byte logical blocks backed by 4096-byte physical sectors or when metadata/DIF adds extra bytes per block.

In SPDK, bdev APIs use blocks for most storage operations. The public getters expose that geometry. The block size is in bytes; the capacity is in logical blocks; a caller computes byte capacity by multiplying the two only after validating that the multiplication cannot overflow in the caller's own type.

• include/spdk/bdev.h:807 declares spdk_bdev_get_block_size().
• include/spdk/bdev.h:830 declares spdk_bdev_get_num_blocks() and documents that valid logical blocks are numbered from 0 through num_blocks - 1.
• include/spdk/bdev_module.h:436 and include/spdk/bdev_module.h:445 show the backend fields blocklen and blockcnt.

The most important beginner rule is this: offsets and lengths in a block API are not byte offsets unless the function name says so. offset_blocks = 10 on a 4096-byte bdev means byte offset 40960, not byte offset 10.

SPDK intentionally provides both byte-oriented and block-oriented entry points. The byte-oriented APIs are convenient at an application boundary, but they still convert to whole blocks before the bdev layer accepts the request. If the offset or length is not a multiple of the bdev block size, the call fails before it reaches the backend module. That is why a caller that thinks in bytes should normalize once at the edge, then keep the rest of the path in blocks.

There is also a second distinction hidden in the words "logical" and "physical." blocklen says what the upper layer may address. phys_blocklen says what the backend reports as the physical block size. A device can legally expose 512-byte logical blocks while preferring, or internally using, 4096-byte physical writes. The logical size is the correctness unit for LBA math; the physical size is a hint about performance and write amplification.

Why This Matters For diskengine And excloud

Cloud volumes need a stable illusion. Tenants and filesystems expect a volume to have a size, support reads and writes at specific offsets, and preserve ordering when the control plane asks for a flush or when a guest issues a barrier. diskengine can compose SPDK bdevs, export them, and reconcile them, but the correctness boundary still starts with the block contract.

When debugging an excloud volume, classify symptoms using the block model first:

• "Read returns old data" may be ordering, flush, cache, or lost completion.
• "Write succeeds but later data is zero" may be unmap/write-zeroes behavior, thin provisioning, or backend replacement.
• "Only 4K writes fail" may be alignment, write unit size, metadata, or atomicity.
• "Latency spikes during random write" may be SSD garbage collection hidden below the block abstraction.

SPDK does not remove those concerns. It gives you sharper tools and fewer kernel layers between the application and the device.

The SPDK bdev Contract In Source

The central structure is struct spdk_bdev in include/spdk/bdev_module.h:420. The fields are the vocabulary of the contract:

• name and aliases identify the device.
• blocklen is the logical block size in bytes.
• phys_blocklen describes the physical block size when it differs.
• io_type_supported records which operation classes the backend accepts.
• blockcnt is the number of logical blocks.
• write_unit_size, optimal_io_boundary, preferred_write_alignment, preferred_write_granularity, and optimal_write_size describe write shape.
• acwu describes the maximum atomic compare-and-write unit, not a normal write atomicity promise.
• max_segment_size, max_num_segments, max_unmap, max_unmap_segments, max_write_zeroes, max_copy, and max_rw_size limit request shape.
• required_alignment says data buffers may need a specific alignment; SPDK may double-buffer when a caller violates it.

These are not just informational fields. They determine whether the bdev layer splits requests, rejects requests, allocates bounce buffers, or passes an operation directly to the module. The comments around include/spdk/bdev_module.h:448 explain that the bdev layer may split writes on write_unit_size or split reads/writes on optimal_io_boundary; the same comments explicitly call out that these flags do not force splitting for unmap, write zeroes, or flush.

The public support check is spdk_bdev_io_type_supported() in include/spdk/bdev.h:752. You should never assume a bdev supports unmap, write zeroes, compare-and-write, zone append, or NVMe passthrough just because the underlying hardware might. The exported bdev may be virtual, layered, or deliberately conservative.

Here is the core of the contract object. This excerpt is not the whole structure; it is the part that turns a name into addressable geometry and declares which operation classes the device accepts:

/* include/spdk/bdev_module.h */
struct spdk_bdev {
	/** Unique name for this block device. */
	char *name;

	/** Unique product name for this kind of block device. */
	char *product_name;

	/** write cache enabled, not used at the moment */
	int write_cache;

	/** Size in bytes of a logical block for the backend */
	uint32_t blocklen;

	/** Size in bytes of a physical block for the backend */
	uint32_t phys_blocklen;

	/** Bitmap of supported io types */
	uint32_t io_type_supported;

	/** Number of blocks */
	uint64_t blockcnt;

Read this as a boundary between two worlds. Above the bdev layer, code should not know whether the storage is NVMe, malloc memory, a file, a RAID volume, or an NVMe-oF namespace. Below it, each module must translate this abstract contract into its own driver, firmware, or memory operations. blocklen and blockcnt are enough to describe the exported byte range, but they are not enough to describe the safe or efficient request shape. That is why the same structure continues with splitting, alignment, and metadata fields.

/* include/spdk/bdev_module.h */
	struct {
		/*
		 * If set to true, the bdev layer will split
		 * WRITE I/O that span the write_unit_size before
		 * submitting them to the bdev module.
		 */
		uint32_t split_on_write_unit : 1;

		/*
		 * If set to true, the bdev layer will split
		 * READ and WRITE I/O that span the optimal_io_boundary before
		 * submitting them to the bdev module.
		 */
		uint32_t split_on_optimal_io_boundary : 1;

		uint32_t md_interleave : 1;
		uint32_t dif_is_head_of_md : 1;
	};

	/** Number of blocks required for write */
	uint32_t write_unit_size;

	/** Atomic compare & write unit */
	uint16_t acwu;

	/**
	 * Specifies an alignment requirement for data buffers associated with an spdk_bdev_io.
	 * 0 = no alignment requirement
	 * >0 = alignment requirement is 2 ^ required_alignment.
	 */
	uint8_t required_alignment;

	uint32_t optimal_io_boundary;
	uint32_t preferred_write_alignment;
	uint32_t preferred_write_granularity;
	uint32_t optimal_write_size;
	uint32_t preferred_unmap_alignment;
	uint32_t preferred_unmap_granularity;
	uint32_t max_unmap;
	uint32_t max_unmap_segments;
	uint32_t max_write_zeroes;
	uint32_t max_rw_size;

The important design choice is that SPDK keeps both mandatory limits and advisory preferences in the same object. A mandatory limit can cause rejection or splitting. A preference may simply be surfaced to a smarter upper layer. For example, max_rw_size is a hard request-shape limit that the bdev layer can split around. optimal_write_size is a performance hint; ignoring it may hurt throughput or media lifetime, but it is not automatically the same thing as an invalid command.

The public API does not ask applications to poke at struct spdk_bdev directly. It exposes small getters and support checks. The getter comments matter because they define the units:

/* include/spdk/bdev.h */
bool spdk_bdev_io_type_supported(struct spdk_bdev *bdev,
				 enum spdk_bdev_io_type io_type);

/**
 * Get block device logical block size.
 *
 * \return Size of logical block for this bdev in bytes.
 */
uint32_t spdk_bdev_get_block_size(const struct spdk_bdev *bdev);

/**
 * Get the write unit size for this bdev.
 *
 * Unit of write unit size is logical block and the minimum of write unit
 * size is one. Write operations must be multiple of write unit size.
 */
uint32_t spdk_bdev_get_write_unit_size(const struct spdk_bdev *bdev);

/**
 * Get size of block device in logical blocks.
 *
 * Logical blocks are numbered from 0 to spdk_bdev_get_num_blocks(bdev) - 1.
 */
uint64_t spdk_bdev_get_num_blocks(const struct spdk_bdev *bdev);

The corresponding implementation is deliberately boring:

/* lib/bdev/bdev.c */
uint32_t
spdk_bdev_get_write_unit_size(const struct spdk_bdev *bdev)
{
	return bdev->write_unit_size;
}

uint64_t
spdk_bdev_get_num_blocks(const struct spdk_bdev *bdev)
{
	return bdev->blockcnt;
}

size_t
spdk_bdev_get_buf_align(const struct spdk_bdev *bdev)
{
	return 1 << bdev->required_alignment;
}

That boringness is useful. The bdev layer is not guessing capacity from a driver at every I/O. The backend module registers an object, the bdev core normalizes defaults, and callers read the exported contract. If the module lies, every upper layer inherits the lie. If the caller ignores the units, SPDK will often reject the request early, but it cannot save a higher-level protocol that computed the wrong LBA before calling into bdev.

The bdev layer also validates the most basic range rule centrally:

/* lib/bdev/bdev.c */
static bool
bdev_io_valid_blocks(struct spdk_bdev *bdev, uint64_t offset_blocks,
		     uint64_t num_blocks)
{
	if (offset_blocks + num_blocks < offset_blocks) {
		return false;
	}

	if (offset_blocks + num_blocks > bdev->blockcnt) {
		return false;
	}

	return true;
}

This small helper is why the chapter keeps repeating "last valid LBA is num_blocks - 1." A request at offset_blocks == blockcnt is already one block past the end, even if num_blocks == 1. The overflow check is equally important: without it, a very large num_blocks could wrap the sum and look like an in-range request.

How A Real Module Fills The Contract

A bdev module owns the backend-specific object and registers a populated struct spdk_bdev with the bdev core. The malloc module is the cleanest first example because the backing store is just memory allocated from SPDK hugepage memory. It still has to advertise the same contract as a hardware-backed bdev:

/* module/bdev/malloc/bdev_malloc.c */
mdisk->disk.product_name = "Malloc disk";

mdisk->disk.write_cache = 1;
mdisk->disk.blocklen = block_size;
mdisk->disk.phys_blocklen = opts->physical_block_size;
mdisk->disk.blockcnt = opts->num_blocks;
mdisk->disk.md_len = opts->md_size;
mdisk->disk.md_interleave = opts->md_interleave;
mdisk->disk.dif_type = opts->dif_type;
mdisk->disk.dif_is_head_of_md = opts->dif_is_head_of_md;

Even a RAM-backed device has logical blocks, a physical block size field, optional metadata, and optional DIF state. The memory implementation does not make those fields fake; it makes them configurable so tests and examples can exercise the same bdev paths that real devices use.

The same module separately declares which I/O types it accepts and wires the backend callbacks into a function table:

/* module/bdev/malloc/bdev_malloc.c */
static bool
bdev_malloc_io_type_supported(void *ctx, enum spdk_bdev_io_type io_type)
{
	switch (io_type) {
	case SPDK_BDEV_IO_TYPE_READ:
	case SPDK_BDEV_IO_TYPE_WRITE:
	case SPDK_BDEV_IO_TYPE_FLUSH:
	case SPDK_BDEV_IO_TYPE_RESET:
	case SPDK_BDEV_IO_TYPE_UNMAP:
	case SPDK_BDEV_IO_TYPE_WRITE_ZEROES:
	case SPDK_BDEV_IO_TYPE_ZCOPY:
	case SPDK_BDEV_IO_TYPE_ABORT:
	case SPDK_BDEV_IO_TYPE_COPY:
		return true;
	default:
		return false;
	}
}

static const struct spdk_bdev_fn_table malloc_fn_table = {
	.destruct = bdev_malloc_destruct,
	.submit_request = bdev_malloc_submit_request,
	.io_type_supported = bdev_malloc_io_type_supported,
	.get_io_channel = bdev_malloc_get_io_channel,
};

This is the bdev object model in miniature. The geometry fields say what address range exists. io_type_supported says which verbs are legal. submit_request is where accepted I/O goes. get_io_channel provides the per-thread path used by the backend. Later runtime chapters explain channels in detail; for this chapter, the key point is that a bdev is not merely a struct full of constants. It is a registered object with callbacks that run on SPDK thread/channel machinery.

The NVMe module fills the same fields from namespace and controller data instead of RPC options:

/* module/bdev/nvme/bdev_nvme.c */
if (cdata->vwc.present) {
	/* Enable if the Volatile Write Cache exists */
	disk->write_cache = 1;
}
if (cdata->oncs.nvmwzsv) {
	disk->max_write_zeroes = UINT16_MAX + 1;
}
disk->blocklen = spdk_nvme_ns_get_extended_sector_size(ns);
disk->blockcnt = spdk_nvme_ns_get_num_sectors(ns);
disk->max_segment_size = spdk_nvme_ctrlr_get_max_xfer_size(ctrlr);
disk->optimal_io_boundary = spdk_nvme_ns_get_optimal_io_boundary(ns);

This excerpt is useful because it shows where the abstraction stops. SPDK does not invent a random block size for NVMe. The bdev is populated from NVMe namespace/controller facts and then exported through the common bdev API. The upper layer sees a struct spdk_bdev; the module still knows it is talking to an NVMe namespace.

Atomicity Is Smaller Than You Think

Atomicity answers: after a crash or power failure, can software observe half of a write? The naive answer is "a sector write is atomic." The useful answer is "read the contract, then still be skeptical."

There are multiple atomicity levels:

• CPU store atomicity: irrelevant once data leaves CPU caches.
• DMA transfer granularity: a device may see a scatter-gather request as multiple memory reads.
• Device media/program granularity: the SSD may program NAND pages or internal units larger than an LBA.
• Controller advertised write unit: the host-visible unit that may constrain legal or reliable writes.
• Filesystem or database transaction: an upper-layer protocol built from writes, flushes, journals, checksums, and recovery.

SPDK exposes two fields that are easy to confuse:

• write_unit_size in include/spdk/bdev_module.h:507 is the number of logical blocks required for a normal write. The public getter comment in include/spdk/bdev.h:815 says write operations must be multiples of that unit. That is a request-shape rule.
• acwu in include/spdk/bdev_module.h:510 is the "Atomic compare & write unit." The public getter in include/spdk/bdev.h:910 calls it the atomic compare-and-write unit size. The bdev core enforces it in the compare-and-write path: lib/bdev/bdev.c:6690 rejects spdk_bdev_comparev_and_writev_blocks() when num_blocks > bdev->acwu.

Do not read acwu as "ordinary writes up to this many blocks are atomically durable." It is the size limit for the fused compare-and-write operation. The NVMe bdev module only populates it when compare-and-write is supported, using namespace/controller atomic compare-and-write data in module/bdev/nvme/bdev_nvme.c:4662 through module/bdev/nvme/bdev_nvme.c:4669.

Normal write atomicity and persistence still depend on the protocol, device guarantees, write cache state, flush behavior, and upper-layer recovery design. write_unit_size can tell you which write shapes are legal. acwu can tell you how large a compare-and-write operation may be. Neither is a blanket transaction guarantee for arbitrary writes.

Misconception to kill: "If a write completion callback fired, the data is on NAND." A write completion means the device accepted and completed the command according to the protocol and its cache policy. If volatile write cache is involved, durability may still require a flush, or for a lower-level protocol path that exposes it, a command-specific forced-unit-access flag.

Alignment And Splitting

Alignment has three separate meanings:

• LBA alignment: the starting block and block count must satisfy some multiple.
• Buffer alignment: the host memory pointer must be aligned for DMA or backend requirements.
• Internal media alignment: the SSD prefers larger sequential shapes even if it accepts smaller legal writes.

SPDK makes alignment visible in bdev fields. required_alignment in include/spdk/bdev_module.h:513 describes buffer alignment and says the bdev layer may double-buffer misaligned I/O. preferred_write_alignment, preferred_write_granularity, and optimal_write_size are performance hints. split_on_write_unit and split_on_optimal_io_boundary are enforcement flags.

A useful diagram in prose:

Application request:
  write 12 blocks at LBA 6

Device rule:
  optimal boundary = 8 blocks

Visual:
  boundary 0          boundary 8          boundary 16
  |-------------------|-------------------|
        request starts here: [6 7 | 8 9 10 11 12 13 14 15 | 16 17]

Possible bdev-layer behavior:
  child A: LBA 6, 2 blocks
  child B: LBA 8, 8 blocks
  child C: LBA 16, 2 blocks

Splitting is a correctness tool and a performance tool. It also changes debugging. One user request may become several module requests and several completions internally, while the user sees one callback.

The bdev core decides whether a read/write request needs splitting by looking at boundary, segment, and size limits. This is not a slow path bolted onto a single driver; it is part of the generic bdev layer:

/* lib/bdev/bdev.c */
static bool
bdev_rw_should_split(struct spdk_bdev_io *bdev_io)
{
	uint32_t io_boundary;
	struct spdk_bdev *bdev = bdev_io->bdev;
	uint32_t max_segment_size = bdev->max_segment_size;
	uint32_t max_size = bdev->max_rw_size;
	int max_segs = bdev->max_num_segments;

	io_boundary = bdev_rw_get_io_boundary(bdev, bdev_io->type);

	if (spdk_likely(!io_boundary && !max_segs &&
			!max_segment_size && !max_size)) {
		return false;
	}

	if (io_boundary) {
		uint64_t start_stripe, end_stripe;

		start_stripe = bdev_io->u.bdev.offset_blocks;
		end_stripe = start_stripe + bdev_io->u.bdev.num_blocks - 1;
		...
		if (start_stripe != end_stripe) {
			return true;
		}
	}

	if (max_size && bdev_io->u.bdev.num_blocks > max_size) {
		return true;
	}

	return false;
}

The omitted lines check whether the request has too many scatter-gather elements or whether any segment is too large. Those are host-memory shape limits, not LBA limits, but they matter for DMA and backend queue construction. The result is that the bdev layer may split one parent I/O for reasons that have nothing to do with the LBA range itself.

The write-unit rule has a second enforcement point during submission:

/* lib/bdev/bdev.c */
if (spdk_unlikely(bdev_io->type == SPDK_BDEV_IO_TYPE_WRITE &&
		  bdev_io->bdev->split_on_write_unit &&
		  bdev_io->u.bdev.num_blocks < bdev_io->bdev->write_unit_size)) {
	SPDK_ERRLOG("IO num_blocks %lu does not match the write_unit_size %u\n",
		    bdev_io->u.bdev.num_blocks, bdev_io->bdev->write_unit_size);
	_bdev_io_complete_in_submit(bdev_ch, bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
	return;
}

This check explains a subtle rule: splitting can make a large write conform to a write-unit boundary, but it cannot make a too-small write magically valid when split_on_write_unit is mandatory. If the device requires four-block writes and the caller submits one block, there is no safe split that preserves the caller's intended update without a read-modify-write policy above the bdev layer.

Flush, NVMe FUA, And Volatile Caches

Storage has at least three places data can be "written":

1. In host memory, before command submission.
2. In controller memory or volatile device cache.
3. In non-volatile media or protected cache.

A flush asks the device to make previously accepted writes durable. FUA, when supported by a protocol or command, asks for a particular write to bypass or commit through volatile cache. A block abstraction that ignores durability semantics can pass tests and still corrupt a filesystem during power loss.

In SPDK bdev, flush is the generic durability primitive. include/spdk/bdev.h:103 through include/spdk/bdev.h:127 lists the generic bdev I/O types and includes SPDK_BDEV_IO_TYPE_FLUSH; it does not include a generic SPDK_BDEV_IO_TYPE_FUA. The bdev API checks flush support through the same spdk_bdev_io_type_supported() mechanism as other types. The implementation path for flush lives in lib/bdev/bdev.c around spdk_bdev_flush() and spdk_bdev_flush_blocks(); later chapters will trace that in detail. For this chapter, the key idea is conceptual: flush is not "write more bytes." It is an ordering and durability command.

FUA is different. In this checkout it appears as an NVMe command flag, SPDK_NVME_IO_FLAGS_FORCE_UNIT_ACCESS, in include/spdk/nvme_spec.h:4875. The NVMe bdev module can pass NVMe command dwords for write commands, for example through the nvme_cdw12 and nvme_cdw13 fields used in module/bdev/nvme/bdev_nvme.c:3337 through module/bdev/nvme/bdev_nvme.c:3349. That is protocol-specific plumbing below the generic bdev contract, not a portable bdev operation every backend must implement.

Beginner rule: if you are writing against generic bdev APIs, think in terms of write-cache query plus flush. If you are deliberately using an NVMe-specific path, then NVMe FUA may be part of the command semantics, but only for a path that exposes and preserves that NVMe flag.

The public API comment states the durability boundary directly:

/* include/spdk/bdev.h */
/**
 * Submit a flush request to the bdev on the given channel. For devices with volatile
 * caches, data is not guaranteed to be persistent until the completion of a flush
 * request. Call spdk_bdev_has_write_cache() to check if the bdev has a volatile cache.
 */
int spdk_bdev_flush(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch,
		    uint64_t offset, uint64_t length,
		    spdk_bdev_io_completion_cb cb, void *cb_arg);

The implementation path first converts bytes to blocks, then checks that the bdev supports flush and that the range is valid:

/* lib/bdev/bdev.c */
int
spdk_bdev_flush_blocks(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch,
		       uint64_t offset_blocks, uint64_t num_blocks,
		       spdk_bdev_io_completion_cb cb, void *cb_arg)
{
	struct spdk_bdev *bdev = spdk_bdev_desc_get_bdev(desc);

	if (!desc->write) {
		return -EBADF;
	}

	if (spdk_unlikely(!bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_FLUSH))) {
		return -ENOTSUP;
	}

	if (!bdev_io_valid_blocks(bdev, offset_blocks, num_blocks)) {
		return -EINVAL;
	}
	...
	bdev_io->type = SPDK_BDEV_IO_TYPE_FLUSH;
	bdev_io_submit(bdev_io);
	return 0;
}

This is a good example of SPDK's synchronous return versus asynchronous completion split. -ENOTSUP, -EINVAL, and -EBADF here mean the request was not accepted and the completion callback will not run. A successful return means the request was accepted into the bdev machinery; the callback later reports whether the flush itself completed successfully.

Edge cases:

• Some virtual bdevs must translate one flush into flushes on multiple base bdevs.
• Some devices complete flush quickly because they have power-loss protection.
• Some devices expose a volatile write cache but lie or behave badly under firmware bugs.
• A flush after an unmap may not mean reads return zero; it means the deallocation command's persistence rules are satisfied.

UNMAP, TRIM, Deallocate, And Write Zeroes

Unmap tells a device that a range no longer contains useful data. SATA calls the idea TRIM; SCSI calls it UNMAP; NVMe uses Dataset Management deallocate and related semantics. SPDK's bdev abstraction has unmap limits: preferred_unmap_alignment, preferred_unmap_granularity, max_unmap, and max_unmap_segments appear in include/spdk/bdev_module.h:538 through include/spdk/bdev_module.h:559.

Write zeroes is different. It asks the device to make future reads return zeroes for a range, often without transferring a zero-filled buffer from the host. SPDK exposes max_write_zeroes in include/spdk/bdev_module.h:561.

Misconception to kill: "Unmap means zero." It may, but it does not have to in every stack. A deallocated read may return zeroes, old data, undefined data, or complete with special semantics depending on protocol, provisioning mode, and bdev implementation. If an upper layer needs zeros, use a zeroing operation whose semantics are actually guaranteed for that path.

SPDK's own API comments make the difference explicit:

/* include/spdk/bdev.h */
/**
 * Submit a write zeroes request to the bdev on the given channel. This command
 * ensures that all bytes in the specified range are set to 00h
 */
int spdk_bdev_write_zeroes_blocks(struct spdk_bdev_desc *desc,
				  struct spdk_io_channel *ch,
				  uint64_t offset_blocks,
				  uint64_t num_blocks,
				  spdk_bdev_io_completion_cb cb,
				  void *cb_arg);

/**
 * Submit an unmap request to the block device. Unmap is sometimes also called trim or
 * deallocate. This notifies the device that the data in the blocks described is no
 * longer valid. Reading blocks that have been unmapped results in indeterminate data.
 */
int spdk_bdev_unmap_blocks(struct spdk_bdev_desc *desc,
			   struct spdk_io_channel *ch,
			   uint64_t offset_blocks,
			   uint64_t num_blocks,
			   spdk_bdev_io_completion_cb cb,
			   void *cb_arg);

The implementation also treats them differently. Write zeroes can be emulated with regular writes when the bdev supports writes but not native write-zeroes. Unmap is not emulated as write zeroes, because that would change its meaning from "these blocks are no longer useful" to "these blocks must read as zero."

/* lib/bdev/bdev.c */
if (!bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_WRITE_ZEROES) &&
    !bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_WRITE)) {
	return -ENOTSUP;
}

bdev_io->type = SPDK_BDEV_IO_TYPE_WRITE_ZEROES;
...
if (bdev_io_type_supported(bdev, SPDK_BDEV_IO_TYPE_WRITE_ZEROES) ||
    bdev_io->internal.f.split) {
	bdev_io_submit(bdev_io);
	return 0;
}

bdev_write_zero_buffer(bdev_io);
return 0;

For storage systems, this distinction is not pedantic. Thin provisioning wants unmap/deallocate so the backend can reclaim space or mark media free. Filesystem initialization, discard-sensitive security logic, and volume reset paths may need actual zeroes. Treating those as interchangeable will eventually produce either wasted work or wrong read-after-operation behavior.

Metadata, DIF, And Protection Information

Some block devices carry extra metadata per block. That metadata can be interleaved with data or stored separately. It may contain Data Integrity Field (DIF) protection information such as guard tags, application tags, or reference tags.

SPDK exposes descriptor-specific metadata queries in include/spdk/bdev.h:658 through include/spdk/bdev.h:719. The bdev structure tracks metadata placement with md_interleave in include/spdk/bdev_module.h:474 and DIF placement with dif_is_head_of_md around include/spdk/bdev_module.h:482.

Beginner trap: a "4096-byte block" may not be only 4096 bytes on the wire or media. The host may manage 4096 bytes of data plus metadata. Passing buffers without considering metadata can make a perfectly aligned data request illegal.

The bdev structure keeps the metadata shape next to the normal geometry:

/* include/spdk/bdev_module.h */
/** Size in bytes of a metadata for the backend */
uint32_t md_len;

/**
 * DIF type for this bdev.
 *
 * Note that this field is valid only if there is metadata.
 */
enum spdk_dif_type dif_type;

/**
 * DIF protection information format for this bdev.
 *
 * Note that this field is valid only if there is metadata and dif_type is
 * not SPDK_DIF_DISABLE.
 */
enum spdk_dif_pi_format dif_pi_format;

/**
 * Specify whether each DIF check type is enabled.
 */
uint32_t dif_check_flags;

The NVMe bdev fills those fields from namespace data:

/* module/bdev/nvme/bdev_nvme.c */
disk->md_len = spdk_nvme_ns_get_md_size(ns);
if (disk->md_len != 0) {
	disk->md_interleave = nsdata->flbas.extended;
	disk->dif_type = (enum spdk_dif_type)spdk_nvme_ns_get_pi_type(ns);
	if (disk->dif_type != SPDK_DIF_DISABLE) {
		disk->dif_is_head_of_md = nsdata->dps.md_start;
		disk->dif_check_flags = bdev_opts->prchk_flags;
		disk->dif_pi_format =
			(enum spdk_dif_pi_format)spdk_nvme_ns_get_pi_format(ns);
	}
}

The practical lesson is that "block size" in an application buffer and "sector size" at the NVMe namespace are not always the same number of bytes. Some APIs expose extended LBA data that includes metadata. Others keep metadata in a separate buffer. SPDK has separate read/write-with-metadata entry points because the location of that metadata affects buffer layout, DMA length, DIF verification, and whether a module can pass the command through to hardware directly.

Edge Cases And Failure Modes

The easiest block-device bugs are arithmetic bugs. They look harmless in tests because small volumes do not overflow and happy-path writes stay far away from the final LBA. Production volumes eventually find the boundary. A correct caller checks both alignment and range before constructing higher-level state around an I/O. A correct bdev module reports a contract that lets the generic bdev layer do the same validation for every caller.

• Out-of-range LBA: the last valid LBA is num_blocks - 1; off-by-one math often writes one block past the end.
• Integer overflow: byte length is num_blocks * block_size; use wide types and validate before multiplying.
• Short writes do not exist in the usual block command model; commands complete or fail, but layered software may split and partially complete internally before surfacing a failure.
• Reset may fail outstanding I/O or delay new I/O.
• Remove/hotplug can invalidate a bdev while descriptors and channels still exist.
• A virtual bdev may have stricter limits than its base bdev.
• Buffer alignment may silently cost performance because of bounce buffers.
• A benchmark that reads deallocated LBAs may measure metadata fast paths instead of NAND reads.
• A workload that never flushes may look fast and still be unsafe for databases.

The most confusing failures are often layered failures. Suppose a RAID bdev accepts a large write, splits it into child I/O, and one child fails after another child has already completed. The application sees one failed parent I/O, but media may already contain a partial update at lower layers. Correct software above bdev treats completion status, flush ordering, metadata, and recovery protocol as one system rather than assuming the block API provides transaction semantics.

Another common trap is testing with null or malloc and assuming all bdevs behave the same. The null bdev is useful because it makes the bdev framework cheap to benchmark, but its support matrix is intentionally different from malloc and NVMe. For example, local source shows null supports read, write, write zeroes, reset, and abort, while returning false for flush and unmap. Code that never checks spdk_bdev_io_type_supported() will pass against one test device and fail against another.

Source Reading Exercise

Read these anchors in order:

1. include/spdk/bdev_module.h:420 through include/spdk/bdev_module.h:575.
2. include/spdk/bdev_module.h:583 through include/spdk/bdev_module.h:608.
3. include/spdk/bdev.h:752 through include/spdk/bdev.h:838.
4. include/spdk/bdev.h:1838 through include/spdk/bdev.h:1984.
5. lib/bdev/bdev.c:5768 through lib/bdev/bdev.c:5783.
6. lib/bdev/bdev.c:6792 through lib/bdev/bdev.c:7025.
7. module/bdev/malloc/bdev_malloc.c:611 through module/bdev/malloc/bdev_malloc.c:705.
8. module/bdev/malloc/bdev_malloc.c:851 through module/bdev/malloc/bdev_malloc.c:860.
9. module/bdev/nvme/bdev_nvme.c:4590 through module/bdev/nvme/bdev_nvme.c:4659.

Answer these while reading:

• Which fields describe logical geometry?
• Which fields describe physical or performance geometry?
• Which fields can force request splitting?
• Which fields are limits rather than hints?
• Which public getters expose fields directly and which expose descriptor-specific views?
• Which failures are returned synchronously, before the callback can run?
• Which bdev operation types can be emulated, and which cannot be safely treated as equivalent?

Operational Lab

No live SPDK system is required.

- write 1 block at LBA 0 - write 4 blocks at LBA 4 - read 256 blocks at LBA 128 - write 8 blocks at LBA 262140

1. Pick a hypothetical bdev with blocklen = 4096, blockcnt = 262144, write_unit_size = 4, and max_rw_size = 128.
2. Compute the byte capacity.
3. Decide whether each request is legal before splitting:
4. For the read of 256 blocks, sketch how a bdev layer could split it if max_rw_size = 128.
5. Explain which failed cases should return an error immediately and which might be transformed.

Expected reasoning: capacity is 262144 * 4096 = 1 GiB; writes must be multiples of 4 blocks; the read may split into two 128-block reads; the final write overruns the device because LBAs 262144 through 262147 do not exist.

Self-Check

1. Why is a logical block not the same as a NAND page?
2. Why can a 512-byte logical block device still prefer 4096-byte writes?
3. What does a flush promise that a write does not necessarily promise?
4. Why is unmap not the same operation as write zeroes?
5. Where does SPDK store the logical block size for a bdev?
6. What can happen when a user buffer violates required_alignment?
7. Why should a benchmark write a device before measuring reads?

References

• Local source: include/spdk/bdev_module.h, especially struct spdk_bdev.
• Local source: include/spdk/bdev.h, especially bdev geometry, metadata, and I/O support getters.
• Local source: lib/bdev/bdev.c, especially request entry points such as read, write, flush, unmap, and write zeroes.
• Local source: module/bdev/malloc/bdev_malloc.c, for a simple module that populates and serves a bdev.
• Local source: module/bdev/nvme/bdev_nvme.c, for a hardware-backed module that maps NVMe namespace/controller data into a bdev.
• SPDK local docs: doc/bdev.md and doc/bdev_module.md.
• SPDK Block Device User Guide: https://spdk.io/doc/bdev.html
• SPDK Block Device Layer Programming Guide: https://spdk.io/doc/bdev_pg.html
• SPDK Writing a Custom Block Device Module: https://spdk.io/doc/bdev_module.html
• SPDK bdev.h generated API reference: https://spdk.io/doc/bdev_8h.html
• NVM Express specifications landing page: https://nvmexpress.org/specifications/