SPDK From First Principles

SPDK deep learning path

Chapter 19: Blobstore

Blobstore is SPDK's small storage engine for "large named chunks of blocks" called blobs. If a normal block device gives you one flat address range, blobstore turns that range...

Source: drafts/lvol-raid/01-blobstore.md

Beginner Mental Model

Blobstore is SPDK's small storage engine for "large named chunks of blocks" called blobs. If a normal block device gives you one flat address range, blobstore turns that range into a collection of independently sized objects. Each blob has metadata, a cluster map, attributes, and optional relationships to snapshots or parents. Blobstore is not a filesystem. It does not have directories, permissions, pathnames, page cache, journaling in the filesystem sense, or POSIX file semantics. It is closer to a lightweight allocator and metadata layer for storage systems that already know they want asynchronous block IO.

The simplest picture is:

base bdev
  |
  | adapted by module/blob/bdev/blob_bdev.c into struct spdk_bs_dev
  v
blobstore
  |
  +-- super block
  +-- used-cluster bitmap
  +-- used-blobid bitmap
  +-- metadata pages
  +-- blob A -> cluster map -> base device LBAs
  +-- blob B -> cluster map -> base device LBAs

A blob is a logical array of IO units. Blobstore maps those IO units to physical clusters. A cluster is the allocation unit. Thin blobs may have logical clusters that are not physically allocated yet. Reads from unallocated clusters resolve through a backing device: a zeroes device for ordinary thin blobs, a snapshot blob for clones, or an external snapshot device for esnap clones.

The important beginner rule is: blobstore IO is asynchronous, but blob metadata ownership is strict. The API documentation in include/spdk/blob.h says functions that do not take an IO channel may be called only on the thread that called spdk_bs_init() or spdk_bs_load(). Functions beginning with spdk_blob_io take a channel created with spdk_bs_alloc_io_channel() and must be called on the thread that created that channel.

flowchart TB bdev[Base bdev] --> bsdev[spdk_bs_dev adapter] bsdev --> bs[Blobstore] bs --> alloc[Cluster allocator] bs --> md[Metadata pages] bs --> blobA[Blob or lvol data] blobA --> cmap[Cluster map] cmap --> baseLba[Base bdev LBAs] clone[Clone blob] --> parent[Snapshot or external snapshot] clone --> zeroes[Zeroes backing for unallocated clusters]

Why This Matters For diskengine/excloud

Cloud volume systems often need fast create, clone, snapshot, resize, and delete without blocking the IO path. Blobstore is the layer that makes those operations cheap enough to compose. lvol uses blobstore to provide logical volumes. vbdev_lvol exposes those lvols as SPDK bdevs, and RAID may sit under or over those bdevs depending on design. If blobstore semantics are misunderstood, higher layers tend to make wrong assumptions:

  • Thin create is not free forever; first writes allocate clusters and can fail with -ENOSPC.
  • Snapshot create is not just "copy metadata"; it freezes IO, swaps cluster maps, sets parent relationships, and persists several pieces of metadata.
  • Delete of a snapshot with clones may rewrite clone metadata and parent relationships.
  • Resize is asynchronous and serialized by locked_operation_in_progress.
  • An lvol metadata object may exist before every external snapshot parent is available, but a degraded esnap lvol does not get a registered lvol bdev. Degraded lvol behavior comes from blobstore and lvol cooperation.

Core Objects

Public API objects:

  • struct spdk_bs_dev in include/spdk/blob.h: the device interface blobstore uses. It has function pointers for read, write, readv, writev, unmap, write_zeroes, flush, copy, is_zeroes, translate_lba, and is_degraded.
  • struct spdk_blob_store in include/spdk/blob.h: opaque publicly, implemented privately in lib/blob/blobstore.h.
  • struct spdk_blob in include/spdk/blob.h: opaque publicly, implemented privately in lib/blob/blobstore.h.
  • spdk_blob_id: a uint64_t identifier.

Private implementation anchors:

  • lib/blob/blobstore.h:struct spdk_blob_store
  • lib/blob/blobstore.h:struct spdk_blob
  • lib/blob/blobstore.h:struct spdk_bs_channel
  • lib/blob/blobstore.h:struct spdk_bs_super_block
  • lib/blob/blobstore.h:struct spdk_blob_md_page
  • lib/blob/blobstore.h:struct spdk_blob_md_descriptor_extent_rle
  • lib/blob/blobstore.h:struct spdk_blob_md_descriptor_extent_table
  • lib/blob/blobstore.h:struct spdk_blob_md_descriptor_extent_page

The private struct spdk_blob_store contains the backing struct spdk_bs_dev, cluster sizing, metadata ranges, bit pools/arrays for allocation, trees/lists of open blobs and snapshots, and external snapshot callbacks. The private struct spdk_blob contains active and clean metadata, the blob ID, parent ID, state, cluster maps, extent pages, xattrs, open ref count, pending persists, and the backing bs_dev used for reads not satisfied by its own allocated clusters.

The reason the public API hides these structures is not only ABI hygiene. Blobstore depends on a small set of invariants that callers must not mutate directly: a blob has a "clean" metadata image that matches disk, an "active" metadata image that is being edited in RAM, and blobstore-wide allocation bitmaps that determine whether a cluster or metadata page may be reused. This excerpt is the shortest useful view of those invariants:

/* lib/blob/blobstore.h */
struct spdk_blob {
	struct spdk_blob_store *bs;

	uint32_t	open_ref;

	spdk_blob_id	id;
	spdk_blob_id	parent_id;

	enum spdk_blob_state		state;

	/* Two copies of the mutable data. One is a version
	 * that matches the last known data on disk (clean).
	 * The other (active) is the current data. Syncing
	 * a blob makes the clean match the active.
	 */
	struct spdk_blob_mut_data	clean;
	struct spdk_blob_mut_data	active;

	bool		invalid;
	bool		data_ro;
	bool		md_ro;

	struct spdk_bs_dev *back_bs_dev;
	struct spdk_xattr_tailq xattrs;
	struct spdk_xattr_tailq xattrs_internal;

	uint32_t frozen_refcnt;
	bool locked_operation_in_progress;
};

The parent_id and back_bs_dev fields are what make thin blobs, clones, and external snapshot clones look like one address space to the reader. The frozen_refcnt and locked_operation_in_progress fields are what let metadata operations temporarily stop conflicting IO without turning the whole library into a lock-heavy filesystem.

Blobstore metadata is descriptor based. A blob metadata page contains descriptors for xattrs, flags, and extent information. Extents can be stored directly as RLE entries or indirectly through an extent table that points to extent pages. That extra indirection exists because large thin blobs would otherwise need to rewrite too much metadata every time a far-away cluster is allocated.

/* lib/blob/blobstore.h */
#define SPDK_MD_DESCRIPTOR_TYPE_EXTENT_RLE 1
#define SPDK_MD_DESCRIPTOR_TYPE_EXTENT_TABLE 5
#define SPDK_MD_DESCRIPTOR_TYPE_EXTENT_PAGE 6

struct spdk_blob_md_descriptor_extent_rle {
	uint8_t		type;
	uint32_t	length;

	struct {
		uint32_t	cluster_idx;
		uint32_t	length; /* In units of clusters */
	} extents[0];
};

struct spdk_blob_md_descriptor_extent_table {
	uint8_t		type;
	uint32_t	length;

	/* Number of data clusters in the blob */
	uint64_t	num_clusters;

	struct {
		uint32_t	page_idx;
		uint32_t	num_pages; /* In units of pages */
	} extent_page[0];
};

The practical consequence is that "the cluster map" is not always one flat array on disk. In memory, readers often reason about it as an array of cluster mappings. On disk, blobstore can choose a compact descriptor format so sparse blobs and large blobs do not pay the same metadata write cost.

Units: Pages, Clusters, IO Units, LBAs

Blobstore uses several units at once:

  • bdev block/LBA: the unit the backing device exposes.
  • IO unit: blobstore's IO address unit. It is derived from device block length.
  • metadata page: fixed blobstore metadata page unit.
  • cluster: allocation unit for blob data.
  • blob logical cluster: index into a blob's cluster map.

Source anchors:

  • lib/blob/blobstore.h:bs_byte_to_lba
  • lib/blob/blobstore.h:bs_dev_byte_to_lba
  • lib/blob/blobstore.h:bs_page_to_lba
  • lib/blob/blobstore.h:bs_md_page_to_lba
  • lib/blob/blobstore.h:bs_cluster_to_io_unit
  • lib/blob/blobstore.h:bs_io_unit_to_cluster
  • lib/blob/blobstore.h:bs_cluster_to_lba
  • lib/blob/blobstore.h:bs_lba_to_cluster
  • lib/blob/blobstore.h:bs_blob_io_unit_to_lba

Beginner misconception to kill: a blob's "size" is not necessarily allocated space. spdk_blob_get_num_clusters() returns logical size in clusters. spdk_blob_get_num_allocated_clusters() returns actual allocated clusters. In a thin blob, these may differ. In a snapshot/clone chain, a read may walk from child to parent rather than consuming an allocated cluster in the child.

The conversion helpers are deliberately simple: they convert magnitudes between units, not ownership. A cluster number can be converted to a device LBA, but that does not mean the cluster belongs to the blob doing the IO. Ownership is checked through the blob's cluster map first.

/* lib/blob/blobstore.h */
static inline uint64_t
bs_cluster_to_io_unit(struct spdk_blob_store *bs, uint32_t cluster)
{
	return (uint64_t)cluster * bs->io_units_per_cluster;
}

static inline uint32_t
bs_io_unit_to_cluster(struct spdk_blob_store *bs, uint64_t io_unit)
{
	assert(io_unit % bs->io_units_per_cluster == 0);

	return io_unit / bs->io_units_per_cluster;
}

static inline uint64_t
bs_cluster_to_lba(struct spdk_blob_store *bs, uint32_t cluster)
{
	assert(bs->cluster_sz / bs->dev->blocklen > 0);

	return (uint64_t)cluster * (bs->cluster_sz / bs->dev->blocklen);
}

This is why the chapter keeps saying "logical cluster" versus "physical cluster." A logical cluster is an index in a blob. A physical cluster is a region on the blobstore device. Thin provisioning is mostly the act of delaying the logical-to-physical mapping until a write forces one to exist.

Device Adaptation: bdev To Blobstore

Blobstore does not directly speak struct spdk_bdev. It speaks struct spdk_bs_dev. The adapter lives in:

  • include/spdk/blob_bdev.h
  • module/blob/bdev/blob_bdev.c

The lvol bdev module uses spdk_bdev_create_bs_dev_ext() to wrap a bdev as a blobstore device. In the lvol examine path, see module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine, which calls spdk_bdev_create_bs_dev_ext(bdev->name, ...) before calling spdk_lvs_load_ext().

The reverse direction also exists: a blob can be exposed as a blobstore device. That is used for snapshot backing:

  • lib/blob/blob_bs_dev.c:bs_create_blob_bs_dev
  • lib/blob/blob_bs_dev.c:blob_bs_dev_read
  • lib/blob/blob_bs_dev.c:blob_bs_is_zeroes
  • lib/blob/blob_bs_dev.c:blob_bs_translate_lba
  • lib/blob/blob_bs_dev.c:blob_bs_is_degraded

Prose diagram:

clone blob read
  |
  | if child cluster is allocated:
  |   read child cluster from base blobstore device
  |
  | if child cluster is unallocated:
  |   read from back_bs_dev
  |
  +-- back_bs_dev may be:
        zeroes device                 -> ordinary thin empty data
        blob-backed bs_dev            -> internal snapshot parent
        external snapshot bs_dev      -> bdev outside this lvolstore

Loading And Initializing A Blobstore

Public entry points:

  • include/spdk/blob.h:spdk_bs_init
  • include/spdk/blob.h:spdk_bs_load
  • include/spdk/blob.h:spdk_bs_grow
  • include/spdk/blob.h:spdk_bs_grow_live
  • include/spdk/blob.h:spdk_bs_destroy
  • include/spdk/blob.h:spdk_bs_unload

Implementation anchors:

  • lib/blob/blobstore.c:spdk_bs_init
  • lib/blob/blobstore.c:spdk_bs_load
  • lib/blob/blobstore.c:bs_alloc
  • lib/blob/blobstore.c:bs_parse_super
  • lib/blob/blobstore.c:bs_recover
  • lib/blob/blobstore.c:bs_load_replay_md
  • lib/blob/blobstore.c:bs_load_replay_md_parse_page
  • lib/blob/blobstore.c:bs_load_write_used_md
  • lib/blob/blobstore.c:spdk_bs_grow
  • lib/blob/blobstore.c:spdk_bs_grow_live

Initialization creates new metadata. Loading reads and validates existing metadata. The load path allocates a struct spdk_bs_load_ctx, reads the superblock, reconstructs in-memory state from metadata pages, recovers if necessary, rebuilds allocation masks, and then completes with a blobstore handle.

The private load context in lib/blob/blobstore.c:struct spdk_bs_load_ctx is worth reading because it shows what blobstore believes it needs to reconstruct the world:

  • super: superblock buffer.
  • mask: used-cluster or used-blobid mask.
  • page: current metadata page.
  • extent_pages: loaded extent pages.
  • used_clusters: load-time allocation view.
  • force_recover: option for recovery behavior.
  • iterator callback fields for walking blobs.

Beginner misconception to kill: load is not merely "read one superblock and return." It may replay metadata page chains and repair masks. Dirty shutdown changes the path.

Official SPDK docs call out two metadata layers: per-blob metadata can be explicitly synchronized, while global blobstore metadata is normally made consistent on unload and rebuilt on the next load after an unclean shutdown. The implementation reflects that distinction by marking the super block dirty before mutating blob metadata.

/* lib/blob/blobstore.c */
static void
bs_mark_dirty(spdk_bs_sequence_t *seq, struct spdk_blob_store *bs,
	      spdk_bs_sequence_cpl cb_fn, void *cb_arg)
{
	struct spdk_bs_mark_dirty *ctx;

	/* Blobstore is already marked dirty */
	if (bs->clean == 0) {
		cb_fn(seq, cb_arg, 0);
		return;
	}

	ctx = calloc(1, sizeof(*ctx));
	if (!ctx) {
		cb_fn(seq, cb_arg, -ENOMEM);
		return;
	}
	ctx->bs = bs;
	ctx->cb_fn = cb_fn;
	ctx->cb_arg = cb_arg;

	ctx->super = spdk_zmalloc(sizeof(*ctx->super), 0x1000, NULL,
				  SPDK_ENV_NUMA_ID_ANY, SPDK_MALLOC_DMA);
	if (!ctx->super) {
		free(ctx);
		cb_fn(seq, cb_arg, -ENOMEM);
		return;
	}

	bs_sequence_read_dev(seq, ctx->super, bs_page_to_lba(bs, 0),
			     bs_byte_to_lba(bs, sizeof(*ctx->super)),
			     bs_mark_dirty_write, ctx);
}

That dirty bit is a recovery affordance. It does not mean every user write is unsafe; it means blobstore should not trust persisted global allocation masks as authoritative until it has replayed per-blob metadata.

Cluster Allocation

Source anchors:

  • lib/blob/blobstore.c:bs_claim_cluster
  • lib/blob/blobstore.c:bs_release_cluster
  • lib/blob/blobstore.c:blob_insert_cluster_on_md_thread
  • lib/blob/blobstore.c:blob_insert_cluster_msg
  • lib/blob/blobstore.c:blob_free_cluster_on_md_thread
  • lib/blob/blobstore.c:blob_free_cluster_msg
  • lib/blob/blobstore.c:blob_write_extent_page

Cluster allocation is the core of thin provisioning. When a thin blob receives a write to an unallocated logical cluster, blobstore claims a physical cluster from the blobstore pool, updates the blob's active metadata, persists the relevant metadata page or extent page, and then performs IO. Releasing a cluster clears the mapping, updates allocation counts, writes metadata, and returns the cluster to the free pool.

For clones, the first write to a cluster is copy-on-write, not just allocate-and-write. Blobstore may need the old contents from the backing snapshot so that a partial-cluster overwrite preserves the bytes outside the caller's write range. The allocation path therefore checks whether the backing range is valid, whether it can use device copy, whether the backing range is known zero, and only then allocates a new physical cluster.

/* lib/blob/blobstore.c */
is_valid_range = blob->back_bs_dev->is_range_valid(blob->back_bs_dev,
		 bs_dev_io_unit_to_lba(blob, blob->back_bs_dev, cluster_start_io_unit),
		 bs_dev_byte_to_lba(blob->back_bs_dev, blob->bs->cluster_sz));

can_copy = is_valid_range && blob_can_copy(blob, cluster_start_io_unit, &copy_src_lba);

is_zeroes = is_valid_range && blob->back_bs_dev->is_zeroes(blob->back_bs_dev,
		bs_dev_io_unit_to_lba(blob, blob->back_bs_dev, cluster_start_io_unit),
		bs_dev_byte_to_lba(blob->back_bs_dev, blob->bs->cluster_sz));
if (blob->parent_id != SPDK_BLOBID_INVALID && !is_zeroes && !can_copy) {
	ctx->buf = spdk_malloc(blob->bs->cluster_sz, blob->back_bs_dev->blocklen,
			       NULL, SPDK_ENV_NUMA_ID_ANY, SPDK_MALLOC_DMA);
	if (!ctx->buf) {
		SPDK_ERRLOG("DMA allocation for cluster of size = %" PRIu32 " failed.\n",
			    blob->bs->cluster_sz);
		free(ctx);
		bs_user_op_abort(op, -ENOMEM);
		return;
	}
}

spdk_spin_lock(&blob->bs->used_lock);
rc = bs_allocate_cluster(blob, cluster_number, &ctx->new_cluster, &ctx->new_extent_page,
			 false);
spdk_spin_unlock(&blob->bs->used_lock);

The spinlock here is intentionally narrow: it protects allocation bookkeeping, not the whole IO. After a cluster is reserved, the metadata-thread path publishes the new mapping. If another operation wins the race and inserts a mapping first, the loser clears and frees its temporary cluster instead of exposing stale data.

The IO path has to be careful at cluster boundaries. A single user IO may cover multiple blob clusters and may need to split. Relevant source anchors:

  • lib/blob/blobstore.c:blob_request_submit_op
  • lib/blob/blobstore.c:blob_request_submit_op_single
  • lib/blob/blobstore.c:blob_request_submit_op_split
  • lib/blob/blobstore.c:blob_request_submit_op_split_next
  • lib/blob/blobstore.c:blob_request_submit_rw_iov

Prose diagram for a write to an unallocated cluster:

spdk_blob_io_write()
  -> blob_request_submit_op()
     -> determine cluster coverage
     -> if one cluster: blob_request_submit_op_single()
     -> if crosses boundary: blob_request_submit_op_split()
     -> if target cluster missing and write needs storage:
          allocate cluster on metadata thread
          update cluster map / extent page
          submit write to bs_dev
     -> completion callback returns bserrno

IO API Surface

Public IO API:

  • include/spdk/blob.h:spdk_blob_io_read
  • include/spdk/blob.h:spdk_blob_io_write
  • include/spdk/blob.h:spdk_blob_io_readv
  • include/spdk/blob.h:spdk_blob_io_writev
  • include/spdk/blob.h:spdk_blob_io_readv_ext
  • include/spdk/blob.h:spdk_blob_io_writev_ext
  • include/spdk/blob.h:spdk_blob_io_unmap
  • include/spdk/blob.h:spdk_blob_io_write_zeroes

Implementation:

  • lib/blob/blobstore.c:spdk_blob_io_read
  • lib/blob/blobstore.c:spdk_blob_io_write
  • lib/blob/blobstore.c:spdk_blob_io_unmap
  • lib/blob/blobstore.c:spdk_blob_io_write_zeroes
  • lib/blob/blobstore.c:spdk_blob_io_readv_ext
  • lib/blob/blobstore.c:spdk_blob_io_writev_ext

Request helpers:

  • lib/blob/request.c:bs_sequence_start_bs
  • lib/blob/request.c:bs_sequence_start_blob
  • lib/blob/request.c:bs_batch_open
  • lib/blob/request.c:bs_user_op_alloc
  • lib/blob/request.c:bs_user_op_execute

Beginner rule: callback completion is the only point at which the caller knows the operation has completed. Returning from spdk_blob_io_write() only means the operation was accepted into the async path or failed immediately by invoking/completing through the callback path depending on the specific helper.

At IO dispatch time, blobstore first asks whether the target range is allocated in the blob itself. Allocated reads and writes go to the blobstore device. Unallocated reads go to the backing device. Unallocated writes allocate/copy a cluster before the user write is submitted. That is the central runtime rule for thin provisioning, internal snapshots, and external snapshots.

/* lib/blob/blobstore.c */
if (is_allocated) {
	/* Read from the blob */
	bs_batch_read_dev(batch, payload, lba, lba_count);
} else {
	/* Read from the backing block device */
	bs_batch_read_bs_dev(batch, blob->back_bs_dev, payload, lba, lba_count);
}

if (is_allocated) {
	/* Write to the blob */
	if (op_type == SPDK_BLOB_WRITE) {
		bs_batch_write_dev(batch, payload, lba, lba_count);
	} else {
		bs_batch_write_zeroes_dev(batch, lba, lba_count);
	}
} else {
	/* Queue this operation and allocate the cluster */
	spdk_bs_user_op_t *op;

	op = bs_user_op_alloc(_ch, &cpl, op_type, blob, payload, 0, offset, length);
	if (!op) {
		cb_fn(cb_arg, -ENOMEM);
		return;
	}

	bs_allocate_and_copy_cluster(blob, _ch, offset, op);
}

The source uses batches for groups of operations that can be issued together and sequences for ordered chains such as "read backing cluster, write new cluster, persist metadata, then complete." This is why blobstore can preserve ordering only where it explicitly builds a sequence; callers should not infer broader IO ordering from asynchronous submission.

Snapshots And Clones

Public API:

  • include/spdk/blob.h:spdk_bs_create_snapshot
  • include/spdk/blob.h:spdk_bs_create_clone
  • include/spdk/blob.h:spdk_blob_is_snapshot
  • include/spdk/blob.h:spdk_blob_is_clone
  • include/spdk/blob.h:spdk_blob_get_parent_snapshot
  • include/spdk/blob.h:spdk_blob_get_clones

Implementation anchors:

  • lib/blob/blobstore.c:spdk_bs_create_snapshot
  • lib/blob/blobstore.c:spdk_bs_create_clone
  • lib/blob/blobstore.c:bs_snapshot_swap_cluster_maps
  • lib/blob/blobstore.c:bs_snapshot_newblob_sync_cpl
  • lib/blob/blobstore.c:bs_snapshot_origblob_sync_cpl
  • lib/blob/blobstore.c:blob_freeze_io
  • lib/blob/blobstore.c:blob_unfreeze_io
  • lib/blob/blobstore.c:bs_blob_list_add
  • lib/blob/blobstore.c:spdk_blob_is_snapshot
  • lib/blob/blobstore.c:spdk_blob_is_clone
  • lib/blob/blobstore.c:spdk_blob_get_parent_snapshot
  • lib/blob/blobstore.c:spdk_blob_get_clones

A snapshot operation converts a mutable blob into a clone of a new read-only snapshot. That sentence is easy to miss. After snapshot creation:

  • The new snapshot blob is read-only.
  • The original blob remains writable but becomes thin-provisioned.
  • The original blob points at the snapshot as backing storage for clusters it has not overwritten since the snapshot.
  • Metadata xattrs record the relationship.
  • IO is frozen while the relationship and cluster maps are changed.

Prose diagram:

Before snapshot:

volume blob V
  cluster map: [A, B, C, D]
  parent: none or previous parent

After snapshot named S:

snapshot blob S, read-only
  cluster map: [A, B, C, D]

volume blob V, writable thin clone
  cluster map: [0, 0, 0, 0]
  parent/backing: S

New write to V cluster 2:
  V allocates E
  V cluster map: [0, 0, E, 0]
  reads cluster 2 from V
  reads clusters 0,1,3 from S

A clone operation creates a new writable thin blob backed by a read-only snapshot. Source readers should compare spdk_bs_create_snapshot and spdk_bs_create_clone. Snapshot creation modifies the original blob; clone creation leaves the snapshot as parent and creates a new child.

The source makes snapshot creation feel less magical: the new snapshot starts as an empty thin blob with the same logical size, IO on the original is frozen, the active cluster maps are swapped, and then metadata is synchronized in stages. The swap is what transfers current data ownership to the new read-only snapshot without copying all data clusters.

/* lib/blob/blobstore.c */
static void
bs_snapshot_swap_cluster_maps(struct spdk_blob *blob1, struct spdk_blob *blob2)
{
	uint64_t *cluster_temp;
	uint64_t num_allocated_clusters_temp;
	uint32_t *extent_page_temp;

	cluster_temp = blob1->active.clusters;
	blob1->active.clusters = blob2->active.clusters;
	blob2->active.clusters = cluster_temp;

	num_allocated_clusters_temp = blob1->active.num_allocated_clusters;
	blob1->active.num_allocated_clusters = blob2->active.num_allocated_clusters;
	blob2->active.num_allocated_clusters = num_allocated_clusters_temp;

	extent_page_temp = blob1->active.extent_pages;
	blob1->active.extent_pages = blob2->active.extent_pages;
	blob2->active.extent_pages = extent_page_temp;
}

The callback sequence around that swap is also the reason snapshot creation is serialized. The original blob sets locked_operation_in_progress, creates a new thin blob, freezes IO, swaps maps, records parent xattrs, creates a blob-backed back_bs_dev, syncs the original as a clone, then marks the new blob read-only and syncs it. Any error in the middle has to put the maps and xattrs back into a recoverable shape.

/* lib/blob/blobstore.c */
ctx->new.blob = newblob;
assert(spdk_blob_is_thin_provisioned(newblob));
assert(spdk_mem_all_zero(newblob->active.clusters,
			 newblob->active.num_clusters * sizeof(*newblob->active.clusters)));
assert(spdk_mem_all_zero(newblob->active.extent_pages,
			 newblob->active.num_extent_pages * sizeof(*newblob->active.extent_pages)));

blob_freeze_io(origblob, bs_snapshot_freeze_cpl, ctx);

External Snapshots

External snapshots allow a blob to use a non-blobstore device as its parent. lvol exposes this through bdev_lvol_clone_bdev and parent-setting RPCs, but the blobstore core owns the metadata relationship.

Public blob anchors:

  • include/spdk/blob.h:spdk_bs_blob_set_external_parent
  • include/spdk/blob.h:spdk_blob_get_esnap_id
  • include/spdk/blob.h:spdk_blob_is_esnap_clone
  • include/spdk/blob.h:spdk_blob_set_esnap_bs_dev
  • include/spdk/blob.h:spdk_blob_get_esnap_bs_dev
  • include/spdk/blob.h:spdk_blob_is_degraded

Implementation anchors:

  • lib/blob/blobstore.c:spdk_bs_blob_set_external_parent
  • lib/blob/blobstore.c:spdk_blob_set_esnap_bs_dev
  • lib/blob/blobstore.c:spdk_blob_get_esnap_bs_dev
  • lib/blob/blobstore.c:spdk_blob_is_degraded
  • lib/blob/blobstore.c:blob_esnap_destroy_bs_dev_channels
  • lib/blob/blobstore.c:blob_esnap_destroy_bs_channel
  • lib/blob/blobstore.c:blob_esnap_get_io_channel

The public callback type spdk_bs_esnap_dev_create in include/spdk/blob.h is central. When blobstore loads an esnap clone, it calls the consumer-provided callback to open the external snapshot as a struct spdk_bs_dev. If the external snapshot is unavailable, the consumer can arrange degraded behavior.

Beginner misconception to kill: an external snapshot is not copied into the lvolstore by default. The clone depends on the external parent until it is inflated, decoupled, shallow-copied elsewhere, or reparented.

On load, blobstore does not know how to open an arbitrary external bdev by itself. It asks its consumer through esnap_bs_dev_create; lvol supplies that callback when loading the lvolstore. The callback can return a real struct spdk_bs_dev, or return success with NULL if the consumer intentionally leaves the external snapshot unopened for now.

/* lib/blob/blobstore.c */
if (bs->esnap_bs_dev_create == NULL) {
	SPDK_NOTICELOG("blob 0x%" PRIx64 " is an esnap clone but the blobstore was opened "
		       "without support for esnap clones\n", blob->id);
	return -ENOTSUP;
}
assert(blob->back_bs_dev == NULL);

rc = blob_get_xattr_value(blob, BLOB_EXTERNAL_SNAPSHOT_ID, &esnap_id, &id_len, true);
if (rc != 0) {
	SPDK_ERRLOG("blob 0x%" PRIx64 " is an esnap clone but has no esnap ID\n", blob->id);
	return -EINVAL;
}

rc = bs->esnap_bs_dev_create(bs->esnap_ctx, blob_ctx, blob, esnap_id, (uint32_t)id_len,
			     &bs_dev);
if (rc != 0) {
	SPDK_DEBUGLOG(blob_esnap, "blob 0x%" PRIx64 ": failed to load back_bs_dev "
		      "with error %d\n", blob->id, rc);
	return rc;
}

blob->back_bs_dev = bs_dev;
blob->parent_id = SPDK_BLOBID_EXTERNAL_SNAPSHOT;

That design keeps blobstore generic. Blobstore stores the relationship and drives IO through spdk_bs_dev; lvol knows how to find and claim the external bdev that relationship names.

Resize And Grow

Blob resize:

  • include/spdk/blob.h:spdk_blob_resize
  • lib/blob/blobstore.c:spdk_blob_resize
  • lib/blob/blobstore.c:blob_resize
  • lib/blob/blobstore.c:bs_resize_freeze_cpl
  • lib/blob/blobstore.c:bs_resize_unfreeze_cpl

Blobstore grow:

  • include/spdk/blob.h:spdk_bs_grow
  • include/spdk/blob.h:spdk_bs_grow_live
  • lib/blob/blobstore.c:spdk_bs_grow
  • lib/blob/blobstore.c:spdk_bs_grow_live
  • lib/blob/blobstore.c:bs_load_try_to_grow
  • lib/blob/blobstore.c:bs_grow_live_load_super_cpl
  • lib/blob/blobstore.c:bs_grow_live_super_write_cpl

Resize is serialized with locked_operation_in_progress. If the blob is metadata-read-only (md_ro), resize fails with -EPERM. If the requested cluster count is unchanged, resize completes successfully without work. If another locked operation is active, resize fails with -EBUSY.

Important edge cases:

  • Growing logical size may not allocate clusters immediately for a thin blob.
  • Shrinking must update metadata so clusters beyond the new end are not considered part of the blob.
  • A clone can grow beyond its parent; backing reads are valid only where the parent has a range. See lib/blob/blob_bs_dev.c:blob_bs_is_range_valid, which explicitly accounts for a backing blob having fewer clusters than the child after expansion.
  • Blobstore grow is about the underlying device becoming larger, not an individual blob becoming larger.

Delete Semantics

Public API:

  • include/spdk/blob.h:spdk_bs_delete_blob

Implementation anchors:

  • lib/blob/blobstore.c:spdk_bs_delete_blob
  • lib/blob/blobstore.c:bs_is_blob_deletable
  • lib/blob/blobstore.c:update_clone_on_snapshot_deletion
  • lib/blob/blobstore.c:delete_snapshot_open_clone_cb
  • lib/blob/blobstore.c:delete_snapshot_freeze_io_cb
  • lib/blob/blobstore.c:delete_snapshot_sync_snapshot_xattr_cpl
  • lib/blob/blobstore.c:delete_snapshot_update_extent_pages
  • lib/blob/blobstore.c:delete_snapshot_sync_clone_cpl
  • lib/blob/blobstore.c:delete_snapshot_sync_snapshot_cpl
  • lib/blob/blobstore.c:bs_delete_blob_finish

Blob deletion has special rules for snapshots:

  • A normal blob with no clones can be removed.
  • A snapshot with more than one clone cannot be removed directly.
  • A snapshot with one clone may be removed by updating the clone to inherit the snapshot's parent/data relationship.
  • An open snapshot cannot be removed except for the internal reference patterns allowed by the deletion logic.
  • Deleting a snapshot may require freezing the clone, copying cluster mappings, changing parent xattrs, handling external snapshot references, and persisting metadata in a power-failure-aware order.

Prose diagram for deleting a snapshot with one clone:

Before:

Parent P -> Snapshot S -> Clone C

Delete S:
  freeze C
  mark S pending removal
  copy needed S cluster mappings into C
  change C parent from S to P, zeroes, or external snapshot depending on S
  sync C metadata
  sync S metadata / remove S from snapshot lists
  unfreeze C

After:

Parent P -> Clone C

Edge Cases And Failure Modes

ENOSPC On Thin Writes

Thin provisioning means creation does not reserve all clusters. First write to a previously unallocated cluster may call bs_claim_cluster and fail if the free cluster pool is exhausted. The visible failure may happen at lvol bdev write time even though lvol creation succeeded earlier.

EBUSY On Concurrent Metadata Operations

Snapshot, resize, parent changes, and some delete paths use locked operations. lib/blob/blobstore.c:spdk_blob_resize checks blob->locked_operation_in_progress and returns -EBUSY if another locked operation is active.

Read-Only Metadata

Snapshots are read-only. lib/blob/blobstore.c:spdk_blob_resize fails with -EPERM when blob->md_ro is true. Snapshot deletion temporarily overrides metadata read-only flags in controlled cleanup paths; that does not mean callers may mutate snapshot metadata freely.

Dirty Load And Recovery

Crash or power loss during metadata updates can leave dirty metadata. The load path has recovery-specific logic:

  • lib/blob/blobstore.c:bs_recover
  • lib/blob/blobstore.c:bs_load_replay_md
  • lib/blob/blobstore.c:bs_load_write_used_md
  • lib/blob/blobstore.c:bs_load_cur_md_page_valid

Blobstore metadata update ordering is designed so load can reason about partially completed operations. For example, snapshot delete uses SNAPSHOT_PENDING_REMOVAL to make recovery possible.

The recovery iterator looks for internal xattrs left behind by interrupted snapshot creation or deletion. It then opens the referenced clone to decide whether to keep the snapshot or remove the corrupted intermediate blob. This is a metadata repair workflow, not user data reconstruction from parity or journaling.

/* lib/blob/blobstore.c */
if (bserrno == 0) {
	/* Examine blob if it is corrupted after power failure. Fix
	 * the ones that can be fixed and remove any other corrupted
	 * ones. If it is not corrupted just process it */
	rc = blob_get_xattr_value(blob, SNAPSHOT_PENDING_REMOVAL, &value, &len, true);
	if (rc != 0) {
		rc = blob_get_xattr_value(blob, SNAPSHOT_IN_PROGRESS, &value, &len, true);
		if (rc != 0) {
			/* Not corrupted - process it and continue with iterating through blobs */
			if (ctx->iter_cb_fn) {
				ctx->iter_cb_fn(ctx->iter_cb_arg, blob, 0);
			}
			bs_blob_list_add(blob);
			spdk_bs_iter_next(ctx->bs, blob, bs_load_iter, ctx);
			return;
		}
	}

	assert(len == sizeof(spdk_blob_id));

	ctx->blob = blob;

	/* Open clone to check if we are able to fix this blob or should we remove it */
	spdk_bs_open_blob(ctx->bs, *(spdk_blob_id *)value, bs_examine_clone, ctx);
	return;
} else if (bserrno == -ENOENT) {
	bserrno = 0;
} else {
	SPDK_ERRLOG("Error in iterating blobs\n");
}

The earlier superblock check chooses this slower path when the clean flag or persisted masks cannot be trusted:

/* lib/blob/blobstore.c */
if (ctx->super->used_blobid_mask_len == 0 || ctx->super->clean == 0 || ctx->force_recover) {
	bs_recover(ctx);
} else {
	bs_load_used_blobids(ctx);
}

External Snapshot Missing

If an esnap clone is loaded but its external parent cannot be opened, the blob/lvol may be degraded. The blobstore public predicate is include/spdk/blob.h:spdk_blob_is_degraded, implemented by lib/blob/blobstore.c:spdk_blob_is_degraded.

IO Channel Lifetime

Blobstore IO APIs require a channel. External snapshot channels may need to be destroyed on all relevant threads. See:

  • lib/blob/blobstore.c:blob_esnap_destroy_bs_dev_channels
  • lib/blob/blobstore.c:blob_esnap_destroy_bs_channel
  • lib/blob/blobstore.c:blob_esnap_get_io_channel

Misconceptions To Kill

  • "Blobstore is a filesystem." It is not. It is an async object allocator over a block device.
  • "Snapshot copies all data." It does not. It changes metadata relationships and shares clusters until overwritten.
  • "Thin provisioning means writes always succeed later." No. Allocation can fail at write time.
  • "Resize allocates data." Logical resize and physical allocation are separate for thin blobs.
  • "A clone owns all bytes it can read." A clone may read from its own clusters, a snapshot, an external snapshot, or zeroes.
  • "Delete only removes a record." Snapshot delete can rewrite clone metadata and parent relationships.
  • "Load is simple." Load may replay metadata, recover dirty state, validate CRC-like metadata, and rebuild allocation masks.

Source Reading Exercise

Read these in order and write down every callback transition:

  1. lib/blob/blobstore.c:spdk_bs_create_snapshot
  2. lib/blob/blobstore.c:bs_snapshot_newblob_sync_cpl
  3. lib/blob/blobstore.c:bs_snapshot_origblob_sync_cpl
  4. lib/blob/blobstore.c:blob_freeze_io
  5. lib/blob/blobstore.c:blob_unfreeze_io

Questions:

  • At what point is the new snapshot marked read-only?
  • Why does snapshot creation swap cluster maps?
  • What must be undone if syncing the new snapshot metadata fails?
  • What prevents concurrent writes while the cluster maps are being rearranged?

Operational Lab

Use test/lvol/snapshot_clone.sh as the high-level lab and test/unit/lib/blob/blob.c/blob_ut.c as the source-level lab.

Suggested lab flow:

  1. Create a malloc bdev.
  2. Create an lvolstore on it.
  3. Create an lvol.
  4. Write recognizable data to different offsets.
  5. Create a snapshot.
  6. Write new data to the original lvol.
  7. Create a clone from the snapshot.
  8. Compare reads from original, snapshot, and clone.
  9. Delete a snapshot with zero, one, and more than one clone, and record expected errors.

If running a full SPDK target is unavailable, trace the expected RPC-to-source path instead:

bdev_lvol_snapshot RPC
  -> module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_snapshot
  -> module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_create_snapshot
  -> lib/lvol/lvol.c:spdk_lvol_create_snapshot
  -> lib/blob/blobstore.c:spdk_bs_create_snapshot

Self-Check

  1. Why can a thin blob have a large logical size but few allocated clusters?
  2. What is the difference between spdk_bs_grow_live() and spdk_blob_resize()?
  3. Why does snapshot creation freeze IO?
  4. Why can deleting a snapshot require updating a clone?
  5. What does spdk_blob_is_degraded() mean for an esnap clone?
  6. Why is struct spdk_bs_dev separate from struct spdk_bdev?
  7. Which functions would you inspect first for an -EBUSY returned by lvol resize?
  8. Which source file adapts a blob into a backing device for clone reads?

References

  • Local API: include/spdk/blob.h
  • Local implementation: lib/blob/blobstore.c
  • Local private structures: lib/blob/blobstore.h
  • Local bdev adapter: module/blob/bdev/blob_bdev.c
  • Local blob-backed bs_dev: lib/blob/blob_bs_dev.c
  • Local official docs source: doc/blob.md, doc/lvol.md, doc/bdev.md
  • Local tests: test/blobstore/blobstore.sh, test/blobstore/blobstore_grow/blobstore_grow.sh, test/lvol/snapshot_clone.sh, test/lvol/thin_provisioning.sh
  • Official SPDK Blobstore Programmer's Guide: <https://spdk.io/doc/blob.html>
  • Official SPDK Logical Volumes documentation: <https://spdk.io/doc/logical_volumes.html>
  • Official SPDK Block Device User Guide: <https://spdk.io/doc/bdev.html>
  • Official SPDK blob.h API reference: <https://spdk.io/doc/blob_8h.html>