SPDK From First Principles

SPDK deep learning path

Chapter 20: lvol And lvol bdev

lvol is SPDK's logical-volume layer on top of blobstore. A logical volume store, or lvolstore, is a blobstore plus lvol-specific metadata. An lvol is a blob plus lvol-specific...

Source: drafts/lvol-raid/02-lvol-and-lvol-bdev.md

Beginner Mental Model

lvol is SPDK's logical-volume layer on top of blobstore. A logical volume store, or lvolstore, is a blobstore plus lvol-specific metadata. An lvol is a blob plus lvol-specific identity, name, UUID, lifecycle state, and helper operations. The lvol bdev module, vbdev_lvol, turns lvols into normal SPDK bdevs so the rest of SPDK can read, write, export, snapshot, clone, resize, and delete them through the bdev abstraction.

Three layers are easy to confuse:

base bdev
  |
  | module/blob/bdev/blob_bdev.c wraps it as struct spdk_bs_dev
  v
blobstore
  |
  | lib/lvol/lvol.c stores lvolstore metadata and lvol blobs
  v
lvol / lvolstore
  |
  | module/bdev/lvol/vbdev_lvol.c registers bdevs for lvols
  v
lvol bdevs visible to bdev users

Blobstore is the allocator and metadata engine. lvol is the volume manager API. vbdev_lvol is the bdev adapter and RPC-facing module.

The official SPDK docs describe the same split from the user side: logical volumes provide variable-size virtual block devices, the logical volume library is built on blobstore, and an lvol bdev translates generic bdev IO into blob operations. Blobstore itself is lower level: it gives applications persistent blobs, xattrs, pages, clusters, asynchronous metadata operations, and IO channels. The bdev layer is higher level: it gives applications one common block-device abstraction, open descriptors, IO channels, aliases, queueing, hot-remove handling, and virtual bdev layering.

That division is why this chapter keeps asking two questions for each operation:

  • What did lvol do to blobstore metadata?
  • What did vbdev_lvol do to make the result visible and usable as a bdev?

Why This Matters For diskengine/excloud

In a cloud disk service, a user-visible volume tends to behave like a bdev: it can be exported over NVMe-oF, attached to a VM through vhost/vfio-user, rate-limited, snapshotted, resized, or deleted. Internally, the fast volume operations are often blobstore operations. lvol is the translation layer between "cloud volume object" and "blobstore blob."

The most important operational consequences:

  • lvol create may produce a bdev only after asynchronous blob creation and bdev registration complete.
  • lvolstore load may auto-discover existing lvols through bdev_examine.
  • lvol delete must unregister the bdev before destroying the underlying blob.
  • lvol resize must update the blob and then notify the bdev layer of the new block count.
  • external-snapshot clones can become degraded if their parent bdev is unavailable.
  • names and UUIDs matter because RPCs often accept either lvolstore name/UUID and bdev name.

Public lvol API

Primary API anchors:

  • include/spdk/lvol.h:spdk_lvs_init
  • include/spdk/lvol.h:spdk_lvs_load
  • include/spdk/lvol.h:spdk_lvs_load_ext
  • include/spdk/lvol.h:spdk_lvs_unload
  • include/spdk/lvol.h:spdk_lvs_destroy
  • include/spdk/lvol.h:spdk_lvol_create
  • include/spdk/lvol.h:spdk_lvol_open
  • include/spdk/lvol.h:spdk_lvol_close
  • include/spdk/lvol.h:spdk_lvol_destroy
  • include/spdk/lvol.h:spdk_lvol_create_snapshot
  • include/spdk/lvol.h:spdk_lvol_create_clone
  • include/spdk/lvol.h:spdk_lvol_create_esnap_clone
  • include/spdk_internal/lvolstore.h:spdk_lvol_resize
  • include/spdk/lvol.h:spdk_lvol_inflate
  • include/spdk/lvol.h:spdk_lvol_decouple_parent
  • include/spdk/lvol.h:spdk_lvol_set_parent
  • include/spdk/lvol.h:spdk_lvol_set_external_parent
  • include/spdk/lvol.h:spdk_lvol_is_degraded

Implementation anchors:

  • lib/lvol/lvol.c:spdk_lvs_init
  • lib/lvol/lvol.c:spdk_lvs_load
  • lib/lvol/lvol.c:spdk_lvs_load_ext
  • lib/lvol/lvol.c:load_next_lvol
  • lib/lvol/lvol.c:lvs_read_uuid
  • lib/lvol/lvol.c:spdk_lvol_create
  • lib/lvol/lvol.c:lvol_create_cb
  • lib/lvol/lvol.c:spdk_lvol_destroy
  • lib/lvol/lvol.c:lvol_delete_blob_cb
  • lib/lvol/lvol.c:spdk_lvol_resize
  • lib/lvol/lvol.c:spdk_lvol_create_snapshot
  • lib/lvol/lvol.c:spdk_lvol_create_clone
  • lib/lvol/lvol.c:spdk_lvol_create_esnap_clone

lvolstore Lifecycle

An lvolstore is initialized on a blobstore device. The public initializer takes a struct spdk_bs_dev *, not a bdev name. The bdev adapter creates that spdk_bs_dev from the base bdev.

Creation path:

bdev_lvol_create_lvstore RPC
  -> module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_create_lvstore
  -> module/bdev/lvol/vbdev_lvol.c:vbdev_lvs_create
  -> module/blob/bdev/blob_bdev.c:spdk_bdev_create_bs_dev_ext
  -> lib/lvol/lvol.c:spdk_lvs_init
  -> lib/blob/blobstore.c:spdk_bs_init

Load/discovery path:

new base bdev appears
  -> lib/bdev/bdev.c:bdev_examine
  -> module/bdev/lvol/vbdev_lvol.c:vbdev_lvs_examine_disk
  -> module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine
  -> module/blob/bdev/blob_bdev.c:spdk_bdev_create_bs_dev_ext
  -> lib/lvol/lvol.c:spdk_lvs_load_ext
  -> lib/blob/blobstore.c:spdk_bs_load
  -> lib/lvol/lvol.c:load_next_lvol
  -> module/bdev/lvol/vbdev_lvol.c:_create_lvol_disk

Source anchors:

  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvs_examine_disk
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine_cb
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine_finish
  • module/bdev/lvol/vbdev_lvol.c:_create_lvol_disk
  • lib/lvol/lvol.c:spdk_lvs_load_ext
  • lib/lvol/lvol.c:load_next_lvol

Creation starts from a base bdev name, but the lvol library does not open bdevs by name. vbdev_lvol first asks the bdev/blob adapter to wrap the base bdev as a struct spdk_bs_dev; only then can spdk_lvs_init() initialize blobstore and write lvolstore metadata.

From lib/lvol/lvol.c:spdk_lvs_init:

if (bs_dev == NULL) {
	SPDK_ERRLOG("Blobstore device does not exist\n");
	return -ENODEV;
}

if (lvs_opts.cluster_sz < bs_dev->blocklen ||
    (lvs_opts.cluster_sz % bs_dev->blocklen) != 0) {
	SPDK_ERRLOG("Cluster size %" PRIu32 " is smaller than blocklen %" PRIu32
		    "Or not an integral multiple\n", lvs_opts.cluster_sz, bs_dev->blocklen);
	return -EINVAL;
}
total_clusters = bs_dev->blockcnt / (lvs_opts.cluster_sz / bs_dev->blocklen);

lvs = lvs_alloc();
...
spdk_uuid_generate(&lvs->uuid);
snprintf(lvs->name, sizeof(lvs->name), "%s", lvs_opts.name);
...
lvs->bs_dev = bs_dev;

SPDK_INFOLOG(lvol, "Initializing lvol store\n");
spdk_bs_init(bs_dev, &opts, lvs_init_cb, lvs_req);

The checks are not cosmetic. lvol sizes are rounded and tracked in blobstore clusters, but blobstore ultimately submits IO in base-device blocks. The cluster size must therefore line up with the backing bdev block size. The lvolstore owns the bs_dev pointer after this point, and the callback chain writes the super blob xattrs for the lvolstore name and UUID.

Discovery inverts the same ownership chain. The bdev module is called for each candidate bdev, creates a bs_dev, asks lvol to load blobstore metadata, then claims the base bdev only after the lvolstore load succeeds.

From module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine:

rc = spdk_bdev_create_bs_dev_ext(bdev->name, vbdev_lvs_base_bdev_event_cb,
				 NULL, &bs_dev);
if (rc < 0) {
	SPDK_INFOLOG(vbdev_lvol, "Cannot create bs dev on %s\n", bdev->name);
	_vbdev_lvs_examine_done(ori_req, rc);
	free(req);
	return;
}

req->base_bdev = bdev;
req->cb_arg = ori_req;

action(bs_dev, _vbdev_lvs_examine_cb, req);

From module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine_cb:

lvserrno = spdk_bs_bdev_claim(lvol_store->bs_dev, &g_lvol_if);
if (lvserrno != 0) {
	SPDK_INFOLOG(vbdev_lvol, "Lvol store base bdev already claimed by another bdev\n");
	ori_req->lvserrno = lvserrno;
	spdk_lvs_unload(lvol_store, _vbdev_lvs_examine_failed, ori_req);
	goto end;
}

lvs_bdev->lvs = lvol_store;
lvs_bdev->bdev = req->base_bdev;

TAILQ_INSERT_TAIL(&g_spdk_lvol_pairs, lvs_bdev, lvol_stores);
...
TAILQ_FOREACH_SAFE(lvol, &lvol_store->lvols, link, tmp) {
	spdk_lvol_open(lvol, _vbdev_lvs_examine_finish, ori_req);
}

The claim matters because the base bdev now backs blobstore metadata and lvol data. The official bdev documentation calls out this virtual-bdev rule: modules that route IO to lower bdevs claim those lower bdevs so outside writers cannot modify them unexpectedly. spdk_bs_bdev_claim() takes a write-one/read-many claim for a read-write blobstore wrapper.

Beginner misconception to kill: lvolstore discovery is not a global scan done by lvol itself. The bdev subsystem calls each module's examine callbacks. vbdev_lvol receives a candidate bdev, tries to load a blobstore/lvolstore from it, claims the base bdev if successful, then creates child bdevs for the lvols.

lvol Metadata And Identity

An lvol has:

  • An lvolstore pointer.
  • A blob ID.
  • A name.
  • A UUID and string form.
  • A unique ID used for bdev naming.
  • A reference count.
  • A struct spdk_blob * when opened.
  • Flags for pending actions or degraded external snapshots.

Source anchors:

  • include/spdk_internal/lvolstore.h:struct spdk_lvol_store
  • include/spdk_internal/lvolstore.h:struct spdk_lvol
  • lib/lvol/lvol.c:lvol_alloc
  • lib/lvol/lvol.c:lvol_get_xattr_value
  • lib/lvol/lvol.c:lvs_verify_lvol_name
  • lib/lvol/lvol.c:lvs_get_lvol_by_blob_id

The lvol name and UUID are persisted as blob xattrs. During load, lib/lvol/lvol.c:load_next_lvol opens each blob, reads xattrs, reconstructs lvol objects, and appends them to the lvolstore's lvol list.

The in-memory objects are deliberately small. The lvolstore points at the blobstore and owns lists of loaded lvol objects. Each lvol points back to its store, remembers the blob ID, and optionally holds an open struct spdk_blob and registered struct spdk_bdev .

From include/spdk_internal/lvolstore.h:

struct spdk_lvol_store {
	struct spdk_bs_dev		*bs_dev;
	struct spdk_blob_store		*blobstore;
	struct spdk_blob		*super_blob;
	spdk_blob_id			super_blob_id;
	struct spdk_uuid		uuid;
	int				lvol_count;
	int				lvols_opened;
	TAILQ_HEAD(, spdk_lvol)		lvols;
	TAILQ_HEAD(, spdk_lvol)		pending_lvols;
	TAILQ_HEAD(, spdk_lvol)		retry_open_lvols;
	...
	struct spdk_thread		*thread;
};

struct spdk_lvol {
	struct spdk_lvol_store		*lvol_store;
	struct spdk_blob		*blob;
	spdk_blob_id			blob_id;
	char				unique_id[SPDK_LVOL_UNIQUE_ID_MAX];
	char				name[SPDK_LVOL_NAME_MAX];
	struct spdk_uuid		uuid;
	char				uuid_str[SPDK_UUID_STRING_LEN];
	struct spdk_bdev		*bdev;
	int				ref_count;
	bool				action_in_progress;
	...
};

This shape explains many edge cases. A loaded lvol can exist before it has a bdev, because blob_id, name, and uuid are enough to represent metadata. A degraded esnap clone can have lvol metadata without a registered lvol bdev. Normal data IO requires a non-degraded blob and a bdev registration; management operations such as listing or delete can sometimes work from metadata alone.

From lib/lvol/lvol.c:load_next_lvol:

blob_id = spdk_blob_get_id(blob);

if (blob_id == lvs->super_blob_id) {
	SPDK_INFOLOG(lvol, "found superblob %"PRIu64"\n", (uint64_t)blob_id);
	spdk_bs_iter_next(bs, blob, load_next_lvol, req);
	return;
}

lvol = calloc(1, sizeof(*lvol));
...
lvol->blob_id = blob_id;
lvol->lvol_store = lvs;

rc = spdk_blob_get_xattr_value(blob, "uuid", (const void **)&attr, &value_len);
...
rc = spdk_blob_get_xattr_value(blob, "name", (const void **)&attr, &value_len);
...
TAILQ_INSERT_TAIL(&lvs->lvols, lvol, link);
lvs->lvol_count++;

The iterator skips the lvolstore super blob, rebuilds lvol identity from per-blob xattrs, and stores only the blob ID. The comment in the source notes that the iterator will close the blob; later spdk_lvol_open() reopens it when vbdev_lvol is ready to create the bdev.

lvol Create

Public API:

  • include/spdk/lvol.h:spdk_lvol_create

Implementation:

  • lib/lvol/lvol.c:spdk_lvol_create
  • lib/lvol/lvol.c:lvol_alloc
  • lib/lvol/lvol.c:lvol_get_xattr_value
  • lib/lvol/lvol.c:lvol_create_cb
  • lib/lvol/lvol.c:lvol_create_open_cb
  • lib/blob/blobstore.c:spdk_bs_create_blob_ext
  • lib/blob/blobstore.c:spdk_bs_open_blob_ext

spdk_lvol_create() verifies the lvolstore and name, allocates an in-memory lvol, prepares blob options, sets lvol xattrs, and calls spdk_bs_create_blob_ext(). When blob creation completes, it opens the blob and moves the lvol from the pending list into the lvolstore lvol list.

From lib/lvol/lvol.c:spdk_lvol_create:

rc = lvs_verify_lvol_name(lvs, name);
if (rc < 0) {
	return rc;
}

bs = lvs->blobstore;
...
lvol = lvol_alloc(lvs, name, thin_provision, clear_method);
...
spdk_blob_opts_init(&opts, sizeof(opts));
opts.thin_provision = thin_provision;
opts.num_clusters = spdk_divide_round_up(sz, spdk_bs_get_cluster_size(bs));
opts.clear_method = lvol->clear_method;
opts.xattrs.count = SPDK_COUNTOF(xattr_names);
opts.xattrs.names = xattr_names;
opts.xattrs.ctx = lvol;
opts.xattrs.get_value = lvol_get_xattr_value;

spdk_bs_create_blob_ext(lvs->blobstore, &opts, lvol_create_cb, req);

The requested size is not stored as a byte count here. It is converted into a blob cluster count, rounded up to the lvolstore cluster size. The lvol_get_xattr_value callback supplies the lvol name and UUID xattrs while blobstore creates the blob. This is why load can reconstruct lvol identity without a separate lvol metadata table.

From lib/lvol/lvol.c:lvol_create_cb:

if (lvolerrno < 0) {
	TAILQ_REMOVE(&req->lvol->lvol_store->pending_lvols, req->lvol, link);
	lvol_free(req->lvol);
	assert(req->cb_fn != NULL);
	req->cb_fn(req->cb_arg, NULL, lvolerrno);
	free(req);
	return;
}

spdk_blob_open_opts_init(&opts, sizeof(opts));
opts.clear_method = req->lvol->clear_method;
opts.esnap_ctx = req->lvol;
bs = req->lvol->lvol_store->blobstore;

spdk_bs_open_blob_ext(bs, blobid, &opts, lvol_create_open_cb, req);

The create callback is still inside the lvol library. It does not register a bdev. It opens the newly created blob and completes with a struct spdk_lvol *. The bdev registration is the adapter's job.

At the bdev layer:

  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_create
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_create_cb
  • module/bdev/lvol/vbdev_lvol.c:_create_lvol_disk

_create_lvol_disk() fills in a struct spdk_bdev: name, aliases, block length, block count, supported operations, fn table, module pointer, product name, and context. Then it registers the bdev.

From module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_create and _vbdev_lvol_create_cb:

rc = spdk_lvol_create(lvs, name, sz, thin_provision, clear_method,
		      _vbdev_lvol_create_cb, req);
...
static void
_vbdev_lvol_create_cb(void *cb_arg, struct spdk_lvol *lvol, int lvolerrno)
{
	struct spdk_lvol_with_handle_req *req = cb_arg;

	if (lvolerrno < 0) {
		goto end;
	}

	lvolerrno = _create_lvol_disk(lvol, true);

end:
	req->cb_fn(req->cb_arg, lvol, lvolerrno);
	free(req);
}

From module/bdev/lvol/vbdev_lvol.c:_create_lvol_disk:

bdev = &lvol_bdev->bdev;
bdev->name = lvol->unique_id;
bdev->product_name = "Logical Volume";
bdev->blocklen = spdk_bs_get_io_unit_size(lvol->lvol_store->blobstore);
total_size = spdk_blob_get_num_clusters(lvol->blob) *
	     spdk_bs_get_cluster_size(lvol->lvol_store->blobstore);
assert((total_size % bdev->blocklen) == 0);
bdev->blockcnt = total_size / bdev->blocklen;
bdev->uuid = lvol->uuid;
...
bdev->ctxt = lvol;
bdev->fn_table = &vbdev_lvol_fn_table;
bdev->module = &g_lvol_if;
...
rc = spdk_bdev_register(bdev);

This is the exact conversion from "lvol object" to "visible SPDK bdev." The bdev name is the lvol unique ID, while the user-friendly alias is added later as lvs_name/lvol_name. The bdev context points back to the lvol, so the IO path can recover the blob pointer from bdev_io->bdev->ctxt.

lvol bdev IO Path

The bdev-facing IO path starts at:

  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_submit_request
  • module/bdev/lvol/vbdev_lvol.c:lvol_read
  • module/bdev/lvol/vbdev_lvol.c:lvol_write
  • module/bdev/lvol/vbdev_lvol.c:lvol_unmap
  • module/bdev/lvol/vbdev_lvol.c:lvol_write_zeroes
  • module/bdev/lvol/vbdev_lvol.c:lvol_reset
  • module/bdev/lvol/vbdev_lvol.c:lvol_seek_data
  • module/bdev/lvol/vbdev_lvol.c:lvol_seek_hole
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_fn_table

Prose diagram:

SPDK bdev user submits write to lvol bdev
  -> lib/bdev/bdev.c:bdev_submit_request
  -> module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_submit_request
  -> module/bdev/lvol/vbdev_lvol.c:lvol_write
  -> lib/blob/blobstore.c:spdk_blob_io_write
  -> lib/blob/blobstore.c:blob_request_submit_op
  -> struct spdk_bs_dev write/writev on base bdev adapter
  -> completion propagates back to spdk_bdev_io_complete()

The lvol bdev does not implement its own allocator. It forwards reads, writes, unmaps, write-zeroes, and seeks to blobstore. That is why understanding blobstore thin allocation and snapshot backing matters before debugging lvol bdev IO.

From module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_submit_request:

static void
vbdev_lvol_submit_request(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)
{
	struct spdk_lvol *lvol = bdev_io->bdev->ctxt;

	switch (bdev_io->type) {
	case SPDK_BDEV_IO_TYPE_READ:
		spdk_bdev_io_get_buf(bdev_io, lvol_get_buf_cb,
				     bdev_io->u.bdev.num_blocks * bdev_io->bdev->blocklen);
		break;
	case SPDK_BDEV_IO_TYPE_WRITE:
		lvol_write(lvol, ch, bdev_io);
		break;
	case SPDK_BDEV_IO_TYPE_UNMAP:
		lvol_unmap(lvol, ch, bdev_io);
		break;
	case SPDK_BDEV_IO_TYPE_WRITE_ZEROES:
		lvol_write_zeroes(lvol, ch, bdev_io);
		break;
	...
	}
}

From the same file's read and write helpers:

lvol_io->ext_io_opts.memory_domain = bdev_io->u.bdev.memory_domain;
lvol_io->ext_io_opts.memory_domain_ctx = bdev_io->u.bdev.memory_domain_ctx;

spdk_blob_io_readv_ext(blob, ch, bdev_io->u.bdev.iovs, bdev_io->u.bdev.iovcnt,
		       start_page, num_pages, lvol_op_comp, bdev_io,
		       &lvol_io->ext_io_opts);
...
spdk_blob_io_writev_ext(blob, ch, bdev_io->u.bdev.iovs, bdev_io->u.bdev.iovcnt,
			start_page, num_pages, lvol_op_comp, bdev_io,
			&lvol_io->ext_io_opts);

The bdev layer calls submit_request on the thread/channel chosen for that IO. vbdev_lvol does minimal translation: it maps bdev block offsets to blob IO-unit offsets and preserves memory-domain information for the lower path. Completion goes back through lvol_op_comp(), which converts blobstore errno values into bdev IO completion status.

The base bdev adapter is the other half of the stack boundary. module/blob/bdev/blob_bdev.c opens a bdev descriptor and exposes it as the spdk_bs_dev interface blobstore expects.

From module/blob/bdev/blob_bdev.c:spdk_bdev_create_bs_dev:

rc = spdk_bdev_open_ext(bdev_name, write, event_cb, event_ctx, &desc);
if (rc != 0) {
	free(b);
	return rc;
}

blob_bdev_init(b, desc);

*bs_dev = &b->bs_dev;
b->write = write;
b->refs = 1;
spdk_spin_init(&b->lock);

From module/blob/bdev/blob_bdev.c:spdk_bs_bdev_claim:

claim_type = blob_bdev->write ? SPDK_BDEV_CLAIM_READ_MANY_WRITE_ONE :
	     SPDK_BDEV_CLAIM_READ_MANY_WRITE_NONE;
rc = spdk_bdev_module_claim_bdev_desc(desc, claim_type, NULL, module);
if (rc != 0) {
	SPDK_ERRLOG("could not claim bs dev\n");
	return rc;
}

So the complete stack has two adapters: blob_bdev.c adapts a bdev into a blobstore device, while vbdev_lvol.c adapts an lvol/blob back into a bdev. That loop is intentional. It lets blobstore manage allocation and metadata while the rest of SPDK only sees normal bdevs.

lvol Snapshots And Clones

Public lvol API:

  • include/spdk/lvol.h:spdk_lvol_create_snapshot
  • include/spdk/lvol.h:spdk_lvol_create_clone
  • include/spdk/lvol.h:spdk_lvol_iter_immediate_clones

Implementation:

  • lib/lvol/lvol.c:spdk_lvol_create_snapshot
  • lib/lvol/lvol.c:spdk_lvol_create_clone
  • lib/lvol/lvol.c:spdk_lvol_iter_immediate_clones
  • lib/blob/blobstore.c:spdk_bs_create_snapshot
  • lib/blob/blobstore.c:spdk_bs_create_clone

bdev/RPC layer:

  • module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_snapshot
  • module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_clone
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_create_snapshot
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_create_clone

Snapshot and clone creation both use the same _vbdev_lvol_create_cb path to create the new lvol bdev after the underlying lvol/blob operation succeeds.

From lib/lvol/lvol.c:spdk_lvol_create_snapshot:

newlvol = lvol_alloc(origlvol->lvol_store, snapshot_name, true,
		     (enum lvol_clear_method)origlvol->clear_method);
...
snapshot_xattrs.count = SPDK_COUNTOF(xattr_names);
snapshot_xattrs.ctx = newlvol;
snapshot_xattrs.names = xattr_names;
snapshot_xattrs.get_value = lvol_get_xattr_value;
req->lvol = newlvol;
req->origlvol = origlvol;
req->cb_fn = cb_fn;
req->cb_arg = cb_arg;

spdk_bs_create_snapshot(lvs->blobstore, spdk_blob_get_id(origblob), &snapshot_xattrs,
			lvol_create_cb, req);

From lib/lvol/lvol.c:spdk_lvol_create_clone:

newlvol = lvol_alloc(lvs, clone_name, true, (enum lvol_clear_method)origlvol->clear_method);
...
clone_xattrs.count = SPDK_COUNTOF(xattr_names);
clone_xattrs.ctx = newlvol;
clone_xattrs.names = xattr_names;
clone_xattrs.get_value = lvol_get_xattr_value;
req->lvol = newlvol;
req->cb_fn = cb_fn;
req->cb_arg = cb_arg;

spdk_bs_create_clone(lvs->blobstore, spdk_blob_get_id(origblob), &clone_xattrs,
		     lvol_create_cb,
		     req);

The lvol layer does not copy user data for ordinary snapshots and clones. It allocates a new lvol wrapper, supplies identity xattrs, and delegates the copy-on-write relationship to blobstore. The SPDK logical volumes documentation describes the user-visible consequence: snapshots are read-only, clones are thin-provisioned, and reads from unallocated clone clusters fall through to the backing snapshot.

At the bdev adapter, snapshots and clones intentionally converge on the same callback as normal create:

spdk_lvol_create_snapshot(lvol, snapshot_name, _vbdev_lvol_create_cb, req);
...
spdk_lvol_create_clone(lvol, clone_name, _vbdev_lvol_create_cb, req);

That is why an RPC such as bdev_lvol_snapshot both creates blobstore snapshot metadata and produces a new bdev name on success.

Beginner misconception to kill: bdev_lvol_snapshot names an lvol bdev and creates another lvol bdev. The snapshot is still a blobstore snapshot under the hood. vbdev_lvol is responsible for making the result visible as a bdev.

External Snapshot Clones

External snapshot support crosses all three layers: blobstore, lvol, and vbdev_lvol.

Public lvol API:

  • include/spdk/lvol.h:spdk_lvol_create_esnap_clone
  • include/spdk/lvol.h:spdk_lvol_set_external_parent

lvol implementation:

  • lib/lvol/lvol.c:spdk_lvol_create_esnap_clone
  • lib/lvol/lvol.c:lvs_esnap_bs_dev_create
  • lib/lvol/lvol.c:spdk_lvs_esnap_missing_add
  • lib/lvol/lvol.c:spdk_lvs_esnap_missing_remove
  • lib/lvol/lvol.c:lvs_esnap_degraded_hotplug
  • lib/lvol/lvol.c:spdk_lvol_is_degraded
  • include/spdk_internal/lvolstore.h:spdk_lvs_notify_hotplug
  • lib/lvol/lvol.c:spdk_lvs_notify_hotplug

vbdev_lvol implementation:

  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_create_bdev_clone
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_esnap_dev_create
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvs_examine_config
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvs_hotplug
  • module/bdev/lvol/vbdev_lvol.c:create_esnap_clone_lvol_disks
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_set_external_parent

External snapshot flow:

bdev_lvol_clone_bdev
  -> open external bdev read-only
  -> parse external bdev UUID
  -> create esnap clone blob with external snapshot ID
  -> lvol create callback opens blob
  -> blobstore asks lvol/vbdev_lvol to create a bs_dev for esnap ID
  -> vbdev_lvol opens the parent bdev and claims it with a shared/read claim
  -> child lvol bdev is registered

Unlike ordinary lvol create and resize, external-snapshot clone creation does not round the requested size up. spdk_lvol_create_esnap_clone() checks size_bytes % cluster_sz and returns -EINVAL unless the clone size is an exact multiple of the lvolstore cluster size. That keeps the external parent relationship aligned to blobstore cluster boundaries from the first metadata record.

If the external bdev is missing during load, lvol tracks the missing esnap in lvs->degraded_lvol_sets_tree. The degraded lvol may be visible through lvol-specific management paths, but _create_lvol_disk() returns early and logs that bdev creation is deferred. When a bdev with the matching UUID appears, vbdev_lvs_examine_config() calls spdk_lvs_notify_hotplug(), and the lvolstore can attempt to attach the external parent and create child bdevs that were previously withheld.

Degraded lvols

Source anchors:

  • include/spdk/lvol.h:spdk_lvol_is_degraded
  • lib/lvol/lvol.c:spdk_lvol_is_degraded
  • include/spdk/blob.h:spdk_blob_is_degraded
  • lib/blob/blobstore.c:spdk_blob_is_degraded
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_get_memory_domains
  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_esnap_dev_create

An lvol is degraded if it has no open blob or the blob is degraded. For an esnap clone, missing external snapshot state can make the blob degraded. A degraded lvol cannot perform normal IO and should not be expected in bdev_get_bdevs, because _create_lvol_disk() skips bdev registration for degraded lvols. It can still be found through lvol metadata paths such as UUID or lvs_name/lvol_name lookup, so delete and close paths have explicit handling to clean it up.

Important edge case: module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_destroy checks spdk_lvol_is_degraded(lvol). If degraded, it closes the lvol instead of unregistering a bdev that may not exist.

Resize

Public API:

  • include/spdk_internal/lvolstore.h:spdk_lvol_resize

Implementation:

  • lib/lvol/lvol.c:spdk_lvol_resize
  • lib/lvol/lvol.c:lvol_blob_resize_cb
  • lib/lvol/lvol.c:lvol_resize_done
  • lib/blob/blobstore.c:spdk_blob_resize

bdev adapter:

  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_resize
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_resize_cb
  • module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_resize

The lvol layer converts bytes to cluster count using the lvolstore cluster size, calls spdk_blob_resize(), then syncs blob metadata. Ordinary lvol resize rounds the requested byte size up to whole blobstore clusters, so a visible lvol bdev can become slightly larger than the requested byte count. The bdev adapter updates the visible bdev block count and notifies the bdev layer after blobstore accepts the resize.

From lib/lvol/lvol.c:spdk_lvol_resize:

struct spdk_blob *blob = lvol->blob;
struct spdk_lvol_store *lvs = lvol->lvol_store;
struct spdk_lvol_req *req;
uint64_t new_clusters = spdk_divide_round_up(sz, spdk_bs_get_cluster_size(lvs->blobstore));

req = calloc(1, sizeof(*req));
...
req->cb_fn = cb_fn;
req->cb_arg = cb_arg;
req->lvol = lvol;

spdk_blob_resize(blob, new_clusters, lvol_blob_resize_cb, req);

The lvol library only knows that the blob should contain a new number of clusters. It does not edit a bdev in place. That separation prevents the visible bdev size from changing before blobstore accepts and persists the resize metadata.

From module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_resize_cb:

total_size = spdk_blob_get_num_clusters(lvol->blob) *
	     spdk_bs_get_cluster_size(lvol->lvol_store->blobstore);
assert((total_size % lvol->bdev->blocklen) == 0);

lvolerrno = spdk_bdev_notify_blockcnt_change(lvol->bdev,
		total_size / lvol->bdev->blocklen);
if (lvolerrno != 0) {
	SPDK_ERRLOG("Could not change num blocks for bdev lvol %s with error no: %d.\n",
		    lvol->name, lvolerrno);
}

This notification is the point where consumers of the bdev can observe the new size. If a target, vhost device, or management plane caches capacity, the correctness question is downstream of this callback: did the consumer subscribe to and handle bdev resize events?

Edge cases:

  • Resize of a read-only snapshot fails at blobstore metadata checks.
  • Resize while another locked blob operation is in progress fails with -EBUSY.
  • Growing a clone beyond its parent is allowed at the lvol/blob level, but backing reads beyond the parent must be treated carefully. See lib/blob/blob_bs_dev.c:blob_bs_is_range_valid.
  • Resize of an exported lvol bdev may require consumers to observe bdev resize events correctly.

Delete

Public API:

  • include/spdk/lvol.h:spdk_lvol_deletable
  • include/spdk/lvol.h:spdk_lvol_destroy

Implementation:

  • lib/lvol/lvol.c:spdk_lvol_deletable
  • lib/lvol/lvol.c:spdk_lvol_destroy
  • lib/lvol/lvol.c:lvol_delete_blob_cb
  • lib/blob/blobstore.c:spdk_bs_delete_blob
  • lib/blob/blobstore.c:bs_is_blob_deletable

bdev adapter:

  • module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_destroy
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_destroy
  • module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_destroy_cb
  • module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_delete

Delete is layered:

bdev_lvol_delete
  -> find bdev by name
  -> map bdev to struct spdk_lvol
  -> module/bdev/lvol/vbdev_lvol.c:vbdev_lvol_destroy
  -> unregister lvol bdev or close degraded lvol
  -> lib/lvol/lvol.c:spdk_lvol_destroy
  -> lib/blob/blobstore.c:spdk_bs_delete_blob

The RPC first resolves a user name into an lvol. Normal lvols can be found through the bdev name or alias. Degraded lvols may not have a registered bdev, so the RPC also tries UUID and lvs_name/lvol_name.

From module/bdev/lvol/vbdev_lvol_rpc.c:rpc_bdev_lvol_delete:

bdev = spdk_bdev_get_by_name(req.name);
if (bdev != NULL) {
	lvol = vbdev_lvol_get_from_bdev(bdev);
	if (lvol != NULL) {
		goto done;
	}
}

/* lvol is degraded, get lvol via UUID */
if (spdk_uuid_parse(&uuid, req.name) == 0) {
	lvol = spdk_lvol_get_by_uuid(&uuid);
	if (lvol != NULL) {
		goto done;
	}
}
...
done:
	vbdev_lvol_destroy(lvol, rpc_bdev_lvol_delete_cb, request);

vbdev_lvol_destroy() removes the bdev surface before the lvol library destroys the blob. This ordering avoids a bdev that still accepts new IO while its blob is being deleted.

From module/bdev/lvol/vbdev_lvol.c:_vbdev_lvol_destroy:

spdk_blob_get_clones(lvol->lvol_store->blobstore, lvol->blob_id, NULL, &count);
if (count > 1) {
	SPDK_ERRLOG("Cannot delete lvol\n");
	cb_fn(cb_arg, -EPERM);
	return;
}
...
if (spdk_lvol_is_degraded(lvol)) {
	spdk_lvol_close(lvol, _vbdev_lvol_destroy_cb, ctx);
	return;
}

spdk_bdev_unregister(lvol->bdev, _vbdev_lvol_destroy_cb, ctx);

From lib/lvol/lvol.c:spdk_lvol_destroy:

if (lvol->ref_count != 0) {
	SPDK_ERRLOG("Cannot destroy lvol %s because it is still open\n", lvol->unique_id);
	cb_fn(cb_arg, -EBUSY);
	return;
}
...
rc = spdk_blob_get_clones(lvs->blobstore, lvol->blob_id, &clone_id, &count);
if (rc == 0 && count == 1) {
	req->clone_lvol = lvs_get_lvol_by_blob_id(lvs, clone_id);
} else if (rc == -ENOMEM) {
	SPDK_INFOLOG(lvol, "lvol %s: cannot destroy: has %" PRIu64 " clones\n",

The adapter rejects snapshots with multiple clones before unregistering the bdev. The lvol library then rejects open lvols and coordinates the blobstore delete. The one-clone case is special because blobstore can update the remaining clone's metadata relationship instead of copying data.

Deletion rules:

  • spdk_lvol_destroy() fails with -EBUSY if the lvol is still open.
  • vbdev_lvol refuses delete when the snapshot has more than one clone and returns -EPERM before unregistering the bdev.
  • The lower blobstore delete check can also reject a snapshot with more than one clone, but that path returns -EBUSY.
  • Blobstore may allow a snapshot with exactly one clone to be deleted by updating the clone.
  • If the lvolstore itself is being removed, lvol bdev deletion follows the lvolstore teardown path rather than the normal RPC delete path.

Misconceptions To Kill

  • "An lvol is a bdev." Not exactly. An lvol is a library object backed by a blob. vbdev_lvol exposes it as a bdev.
  • "Creating an lvolstore creates lvol bdevs immediately." It creates the lvolstore. lvol bdevs appear when lvols are created or loaded and registered.
  • "lvol resize is just changing a bdev field." It resizes the underlying blob and then updates the bdev.
  • "lvol snapshots copy data." They call blobstore snapshot logic, which is metadata/COW based.
  • "External snapshot clones are independent volumes." They depend on an external parent until decoupled or inflated.
  • "If an external snapshot parent is missing, the lvolstore cannot load at all." The lvol layer can track degraded lvols and hotplug the parent later.

Source Reading Exercise

Trace lvolstore auto-discovery:

  1. lib/bdev/bdev.c:bdev_examine
  2. module/bdev/lvol/vbdev_lvol.c:vbdev_lvs_examine_disk
  3. module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine
  4. lib/lvol/lvol.c:spdk_lvs_load_ext
  5. lib/lvol/lvol.c:load_next_lvol
  6. module/bdev/lvol/vbdev_lvol.c:_vbdev_lvs_examine_cb
  7. module/bdev/lvol/vbdev_lvol.c:_create_lvol_disk

Write down:

  • Where the base bdev is claimed.
  • Where lvol xattrs are read.
  • Where each lvol bdev is registered.
  • Where spdk_bdev_module_examine_done() is called.

Operational Lab

Use local test scripts as lab guides:

  • test/lvol/basic.sh
  • test/lvol/resize.sh
  • test/lvol/snapshot_clone.sh
  • test/lvol/external_snapshot.sh
  • test/lvol/hotremove.sh

Suggested RPC lab:

1. Start SPDK app with JSON-RPC enabled.
2. Create a malloc bdev.
3. Create an lvolstore on the malloc bdev.
4. Create a thin lvol.
5. Run bdev_get_bdevs and identify:
   - base malloc bdev
   - lvol bdev
   - claim state of the base bdev
6. Snapshot the lvol.
7. Clone the snapshot.
8. Resize the clone.
9. Delete clone, snapshot, original in different orders and record expected failures.
10. Restart the app and verify lvolstore/lvol bdevs are recreated through examine.

Debug prompt:

  • If an lvol bdev does not appear after restart, check whether vbdev_lvs_examine_disk() ran, whether spdk_lvs_load_ext() succeeded, whether spdk_bs_bdev_claim() failed, and whether _create_lvol_disk() returned an error.

Self-Check

  1. What is the difference between spdk_lvol_create() and vbdev_lvol_create()?
  2. Why does vbdev_lvol_submit_request() call blobstore IO functions?
  3. Which source function turns an lvol into a bdev?
  4. What has to happen before an lvolstore loaded from disk can expose lvol bdevs?
  5. Why can external snapshot clones become degraded?
  6. What prevents deleting an open lvol?
  7. Why does deleting a snapshot with one clone differ from deleting a snapshot with two clones?
  8. Which path handles bdev resize notification after lvol resize?

References

  • Official SPDK logical volumes documentation: <https://spdk.io/doc/logical_volumes.html>
  • Official SPDK Blobstore Programmer's Guide: <https://spdk.io/doc/blob.html>
  • Official SPDK Block Device Layer Programming Guide: <https://spdk.io/doc/bdev_pg.html>
  • Official SPDK Writing a Custom Block Device Module guide: <https://spdk.io/doc/bdev_module.html>
  • Official SPDK blob_bdev.h API reference: <https://spdk.io/doc/blob__bdev_8h.html>
  • Local API: include/spdk/lvol.h
  • Local lvol implementation: lib/lvol/lvol.c
  • Local lvol bdev implementation: module/bdev/lvol/vbdev_lvol.c
  • Local lvol RPCs: module/bdev/lvol/vbdev_lvol_rpc.c
  • Local bdev-backed blobstore adapter: module/blob/bdev/blob_bdev.c
  • Local tests: test/lvol/basic.sh, test/lvol/resize.sh, test/lvol/snapshot_clone.sh, test/lvol/external_snapshot.sh, test/lvol/hotremove.sh