SPDK From First Principles

SPDK deep learning path

Chapter 16: QoS, Reset, Remove, Hotplug, Events

By the end of this chapter you should be able to explain the non-happy-path machinery around bdev I/O: rate limits, reset, unregister, hotremove, media-management events, and...

Source: drafts/bdev-nvme/16-qos-reset-remove-hotplug-events.md

Reader Promise

By the end of this chapter you should be able to explain the non-happy-path machinery around bdev I/O: rate limits, reset, unregister, hotremove, media-management events, and quiesce. These are the paths that matter when production systems hang, drain, reconnect, delete, or rebalance.

The happy path is "submit I/O, complete I/O." Real systems spend a lot of engineering effort on what happens while someone is deleting a volume, resetting a controller, changing a namespace, limiting throughput, or handling a device disappearing. The bdev layer is where SPDK centralizes much of that coordination so every module does not have to rediscover the same lifecycle rules.

Official SPDK references used for this chapter:

  • SPDK Block Device User Guide: https://spdk.io/doc/bdev.html.
  • SPDK Block Device Layer Programming Guide: https://spdk.io/doc/bdev_pg.html.
  • SPDK JSON-RPC reference: https://spdk.io/doc/jsonrpc.html.
  • SPDK bdev public API Doxygen: https://spdk.io/doc/bdev_8h.html.
  • SPDK bdev module API Doxygen: https://spdk.io/doc/bdev__module_8h.html.

The key point from the programming guide is that bdev calls run on SPDK threads, descriptors are the open handles, and each thread uses its own I/O channel. The guide also describes reset as a bdev-layer operation that pauses other channels, forwards one reset request to the module, waits for completion, then resumes the channels. The API docs for spdk_bdev_open_ext() say removal events are delivered through the descriptor event callback and that, on removal, the descriptor must be closed manually so unregister can proceed.

The Shared Mental Model

A bdev channel is the per-thread execution object behind a descriptor. On the normal path, an I/O is added to the channel's submitted list and forwarded to the module. On the paths in this chapter, the same I/O may instead sit in one of several side queues:

  • qos_queued_io: waiting for rate-limit quota.
  • shared nomem_io: waiting for backend resources.
  • io_locked: blocked by a locked or quiesced LBA range.
  • queued_resets: waiting for the reset already in progress to finish.

These queues exist for different reasons, but they share one design goal: do not block the SPDK thread. SPDK tries to finish the API call quickly, then uses pollers, messages, completion callbacks, and channel iteration to make progress later.

flowchart LR A[caller submits bdev_io] --> B{channel state} B -->|normal| C[submit_request in module] B -->|QoS enabled| D[qos_queued_io] B -->|reset in progress| E[complete aborted] B -->|range locked| F[io_locked] D -->|quota refilled| C F -->|unquiesce/unlock| A C --> G[I/O completion]

QoS: Rate Limiting At The bdev Layer

SPDK bdev QoS is per-bdev rate limiting implemented in the bdev core. It is not an NVMe feature and it does not change a hardware submission queue directly. It queues I/O before module submission, then releases queued work as quota becomes available.

The JSON-RPC API exposes four limits:

  • rw_ios_per_sec: combined read/write IOPS.
  • rw_mbytes_per_sec: combined read/write MiB/s.
  • r_mbytes_per_sec: read MiB/s.
  • w_mbytes_per_sec: write MiB/s.

The official JSON-RPC docs describe bdev_set_qos_limit with those same fields and define 0 as unlimited. The RPC handler opens the bdev read-only, verifies that at least one limit was specified, and calls the bdev core.

/* lib/bdev/bdev_rpc.c */
struct rpc_bdev_set_qos_limit {
	char		*name;
	uint64_t	limits[SPDK_BDEV_QOS_NUM_RATE_LIMIT_TYPES];
};

static const struct spdk_json_object_decoder rpc_bdev_set_qos_limit_decoders[] = {
	{"name", offsetof(struct rpc_bdev_set_qos_limit, name), spdk_json_decode_string},
	{
		"rw_ios_per_sec", offsetof(struct rpc_bdev_set_qos_limit,
					   limits[SPDK_BDEV_QOS_RW_IOPS_RATE_LIMIT]),
		spdk_json_decode_uint64, true
	},
	{
		"rw_mbytes_per_sec", offsetof(struct rpc_bdev_set_qos_limit,
					      limits[SPDK_BDEV_QOS_RW_BPS_RATE_LIMIT]),
		spdk_json_decode_uint64, true
	},
	{
		"r_mbytes_per_sec", offsetof(struct rpc_bdev_set_qos_limit,
					     limits[SPDK_BDEV_QOS_R_BPS_RATE_LIMIT]),
		spdk_json_decode_uint64, true
	},
	{
		"w_mbytes_per_sec", offsetof(struct rpc_bdev_set_qos_limit,
					     limits[SPDK_BDEV_QOS_W_BPS_RATE_LIMIT]),
		spdk_json_decode_uint64, true
	},
};

The core normalizes the user-facing values before enabling or updating the poller. IOPS limits are already "per second"; byte limits arrive as MiB/s from RPC and are converted to bytes/s. The code also rounds up to the minimum supported granularity. This is why a user can request a slightly odd number and see a log saying SPDK rounded it.

/* lib/bdev/bdev.c */
for (i = 0; i < SPDK_BDEV_QOS_NUM_RATE_LIMIT_TYPES; i++) {
	if (limits[i] == SPDK_BDEV_QOS_LIMIT_NOT_DEFINED) {
		continue;
	}

	if (limits[i] > 0) {
		disable_rate_limit = false;
	}

	if (bdev_qos_is_iops_rate_limit(i) == true) {
		min_limit_per_sec = SPDK_BDEV_QOS_MIN_IOS_PER_SEC;
	} else {
		if (limits[i] > SPDK_BDEV_QOS_MAX_MBYTES_PER_SEC) {
			SPDK_WARNLOG("Requested rate limit %" PRIu64 " will result in uint64_t overflow, "
				     "reset to %" PRIu64 "\n", limits[i], SPDK_BDEV_QOS_MAX_MBYTES_PER_SEC);
			limits[i] = SPDK_BDEV_QOS_MAX_MBYTES_PER_SEC;
		}
		/* Change from megabyte to byte rate limit */
		limits[i] = limits[i] * 1024 * 1024;
		min_limit_per_sec = SPDK_BDEV_QOS_MIN_BYTES_PER_SEC;
	}

	limit_set_complement = limits[i] % min_limit_per_sec;
	if (limit_set_complement) {
		SPDK_ERRLOG("Requested rate limit %" PRIu64 " is not a multiple of %" PRIu64 "\n",
			    limits[i], min_limit_per_sec);
		limits[i] += min_limit_per_sec - limit_set_complement;
		SPDK_ERRLOG("Round up the rate limit to %" PRIu64 "\n", limits[i]);
	}
}

QoS does not apply to every bdev operation. The helper below treats normal reads and writes, NVMe I/O passthrough, and zcopy starts as rate-limited traffic. A zcopy end does not move data in the same way, so it is excluded.

/* lib/bdev/bdev.c */
static bool
bdev_qos_io_to_limit(struct spdk_bdev_io *bdev_io)
{
	switch (bdev_io->type) {
	case SPDK_BDEV_IO_TYPE_NVME_IO:
	case SPDK_BDEV_IO_TYPE_NVME_IO_MD:
	case SPDK_BDEV_IO_TYPE_READ:
	case SPDK_BDEV_IO_TYPE_WRITE:
		return true;
	case SPDK_BDEV_IO_TYPE_ZCOPY:
		if (bdev_io->u.bdev.zcopy.start) {
			return true;
		} else {
			return false;
		}
	default:
		return false;
	}
}

The actual queue decision is quota accounting. Each configured limit has a queue_io function. If one limit says "no quota," SPDK rewinds the quota already charged against earlier limits for the same I/O and leaves the I/O queued.

/* lib/bdev/bdev.c */
static bool
bdev_qos_queue_io(struct spdk_bdev_qos *qos, struct spdk_bdev_io *bdev_io)
{
	int i;

	if (bdev_qos_io_to_limit(bdev_io) == true) {
		for (i = 0; i < SPDK_BDEV_QOS_NUM_RATE_LIMIT_TYPES; i++) {
			if (!qos->rate_limits[i].queue_io) {
				continue;
			}

			if (qos->rate_limits[i].queue_io(&qos->rate_limits[i],
							 bdev_io) == true) {
				for (i -= 1; i >= 0 ; i--) {
					if (!qos->rate_limits[i].queue_io) {
						continue;
					}

					qos->rate_limits[i].rewind_quota(&qos->rate_limits[i], bdev_io);
				}
				return true;
			}
		}
	}

	return false;
}

When QoS is active, _bdev_io_submit() does not immediately call the module. It puts the I/O on qos_queued_io and then immediately tries to drain that queue. This means an I/O can still pass through right away when quota is available; the queue is the entry point for accounting, not proof that the I/O will wait for a full timeslice.

/* lib/bdev/bdev.c */
if (bdev_ch->flags & BDEV_CH_RESET_IN_PROGRESS) {
	_bdev_io_complete_in_submit(bdev_ch, bdev_io, SPDK_BDEV_IO_STATUS_ABORTED);
} else if (bdev_ch->flags & BDEV_CH_QOS_ENABLED) {
	if (spdk_unlikely(bdev_io->type == SPDK_BDEV_IO_TYPE_ABORT) &&
	    bdev_abort_queued_io(&bdev_ch->qos_queued_io, bdev_io->u.abort.bio_to_abort)) {
		_bdev_io_complete_in_submit(bdev_ch, bdev_io, SPDK_BDEV_IO_STATUS_SUCCESS);
	} else {
		TAILQ_INSERT_TAIL(&bdev_ch->qos_queued_io, bdev_io, internal.link);
		bdev_qos_io_submit(bdev_ch, bdev->internal.qos);
	}
} else {
	SPDK_ERRLOG("unknown bdev_ch flag %x found\n", bdev_ch->flags);
	_bdev_io_complete_in_submit(bdev_ch, bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
}

The poller is the refiller. Every QoS timeslice, it accounts for any previous overrun, adds fresh quota, and iterates bdev channels to submit queued I/O. The overrun handling is deliberate: an I/O larger than one timeslice quota may be allowed occasionally, then charged against a later timeslice instead of being stuck forever.

/* lib/bdev/bdev.c */
static int
bdev_channel_poll_qos(void *arg)
{
	struct spdk_bdev *bdev = arg;
	struct spdk_bdev_qos *qos = bdev->internal.qos;
	uint64_t now = spdk_get_ticks();
	int i;
	int64_t remaining_last_timeslice;

	if (spdk_unlikely(qos->thread == NULL)) {
		/* Old QoS was unbound to remove and new QoS is not enabled yet. */
		return SPDK_POLLER_IDLE;
	}

	if (now < (qos->last_timeslice + qos->timeslice_size)) {
		return SPDK_POLLER_IDLE;
	}

	for (i = 0; i < SPDK_BDEV_QOS_NUM_RATE_LIMIT_TYPES; i++) {
		remaining_last_timeslice = __atomic_exchange_n(&qos->rate_limits[i].remaining_this_timeslice,
				   0, __ATOMIC_RELAXED);
		if (remaining_last_timeslice < 0) {
			__atomic_store_n(&qos->rate_limits[i].remaining_this_timeslice,
					 remaining_last_timeslice, __ATOMIC_RELAXED);
		}
	}

	while (now >= (qos->last_timeslice + qos->timeslice_size)) {
		qos->last_timeslice += qos->timeslice_size;
		for (i = 0; i < SPDK_BDEV_QOS_NUM_RATE_LIMIT_TYPES; i++) {
			__atomic_add_fetch(&qos->rate_limits[i].remaining_this_timeslice,
					   qos->rate_limits[i].max_per_timeslice, __ATOMIC_RELAXED);
		}
	}

	spdk_bdev_for_each_channel(bdev, bdev_channel_submit_qos_io, qos,
				   bdev_channel_submit_qos_io_done);

	return SPDK_POLLER_BUSY;
}

QoS edge cases to keep in your head:

  • A concurrent QoS change returns -EAGAIN; the bdev has only one qos_mod_in_progress slot.
  • A byte limit may be capped before conversion to avoid overflow.
  • Disabling QoS resubmits queued I/O instead of dropping it.
  • Reset aborts QoS queued I/O on each channel.
  • QoS is lazy around channels: a QoS poller needs a selected channel/thread, and teardown may need to run on that thread.

Reset: Freeze, Drain, Submit, Unfreeze

The public reset API looks small:

/* lib/bdev/bdev.c */
int
spdk_bdev_reset(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch,
		spdk_bdev_io_completion_cb cb, void *cb_arg)
{
	struct spdk_bdev *bdev = spdk_bdev_desc_get_bdev(desc);
	struct spdk_bdev_io *bdev_io;
	struct spdk_bdev_channel *channel = __io_ch_to_bdev_ch(ch);

	bdev_io = bdev_channel_get_io(channel);
	if (!bdev_io) {
		return -ENOMEM;
	}

	bdev_io->internal.ch = channel;
	bdev_io->internal.desc = desc;
	bdev_io->internal.submit_tsc = spdk_get_ticks();
	bdev_io->type = SPDK_BDEV_IO_TYPE_RESET;
	bdev_io_init(bdev_io, bdev, cb_arg, cb);

	bdev_start_reset(bdev_io);
	return 0;
}

The work behind that call is not "send one reset to hardware." Reset is a bdev-wide coordination operation. The core serializes resets, freezes every channel, aborts queued work that will not reach the module, optionally waits for outstanding I/O to drain, submits one reset request to the module, and finally unfreezes the channels.

The reset_io_drain_timeout field explains why a reset may be skipped. The comment in struct spdk_bdev calls out the shared-lower-device case: if several upper bdevs share one underlying bdev, a nonzero timeout gives outstanding I/O a chance to finish so SPDK can avoid sending an empty disruptive reset to the lower device.

/* include/spdk/bdev_module.h */
/* Upon receiving a reset request, this is the amount of time in seconds
 * to wait for all I/O to complete before moving forward with the reset.
 * If all I/O completes prior to this time out, the reset will be skipped.
 * A value of 0 is special and will always send resets immediately, even
 * if there is no I/O outstanding.
 */
uint16_t reset_io_drain_timeout;

bdev_start_reset() puts the reset I/O on the submitted list and takes a channel reference. That reference is lifecycle protection: reset completion may happen later, and the channel must not disappear while the reset is being coordinated. The spinlock-protected reset_in_progress pointer is the serialization point. Later reset requests are not forwarded to the module; they are queued and later completed with the same status as the active reset.

/* lib/bdev/bdev.c */
static void
bdev_start_reset(struct spdk_bdev_io *bdev_io)
{
	struct spdk_io_channel *io_ch = spdk_io_channel_from_ctx(bdev_io->internal.ch);
	struct spdk_bdev *bdev = bdev_io->bdev;
	bool freeze_channel = false;

	bdev_ch_add_to_io_submitted(bdev_io);

	bdev_io->u.reset.ch_ref = spdk_io_channel_ref(io_ch);

	spdk_spin_lock(&bdev->internal.spinlock);
	if (bdev->internal.reset_in_progress == NULL) {
		bdev->internal.reset_in_progress = bdev_io;
		freeze_channel = true;
	} else {
		TAILQ_INSERT_TAIL(&bdev->internal.queued_resets, bdev_io, internal.link);
	}
	spdk_spin_unlock(&bdev->internal.spinlock);

	if (freeze_channel) {
		spdk_bdev_for_each_channel(bdev, bdev_reset_freeze_channel, bdev_io,
					   bdev_reset_freeze_channel_done);
	}
}

Freezing a channel sets BDEV_CH_RESET_IN_PROGRESS. New I/O that arrives after this point is completed as aborted by _bdev_io_submit(). The freeze pass also aborts I/O waiting in resource queues. That matters because queued work has not reached the module, so the module cannot complete or abort it during its hardware reset.

/* lib/bdev/bdev.c */
static void
bdev_reset_freeze_channel(struct spdk_bdev_channel_iter *i, struct spdk_bdev *bdev,
			  struct spdk_io_channel *ch, void *_ctx)
{
	struct spdk_bdev_channel	*channel;
	struct spdk_bdev_mgmt_channel	*mgmt_channel;
	struct spdk_bdev_shared_resource *shared_resource;

	channel = __io_ch_to_bdev_ch(ch);
	shared_resource = channel->shared_resource;
	mgmt_channel = shared_resource->mgmt_ch;

	channel->flags |= BDEV_CH_RESET_IN_PROGRESS;

	bdev_abort_all_nomem_io(channel);
	bdev_abort_all_buf_io(mgmt_channel, channel);

	if ((channel->flags & BDEV_CH_QOS_ENABLED) != 0) {
		bdev_abort_all_queued_io(&channel->qos_queued_io, channel);
	}

	spdk_bdev_for_each_channel_continue(i, 0);
}

The drain check looks across channels. If any channel still has submitted I/O, memory-domain work, or accel work, reset waits until the timeout. This is an all-channel busy check, not just the reset submitter's channel.

/* lib/bdev/bdev.c */
static void
bdev_reset_check_outstanding_io(struct spdk_bdev_channel_iter *i, struct spdk_bdev *bdev,
				struct spdk_io_channel *io_ch, void *_ctx)
{
	struct spdk_bdev_channel *cur_ch = __io_ch_to_bdev_ch(io_ch);
	int status = 0;

	if (cur_ch->io_outstanding > 0 ||
	    !TAILQ_EMPTY(&cur_ch->io_memory_domain) ||
	    !TAILQ_EMPTY(&cur_ch->io_accel_exec)) {
		status = -EBUSY;
	}
	spdk_bdev_for_each_channel_continue(i, status);
}

The timeout branch is narrower than that first scan. After the drain timeout, the completion path looks at the reset I/O's channel and checks whether that channel still has memory-domain or accel work. If not, reset is submitted even though ordinary outstanding I/O was the reason the all-channel scan stayed busy until timeout. If the reset I/O's channel still has memory-domain or accel work, reset fails because this layer cannot safely abort those operations.

/* lib/bdev/bdev.c */
if (status == -EBUSY) {
	if (spdk_get_ticks() < bdev_io->u.reset.wait_poller.stop_time_tsc) {
		bdev_io->u.reset.wait_poller.poller = SPDK_POLLER_REGISTER(
			bdev_reset_poll_for_outstanding_io, bdev_io,
			BDEV_RESET_CHECK_OUTSTANDING_IO_PERIOD_IN_USEC);
	} else {
		if (TAILQ_EMPTY(&ch->io_memory_domain) &&
		    TAILQ_EMPTY(&ch->io_accel_exec)) {
			bdev_io_submit_reset(bdev_io);
		} else {
			spdk_bdev_io_complete(bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
		}
	}
}

That distinction matters when debugging reset hangs. The wait loop asks "is any channel still busy?" The post-timeout memory-domain/accel decision is made with the channel attached to the reset I/O.

Completion also happens through bdev core. When a module completes a reset I/O, spdk_bdev_io_complete() detects the reset type and iterates channels to unfreeze them. Then bdev_reset_complete() copies the active reset status to every queued reset and completes those reset I/Os on their original threads.

/* lib/bdev/bdev.c */
void
spdk_bdev_io_complete(struct spdk_bdev_io *bdev_io, enum spdk_bdev_io_status status)
{
	struct spdk_bdev *bdev = bdev_io->bdev;

	bdev_io->internal.status = status;

	if (spdk_unlikely(bdev_io->type == SPDK_BDEV_IO_TYPE_RESET)) {
		assert(bdev_io == bdev->internal.reset_in_progress);
		spdk_bdev_for_each_channel(bdev, bdev_unfreeze_channel, bdev_io,
					   bdev_reset_complete);
		return;
	}

	/* normal completion path continues here */
}
/* lib/bdev/bdev.c */
static void
bdev_reset_complete(struct spdk_bdev *bdev, void *_ctx, int status)
{
	struct spdk_bdev_io *bdev_io = _ctx;
	bdev_io_tailq_t queued_resets;
	struct spdk_bdev_io *queued_reset;

	assert(bdev_io == bdev->internal.reset_in_progress);

	TAILQ_INIT(&queued_resets);

	spdk_spin_lock(&bdev->internal.spinlock);
	TAILQ_SWAP(&bdev->internal.queued_resets, &queued_resets,
		   spdk_bdev_io, internal.link);
	bdev->internal.reset_in_progress = NULL;
	spdk_spin_unlock(&bdev->internal.spinlock);

	while (!TAILQ_EMPTY(&queued_resets)) {
		queued_reset = TAILQ_FIRST(&queued_resets);
		TAILQ_REMOVE(&queued_resets, queued_reset, internal.link);
		queued_reset->internal.status = bdev_io->internal.status;
		spdk_thread_send_msg(spdk_bdev_io_get_thread(queued_reset),
				     _bdev_reset_complete, queued_reset);
	}

	_bdev_reset_complete(bdev_io);
}

Reset edge cases:

  • Reset while reset is already active does not create concurrent module resets; it queues behind the active one.
  • New I/O during reset is aborted before module submission.
  • Queued NOMEM, iobuf, and QoS I/O are aborted because they are still core-owned.
  • If the drain timeout is nonzero and all I/O drains before timeout, SPDK can complete reset successfully without forwarding it to the module.
  • If memory-domain or accel work remains after timeout, reset fails instead of pretending it can safely abort that work.
  • A virtual bdev's module still decides what SPDK_BDEV_IO_TYPE_RESET means for its bases.

Remove And Unregister

Remove has two related ideas:

  • A module or control plane intentionally unregisters a bdev.
  • A lower device disappears and upper users receive a remove event.

The bdev module API docs say spdk_bdev_unregister() notifies open descriptors of hotremoval and asks upper layers to close their descriptors. Actual unregistration may be deferred until descriptors are closed. The same docs currently mark calling unregister from arbitrary threads as deprecated and say it should be called from the SPDK app thread. spdk_bdev_unregister_by_name() is recommended for external deletion because it opens the named bdev, checks module ownership, calls unregister, and closes its temporary descriptor.

The first important core behavior is that descriptor event callbacks are deferred. SPDK posts an event message instead of invoking a remove callback while unregister still holds internal state. That avoids recursive unregister/close surprises.

/* lib/bdev/bdev.c */
static void
_remove_notify(void *arg)
{
	struct spdk_bdev_desc *desc = arg;

	_event_notify(desc, SPDK_BDEV_EVENT_REMOVE);
}

static int
bdev_unregister_unsafe(struct spdk_bdev *bdev)
{
	struct spdk_bdev_desc	*desc, *tmp;
	int			rc = 0;

	assert(spdk_spin_held(&g_bdev_mgr.spinlock));
	assert(spdk_spin_held(&bdev->internal.spinlock));

	/* Notify each descriptor about hotremoval */
	TAILQ_FOREACH_SAFE(desc, &bdev->internal.open_descs, link, tmp) {
		rc = -EBUSY;
		event_notify(desc, _remove_notify);
	}

	if (bdev->internal.qos_mod_in_progress) {
		rc = -EBUSY;
	}

	/* If there are no descriptors, proceed removing the bdev */
	if (rc == 0) {
		bdev_alias_del_all(bdev, bdev_name_del_unsafe);
		TAILQ_REMOVE(&g_bdev_mgr.bdevs, bdev, internal.link);
		bdev_name_del_unsafe(&bdev->internal.bdev_name);
		spdk_notify_send("bdev_unregister", spdk_bdev_get_name(bdev));
	}

	return rc;
}

The public unregister path sets status, stores the callback and callback thread, stops queue-depth sampling, aborts queued work on every channel, and then calls the unsafe removal logic after channel iteration. If descriptors remain open, bdev_unregister_unsafe() returns -EBUSY; destruction is deferred.

/* lib/bdev/bdev.c */
void
spdk_bdev_unregister(struct spdk_bdev *bdev, spdk_bdev_unregister_cb cb_fn, void *cb_arg)
{
	struct spdk_thread	*thread;

	thread = spdk_get_thread();
	if (!thread) {
		if (cb_fn != NULL) {
			cb_fn(cb_arg, -ENOTSUP);
		}
		return;
	}

	spdk_spin_lock(&g_bdev_mgr.spinlock);
	if (bdev->internal.status == SPDK_BDEV_STATUS_UNREGISTERING ||
	    bdev->internal.status == SPDK_BDEV_STATUS_REMOVING) {
		spdk_spin_unlock(&g_bdev_mgr.spinlock);
		if (cb_fn) {
			cb_fn(cb_arg, -EBUSY);
		}
		return;
	}

	spdk_spin_lock(&bdev->internal.spinlock);
	bdev->internal.status = SPDK_BDEV_STATUS_UNREGISTERING;
	bdev->internal.unregister_cb = cb_fn;
	bdev->internal.unregister_ctx = cb_arg;
	bdev->internal.unregister_td = thread;
	spdk_spin_unlock(&bdev->internal.spinlock);
	spdk_spin_unlock(&g_bdev_mgr.spinlock);

	spdk_bdev_set_qd_sampling_period(bdev, 0);

	spdk_bdev_for_each_channel(bdev, bdev_unregister_abort_channel, bdev,
				   bdev_unregister);
}

For code that receives a name from RPC or a module-specific delete path, spdk_bdev_unregister_by_name() avoids deleting someone else's bdev by checking the module pointer.

/* lib/bdev/bdev.c */
int
spdk_bdev_unregister_by_name(const char *bdev_name, struct spdk_bdev_module *module,
			     spdk_bdev_unregister_cb cb_fn, void *cb_arg)
{
	struct spdk_bdev_desc *desc;
	struct spdk_bdev *bdev;
	int rc;

	rc = spdk_bdev_open_ext(bdev_name, false, _tmp_bdev_event_cb, NULL, &desc);
	if (rc != 0) {
		return rc;
	}

	bdev = spdk_bdev_desc_get_bdev(desc);

	if (bdev->module != module) {
		spdk_bdev_close(desc);
		return -ENODEV;
	}

	spdk_bdev_unregister(bdev, cb_fn, cb_arg);
	spdk_bdev_close(desc);

	return 0;
}

Misconception to kill: remove notification does not magically close every descriptor. The descriptor owner must stop submitting I/O and close its descriptor. Otherwise SPDK cannot finish destruction.

Descriptor Event Callback: What A Virtual Bdev Does

Virtual bdev modules depend on descriptor events. The passthru sample is intentionally simple: it opens the base bdev with an event callback, and if that base is removed it unregisters the virtual bdev built on top.

/* module/bdev/passthru/vbdev_passthru.c */
static void
vbdev_passthru_base_bdev_hotremove_cb(struct spdk_bdev *bdev_find)
{
	struct vbdev_passthru *pt_node, *tmp;

	TAILQ_FOREACH_SAFE(pt_node, &g_pt_nodes, link, tmp) {
		if (bdev_find == pt_node->base_bdev) {
			spdk_bdev_unregister(&pt_node->pt_bdev, NULL, NULL);
		}
	}
}

static void
vbdev_passthru_base_bdev_event_cb(enum spdk_bdev_event_type type, struct spdk_bdev *bdev,
				  void *event_ctx)
{
	switch (type) {
	case SPDK_BDEV_EVENT_REMOVE:
		vbdev_passthru_base_bdev_hotremove_cb(bdev);
		break;
	default:
		SPDK_NOTICELOG("Unsupported bdev event: type %d\n", type);
		break;
	}
}

The same module also shows the "base bdev appears later" pattern. It records the requested base/virtual names even if the base does not exist yet, then examines every newly registered bdev and tries to finish construction.

/* module/bdev/passthru/vbdev_passthru.c */
int
bdev_passthru_create_disk(const char *bdev_name, const char *vbdev_name,
			  const struct spdk_uuid *uuid)
{
	int rc;

	rc = vbdev_passthru_insert_name(bdev_name, vbdev_name, uuid);
	if (rc) {
		return rc;
	}

	rc = vbdev_passthru_register(bdev_name);
	if (rc == -ENODEV) {
		SPDK_NOTICELOG("vbdev creation deferred pending base bdev arrival\n");
		rc = 0;
	}

	return rc;
}

static void
vbdev_passthru_examine(struct spdk_bdev *bdev)
{
	vbdev_passthru_register(bdev->name);

	spdk_bdev_module_examine_done(&passthru_if);
}

That is the bdev-layer version of hotplug for virtual devices. It is not PCIe hotplug. It is a module reacting to the bdev graph changing.

Delay is the same kind of graph reaction, but it has one extra reset concern: it may have already received base completions and be holding the original virtual I/O in a delayed completion queue. On reset, vbdev_delay_reset_channel() walks each delay channel and aborts queued delayed I/O before vbdev_delay_reset_dev() forwards reset to the base bdev. That is why virtual modules with their own queues need reset logic beyond "pass reset down."

NVMe Hotplug And Hotremove

At the bdev level, hotremove becomes remove events and unregister. At the NVMe module level, PCIe hotplug and removal are detected by pollers around the NVMe library. The JSON-RPC front door is bdev_nvme_set_hotplug.

/* module/bdev/nvme/bdev_nvme_rpc.c */
struct rpc_bdev_nvme_hotplug {
	bool enabled;
	uint64_t period_us;
};

static const struct spdk_json_object_decoder rpc_bdev_nvme_set_hotplug_decoders[] = {
	{"enable", offsetof(struct rpc_bdev_nvme_hotplug, enabled), spdk_json_decode_bool, false},
	{"period_us", offsetof(struct rpc_bdev_nvme_hotplug, period_us), spdk_json_decode_uint64, true},
};

static void
rpc_bdev_nvme_set_hotplug(struct spdk_jsonrpc_request *request,
			  const struct spdk_json_val *params)
{
	struct rpc_bdev_nvme_hotplug req = {false, 0};
	int rc;

	if (spdk_json_decode_object(params, rpc_bdev_nvme_set_hotplug_decoders,
				    SPDK_COUNTOF(rpc_bdev_nvme_set_hotplug_decoders), &req)) {
		rc = -EINVAL;
		goto invalid;
	}

	rc = bdev_nvme_set_hotplug(req.enabled, req.period_us);
	if (rc) {
		goto invalid;
	}

	spdk_jsonrpc_send_bool_response(request, true);
	return;
invalid:
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS, spdk_strerror(-rc));
}

The implementation is primary-process sensitive. Enabling hotplug from a secondary process returns -EPERM, because PCIe probing and ownership are not just ordinary shared state. Disabling hotplug does not mean "never notice removals"; the code switches to the remove poller so attached devices can still be checked.

/* module/bdev/nvme/bdev_nvme.c */
int
bdev_nvme_set_hotplug(bool enabled, uint64_t period_us)
{
	assert(spdk_thread_is_app_thread(NULL));

	if (enabled == true && !spdk_process_is_primary()) {
		return -EPERM;
	}

	period_us = period_us == 0 ? NVME_HOTPLUG_POLL_PERIOD_DEFAULT : period_us;
	period_us = spdk_min(period_us, NVME_HOTPLUG_POLL_PERIOD_MAX);

	spdk_poller_unregister(&g_hotplug_poller);
	if (enabled) {
		g_hotplug_poller = SPDK_POLLER_REGISTER(bdev_nvme_hotplug, NULL, period_us);
	} else {
		g_hotplug_poller = SPDK_POLLER_REGISTER(bdev_nvme_remove_poller, NULL,
							NVME_HOTPLUG_POLL_PERIOD_DEFAULT);
	}

	g_nvme_hotplug_poll_period_us = period_us;
	g_nvme_hotplug_enabled = enabled;
	return 0;
}

The hotplug poller starts asynchronous PCIe probing. The remove poller scans already attached PCIe devices. When lower NVMe library callbacks discover add/remove effects, the NVMe bdev module creates or removes bdevs, which then flows back into the bdev event/unregister machinery described above.

/* module/bdev/nvme/bdev_nvme.c */
static int
bdev_nvme_hotplug(void *arg)
{
	struct spdk_nvme_transport_id trid_pcie;

	if (g_hotplug_probe_ctx) {
		return SPDK_POLLER_BUSY;
	}

	memset(&trid_pcie, 0, sizeof(trid_pcie));
	spdk_nvme_trid_populate_transport(&trid_pcie, SPDK_NVME_TRANSPORT_PCIE);

	g_hotplug_probe_ctx = spdk_nvme_probe_async(&trid_pcie, NULL,
			      hotplug_probe_cb, attach_cb, NULL);

	if (g_hotplug_probe_ctx) {
		assert(g_hotplug_probe_poller == NULL);
		g_hotplug_probe_poller = SPDK_POLLER_REGISTER(bdev_nvme_hotplug_probe, NULL, 1000);
	}

	return SPDK_POLLER_BUSY;
}

static int
bdev_nvme_remove_poller(void *ctx)
{
	struct spdk_nvme_transport_id trid_pcie;

	if (TAILQ_EMPTY(&g_nvme_bdev_ctrlrs)) {
		spdk_poller_unregister(&g_hotplug_poller);
		return SPDK_POLLER_IDLE;
	}

	memset(&trid_pcie, 0, sizeof(trid_pcie));
	spdk_nvme_trid_populate_transport(&trid_pcie, SPDK_NVME_TRANSPORT_PCIE);

	if (spdk_nvme_scan_attached(&trid_pcie)) {
		SPDK_ERRLOG_RATELIMIT("spdk_nvme_scan_attached() failed\n");
	}

	return SPDK_POLLER_BUSY;
}

The JSON-RPC docs also expose bdev_nvme_detach_controller, which intentionally detaches a controller and deletes associated bdevs, and bdev_nvme_reset_controller, which resets an NVMe controller. Those controller-level RPCs are related to this chapter but not identical to bdev reset: they live in the NVMe module/control plane, while spdk_bdev_reset() is the generic bdev I/O type coordinated by bdev core.

Media Events

Some bdevs expose media-management events. The public bdev API docs for spdk_bdev_get_media_events() say it can only be called from the context of SPDK_BDEV_EVENT_MEDIA_MANAGEMENT, and the module API docs say a module pushes events and then calls spdk_bdev_notify_media_management() to notify descriptors with pending events.

The core model is descriptor-owned buffering. spdk_bdev_push_media_events() finds a writable descriptor with a media event buffer, moves events from that descriptor's free queue to its pending queue, and returns the number pushed. If no suitable descriptor exists, it returns -ENODEV.

/* lib/bdev/bdev.c */
int
spdk_bdev_push_media_events(struct spdk_bdev *bdev, const struct spdk_bdev_media_event *events,
			    size_t num_events)
{
	struct spdk_bdev_desc *desc;
	struct media_event_entry *entry;
	size_t event_id;
	int rc = 0;

	assert(bdev->media_events);

	spdk_spin_lock(&bdev->internal.spinlock);
	TAILQ_FOREACH(desc, &bdev->internal.open_descs, link) {
		if (desc->write) {
			break;
		}
	}

	if (desc == NULL || desc->media_events_buffer == NULL) {
		rc = -ENODEV;
		goto out;
	}

	for (event_id = 0; event_id < num_events; ++event_id) {
		entry = TAILQ_FIRST(&desc->free_media_events);
		if (entry == NULL) {
			break;
		}

		TAILQ_REMOVE(&desc->free_media_events, entry, tailq);
		TAILQ_INSERT_TAIL(&desc->pending_media_events, entry, tailq);
		entry->event = events[event_id];
	}

	rc = event_id;
out:
	spdk_spin_unlock(&bdev->internal.spinlock);
	return rc;
}

Notification is separate from pushing. That separation lets a module batch events into a descriptor buffer, then deliver one descriptor event to tell the opener to call spdk_bdev_get_media_events().

/* lib/bdev/bdev.c */
size_t
spdk_bdev_get_media_events(struct spdk_bdev_desc *desc, struct spdk_bdev_media_event *events,
			   size_t max_events)
{
	struct media_event_entry *entry;
	size_t num_events = 0;

	for (; num_events < max_events; ++num_events) {
		entry = TAILQ_FIRST(&desc->pending_media_events);
		if (entry == NULL) {
			break;
		}

		events[num_events] = entry->event;
		TAILQ_REMOVE(&desc->pending_media_events, entry, tailq);
		TAILQ_INSERT_TAIL(&desc->free_media_events, entry, tailq);
	}

	return num_events;
}

void
spdk_bdev_notify_media_management(struct spdk_bdev *bdev)
{
	struct spdk_bdev_desc *desc;

	spdk_spin_lock(&bdev->internal.spinlock);
	TAILQ_FOREACH(desc, &bdev->internal.open_descs, link) {
		if (!TAILQ_EMPTY(&desc->pending_media_events)) {
			event_notify(desc, _media_management_notify);
		}
	}
	spdk_spin_unlock(&bdev->internal.spinlock);
}

Misconception to kill: media events are not I/O completions. They are descriptor events. The opener receives an event callback and then drains pending events from the descriptor.

Quiesce And Locked Ranges

Quiesce lets the registering module temporarily stop I/O for a whole bdev or for an LBA range. The official module API docs state two important rules:

  • Only the module that registered the bdev may call quiesce/unquiesce.
  • spdk_bdev_unquiesce_range() must match exactly a previously quiesced range.

Quiesce is built on the same locked-range machinery used by explicit LBA locks. When an I/O arrives, bdev_io_submit() checks the channel's locked ranges before adding the I/O to the submitted list. A blocked I/O is put on io_locked and retried later.

/* lib/bdev/bdev.c */
void
bdev_io_submit(struct spdk_bdev_io *bdev_io)
{
	struct spdk_bdev_channel *ch = bdev_io->internal.ch;

	assert(bdev_io->internal.status == SPDK_BDEV_IO_STATUS_PENDING);

	if (!bdev_io->internal.f.child_io && !TAILQ_EMPTY(&ch->locked_ranges)) {
		struct lba_range *range;

		TAILQ_FOREACH(range, &ch->locked_ranges, tailq) {
			if (bdev_io_range_is_locked(bdev_io, range)) {
				TAILQ_INSERT_TAIL(&ch->io_locked, bdev_io, internal.ch_link);
				return;
			}
		}
	}

	bdev_ch_add_to_io_submitted(bdev_io);
	bdev_io->internal.submit_tsc = spdk_get_ticks();
	_bdev_io_submit(bdev_io);
}

The overlap logic is conservative. For NVMe passthrough, the bdev layer does not decode the command; it assumes worst-case overlap with the locked range. For reads, an ordinary lock may allow reads, but a quiesce range blocks reads too. Writes and modifying commands are blocked when they overlap unless they come from the same channel/context that owns the lock.

/* lib/bdev/bdev.c */
static bool
bdev_io_range_is_locked(struct spdk_bdev_io *bdev_io, struct lba_range *range)
{
	struct spdk_bdev_channel *ch = bdev_io->internal.ch;
	struct lba_range r;

	switch (bdev_io->type) {
	case SPDK_BDEV_IO_TYPE_NVME_IO:
	case SPDK_BDEV_IO_TYPE_NVME_IO_MD:
		return true;
	case SPDK_BDEV_IO_TYPE_READ:
		if (!range->quiesce) {
			return false;
		}
	/* fallthrough */
	case SPDK_BDEV_IO_TYPE_WRITE:
	case SPDK_BDEV_IO_TYPE_WRITE_UNCORRECTABLE:
	case SPDK_BDEV_IO_TYPE_UNMAP:
	case SPDK_BDEV_IO_TYPE_WRITE_ZEROES:
	case SPDK_BDEV_IO_TYPE_ZCOPY:
	case SPDK_BDEV_IO_TYPE_COPY:
		r.offset = bdev_io->u.bdev.offset_blocks;
		r.length = bdev_io->u.bdev.num_blocks;
		if (!bdev_lba_range_overlapped(range, &r)) {
			return false;
		} else if (range->owner_ch == ch && range->locked_ctx == bdev_io->internal.caller_ctx) {
			return false;
		} else {
			return true;
		}
	default:
		return false;
	}
}

The public quiesce helpers are thin wrappers around _spdk_bdev_quiesce(). Full-bdev quiesce is just a range from block 0 through bdev->blockcnt.

/* lib/bdev/bdev.c */
int
spdk_bdev_quiesce(struct spdk_bdev *bdev, struct spdk_bdev_module *module,
		  spdk_bdev_quiesce_cb cb_fn, void *cb_arg)
{
	return _spdk_bdev_quiesce(bdev, module, 0, bdev->blockcnt, cb_fn, cb_arg, false);
}

int
spdk_bdev_unquiesce(struct spdk_bdev *bdev, struct spdk_bdev_module *module,
		    spdk_bdev_quiesce_cb cb_fn, void *cb_arg)
{
	return _spdk_bdev_quiesce(bdev, module, 0, bdev->blockcnt, cb_fn, cb_arg, true);
}

int
spdk_bdev_quiesce_range(struct spdk_bdev *bdev, struct spdk_bdev_module *module,
			uint64_t offset, uint64_t length,
			spdk_bdev_quiesce_cb cb_fn, void *cb_arg)
{
	return _spdk_bdev_quiesce(bdev, module, offset, length, cb_fn, cb_arg, false);
}

The exact-match rule is visible in the unquiesce branch: SPDK searches the registering module's quiesced_ranges list for the same bdev, offset, and length. If it cannot find that range, unquiesce returns -EINVAL.

/* lib/bdev/bdev.c */
if (unquiesce) {
	struct lba_range *range;

	spdk_spin_lock(&module->internal.spinlock);
	TAILQ_FOREACH(range, &module->internal.quiesced_ranges, tailq_module) {
		if (range->bdev == bdev && range->offset == offset && range->length == length) {
			TAILQ_REMOVE(&module->internal.quiesced_ranges, range, tailq_module);
			break;
		}
	}
	spdk_spin_unlock(&module->internal.spinlock);

	if (range == NULL) {
		SPDK_ERRLOG("The range to unquiesce was not found.\n");
		return -EINVAL;
	}

	quiesce_ctx = range->locked_ctx;
	quiesce_ctx->cb_fn = cb_fn;
	quiesce_ctx->cb_arg = cb_arg;

	rc = _bdev_unlock_lba_range(bdev, offset, length, bdev_unquiesce_range_unlocked, quiesce_ctx);
}

Unlock resubmits queued I/O by moving io_locked to a temporary list and calling bdev_io_submit() again. That retry path is intentionally simple: it retries all locked I/O on the channel rather than trying to micro-optimize only I/O overlapping the just-unlocked range.

/* lib/bdev/bdev.c */
TAILQ_INIT(&io_locked);
TAILQ_SWAP(&ch->io_locked, &io_locked, spdk_bdev_io, internal.ch_link);
while (!TAILQ_EMPTY(&io_locked)) {
	bdev_io = TAILQ_FIRST(&io_locked);
	TAILQ_REMOVE(&io_locked, bdev_io, internal.ch_link);
	bdev_io_submit(bdev_io);
}

Combined Failure Modes

The hard production bugs often involve more than one mechanism:

  • QoS plus reset: reset aborts qos_queued_io, so users may see aborted I/O that never reached the module.
  • Remove plus open descriptors: unregister sends remove events but final destruction waits for descriptors to close.
  • Remove plus reset: bdev core defers destruction if reset_in_progress is still set, then reset completion checks whether the bdev can be destroyed.
  • Quiesce plus unregister: unregister waits if locked ranges exist; unlock continues unregister when the range list becomes empty.
  • Media events plus missing writable descriptor: pushing events fails with -ENODEV; there is no descriptor-owned buffer to hold them.
  • NVMe hotplug disabled: insertion polling is disabled, but removal polling may still run.
  • NVMe passthrough plus quiesce: the core assumes passthrough overlaps a locked range because it does not decode the command.

Misconceptions To Kill

  • "Delete means freed immediately." No. Open descriptors, reset, QoS changes, locked ranges, and async destruct can delay final free.
  • "Hotremove is only a PCIe problem." No. Any base bdev removal matters to virtual bdevs.
  • "QoS changes hardware queue depth." No. It queues at the bdev layer before module submission.
  • "Reset only affects the caller's channel." No. bdev reset iterates all channels.
  • "Quiesce is for applications." No. The module API says only the registering module may call it.
  • "Media events are completions." No. They are descriptor events, separate from I/O completion callbacks.
  • "A virtual bdev automatically disappears when its base disappears." No. The virtual module must handle SPDK_BDEV_EVENT_REMOVE and unregister itself.

Source Reading Exercise

Read these paths in order:

  1. QoS front door: lib/bdev/bdev_rpc.c:rpc_bdev_set_qos_limit() -> lib/bdev/bdev.c:spdk_bdev_set_qos_rate_limits().
  2. QoS data path: lib/bdev/bdev.c:_bdev_io_submit() -> lib/bdev/bdev.c:bdev_qos_queue_io() -> lib/bdev/bdev.c:bdev_channel_poll_qos().
  3. Reset: lib/bdev/bdev.c:spdk_bdev_reset() -> lib/bdev/bdev.c:bdev_start_reset() -> lib/bdev/bdev.c:bdev_reset_freeze_channel() -> lib/bdev/bdev.c:spdk_bdev_io_complete() -> lib/bdev/bdev.c:bdev_reset_complete().
  4. Remove: lib/bdev/bdev.c:spdk_bdev_unregister() -> lib/bdev/bdev.c:bdev_unregister_unsafe() -> lib/bdev/bdev.c:spdk_bdev_close().
  5. Virtual hotremove: module/bdev/passthru/vbdev_passthru.c:vbdev_passthru_base_bdev_event_cb().
  6. NVMe hotplug: module/bdev/nvme/bdev_nvme_rpc.c:rpc_bdev_nvme_set_hotplug() -> module/bdev/nvme/bdev_nvme.c:bdev_nvme_set_hotplug() -> module/bdev/nvme/bdev_nvme.c:bdev_nvme_hotplug() -> module/bdev/nvme/bdev_nvme.c:bdev_nvme_remove_poller().
  7. Media events: lib/bdev/bdev.c:spdk_bdev_push_media_events() -> lib/bdev/bdev.c:spdk_bdev_notify_media_management() -> lib/bdev/bdev.c:spdk_bdev_get_media_events().
  8. Quiesce: lib/bdev/bdev.c:spdk_bdev_quiesce_range() -> lib/bdev/bdev.c:_spdk_bdev_quiesce() -> lib/bdev/bdev.c:bdev_io_range_is_locked() -> lib/bdev/bdev.c:_bdev_unlock_lba_range().