The Reader Contract: How To Learn SPDK Without Drowning

The problem with reading SPDK like normal application code

If you read SPDK as if it were a request/response web server, it will hurt. A web handler often receives a request, calls functions, returns a response, and the stack unwinds. SPDK code often receives an event, allocates a context object, submits work, returns immediately, and finishes later in a callback that may run on the same spdk_thread or a different one. The call stack is not the story. The state machine is the story.

This is why the code is full of small structs named ctx, req, task, iter, cb_arg, and *_ctx. Those objects are the synthetic stack frames for async C. They carry the variables that a blocking implementation would have kept on the stack.

The official SPDK concurrency documentation describes the same model from the runtime side: SPDK tries to avoid shared mutable data protected by locks, assigns data to an owning thread when possible, and sends a small message to that owner when another thread needs work done. It also calls spdk_thread a lightweight, stackless thread of execution. That word "stackless" is the key to reading the code. If a function returns before the work is done, then the real continuation cannot be on the C stack. It must be in a heap object, a queued message, a poller, a completion callback, or some combination of those.

This book's reader contract is therefore simple: every time a line of SPDK code feels mysterious, stop trying to make the call stack explain it. Instead, identify the object, the owner, the current thread, the callback path, and the cleanup path. Those five facts usually explain the code's shape.

The public thread API says this directly:

/* include/spdk/thread.h */
/**
 * Send a message to the given thread.
 *
 * The message will be sent asynchronously - i.e. spdk_thread_send_msg will always return
 * prior to `fn` being called.
 *
 * Errors are handled internally and are fatal. Calling code can skip checking the return
 * value as it has been left only for compatibility.
 *
 * \param thread The target thread.
 * \param fn This function will be called on the given thread.
 * \param ctx This context will be passed to fn when called.
 *
 * \return 0 left for API compatibility
 */
int spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx);

That one comment prevents a common beginner mistake. spdk_thread_send_msg() returning 0 does not mean fn(ctx) has already run. It means the runtime accepted responsibility for arranging that future callback. The object behind ctx must remain valid until the callback runs, and the callback must know whether it owns that object and when to free it.

The five-question reading loop

Every time you enter a new SPDK function, ask these questions.

1. What object is this about?

Examples:

struct spdk_bdev
struct spdk_bdev_io
struct spdk_bdev_desc
struct spdk_io_channel
struct spdk_thread
struct spdk_poller
struct spdk_nvme_ctrlr
struct spdk_nvme_qpair
struct spdk_lvol
struct spdk_blob
struct spdk_nvmf_request
struct spdk_nvmf_qpair

Do not start with helpers. Start with the object. Find its struct definition. Read the fields. Look for embedded TAILQ_ENTRY, RB_ENTRY, reference counts, state enums, callbacks, and owner pointers.

This is not just a navigation trick. In SPDK, the object usually tells you what kind of lifetime you are looking at. A struct spdk_bdev represents a registered block device and its module-facing identity. A struct spdk_bdev_io represents one request flowing through the bdev layer. A struct spdk_io_channel represents per-thread access to an I/O device. A struct spdk_poller represents repeated work on a thread. If you confuse those lifetimes, the code will look arbitrary.

For example, when you see struct spdk_bdev_io *bdev_io, treat it as the request object. It carries the operation type, the bdev, the caller's callback, the caller's callback argument, the channel, status, and internal bookkeeping. The module does not usually allocate this object itself. The bdev layer allocates and initializes it, then calls the module's submit_request function. That means the module's obligation is not "return a value." Its obligation is "eventually complete this request exactly once."

2. Who owns it?

Ownership in SPDK usually means one or more of:

The current spdk_thread owns access to mutable state.
A module owns a bdev it registered.
A descriptor owns an open claim.
A channel owns per-thread resources.
A controller owns qpairs and namespaces.
A request owns child IOs until completion.
A callback context owns heap memory until the terminal callback frees it.

Ownership bugs are the source of many hard failures. A wrong-thread channel put is not a style issue. It is a correctness issue.

The practical test for ownership is: who is allowed to mutate this field right now? SPDK often answers "the current spdk_thread" instead of "whoever has a pointer." A pointer proves reachability, not permission. If code needs to mutate an object owned by another thread, it should send a message to that thread or use an API that does that internally.

spdk_io_channel is the clearest example. The SPDK concurrency docs describe it as per-thread context associated with an spdk_io_device; the bdev module guide says the bdev layer calls get_io_channel once per thread, caches the result, and passes that thread's channel to submit_request. That is why channel create/destroy functions, channel iteration, and channel puts are so sensitive to thread context. A channel is not just a handle. It is a thread-local lane of access.

When ownership transfers across a callback boundary, write down the rule in plain English before you trust your understanding. Example: "The bdev layer owns bdev_io until it invokes my completion callback; inside the callback I may inspect it and must call spdk_bdev_free_io() when the public API requires that." The exact free rule varies by API, so do not generalize blindly. The habit is what matters.

3. Is this synchronous or asynchronous?

SPDK functions often return an integer that only tells you whether submission succeeded. The actual operation completes later. Examples:

bdev IO submission returns before IO completion.
lvol create/delete/resize uses callbacks.
blobstore metadata operations use callbacks.
subsystem initialization is async and must call the next init step.
config replay sends JSON-RPC requests and waits for responses through a poller.

If a function takes a callback and callback argument, assume the return value is not the final result unless the documentation explicitly says otherwise.

This distinction is the source of many wrong mental models:

Submission success means the request was accepted for execution.
Operation success means the terminal callback reported success.
Cleanup success means all references and temporary resources were released after completion.

Those are three different facts. One request can pass the first and fail the second. Another can fail before submission and still require a callback so the caller's state machine can make progress. Another can complete successfully but leak a context object if the terminal path forgets to free it.

The bdev completion callback type makes the operation-result boundary visible:

/* include/spdk/bdev.h */
/**
 * Block device completion callback.
 *
 * \param bdev_io Block device I/O that has completed.
 * \param success True if I/O completed successfully or false if it failed;
 * additional error information may be retrieved from bdev_io by calling
 * spdk_bdev_io_get_nvme_status() or spdk_bdev_io_get_scsi_status().
 * \param cb_arg Callback argument specified when bdev_io was submitted.
 */
typedef void (*spdk_bdev_io_completion_cb)(struct spdk_bdev_io *bdev_io,
		bool success,
		void *cb_arg);

Notice what the callback receives: the completed request, a simplified success boolean, and the caller's original context. That context is the manual stack frame. If the caller submitted several child I/Os, the context may include an outstanding count. If it submitted one I/O as part of a larger state machine, the context may include the next state. If the callback is terminal, it may free the context. The callback is not an afterthought; it is where the synchronous version of the function would have resumed.

4. Which thread must run the next step?

SPDK avoids locks by moving work to the owner thread. That means you must trace messages:

spdk_thread_send_msg(thread, fn, ctx);

This does not call fn immediately unless you are using a helper that explicitly executes inline for the current thread. It enqueues a message to another spdk_thread. The callback runs when that target thread is polled.

The implementation reinforces the API comment. spdk_thread_send_msg() allocates or reuses a message object, stores the function pointer and context pointer in it, enqueues it on the target thread's ring, and notifies the target thread. It does not run the function:

/* lib/thread/thread.c */
msg->fn = fn;
msg->arg = ctx;

rc = spdk_ring_enqueue(thread->messages, (void **)&msg, 1, NULL);
if (rc != 1) {
	SPDK_ERRLOG("msg could not be enqueued\n");
	abort();
}

thread_send_msg_notification(thread);

return 0;

The other half of the story is spdk_thread_poll(). Polling temporarily makes the target spdk_thread current in thread-local storage, runs messages and pollers through thread_poll(), handles exit work, updates statistics, and restores the previous current thread:

/* lib/thread/thread.c */
spdk_thread_poll(struct spdk_thread *thread, uint32_t max_msgs, uint64_t now)
{
	struct spdk_thread *orig_thread;
	int rc;

	orig_thread = _get_thread();
	tls_thread = thread;

	if (now == 0) {
		now = spdk_get_ticks();
	}

	if (spdk_likely(!thread->in_interrupt)) {
		rc = thread_poll(thread, max_msgs, now);
		if (spdk_unlikely(thread->state == SPDK_THREAD_STATE_EXITING)) {
			thread_exit(thread, now);
		}
	} else {
		rc = spdk_fd_group_wait(thread->fgrp, 0);
	}

	thread_update_stats(thread, spdk_get_ticks(), now, rc);
	tls_thread = orig_thread;

	return rc;
}

That temporary tls_thread = thread assignment is why many assertions read like "this function must run on the owning SPDK thread." The code is not asking which pthread happens to be executing. It is asking which SPDK lightweight thread is current during this poll slice.

When you trace a bug, draw the message path as a sequence:

current callback -> spdk_thread_send_msg(owner, next_fn, ctx)
owner poll slice -> next_fn(ctx)
next_fn -> completes, sends another message, or arms a poller

If a step is missing, the state machine stalls. If a step runs on the wrong thread, the assertions may fire or the per-thread data may be corrupted.

5. Who completes and who frees?

Every async path needs a terminal event. Find it.

For bdev IO, the terminal event is usually:

spdk_bdev_io_complete(bdev_io, status);

For JSON-RPC, it may be:

spdk_jsonrpc_send_result(request, w);

For subsystem init:

spdk_subsystem_init_next(rc);

For channel iteration:

spdk_for_each_channel_continue(i, status);

When debugging a hang, ask: which required completion was never called?

Completion is usually more than a callback invocation. It often updates accounting, removes the request from an outstanding list, handles retries, resumes waiters, or sends the callback to the original thread. In bdev, the public module-facing completion API starts by validating that the I/O is still pending and then records the terminal status:

/* lib/bdev/bdev.c */
void
spdk_bdev_io_complete(struct spdk_bdev_io *bdev_io, enum spdk_bdev_io_status status)
{
	struct spdk_bdev *bdev = bdev_io->bdev;
	struct spdk_bdev_channel *bdev_ch = bdev_io->internal.ch;
	struct spdk_bdev_shared_resource *shared_resource = bdev_ch->shared_resource;

	if (spdk_unlikely(bdev_io->internal.status != SPDK_BDEV_IO_STATUS_PENDING)) {
		SPDK_ERRLOG("Unexpected completion on IO from %s module, status was %s\n",
			    spdk_bdev_get_module_name(bdev),
			    bdev_io_status_get_string(bdev_io->internal.status));
		assert(false);
	}
	bdev_io->internal.status = status;

	if (spdk_unlikely(bdev_io->type == SPDK_BDEV_IO_TYPE_RESET)) {
		assert(bdev_io == bdev->internal.reset_in_progress);
		spdk_bdev_for_each_channel(bdev, bdev_unfreeze_channel, bdev_io,
					   bdev_reset_complete);
		return;
	}

That status != PENDING assertion is a useful reading clue. The bdev layer expects exactly one terminal completion for each submitted I/O. Double-completion is not harmless. Missing completion is not harmless. The first corrupts ownership; the second strands the caller's state machine.

The final callback handoff asserts that the code is back on the original bdev I/O thread before invoking the caller's callback:

/* lib/bdev/bdev.c */
static inline void
_bdev_io_complete(void *ctx)
{
	struct spdk_bdev_io *bdev_io = ctx;

	if (spdk_unlikely(bdev_io_use_accel_sequence(bdev_io))) {
		assert(bdev_io->internal.status != SPDK_BDEV_IO_STATUS_SUCCESS);
		spdk_accel_sequence_abort(bdev_io->internal.accel_sequence);
	}

	assert(bdev_io->internal.cb != NULL);
	assert(spdk_get_thread() == spdk_bdev_io_get_thread(bdev_io));

	bdev_io->internal.cb(bdev_io, bdev_io->internal.status == SPDK_BDEV_IO_STATUS_SUCCESS,
			     bdev_io->internal.caller_ctx);
}

This is the callback contract in code form: complete once, on the right thread, with the caller's context preserved.

Why edge cases matter more than happy path

Storage systems spend most of their complexity on edge cases:

device disappears
controller resets
queue is full
CQ has no free slots
allocation returns -ENOMEM
bdev is being removed
descriptor is still open
lvolstore metadata is not loaded yet
duplicate name after restart
reset is draining outstanding IO
channel destruction is deferred
config replay issues an RPC in the wrong phase

A happy-path-only tutorial is actively dangerous because it teaches the wrong shape. SPDK code is built around preserving invariants when these edge cases happen.

The edge case is often the real design. Consider a bdev I/O that cannot allocate a buffer immediately. The correct behavior is not necessarily "fail now." It might be "queue on a wait list and retry later." Consider a reset. The correct behavior is not "mark reset done." It might be "freeze channels, complete or abort outstanding I/O, unfreeze channels, then complete reset." Consider channel iteration. The correct behavior is not "loop through channels in a for loop." The channels live on different SPDK threads, so the iterator has to send messages and wait for each per-thread callback to explicitly continue.

Here is the channel-iteration shape:

/* lib/thread/thread.c */
if (thr_link != NULL) {
	i->dev->for_each_count++;
	i->cur_thread = thr_link->thread;
	spdk_thread_send_msg(i->cur_thread, _call_channel, i);
	pthread_mutex_unlock(&g_devlist_mutex);
	return;
}

end:
pthread_mutex_unlock(&g_devlist_mutex);

spdk_thread_send_msg(i->orig_thread, _call_completion, i);

And here is the continuation that advances to the next owning thread or returns to the original thread:

/* lib/thread/thread.c */
void
spdk_for_each_channel_continue(struct spdk_io_channel_iter *i, int status)
{
	struct spdk_thread *thread;
	struct io_device *dev;

	assert(i->cur_thread == spdk_get_thread());

	i->status = status;

	pthread_mutex_lock(&g_devlist_mutex);
	dev = i->dev;
	if (status) {
		goto end;
	}

	thread = io_dev_get_next_thread(i->dev, i->cur_thread);
	if (thread != NULL) {
		i->cur_thread = thread;
		spdk_thread_send_msg(i->cur_thread, _call_channel, i);
		pthread_mutex_unlock(&g_devlist_mutex);
		return;
	}

Now the hang question is concrete. If a channel callback never calls spdk_for_each_channel_continue(), the iterator never sends the next _call_channel message and never sends _call_completion back to the original thread. The system is not "slow"; the state machine is missing its transition.

Read error paths with this same discipline. For every return, ask whether this function has already completed the request, queued it for later, transferred ownership to another object, or left completion to its caller. A short function with three early returns can be harder than a long function if each return has a different ownership outcome.

A complete small example: null bdev

The null bdev module is a good first source tour because it has the same bdev contracts as a hardware-backed module without a device-specific transport. The official bdev module guide explicitly recommends it as a starting point for custom modules.

The module registers itself and provides a function table. The important reader move is to connect submit_request in the table to the function that must eventually complete each I/O:

/* module/bdev/null/bdev_null.c */
SPDK_BDEV_MODULE_REGISTER(null, &null_if)

static const struct spdk_bdev_fn_table null_fn_table = {
	.destruct		= bdev_null_destruct,
	.submit_request		= bdev_null_submit_request,
	.io_type_supported	= bdev_null_io_type_supported,
	.get_io_channel		= bdev_null_get_io_channel,
	.write_config_json	= bdev_null_write_config_json,
};

Inside bdev_null_submit_request(), a read with no caller-provided buffer gets pointed at the module's shared read buffer when the size is allowed. Unsupported or invalid cases complete with failure. Supported cases are inserted onto the per-channel queue:

/* module/bdev/null/bdev_null.c */
case SPDK_BDEV_IO_TYPE_READ:
	if (bdev_io->u.bdev.iovs[0].iov_base == NULL) {
		assert(bdev_io->u.bdev.iovcnt == 1);
		if (spdk_likely(bdev_io->u.bdev.num_blocks * bdev_io->bdev->blocklen <=
				SPDK_BDEV_LARGE_BUF_MAX_SIZE)) {
			bdev_io->u.bdev.iovs[0].iov_base = g_null_read_buf;
			bdev_io->u.bdev.iovs[0].iov_len = bdev_io->u.bdev.num_blocks * bdev_io->bdev->blocklen;
		} else {
			SPDK_ERRLOG("Overflow occurred. Read I/O size %" PRIu64 " was larger than permitted %d\n",
				    bdev_io->u.bdev.num_blocks * bdev_io->bdev->blocklen,
				    SPDK_BDEV_LARGE_BUF_MAX_SIZE);
			spdk_bdev_io_complete(bdev_io, SPDK_BDEV_IO_STATUS_FAILED);
			return;
		}
	}
	TAILQ_INSERT_TAIL(&ch->io, null_io, link);
	break;

That TAILQ_INSERT_TAIL is not completion. It is submission into the module's per-channel work queue. The module still owes the bdev layer a terminal spdk_bdev_io_complete().

The terminal event comes from the channel poller:

/* module/bdev/null/bdev_null.c */
static int
null_io_poll(void *arg)
{
	struct null_io_channel		*ch = arg;
	TAILQ_HEAD(, null_bdev_io)	io;
	struct null_bdev_io		*null_io;

	TAILQ_INIT(&io);
	TAILQ_SWAP(&ch->io, &io, null_bdev_io, link);

	if (TAILQ_EMPTY(&io)) {
		return SPDK_POLLER_IDLE;
	}

	while (!TAILQ_EMPTY(&io)) {
		null_io = TAILQ_FIRST(&io);
		TAILQ_REMOVE(&io, null_io, link);
		spdk_bdev_io_complete(spdk_bdev_io_from_ctx(null_io), SPDK_BDEV_IO_STATUS_SUCCESS);
	}

	return SPDK_POLLER_BUSY;
}

This tiny path contains the whole reader contract:

Object: bdev_io is the request; null_io is the module's per-I/O driver context; ch is the per-thread channel.
Owner: the channel poller owns the ch->io queue while it drains it.
Async boundary: submit_request may return after queueing the I/O.
Next thread: the poller runs when the SPDK thread owning the channel is polled.
Terminal event: spdk_bdev_io_complete() returns the result to the bdev layer, which invokes the caller's completion callback on the expected thread.

The source tour method

When you read a file, do this:

Find the public entry points.
Find the state structs.
Find the registration macro.
Find the callback types.
Find the completion function.
Find error cleanup labels.
Find reset/remove/shutdown paths.
Find tests.

For example, in a bdev module:

public RPC handler in *_rpc.c
module registration with SPDK_BDEV_MODULE_REGISTER
bdev function table with submit_request
channel create/destroy
base bdev event callback if virtual
spdk_bdev_io_complete
destruct callback

Add two more passes after that first mechanical tour.

First, make an ownership map. It can be a three-column note:

object                owner / valid thread                 terminal action
spdk_bdev_io          bdev layer, then module in submit     spdk_bdev_io_complete()
module io ctx         module while request is outstanding   implicit with bdev_io/free path
spdk_io_channel       current SPDK thread                   spdk_put_io_channel()
callback ctx          submitting state machine              terminal callback frees or advances

Second, make a failure map. For each allocation, queue insertion, device state check, or unsupported operation, record whether the path completes the request, queues it, retries it, or returns an error to the caller. This catches the most common misunderstanding: an early return is not automatically a leak or a bug. It is only suspicious if no one now owns the next transition.

The callback-context rule

A callback context is a manual stack frame. Treat it like a small heap-allocated activation record with three responsibilities:

It preserves local variables across the async gap.
It identifies the next state or terminal callback.
It makes cleanup possible even when the path fails early.

Good context structs are boring. They hold pointers to the objects they need, a callback and callback argument, counters for child work, and sometimes a status field. Bad context structs hide ownership by mixing long-lived objects with one-shot state or by freeing themselves in more than one branch.

When reading a callback chain, find the allocation site and write down all terminal frees. If there are two terminal callbacks, ask why there are two. If there is no terminal free, look for reference-counted ownership or an object embedded in a larger parent. If neither exists, you may have found a leak.

The return-value rule

Do not ask "did this function succeed?" until you know what kind of function it is.

For synchronous helpers, the return value may be the result. For submission functions, the return value may only mean the request entered the async machinery. For message sends in SPDK's thread abstraction, the public API says the return value is left for compatibility and errors are fatal internally. For pollers, the return value is not a user result at all; it tells the scheduler whether the poller did work.

This is why examples in later chapters will separate the words "submitted," "completed," "failed," "aborted," "freed," and "destroyed." Those words are not synonyms.

Source reading path for this chapter

Read these in order before diving into a large subsystem:

include/spdk/thread.h: read spdk_msg_fn, spdk_poller_fn, spdk_thread_send_msg(), spdk_thread_poll(), and spdk_for_each_channel_continue().
lib/thread/thread.c: read spdk_thread_send_msg(), spdk_thread_poll(), spdk_for_each_channel(), and spdk_for_each_channel_continue().
include/spdk/bdev.h: read spdk_bdev_io_completion_cb and the public bdev I/O submission APIs near the operation you care about.
lib/bdev/bdev.c: read bdev_io_init(), bdev_io_do_submit(), spdk_bdev_io_complete(), _bdev_io_complete(), and one split or reset path.
module/bdev/null/bdev_null.c: read null_fn_table, bdev_null_submit_request(), channel creation, poller registration, null_io_poll(), and destruct.
Only then move to a hardware-backed module such as NVMe bdev. The contracts are easier to see after the null module.

Keep the official SPDK "Message Passing and Concurrency" and "Writing a Custom Block Device Module" pages open while reading. They explain the intent; the source shows the enforcement.

Labs

Lab 1: classify a function

Open lib/thread/thread.c and find spdk_thread_poll. Answer:

What object does it operate on?
Does it block?
What callbacks can it execute?
What does its return value mean?
What state does it temporarily set?

Expected shape of the answer: it operates on one struct spdk_thread; it does not block in normal poll mode; it can execute queued messages and pollers through thread_poll(); its return value is scheduler/accounting information about work done, not an application operation result; and it temporarily sets tls_thread so APIs like spdk_get_thread() see the polled SPDK thread.

Lab 2: classify a diskengine RPC wrapper

Open /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go. Pick BdevNvmeAttachController.

What JSON-RPC method is sent?
What does the Go wrapper consider success?
Does success mean the resulting bdev is fully examined?
Where in SPDK is that RPC handler registered?

Expected shape of the answer: separate JSON-RPC transport success from SPDK operation success and from later bdev examine state. If the wrapper returns after receiving an RPC result, do not assume every asynchronous SPDK side effect has become visible unless the RPC contract says so. Then trace the RPC method into SPDK's RPC registration and handler.

Lab 3: find the terminal callback

Open module/bdev/null/bdev_null.c. Find the path for a read. Where does the module eventually call spdk_bdev_io_complete? What would happen if it forgot?

Expected shape of the answer: bdev_null_submit_request() queues a supported read on the channel's io tailq; null_io_poll() drains that queue and calls spdk_bdev_io_complete(..., SPDK_BDEV_IO_STATUS_SUCCESS). If it forgot, the caller's bdev completion callback would never run, outstanding accounting would not unwind, and any state machine waiting for that I/O would stall.

Lab 4: explain a channel-iteration hang

Open lib/thread/thread.c and read spdk_for_each_channel() plus spdk_for_each_channel_continue(). Suppose the per-channel callback returns without calling spdk_for_each_channel_continue().

Which object is stranded?
Which thread was supposed to receive the next message?
Which completion callback will not run?
Why is this a state-machine bug rather than a CPU scheduling bug?

Self-check

Why is a callback context object like a manual stack frame?
Why is "who frees this" as important as "what does this do"?
What is the difference between submission success and operation success?
Why can a missing spdk_for_each_channel_continue hang a system?
Why is a pointer not the same thing as permission to mutate an object?
Why does spdk_thread_poll() temporarily set the current SPDK thread?
In the null bdev read path, where is the async boundary and where is the terminal event?