Orientation: SPDK Is Not Magic | SPDK From First Principles

The honest starting point

SPDK feels like a black box because it sits at the intersection of several things that are usually hidden from application developers: SSD firmware behavior, NVMe queue mechanics, PCIe device access, DMA memory, Linux driver binding, event-loop scheduling, asynchronous C callbacks, and a large plugin-based storage framework. If you start by opening lib/bdev/bdev.c at a random line, the code looks like a pile of callbacks and intrusive lists. If you start at the bottom and climb one layer at a time, it becomes a machine.

This book is written for that second path. The goal is not to memorize every SPDK function. The goal is to build enough mental machinery that when you read a source file you know what questions to ask:

What object is this function manipulating?
Which thread owns that object?
Is this code running in a reactor event, an spdk_thread message, a poller, an RPC handler, or a completion callback?
Is the work synchronous, or did the function merely submit async work?
Who completes the operation?
Who frees the memory?
What happens during reset, remove, -ENOMEM, hotplug, or shutdown?

Once those questions become automatic, SPDK source stops being mysterious. It remains dense, but it becomes readable because you can sort each function into a small number of roles: create an object, register an object, submit work, move work to an owner thread, poll progress, complete work, or tear an object down.

The first uncomfortable idea is that the call stack is often not the story. In normal blocking C, a function call may describe the full operation from start to finish. In SPDK, a function often allocates a context, submits work, returns, and relies on a later poller or callback to finish. This is why SPDK code is full of tiny context structs, callback function pointers, TAILQ lists, reference counts, and state flags. Those pieces are not decoration. They are the runtime's replacement for the blocking stack frame.

The second uncomfortable idea is that "userspace" does not mean "simple application code." SPDK moves performance-critical driver and storage-stack logic into a userspace process. That process still depends on the operating system for process isolation, VFIO, the IOMMU, hugepage setup, scheduling, signals, and filesystems used for ordinary files. But once the hot path is running, SPDK is intentionally trying to avoid syscalls, interrupts, sleeping locks, and scheduler wakeups in the IO path.

The daemon you actually run

In diskengine deployments, SPDK is usually not something your Go process links as an ordinary library. You run an SPDK application, most often a target-style daemon, and configure it through JSON-RPC. diskengine keeps desired state in its own world and asks SPDK to create, delete, export, or inspect C runtime objects.

That distinction matters. diskengine can decide that a volume should exist, but SPDK owns the struct spdk_bdev, struct spdk_io_channel, NVMe controller handles, lvolstore state, pollers, and transport objects. A successful JSON-RPC response means SPDK accepted or completed that method according to that method's contract. It does not automatically mean every downstream discovery, hotplug, reset, or VM-facing queue has fully converged. Later chapters will be precise about those contracts. For now, keep the separation clear:

diskengine owns orchestration intent and talks over JSON-RPC.
SPDK owns the userspace storage engine and hot IO path.
Linux still owns the process, memory-management mechanisms, IOMMU/VFIO plumbing, and non-SPDK system services.
Devices and remote peers own their side of NVMe, NVMe-oF, vhost, or vfio-user protocol behavior.

The generic SPDK target entry point is intentionally small:

/* app/spdk_tgt/spdk_tgt.c */
int
main(int argc, char **argv)
{
	struct spdk_app_opts opts = {};
	int rc;

	spdk_app_opts_init(&opts, sizeof(opts));
	opts.name = "spdk_tgt";
	if ((rc = spdk_app_parse_args(argc, argv, &opts, g_spdk_tgt_get_opts_string,
				      NULL, spdk_tgt_parse_arg, spdk_tgt_usage)) !=
	    SPDK_APP_PARSE_ARGS_SUCCESS) {
		return rc;
	}

	rc = spdk_app_start(&opts, spdk_tgt_started, NULL);
	spdk_app_fini();

	return rc;
}

This is the first useful orientation point: spdk_tgt is not a giant main() that manually constructs every bdev and transport. It prepares application options, then enters the SPDK event framework through spdk_app_start(). From there, command-line options, environment setup, reactors, RPC setup, subsystem initialization, and the application's start callback take over.

In lib/event/app.c, startup is guarded before any storage graph is built. The code checks the option structure, requires an application name, rejects a --wait-for-rpc configuration without an RPC server, picks a default reactor mask if needed, and configures logging before proceeding into lower-level setup. Those checks are not glamorous, but they explain many early failures: if the event framework cannot establish a coherent process-level runtime, no bdev or transport code should run yet.

The later startup handoff shows why --wait-for-rpc changes the mental model:

/* lib/event/app.c */
static void
app_do_spdk_subsystem_init(int rc, void *arg1)
{
	struct spdk_rpc_opts opts;

	if (rc) {
		spdk_app_stop(rc);
		return;
	}

	if (g_spdk_app.rpc_addr) {
		opts.size = SPDK_SIZEOF(&opts, log_level);
		opts.log_file = g_spdk_app.rpc_log_file;
		opts.log_level = g_spdk_app.rpc_log_level;

		rc = spdk_rpc_initialize(g_spdk_app.rpc_addr, &opts);
		if (rc) {
			spdk_app_stop(rc);
			return;
		}
		if (g_delay_subsystem_init) {
			return;
		}
		spdk_rpc_server_pause(g_spdk_app.rpc_addr);
	} else {
		SPDK_DEBUGLOG(app_rpc, "RPC server not started\n");
	}
	spdk_subsystem_init(app_subsystem_init_done, NULL);
}

If delayed subsystem initialization is enabled, this function starts the RPC server and returns before spdk_subsystem_init(). That is why "the process is listening for RPC" and "all SPDK subsystems are initialized" are different states. The point of --wait-for-rpc is to let a controller process provide configuration before runtime subsystems finish initialization. The edge case is obvious once you see this source: an RPC issued in the wrong phase may be rejected or may be a setup-phase RPC rather than a runtime operation.

The full stack you are trying to understand

At the highest level, your diskengine world uses SPDK as a storage engine daemon. diskengine is Go code. It does not link libspdk directly in the ordinary in-process sense. It speaks JSON-RPC over a Unix socket. SPDK owns the C runtime objects: bdevs, lvolstores, NVMe controllers, NVMe-oF subsystems, vhost controllers, pollers, threads, and channels.

The full path looks like this:

A guest VM issues storage IO.
QEMU exposes that IO to a host-side backend such as SPDK vhost-blk or vfio-user NVMe.
SPDK turns guest queue activity into bdev IO.
A RAID bdev may mirror or split that IO across base bdevs.
The base bdevs may be remote NVMe-oF controllers attached over RDMA.
On storage nodes, those remote exports are lvol bdevs.
lvol bdevs are blobs inside a blobstore.
Blobstore maps blob clusters onto a base bdev.
The base bdev may be a physical NVMe namespace.
The NVMe library turns IO into NVMe commands, puts them in submission queues, rings doorbells, polls completions, and returns completion callbacks back up the stack.

That is one path. SPDK also includes iSCSI, accel, crypto, malloc bdevs, null bdevs, passthru bdevs, TCP transports, RDMA transports, and many more pieces. The book focuses on the pieces you need for diskengine and for reading/extending the C source.

The shortest useful diagram is not a class hierarchy. It is a path of ownership handoffs:

flowchart LR VM[Guest VM IO] --> QEMU[QEMU queue/backend] QEMU --> Transport[SPDK vhost or vfio-user] Transport --> Bdev[Bdev IO] Bdev --> Virtual[RAID/lvol/other vbdev] Virtual --> Base[Base bdev] Base --> NVMe[NVMe library] NVMe --> SSD[Controller and namespace] Diskengine[diskengine JSON-RPC] -. configures .-> Bdev Diskengine -. configures .-> Transport

The dashed arrows are control-plane actions. The solid arrows are data-plane IO. Confusing those two planes causes bad debugging. A JSON-RPC method may create a bdev, export a namespace, or ask for status, but the guest's read and write completions are driven by SPDK pollers, channels, bdev module callbacks, NVMe queue completions, and transport-specific queue handling.

The four mental models

1. The hardware model

An SSD is not a magic byte array. It is a controller plus NAND flash. NAND has pages and erase blocks. It cannot overwrite in place the way RAM does. The SSD controller maintains a flash translation layer that maps host logical block addresses to physical flash locations. This is why write amplification, garbage collection, wear leveling, latency cliffs, TRIM/UNMAP, and power-loss behavior matter.

NVMe is the protocol host software uses to talk to modern SSDs. NVMe is a queue protocol. The host writes commands into submission queues in host memory. The device reads them with DMA, transfers payload data with DMA, writes completions into completion queues, and the host observes those completions. The driver is not "calling the SSD" like a normal function. It is publishing command descriptors into memory and ringing doorbells so the controller knows there is work.

This matters for SPDK because the fastest code path is organized around queues and polling, not around blocking syscalls. If a queue is full, if a controller resets, or if a device is removed while IO is outstanding, the software must preserve ownership and completion rules while the hardware state changes underneath it. Later chapters explain the NVMe queue machine in detail; for now, remember that SPDK's async style mirrors the device model.

2. The kernel-bypass model

Normal storage IO goes through Linux system calls, kernel block layers, kernel drivers, interrupts, wakeups, and copies. SPDK moves the driver and storage stack into userspace. It uses DPDK and VFIO to set up hugepage-backed DMA memory and direct device access. It polls instead of waiting for interrupts. This burns CPU to remove latency variance and kernel crossings.

This does not mean the kernel is gone. The kernel still provides process isolation, IOMMU support, VFIO, memory management, scheduling, and filesystems for normal files. SPDK removes the kernel from the hot storage data path after setup.

The official SPDK structural overview describes the repository in the same split you will see locally: C libraries under lib, public APIs under include/spdk, and runnable applications under app. It also calls out the env abstraction because SPDK needs operations POSIX does not provide, such as PCI enumeration and DMA-safe memory allocation. That is the shape to keep in your head: application code at the top, portable SPDK library APIs in the middle, and an environment layer below them for platform-specific capabilities.

3. The SPDK runtime model

SPDK is cooperative, event-driven C. Reactors are pinned OS threads. spdk_thread is a lightweight SPDK execution context scheduled on a reactor. Pollers are callbacks that run repeatedly. Messages are callbacks sent to another spdk_thread. io_channel is per-thread device state that allows hot paths to avoid locks.

The rule is simple and brutal: do not block. If a poller blocks, it stalls the reactor. If a callback waits synchronously for work that needs the same thread, it can deadlock. If you touch an object from the wrong thread, debug builds often assert because SPDK would rather crash than silently corrupt state.

The official concurrency guide states the core design directly: SPDK often assigns data to a single thread and asks other threads to pass messages to that owner. The local thread API documents the asynchronous contract:

/* include/spdk/thread.h */
/*
 * The message will be sent asynchronously - i.e. spdk_thread_send_msg will always return
 * prior to `fn` being called.
 *
 * Errors are handled internally and are fatal. Calling code can skip checking the return
 * value as it has been left only for compatibility.
 *
 * \param thread The target thread.
 * \param fn This function will be called on the given thread.
 * \param ctx This context will be passed to fn when called.
 *
 * \return 0 left for API compatibility
 */
int spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx);

The implementation is short enough to be worth reading early:

/* lib/thread/thread.c */
int
spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx)
{
	struct spdk_thread *local_thread;
	struct spdk_msg *msg;
	int rc;

	local_thread = _get_thread();

	msg = NULL;
	if (local_thread != NULL) {
		if (local_thread->msg_cache_count > 0) {
			msg = SLIST_FIRST(&local_thread->msg_cache);
			assert(msg != NULL);
			SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
			local_thread->msg_cache_count--;
		}
	}

	if (msg == NULL) {
		msg = spdk_mempool_get(g_spdk_msg_mempool);
		if (!msg) {
			SPDK_ERRLOG("msg could not be allocated\n");
			abort();
		}
	}

	msg->fn = fn;
	msg->arg = ctx;

	rc = spdk_ring_enqueue(thread->messages, (void **)&msg, 1, NULL);
	if (rc != 1) {
		SPDK_ERRLOG("msg could not be enqueued\n");
		abort();
	}

	thread_send_msg_notification(thread);

	return 0;
}

This excerpt is important because it shows three SPDK habits at once. First, a message is just a function pointer plus context. Second, allocation or enqueue failure here is fatal because the runtime treats message delivery as infrastructure, not optional application work. Third, the target function is not called inline. It runs only when the target spdk_thread is polled.

When reading source, ask whether the current function is running on the thread that owns the object. If not, look for spdk_thread_send_msg(), a channel iterator, or an event helper. Wrong-thread bugs in SPDK are often hidden as innocent-looking field access.

4. The bdev graph model

The bdev layer is SPDK's common block-device abstraction. A physical NVMe namespace is a bdev. An lvol is a bdev. A RAID device is a bdev. A vhost controller consumes a bdev. NVMe-oF exports bdevs as namespaces. A virtual bdev is just a bdev that forwards or transforms IO to one or more base bdevs.

Once you see the world as a bdev graph, diskengine becomes easier to reason about. It is constantly reconciling desired database state against actual SPDK bdev graph state.

The official bdev user guide describes bdev as SPDK's equivalent of the operating-system block layer above device drivers. That comparison is useful, but SPDK's implementation is modular and callback-driven rather than a clone of the Linux block layer. The backend contract is visible in include/spdk/bdev_module.h:

/* include/spdk/bdev_module.h */
/**
 * Function table for a block device backend.
 *
 * The backend block device function table provides a set of APIs to allow
 * communication with a backend. The main commands are read/write API
 * calls for I/O via submit_request.
 */
struct spdk_bdev_fn_table {
	/** Destroy the backend block device object. If the destruct process
	 *  for the bdev is asynchronous, return 1 from this function, and
	 *  then call spdk_bdev_destruct_done() once the async work is
	 *  complete. If the destruct process is synchronous, return 0 if
	 *  successful, or <0 if unsuccessful.
	 */
	int (*destruct)(void *ctx);

	/** Process the IO. */
	void (*submit_request)(struct spdk_io_channel *ch, struct spdk_bdev_io *);

	/** Check if the block device supports a specific I/O type. */
	bool (*io_type_supported)(void *ctx, enum spdk_bdev_io_type);

Every bdev module plugs into this shape. The generic bdev layer owns the common object model, descriptors, channels, splitting, QoS checks, reset handling, tracing, and submission flow. The module owns the backend-specific behavior. For an NVMe bdev, backend behavior means translating the bdev IO into NVMe commands. For a RAID bdev, it may mean splitting or mirroring to child bdevs. For a null bdev, it may mean completing synthetic IO without touching hardware.

The generic handoff point in lib/bdev/bdev.c is deliberately direct:

/* lib/bdev/bdev.c */
static inline void
bdev_submit_request(struct spdk_bdev *bdev, struct spdk_io_channel *ioch,
		    struct spdk_bdev_io *bdev_io)
{
	/* After a request is submitted to a bdev module, the ownership of an accel sequence
	 * associated with that bdev_io is transferred to the bdev module. So, clear the internal
	 * sequence pointer to make sure we won't touch it anymore. */
	if ((bdev_io->type == SPDK_BDEV_IO_TYPE_WRITE ||
	     bdev_io->type == SPDK_BDEV_IO_TYPE_READ) && bdev_io->u.bdev.accel_sequence != NULL) {
		assert(!bdev_io_needs_sequence_exec(bdev_io));
		bdev_io->internal.f.has_accel_sequence = false;
	}

	/* The generic bdev layer should not pass an I/O with a dif_check_flags set that
	 * the underlying bdev does not support. Add an assert to check this.
	 */
	assert((bdev_io->type != SPDK_BDEV_IO_TYPE_WRITE &&
		bdev_io->type != SPDK_BDEV_IO_TYPE_READ) ||
	       ((bdev_io->u.bdev.dif_check_flags & bdev->dif_check_flags) ==
		bdev_io->u.bdev.dif_check_flags));

	bdev->fn_table->submit_request(ioch, bdev_io);
}

Notice the ownership transfer comment. That is the kind of comment to slow down for. After the generic layer calls the module's submit_request, the module is responsible for eventually completing the spdk_bdev_io or forwarding it into another async path that will. If a module forgets to complete an IO, the upper layer does not magically recover. It waits.

The null bdev module is a small landmark because it shows both registration and backend submission without requiring hardware:

/* module/bdev/null/bdev_null.c */
static struct spdk_bdev_module null_if = {
	.name = "null",
	.module_init = bdev_null_initialize,
	.module_fini = bdev_null_finish,
	.async_fini = true,
	.get_ctx_size = bdev_null_get_ctx_size,
};

SPDK_BDEV_MODULE_REGISTER(null, &null_if)

static void
bdev_null_submit_request(struct spdk_io_channel *_ch, struct spdk_bdev_io *bdev_io)
{
	struct null_bdev_io *null_io = (struct null_bdev_io *)bdev_io->driver_ctx;
	struct null_io_channel *ch = spdk_io_channel_get_ctx(_ch);
	struct spdk_bdev *bdev = bdev_io->bdev;
	struct spdk_dif_ctx dif_ctx;
	struct spdk_dif_error err_blk;
	int rc;
	struct spdk_dif_ctx_init_ext_opts dif_opts;

That snippet starts the module side of the contract. It receives the per-thread channel, recovers module-private IO context from driver_ctx, and operates on bdev_io. Later in that file, the module completes the IO. Even a fake device follows the same basic contract: register a module, provide a function table, receive bdev IO, and complete it.

Subsystems: why startup is ordered

SPDK has many independently built libraries and modules, but a daemon still needs a coherent initialization order. The event subsystem mechanism gives modules a way to register themselves and declare dependencies. The registration macro is simple:

/* include/spdk_internal/init.h */
struct spdk_subsystem {
	const char *name;
	/* User must call spdk_subsystem_init_next() when they are done with their initialization. */
	void (*init)(void);
	void (*fini)(void);

	void (*write_config_json)(struct spdk_json_write_ctx *w);
	TAILQ_ENTRY(spdk_subsystem) tailq;
};

struct spdk_subsystem_depend {
	const char *name;
	const char *depends_on;
	TAILQ_ENTRY(spdk_subsystem_depend) tailq;
};

#define SPDK_SUBSYSTEM_REGISTER(_name) \
	__attribute__((constructor)) static void _name ## _register(void)	\
	{									\
		spdk_add_subsystem(&_name);					\
	}

The important part is not the macro trick itself. The important part is that subsystems form a dependency graph before initialization runs. For example, local source shows nvmf depends on bdev, keyring, and sock; vhost_blk depends on bdev; and bdev depends on lower services such as accel, keyring, vmd, sock, and iobuf. That is why a storage daemon is not just "start RPC and make bdevs." It must initialize lower services first, then consumers.

The edge case follows from the comment in struct spdk_subsystem: a subsystem that starts async initialization must call spdk_subsystem_init_next() when it is done. If it forgets, startup hangs in the middle of the dependency chain. If it calls next too early, later subsystems may run against partially initialized state. This is a common SPDK reading pattern: the tiny callback contract carries the correctness of the whole flow.

What to read locally

Use these files as landmarks:

README.md for SPDK's own high-level promise.
doc/overview.md, doc/concurrency.md, and doc/bdev.md for local copies of the official explanations.
app/spdk_tgt/spdk_tgt.c for the generic target entry point.
lib/event/app.c for app startup, RPC startup, delayed subsystem initialization, and shutdown.
include/spdk_internal/init.h and module/event/subsystems/* for subsystem registration and dependency declarations.
lib/event/reactor.c for reactor loops.
lib/thread/thread.c and include/spdk/thread.h for spdk_thread, messages, pollers, and channels.
lib/bdev/bdev.c, include/spdk/bdev.h, and include/spdk/bdev_module.h for bdev.
module/bdev/null/bdev_null.c for a small bdev module.
module/bdev/nvme/ for the NVMe bdev module.
lib/nvme/ for the initiator-side NVMe library.
lib/nvmf/ and module/event/subsystems/nvmf/ for the target-side NVMe-oF library and subsystem wiring.
lib/blob/, lib/lvol/, and module/bdev/lvol/ for blobstore and lvol.
module/bdev/raid/ for RAID.
lib/vhost/, module/event/subsystems/vhost_blk/, and lib/nvmf/vfio_user.c for VM-facing transports.
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/ for diskengine's JSON-RPC client.

Read those files in this order if you are new:

app/spdk_tgt/spdk_tgt.c: see how little the application entry point does.
lib/event/app.c: follow spdk_app_start(), app_do_spdk_subsystem_init(), and app_subsystem_init_done().
include/spdk_internal/init.h: learn what a subsystem promises.
module/event/subsystems/bdev/bdev.c: see the bdev subsystem enter the event framework.
include/spdk/thread.h and lib/thread/thread.c: learn the message and poller vocabulary.
include/spdk/bdev_module.h: read the bdev backend contract.
module/bdev/null/bdev_null.c: trace a small module from registration to IO completion.
lib/bdev/bdev.c: return to the large file after the smaller landmarks make the vocabulary familiar.

Primary external references

Keep these open while studying:

How to use this book

Read Part 1 even if you are tempted to jump to SPDK. If you do not understand why DMA memory must be special, why NVMe is queue-based, or why NAND cannot overwrite in place, many SPDK choices will look arbitrary.

When a chapter shows a source path, open it in the repo. Do not only read the quoted excerpts. The excerpt is the doorway. The file is the lesson.

When a lab asks you to predict behavior, actually predict before reading the answer or running a command. SPDK debugging is mostly state classification. You get better by forcing yourself to name the state before poking it.

When reading an SPDK source file, keep four columns in your notes:

Object: the struct whose lifetime matters.
Owner: the spdk_thread, module, descriptor, or subsystem that may mutate it.
Progress: the poller, message, queue completion, or callback that advances it.
Terminal event: the callback, completion, unregister, or free that ends the operation.

For example, a bdev write has a struct spdk_bdev_io object. Its channel is per-thread state. The generic bdev layer may split it, queue it, trace it, or pass it to a module. The module eventually completes it or forwards it to lower IO that will complete. The terminal event is not the function returning from submission; it is the eventual completion callback.

Failure modes to remember from day one

SPDK failures often look strange until you classify which invariant was broken.

A blocked poller is a local bug with global symptoms. If the reactor runs one callback that spins or waits synchronously, other pollers on that reactor do not make progress. That can look like an NVMe, RPC, or bdev hang even when the root cause is one callback refusing to yield.

A missing completion is worse than an error completion. If a bdev module detects failure and completes the IO with failed status, upper layers can unwind. If it loses the IO or returns from submit_request without arranging completion, the request hangs.

A wrong-thread access can be silent in release builds and fatal in debug builds. SPDK's model is not "protect every field with a mutex." It is "mutate this object on its owner thread or pass a message to that owner." When you see a field access, ask why the current thread is allowed to do it.

An RPC success can be narrower than the operational state you care about. Some RPCs create configuration objects. Some trigger asynchronous work. Some report state that can change immediately after the response. A controller like diskengine must often follow a successful RPC with polling, inspection, or reconciliation against expected graph state.

Reset and remove paths are part of the design, not afterthoughts. A physical NVMe controller can reset, a remote path can disappear, a bdev can be removed while descriptors exist, or a VM-facing queue can disconnect. The happy path teaches vocabulary. The edge paths teach correctness.

Self-check

Can you explain why diskengine is not "using an SPDK library" in the normal linked-library sense?
Can you name the four core mental models: hardware, kernel bypass, runtime, bdev graph?
Can you point to the source file that owns reactor polling?
Can you point to the source file that owns bdev IO routing?
Can you explain why "RPC returned success" may not mean "the storage graph is fully converged"?
Can you explain why spdk_thread_send_msg() returning 0 does not mean the target function already ran?
Can you explain what a bdev module promises when it implements submit_request?
Can you name one startup state where the RPC server exists but runtime subsystems are not initialized yet?