SPDK From First Principles

SPDK deep learning path

Chapter 7: Linux Storage Path vs SPDK Path

A first-principles comparison of the normal Linux storage stack and SPDK's userspace, polling, DMA-first storage path.

Source: content/chapters/07-linux-storage-path-vs-spdk-path.md

Reader Promise

By the end of this chapter you should be able to explain why SPDK exists without hand-waving. You should be able to draw the normal Linux path for a read or write, draw the SPDK path for the same operation, and name the tradeoffs instead of repeating "kernel slow, SPDK fast". That phrase is not good enough. The real difference is ownership of queues, memory, threads, interrupts, context switches, batching, and failure handling.

The Linux stack is a general-purpose storage operating system. SPDK is a specialized userspace storage runtime. Linux optimizes for protection, sharing, fairness, device diversity, filesystems, page cache, suspend/resume, hotplug, and decades of applications. SPDK optimizes for a narrower target: high-throughput, low-latency storage services that can reserve CPU cores, hugepage memory, and direct device ownership.

This chapter is deliberately about the data path, not about benchmark slogans. The data path is the chain of code and hardware state that turns "read these bytes" into a command in an NVMe submission queue and later turns a completion entry into application-visible progress. Once you can name the owner of every queue, buffer, and callback in that chain, the Linux/SPDK tradeoff becomes much less mysterious.

The Normal Linux Read Path

Imagine an application calls pread(fd, buf, 4096, offset). If the file is on a filesystem backed by an NVMe SSD, the path is roughly:

application thread
  |
  | syscall boundary
  v
kernel VFS
  |
  | file, inode, page cache, filesystem mapping
  v
filesystem
  |
  | logical file blocks -> block device sectors
  v
Linux block layer
  |
  | request allocation, scheduler/merge, bio/request mapping
  v
NVMe kernel driver
  |
  | submit command to hardware queue
  v
NVMe SSD
  |
  | interrupt or poll completion
  v
kernel completion path
  |
  | wake task, copy/map data, return from syscall
  v
application thread

That stack is not dumb. It provides a lot:

  • Permission checks and process isolation.
  • Filesystem semantics.
  • Page cache.
  • Device sharing.
  • Request accounting.
  • Cgroup and IO scheduling policy.
  • Kernel driver recovery.
  • Hotplug and power-management integration.
  • A stable POSIX programming model.

For many systems, this is exactly what you want.

The important word in that diagram is not "kernel"; it is "shared". A normal Linux block device is usually a shared operating-system resource. The kernel driver owns the controller, the block layer arbitrates requests from many tasks, the filesystem protects metadata invariants, and the scheduler decides when sleeping tasks should run again. That design lets a database, shell command, backup agent, container runtime, and monitoring daemon all use the same machine without agreeing on a storage runtime.

That sharing is also why the Linux path has more crossing points. The application crosses into the kernel with a syscall. The filesystem converts file offsets into block ranges. The block layer converts bios into device requests. The kernel NVMe driver converts those requests into NVMe commands. Completion then walks back through kernel completion code before the original task can continue. Each crossing point carries state that makes the system robust for general-purpose use.

What Costs Show Up In The Kernel Path

The costs are not one single thing. They are a pile of small costs that become visible at high IOPS:

  • A syscall crosses from userspace into kernel mode.
  • The filesystem may consult metadata.
  • The page cache may copy, map, dirty, reclaim, or bypass pages.
  • The block layer allocates and transforms request objects.
  • The scheduler may merge or reorder requests.
  • The NVMe driver submits to hardware queues owned by the kernel.
  • Interrupt handling may move completion work onto a CPU that is not the original submitter.
  • The sleeping application may need to be woken.
  • Locks, atomics, memory barriers, and shared queues protect many users and devices.

None of these are inherently bad. They buy generality. But if your storage server already owns the disk, already speaks an async protocol, already uses direct buffers, and already dedicates CPU to IO, some of that generality is overhead.

The most subtle cost is not the raw instruction count. It is the loss of locality and ownership. If one CPU submits, another CPU handles an interrupt, a scheduler wakes the original task later, and the application then resubmits more work, the hardware queue is no longer a simple extension of one userspace state machine. SPDK's design is an answer to that specific problem.

The SPDK Path

SPDK moves the device driver, queue ownership, and polling loop into userspace. For an NVMe read through SPDK's bdev layer, the shape is closer to:

storage service callback
  |
  | spdk_bdev_read()
  v
bdev core
  |
  | allocate/route/split/QoS if needed
  v
bdev module, such as bdev_nvme
  |
  | submit to per-thread NVMe qpair
  v
NVMe submission queue in DMA-safe hugepage memory
  |
  | MMIO doorbell
  v
NVMe SSD
  |
  | completion written into host memory
  v
SPDK poller observes completion
  |
  | callback chain runs on SPDK thread
  v
storage service continuation

The crucial differences:

  • The hot path is async. You submit and later receive a callback.
  • The application usually does not block.
  • Completion is usually found by polling, not by sleeping and waiting for an interrupt.
  • Direct I/O data buffers are allocated from DMA-safe memory.
  • Device queues are owned by the SPDK process through VFIO/UIO-style binding.
  • Per-core or per-thread resources avoid many cross-core locks.

The official SPDK NVMe driver guide describes the same model from the driver side: the NVMe driver is a C library linked into the application, performs direct zero-copy transfers, does not spawn its own threads, and acts only when the application calls into it. SPDK's userspace-driver documentation also emphasizes that a userspace driver maps the PCI BAR into the process through VFIO or UIO and performs MMIO directly. DPDK's EAL documentation supplies the lower-level environment idea: EAL gets access to hardware and memory resources and presents them to libraries and applications. Linux VFIO documentation explains the safety mechanism behind direct userspace device access: VFIO exposes devices to userspace in an IOMMU-protected framework.

The result is not "no kernel". It is a different division of labor. Linux still provides the process, virtual memory, scheduling, VFIO, IOMMU, and filesystems used by the control plane. SPDK takes over the storage hot path inside that process.

Where SPDK Starts Owning The Path

An SPDK application does not begin with a blocking read() loop. It begins by creating a runtime: environment setup, memory setup, core selection, reactor launch, subsystem initialization, and then user code. In this repository, spdk_app_start() is the front door used by event-framework applications.

/* lib/event/app.c */
if (!(opts->lcore_map || opts->reactor_mask)) {
	/* Set default CPU mask */
	opts->reactor_mask = SPDK_APP_DPDK_DEFAULT_CORE_MASK;
}

if (opts->interrupt_mode) {
	spdk_interrupt_mode_enable();
}

memset(&g_spdk_app, 0, sizeof(g_spdk_app));

g_spdk_app.rpc_addr = opts->rpc_addr;
g_spdk_app.shm_id = opts->shm_id;
g_spdk_app.shutdown_cb = opts->shutdown_cb;
g_spdk_app.rc = 0;
g_spdk_app.stopped = false;

if (app_setup_env(g_env_was_setup ? NULL : opts) < 0) {
	return 1;
}

This excerpt shows that the application framework is not just a helper around main(). It decides the reactor core mask when the user did not provide one, optionally enables interrupt mode, stores process-wide application state, and calls app_setup_env(). From this point forward, the storage service is no longer an ordinary process that occasionally asks the kernel to do IO. It is a process preparing to own CPU placement and memory resources for its own IO runtime.

app_setup_env() converts application options into environment options. Those options include the application name, CPU mask or lcore map, shared-memory id, memory size, hugepage directory, PCI allow/block lists, IOVA mode, and NUMA policy.

/* lib/event/app.c */
env_opts.opts_size = sizeof(env_opts);
spdk_env_opts_init(&env_opts);

env_opts.name = opts->name;
env_opts.core_mask = opts->reactor_mask;
env_opts.lcore_map = opts->lcore_map;
env_opts.shm_id = opts->shm_id;
env_opts.mem_size = opts->mem_size;
env_opts.hugedir = opts->hugedir;
env_opts.no_pci = opts->no_pci;
env_opts.pci_blocked = opts->pci_blocked;
env_opts.pci_allowed = opts->pci_allowed;
env_opts.iova_mode = opts->iova_mode;
env_opts.no_huge = opts->no_huge;
env_opts.enforce_numa = opts->enforce_numa;

rc = spdk_env_init(&env_opts);

The environment layer is where the SPDK path becomes a DMA-capable path. In this checkout, SPDK's DPDK-backed environment builds a DPDK EAL command line and calls rte_eal_init().

/* lib/env_dpdk/init.c */
rc = build_eal_cmdline(opts);
if (rc < 0) {
	SPDK_ERRLOG("Invalid arguments to initialize DPDK\n");
	return -EINVAL;
}

dpdk_args = calloc(g_eal_cmdline_argcount, sizeof(char *));
if (dpdk_args == NULL) {
	SPDK_ERRLOG("Failed to allocate dpdk_args\n");
	return -ENOMEM;
}
memcpy(dpdk_args, g_eal_cmdline, sizeof(char *) * g_eal_cmdline_argcount);

fflush(stdout);
orig_optind = optind;
optind = 1;
rc = rte_eal_init(g_eal_cmdline_argcount, dpdk_args);
optind = orig_optind;

Why this matters: DPDK EAL is the layer that gives SPDK access to hugepage-backed memory, CPU topology, PCI devices, timers, and related low-level resources. SPDK then wraps that environment in APIs such as spdk_dma_malloc() and spdk_dma_zmalloc(), which are the kinds of buffers a userspace driver can hand to hardware.

/* lib/env_dpdk/env.c */
void *
spdk_dma_malloc_socket(size_t size, size_t align, uint64_t *unused, int numa_id)
{
	return spdk_malloc(size, align, unused, numa_id, (SPDK_MALLOC_DMA | SPDK_MALLOC_SHARE));
}

void *
spdk_dma_zmalloc_socket(size_t size, size_t align, uint64_t *unused, int numa_id)
{
	return spdk_zmalloc(size, align, unused, numa_id, (SPDK_MALLOC_DMA | SPDK_MALLOC_SHARE));
}

The ownership point is simple: the Linux path can often accept ordinary user buffers because the kernel mediates copying, pinning, mapping, or direct IO details. The SPDK fast path wants I/O data buffers that are already suitable for device DMA from the userspace driver model. That does not mean every C allocation in an SPDK process must come from spdk_dma_malloc(): control-plane structs, JSON text, logs, and ordinary bookkeeping can use normal memory unless a specific API says otherwise. When a storage service ignores the data-buffer distinction, the failure usually appears much later as an allocation failure, an IOMMU translation problem, or a controller command that cannot be built safely.

Device Binding Is Part Of The Data Path

SPDK setup is operationally visible because the controller must be bound to a userspace-capable driver before SPDK can drive it directly. scripts/setup.sh is not just a convenience script; it reflects the prerequisites for kernel bypass.

# scripts/setup.sh
elif is_iommu_enabled; then
	driver_name=vfio-pci
	# Just in case, attempt to load VFIO_IOMMU_TYPE1 module into the kernel - this
	# should be done automatically by modprobe since this particular module should
	# be a part of vfio-pci dependencies, however, on some distros, it seems that
	# it's not the case. See #1689.
	if modinfo vfio_iommu_type1 > /dev/null; then
		modprobe vfio_iommu_type1
	fi
elif ! check_for_driver uio_pci_generic || modinfo uio_pci_generic > /dev/null 2>&1; then
	driver_name=uio_pci_generic

With VFIO, the kernel is still involved, but not as the NVMe block driver. The Linux VFIO documentation describes VFIO as an IOMMU/device-agnostic framework that exposes direct device access to userspace in a protected environment. In practice, if a controller is bound to vfio-pci, it normally disappears from lsblk because the kernel NVMe block driver no longer owns it. That is expected for a local PCIe NVMe controller used directly by SPDK.

Hugepages are the other visible setup step. SPDK needs memory that can be mapped for DMA and shared with DPDK's allocator.

# scripts/setup.sh
function configure_linux() {
	configure_linux_pci
	hugetlbfs_mounts=$(linux_hugetlbfs_mounts)

	if [ -z "$hugetlbfs_mounts" ]; then
		hugetlbfs_mounts=/mnt/huge
		echo "Mounting hugetlbfs at $hugetlbfs_mounts"
		mkdir -p "$hugetlbfs_mounts"
		mount -t hugetlbfs nodev "$hugetlbfs_mounts"
	fi

	configure_linux_hugepages

This setup changes the troubleshooting model. A Linux-path failure might be "the block device exists but the filesystem cannot mount." An SPDK-path failure might be "the process cannot allocate the right hugepage memory" or "the PCI device is still owned by the kernel driver" or "the VFIO group is not viable because another device in the IOMMU group is still attached to a host driver."

The Polling Thread Is The Completion Engine

In the Linux path, it is normal for the application thread to sleep and later be woken by kernel completion work. In the SPDK event framework, a reactor is a POSIX thread tied to a core. That reactor runs SPDK threads, and those SPDK threads run pollers, messages, and completions.

/* lib/event/reactor.c */
_reactor_run(struct spdk_reactor *reactor)
{
	struct spdk_thread	*thread;
	struct spdk_lw_thread	*lw_thread, *tmp;
	uint64_t		now;
	int			rc;

	event_queue_run_batch(reactor);

	if (spdk_unlikely(TAILQ_EMPTY(&reactor->threads))) {
		now = spdk_get_ticks();
		reactor->idle_tsc += now - reactor->tsc_last;
		reactor->tsc_last = now;
		return;
	}

	TAILQ_FOREACH_SAFE(lw_thread, &reactor->threads, link, tmp) {
		thread = spdk_thread_get_from_ctx(lw_thread);
		rc = spdk_thread_poll(thread, 0, reactor->tsc_last);

The reactor owns CPU time. Each lightweight thread on the reactor gets polled. A return value of zero means no useful work was found; a positive value means work was done. That is why a poll-mode application can use CPU while "idle": it is actively checking for completions and messages so it can avoid interrupt and wakeup latency when work arrives.

The loop around _reactor_run() makes that ownership explicit:

/* lib/event/reactor.c */
static int
reactor_run(void *arg)
{
	struct spdk_reactor	*reactor = arg;
	char			thread_name[32];

	SPDK_NOTICELOG("Reactor started on core %u\n", reactor->lcore);

	snprintf(thread_name, sizeof(thread_name), "reactor_%u", reactor->lcore);
	_set_thread_name(thread_name);

	reactor->tsc_last = spdk_get_ticks();

	while (1) {
		if (spdk_unlikely(reactor->in_interrupt)) {
			reactor_interrupt_run(reactor);
		} else {
			_reactor_run(reactor);
		}

The NVMe library follows the same rule: completions happen when application code asks for completions to be processed. The SPDK NVMe documentation calls the library passive: it does not create a private completion thread for you. A caller must periodically process completions on the qpair or poll group.

/* lib/nvme/nvme_qpair.c */
spdk_nvme_qpair_process_completions(struct spdk_nvme_qpair *qpair, uint32_t max_completions)
{
	int32_t ret;
	struct nvme_request *req, *tmp;

	if (spdk_unlikely(qpair->ctrlr->is_failed &&
			  nvme_qpair_get_state(qpair) != NVME_QPAIR_DISCONNECTING)) {
		if (qpair->ctrlr->is_removed) {
			nvme_qpair_set_state(qpair, NVME_QPAIR_DESTROYING);
			nvme_qpair_abort_all_queued_reqs(qpair);
			nvme_transport_qpair_abort_reqs(qpair);
		}
		return -ENXIO;
	}

	qpair->in_completion_context = 1;
	ret = nvme_transport_qpair_process_completions(qpair, max_completions);

This is the callback ownership rule for SPDK: submit on the right SPDK thread with the right channel, then make progress by polling that thread or qpair. If the reactor stops polling because user code blocks in a filesystem call, sleeps on a mutex, waits for RPC synchronously, or burns CPU in a long calculation, completions tied to that reactor can stop making progress.

Bdev Read: Logical IO Before Hardware IO

Most SPDK applications should not talk directly to struct spdk_nvme_qpair for every storage operation. They usually use the bdev layer because it provides a common block-device abstraction over NVMe, malloc disks, AIO, lvol, RAID, crypto, compression, and other modules. The bdev layer is where a logical read becomes a struct spdk_bdev_io.

/* lib/bdev/bdev.c */
static int
bdev_read_blocks_with_md(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch, void *buf,
			 void *md_buf, uint64_t offset_blocks, uint64_t num_blocks,
			 spdk_bdev_io_completion_cb cb, void *cb_arg)
{
	struct spdk_bdev *bdev = spdk_bdev_desc_get_bdev(desc);
	struct spdk_bdev_io *bdev_io;
	struct spdk_bdev_channel *channel = __io_ch_to_bdev_ch(ch);

	if (!bdev_io_valid_blocks(bdev, offset_blocks, num_blocks)) {
		return -EINVAL;
	}

	bdev_io = bdev_channel_get_io(channel);
	if (!bdev_io) {
		return -ENOMEM;
	}

	bdev_io->internal.ch = channel;
	bdev_io->internal.desc = desc;
	bdev_io->type = SPDK_BDEV_IO_TYPE_READ;
	bdev_io->u.bdev.iovs = &bdev_io->iov;
	bdev_io->u.bdev.iovs[0].iov_base = buf;
	bdev_io->u.bdev.num_blocks = num_blocks;
	bdev_io->u.bdev.offset_blocks = offset_blocks;
	bdev_io_init(bdev_io, bdev, cb_arg, cb);

	bdev_io_submit(bdev_io);
	return 0;
}

The application owns the callback and callback argument. The bdev layer owns the spdk_bdev_io object while the operation is in flight. The spdk_io_channel is the caller's per-thread handle into the bdev; it is not a generic global object that any thread can use freely. This per-thread channel model is one of the ways SPDK avoids turning every IO submission into a shared-lock problem.

spdk_bdev_read() is a convenience wrapper. It converts byte offsets to blocks and then calls the block-oriented path:

/* lib/bdev/bdev.c */
int
spdk_bdev_read(struct spdk_bdev_desc *desc, struct spdk_io_channel *ch,
	       void *buf, uint64_t offset, uint64_t nbytes,
	       spdk_bdev_io_completion_cb cb, void *cb_arg)
{
	uint64_t offset_blocks, num_blocks;

	if (bdev_bytes_to_blocks(desc, offset, &offset_blocks, nbytes, &num_blocks) != 0) {
		return -EINVAL;
	}

	return spdk_bdev_read_blocks(desc, ch, buf, offset_blocks, num_blocks, cb, cb_arg);
}

Submitting the bdev IO is still not necessarily a direct device command. The bdev core may trace the IO, split it, queue it behind a locked LBA range, apply QoS, or fail it during reset.

/* lib/bdev/bdev.c */
void
bdev_io_submit(struct spdk_bdev_io *bdev_io)
{
	struct spdk_bdev_channel *ch = bdev_io->internal.ch;

	assert(bdev_io->internal.status == SPDK_BDEV_IO_STATUS_PENDING);

	if (!bdev_io->internal.f.child_io && !TAILQ_EMPTY(&ch->locked_ranges)) {
		struct lba_range *range;

		TAILQ_FOREACH(range, &ch->locked_ranges, tailq) {
			if (bdev_io_range_is_locked(bdev_io, range)) {
				TAILQ_INSERT_TAIL(&ch->io_locked, bdev_io, internal.ch_link);
				return;
			}
		}
	}

	bdev_ch_add_to_io_submitted(bdev_io);
	bdev_io->internal.submit_tsc = spdk_get_ticks();

	if (bdev_io->internal.f.split) {
		bdev_io_split(bdev_io);
		return;
	}

	_bdev_io_submit(bdev_io);
}

This is where the "SPDK removes the kernel block layer" statement needs precision. SPDK removes the Linux block layer from this hot path, but it still has its own bdev core because a storage service needs policy and composition. The difference is that this policy is inside the SPDK process and uses SPDK's thread/channel rules.

The final dispatch step calls the module's submit path unless the channel is in a special state:

/* lib/bdev/bdev.c */
static void
_bdev_io_submit(struct spdk_bdev_io *bdev_io)
{
	struct spdk_bdev *bdev = bdev_io->bdev;
	struct spdk_bdev_channel *bdev_ch = bdev_io->internal.ch;

	if (spdk_likely(bdev_ch->flags == 0)) {
		bdev_io_do_submit(bdev_ch, bdev_io);
		return;
	}

	if (bdev_ch->flags & BDEV_CH_RESET_IN_PROGRESS) {
		_bdev_io_complete_in_submit(bdev_ch, bdev_io, SPDK_BDEV_IO_STATUS_ABORTED);
	} else if (bdev_ch->flags & BDEV_CH_QOS_ENABLED) {
		TAILQ_INSERT_TAIL(&bdev_ch->qos_queued_io, bdev_io, internal.link);
		bdev_qos_io_submit(bdev_ch, bdev->internal.qos);
	}
}

Now the logical IO has reached a module boundary. For an NVMe bdev, the next owner is the NVMe bdev module.

NVMe Bdev: Logical IO Becomes A Qpair Command

The NVMe bdev module receives the bdev IO on a bdev channel. It finds an IO path for that channel, records trace state, and then dispatches by IO type.

/* module/bdev/nvme/bdev_nvme.c */
static void
bdev_nvme_submit_request(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)
{
	struct nvme_bdev_channel *nbdev_ch = spdk_io_channel_get_ctx(ch);
	struct nvme_bdev_io *nbdev_io = (struct nvme_bdev_io *)bdev_io->driver_ctx;

	if (spdk_likely(nbdev_io->submit_tsc == 0)) {
		nbdev_io->submit_tsc = spdk_bdev_io_get_submit_tsc(bdev_io);
	} else {
		nbdev_io->submit_tsc = spdk_get_ticks();
	}

	spdk_trace_record(TRACE_BDEV_NVME_IO_START, 0, 0, (uintptr_t)nbdev_io, (uintptr_t)bdev_io);
	nbdev_io->io_path = bdev_nvme_find_io_path(nbdev_ch);
	if (spdk_unlikely(!nbdev_io->io_path)) {
		if (!bdev_nvme_io_type_is_admin(bdev_io->type)) {
			bdev_nvme_io_complete(nbdev_io, -ENXIO);
			return;
		}
	}

	_bdev_nvme_submit_request(nbdev_ch, bdev_io);
}

The io_path is the bridge from logical bdev state to an NVMe namespace and qpair. Losing that path is not the same as a short read from a POSIX file. It means the module cannot find an available NVMe path for this IO, so it completes the bdev IO with an error.

For reads, _bdev_nvme_submit_request() calls bdev_nvme_readv() when the caller already supplied buffers. If the buffer is missing, the module asks the bdev layer to get one asynchronously and returns for now.

/* module/bdev/nvme/bdev_nvme.c */
switch (bdev_io->type) {
case SPDK_BDEV_IO_TYPE_READ:
	if (bdev_io->u.bdev.iovs && bdev_io->u.bdev.iovs[0].iov_base) {

		rc = bdev_nvme_readv(nbdev_io,
				     bdev_io->u.bdev.iovs,
				     bdev_io->u.bdev.iovcnt,
				     bdev_io->u.bdev.md_buf,
				     bdev_io->u.bdev.num_blocks,
				     bdev_io->u.bdev.offset_blocks,
				     bdev_io->u.bdev.dif_check_flags,
				     bdev_io->u.bdev.memory_domain,
				     bdev_io->u.bdev.memory_domain_ctx,
				     bdev_io->u.bdev.accel_sequence);
	} else {
		spdk_bdev_io_get_buf(bdev_io, bdev_nvme_get_buf_cb,
				     bdev_io->u.bdev.num_blocks * bdev->blocklen);
		rc = 0;
	}
	break;

The actual read function extracts the NVMe namespace and qpair from the selected IO path, then submits a command through the NVMe library. The completion callback is bdev_nvme_readv_done, and the callback context is the NVMe bdev IO.

/* module/bdev/nvme/bdev_nvme.c */
static int
bdev_nvme_readv(struct nvme_bdev_io *bio, struct iovec *iov, int iovcnt,
		void *md, uint64_t lba_count, uint64_t lba, uint32_t flags,
		struct spdk_memory_domain *domain, void *domain_ctx,
		struct spdk_accel_sequence *seq)
{
	struct spdk_nvme_ns *ns = bio->io_path->nvme_ns->ns;
	struct spdk_nvme_qpair *qpair = bio->io_path->qpair->qpair;
	int rc;

	bio->iovs = iov;
	bio->iovcnt = iovcnt;
	bio->iovpos = 0;
	bio->iov_offset = 0;

	if (domain != NULL || seq != NULL) {
		/* extended read path omitted */
	} else if (iovcnt == 1) {
		rc = spdk_nvme_ns_cmd_read_with_md(ns, qpair, iov[0].iov_base,
						   md, lba, lba_count, bdev_nvme_readv_done,
						   bio, flags, 0, 0);
	} else {
		rc = spdk_nvme_ns_cmd_readv_with_md(ns, qpair, lba, lba_count,
						    bdev_nvme_readv_done, bio, flags,
						    bdev_nvme_queued_reset_sgl,
						    bdev_nvme_queued_next_sge, md, 0, 0);
	}

This is the moment the SPDK path differs most sharply from Linux. There is no syscall here, no kernel block request, and no kernel NVMe driver submitting on behalf of the process. The userspace module has selected a qpair and asked the SPDK NVMe library to build a command. Later, a poller processes completions on that qpair, the NVMe completion callback runs, the NVMe bdev IO completes, and the original bdev callback runs on the SPDK thread that owns the channel.

The callback chain is therefore:

application bdev callback
  ^
  | bdev completion
NVMe bdev completion helper
  ^
  | bdev_nvme_readv_done(bio, cpl)
SPDK NVMe qpair completion processing
  ^
  | controller writes completion entry to host memory
NVMe SSD

That chain is why blocking inside an SPDK completion callback is dangerous. You are not just blocking one application request; you may be blocking the thread that must poll more completions and run more callbacks.

What SPDK Removes

SPDK can remove or reduce:

  • Syscall overhead on the IO hot path.
  • Kernel block-layer request scheduling.
  • Interrupt and wakeup overhead in the common poll-mode path.
  • Extra copies when the application already has DMA-safe data buffers.
  • Kernel-driver queue sharing between unrelated processes.
  • Lock contention from multi-tenant kernel abstractions.

This is why SPDK is attractive for storage appliances, NVMe-oF targets, userspace vhost targets, virtual block device stacks, and cloud volume services.

What SPDK Adds

SPDK does not make complexity disappear. It moves complexity into the userspace storage service:

  • The process must reserve hugepages.
  • The process must bind devices away from kernel drivers.
  • The process must obey strict thread-affinity rules.
  • The process must avoid blocking poller threads.
  • The process must manage async cleanup, resets, removals, and reconnects.
  • The process must expose its own control plane.
  • The process must be monitored like a storage operating system.

This matters for diskengine. If diskengine tells SPDK to create a volume, attach an NVMe-oF controller, or export a bdev, diskengine is depending on an external userspace storage runtime. The failure modes are not just Linux file errors. They include JSON-RPC replay errors, bdev examine delays, VFIO binding problems, reactor stalls, qpair resets, and metadata operations in blobstore/lvol.

Side-By-Side Mental Model

Linux path                           SPDK path
----------                           ---------
application calls read/write         service calls async SPDK API
kernel owns device driver            SPDK process owns device driver
kernel manages NVMe queues           SPDK manages NVMe queues
interrupts wake sleepers             pollers find completions
general scheduling/fairness          dedicated cores and explicit queues
page cache often involved            DMA-safe data buffers preferred
blocking API is common               callback state machines are normal
OS handles broad policy              storage service must own policy

When The Linux Path Is Better

SPDK is not automatically the right answer. The Linux path may be better when:

  • You need normal filesystems and POSIX semantics.
  • You need ordinary process isolation and device sharing.
  • IO rate is modest and development simplicity matters more.
  • CPU cores cannot be dedicated to polling.
  • Operational teams already understand kernel storage better.
  • Latency targets are not tight enough to justify SPDK complexity.
  • The workload benefits from the page cache.
  • You need mature kernel features such as broad hardware quirks, power management, or conventional multipath integration.

The worst SPDK design is a system that pays all of SPDK's complexity costs but does not use its queue ownership, polling, batching, or async model.

When SPDK Is The Right Tool

SPDK starts to make sense when:

  • You are building a storage service rather than an ordinary application.
  • The service owns the storage devices or remote NVMe-oF connections.
  • The workload is high IOPS or latency-sensitive.
  • You can dedicate CPU cores.
  • The data path is already asynchronous.
  • You can allocate DMA-safe data buffers and avoid page-cache semantics.
  • The control plane can tolerate async operations and explicit failure handling.
  • You need userspace composition of bdevs, lvol, RAID, vhost, NVMe-oF, or vfio-user.

This is why it fits diskengine. diskengine is not trying to be cp or sqlite on a laptop filesystem. It is orchestrating cloud volumes, bdev stacks, NVMe-oF exports, vhost/vfio-user exposure, snapshots, RAID, and recovery loops.

Source Reading Path

Read these files in this order, with one question in mind at every step: who owns the thread, who owns the queue, and who calls the completion?

  1. scripts/setup.sh: start with configure_linux_pci() and configure_linux(). This is where host setup chooses vfio-pci or UIO-style drivers and ensures hugepages are mounted and allocated.
  2. lib/event/app.c: read spdk_app_start() and app_setup_env(). This is where a normal process becomes an SPDK event-framework process with selected cores, app state, and environment options.
  3. lib/env_dpdk/init.c and lib/env_dpdk/env.c: read spdk_env_init(), build_eal_cmdline(), and the spdk_dma_*() allocators. This connects SPDK's environment API to DPDK EAL and DMA-capable memory.
  4. lib/event/reactor.c: read reactor_run() and _reactor_run(). This is the polling loop that replaces "sleep until kernel wakes me" with "keep checking SPDK threads for work."
  5. lib/thread/thread.c: read spdk_thread_send_msg() and spdk_thread_poll(). This explains how work moves between SPDK threads without treating every object as globally lockable.
  6. lib/bdev/bdev.c: read spdk_bdev_read(), bdev_read_blocks_with_md(), bdev_io_submit(), and _bdev_io_submit(). This is the logical block IO path before any NVMe command exists.
  7. module/bdev/nvme/bdev_nvme.c: read bdev_nvme_submit_request(), _bdev_nvme_submit_request(), and bdev_nvme_readv(). This is where the bdev IO becomes an NVMe namespace/qpair submission.
  8. lib/nvme/nvme.c and lib/nvme/nvme_qpair.c: read nvme_wait_for_completion_poll() and spdk_nvme_qpair_process_completions(). This is where polling turns completion queue entries into callbacks.

Edge Cases And Failure Modes

Polling burns CPU because the reactor is doing useful negative work: it is repeatedly checking queues and pollers so future completions can be handled without interrupt wakeup latency. That can be the correct tradeoff for a storage appliance, but it is a poor default for a lightly used process on a shared host. If an operator expects idle SPDK cores to look like idle kernel-block applications, the monitoring interpretation will be wrong.

Blocking is poisonous because a reactor is not just "your current request." It may own multiple SPDK threads, message queues, bdev channels, NVMe qpairs, and completion callbacks. A filesystem call, synchronous RPC wait, sleep, or long CPU loop inside a reactor callback can stall unrelated IOs that need the same reactor to poll. When debugging unexplained tail latency, ask first whether the owner thread is still polling.

Wrong memory can fail late. Ordinary process memory is not automatically appropriate for direct userspace DMA data buffers. Some APIs can allocate or bounce through safe buffers, but the fast path assumes I/O data buffers and metadata are compatible with the device and memory-domain rules. If a failure appears as -ENOMEM, an IOMMU mapping problem, or a device submission failure, inspect the buffer source, hugepage setup, NUMA placement, and memory-domain path before assuming the SSD is bad.

Wrong device binding changes what Linux can see. If the NVMe controller is bound to vfio-pci, lsblk usually will not show it because the kernel NVMe block driver no longer owns it. That is expected for direct local SPDK use. The opposite failure is also common: Linux still owns the controller, so SPDK cannot claim it as a userspace PCIe device.

Kernel bypass changes observability. iostat reports kernel block-device activity, so it may be quiet while SPDK drives a VFIO-bound NVMe controller heavily. Use SPDK logs, RPCs, bdev stats, tracepoints, spdk_top, DPDK telemetry where relevant, and process-level CPU counters. The absence of Linux block IO is not proof that no storage IO is happening.

Page cache assumptions break because SPDK bdev IO is block IO. There is no automatic filesystem metadata path, buffered read cache, or POSIX writeback policy in the bdev layer. If a service wants caching, crash consistency, snapshots, checksums, or metadata transactions, it must get them from higher SPDK layers such as blobstore/lvol or from the service's own design.

Crash semantics move upward. If the SPDK process dies, the kernel does not quietly keep exporting the same userspace bdevs. NVMe-oF connections, vhost devices, vfio-user endpoints, in-flight bdev IOs, and control-plane state all need a recovery story. For diskengine-style systems, that means configuration replay, idempotent RPCs, volume ownership rules, and clear handling for partial startup.

Misconceptions To Kill

  • "SPDK is just faster read()." No. It is a different runtime and driver model.
  • "Polling is always better." No. Polling trades CPU for latency and throughput.
  • "Kernel storage is obsolete." No. Kernel storage is the right default for many systems.
  • "Userspace means unsafe." Not exactly. VFIO and IOMMU exist to make userspace DMA device access controlled, but the application still has more responsibility.
  • "SPDK means no kernel at all." No. Linux still provides process isolation, memory management, VFIO/IOMMU infrastructure, networking support, and scheduling of the userspace process.

Lab: Trace One Read Both Ways

  1. Pick one IO: a 4 KiB read at LBA 100.
  2. Draw the Linux path from application to NVMe driver completion.
  3. Draw the SPDK path from spdk_bdev_read() to completion callback.
  4. Mark every place where a thread can sleep.
  5. Mark every place where ownership crosses from one subsystem to another.
  6. Mark where the data buffer must be DMA-safe in the SPDK path.

Source Reading Exercise

Open lib/bdev/bdev.c and find where a bdev IO is submitted. Then open module/bdev/nvme/bdev_nvme.c and find where the NVMe bdev module submits to an NVMe qpair. Write down:

  • The object representing the logical IO.
  • The object representing the per-thread channel.
  • The callback that finishes the IO.
  • The function that would be unsafe to block inside.

Operational Exercise

On a real host, classify a storage problem into Linux-path or SPDK-path first:

  • If lsblk does not show the NVMe device after binding to VFIO, is that a bug?
  • If SPDK can see the controller but iostat is quiet during heavy IO, is that a bug?
  • If CPU usage stays high while idle, is that a bug or an expected polling tradeoff?
  • If an SPDK reactor blocks in a filesystem call, which unrelated IOs might stall?

References

  • SPDK documentation index: https://spdk.io/doc/
  • SPDK message passing and concurrency: https://spdk.io/doc/concurrency.html
  • SPDK event framework: https://spdk.io/doc/event.html
  • SPDK user space drivers: https://spdk.io/doc/userspace.html
  • SPDK NVMe driver guide: https://spdk.io/doc/nvme.html
  • SPDK NVMe submission overview: https://spdk.io/doc/nvme_spec.html
  • SPDK block device guide: https://spdk.io/doc/bdev.html
  • SPDK block device programming guide: https://spdk.io/doc/bdev_pg.html
  • DPDK Environment Abstraction Layer: https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html
  • Linux VFIO documentation: https://docs.kernel.org/driver-api/vfio.html

Self-Check

  • Why can the Linux path be better even if SPDK can be faster?
  • What does polling remove, and what does it cost?
  • Why does SPDK care about hugepages and DMA-safe data buffers?
  • What is the difference between a blocking syscall path and an async callback path?
  • Why does diskengine need to treat SPDK as a storage runtime rather than a library-shaped black box?