Chapter 12: Memory, Iobuf, Mempools, Zero Copy

Reader Promise

By the end of this chapter, a beginner should be able to explain SPDK's memory categories, why DMA-safe allocation is different from ordinary allocation, how mempools and iobuf reduce hot-path allocation, how iobuf wait queues handle NOMEM pressure, and what "zero copy" really means in SPDK contexts.

The chapter also kills a dangerous myth: zero copy does not mean "no memory management." It usually means memory ownership, alignment, lifetime, and device compatibility become more strict.

Mental Model

SPDK memory choices answer three questions:

Who owns this memory?
Can hardware or another process safely access it?
What happens when memory is temporarily unavailable?

Common categories:

ordinary C heap: okay for control-plane metadata, not generally for direct device DMA data buffers
SPDK DMA memory: allocated through spdk_dma_*, suitable for many direct device data paths
mempool objects: fixed-size reusable objects, often for messages or I/O descriptors
memzones: named shared/aligned regions
iobuf buffers: shared runtime data buffers with per-thread caches and wait queues
memory domains: abstraction for memory owned by another DMA-capable domain such as RDMA

The distinction matters because SPDK is not only choosing where bytes live. It is choosing which subsystem is allowed to keep a pointer, whether a device can legally DMA to the backing pages, whether a later completion callback still owns the buffer, and what the code does when the fast path cannot obtain a buffer immediately. In a kernel block stack those details are hidden behind kernel memory pinning, bio pages, and driver-private queues. In SPDK they are explicit user-space contracts.

Official SPDK and DPDK docs line up with this model. SPDK's DMA memory guide explains that data buffers passed to SPDK direct DMA paths need stable physical placement and addresses that can be translated for the device; SPDK relies on DPDK hugepage-backed allocation for that. The SPDK system configuration guide then connects the memory story to VFIO, IOMMU groups, hugetlbfs access, and RLIMIT_MEMLOCK. The DPDK EAL guide adds the lower-level detail: EAL reserves hugepage-backed memory, exposes memzones and memory pools, and in dynamic mode can grow or shrink hugepage use as allocations occur.

Source Anchors

include/spdk/env.h: spdk_malloc(), spdk_zmalloc(), spdk_dma_malloc(), spdk_dma_zmalloc(), spdk_dma_free(), spdk_mempool_create(), spdk_mempool_get(), spdk_mempool_put(), spdk_memzone_reserve(), spdk_vtophys()
lib/env_dpdk/env.c: spdk_malloc(), spdk_zmalloc(), spdk_dma_malloc_socket(), spdk_dma_zmalloc_socket(), spdk_mempool_create_ctor(), spdk_mempool_get(), spdk_mempool_put()
lib/env_dpdk/memory.c: vtophys_init(), spdk_vtophys(), vtophys_notify(), vtophys_iommu_init()
include/spdk/dma.h: struct spdk_memory_domain, spdk_memory_domain_create(), spdk_memory_domain_set_translation(), spdk_memory_domain_translate_data(), spdk_memory_domain_transfer_data(), spdk_memory_domain_get_system_domain()
include/spdk/thread.h: struct spdk_iobuf_opts, struct spdk_iobuf_channel, spdk_iobuf_initialize(), spdk_iobuf_finish(), spdk_iobuf_register_module(), spdk_iobuf_channel_init(), spdk_iobuf_get(), spdk_iobuf_put(), spdk_iobuf_entry_abort(), spdk_iobuf_get_stats()
lib/thread/iobuf.c: spdk_iobuf_initialize(), spdk_iobuf_set_opts(), spdk_iobuf_channel_init(), spdk_iobuf_channel_fini(), spdk_iobuf_get(), spdk_iobuf_put(), spdk_iobuf_for_each_entry(), spdk_iobuf_entry_abort(), spdk_iobuf_get_stats()
lib/nvmf/transport.c: spdk_iobuf_register_module() use, spdk_iobuf_channel_init() use, spdk_iobuf_get() use, spdk_iobuf_put() use, nvmf_request_iobuf_get_cb()
include/spdk/nvmf.h: opts->no_srq and opts->zero_copy related target options, including "Use zero-copy operations if the underlying bdev supports them"
include/spdk_internal/sock_module.h: zerocopy_threshold for socket implementations

DMA-Safe Allocation

SPDK APIs that pass data buffers to direct device or transport DMA paths often require DMA-safe buffers. This rule is about I/O payload buffers, not every allocation in an SPDK process. Ordinary C structs, JSON strings, configuration objects, and other control-plane data can still live on the normal heap unless a specific API says otherwise. The public API is in include/spdk/env.h:

spdk_dma_malloc()
spdk_dma_malloc_socket()
spdk_dma_zmalloc()
spdk_dma_zmalloc_socket()
spdk_dma_realloc()
spdk_dma_free()

In this DPDK env, DMA allocation is not a separate custom allocator. It is a constrained call into the regular SPDK env allocator with flags saying "DMA-capable" and "shareable." The base allocator then enforces a nonzero flag set, rejects the old unused physical-address parameter, rounds alignment up to at least RTE_CACHE_LINE_SIZE, and calls DPDK's socket-aware allocator.

/* lib/env_dpdk/env.c */
void *
spdk_zmalloc(size_t size, size_t align, uint64_t *unused, int numa_id, uint32_t flags)
{
	void *buf;

	if (flags == 0 || unused != NULL) {
		return NULL;
	}

	align = spdk_max(align, RTE_CACHE_LINE_SIZE);
	buf = rte_zmalloc_socket(NULL, size, align, numa_id);
	if (buf == NULL && !g_enforce_numa && numa_id != SOCKET_ID_ANY) {
		buf = rte_zmalloc_socket(NULL, size, align, SOCKET_ID_ANY);
	}
	return buf;
}

The DMA wrappers are intentionally small:

/* lib/env_dpdk/env.c */
void *
spdk_dma_malloc_socket(size_t size, size_t align, uint64_t *unused, int numa_id)
{
	return spdk_malloc(size, align, unused, numa_id, (SPDK_MALLOC_DMA | SPDK_MALLOC_SHARE));
}

void *
spdk_dma_zmalloc_socket(size_t size, size_t align, uint64_t *unused, int numa_id)
{
	return spdk_zmalloc(size, align, unused, numa_id, (SPDK_MALLOC_DMA | SPDK_MALLOC_SHARE));
}

This is why "DMA-safe" is a property of the allocation path, not a type annotation. A void coming from malloc() and a void coming from spdk_dma_zmalloc() look identical to C, but the latter came from the env layer that can cooperate with DPDK, hugepages, IOVA mapping, and SPDK's address translation assumptions.

The unused parameter must be NULL. The implementation returns NULL if it is not.

Beginner rule:

Use the allocation family expected by the API you call. Do not pass a stack buffer or ordinary heap buffer as a direct I/O data buffer just because the type is void *. Some bdev paths can allocate iobuf-backed or bounce buffers when they need staging, but that fallback is path-specific and should not be treated as permission to ignore the data-buffer contract.

Alignment

Alignment appears everywhere in storage:

cache-line alignment avoids false sharing and supports CPU efficiency
device descriptors may require specific alignment
metadata/DIF layouts may require block or protection information boundaries
hugepage-backed memory simplifies translation and pinning

lib/env_dpdk/env.c:spdk_malloc() and spdk_zmalloc() apply at least RTE_CACHE_LINE_SIZE alignment. lib/thread/iobuf.c:spdk_iobuf_initialize() rounds small and large iobuf sizes up to IOBUF_ALIGNMENT.

Misconception to kill:

"If the address is aligned, it is DMA-safe." No. Alignment is one requirement. DMA safety also needs appropriate allocation, mapping, lifetime, and address translation.

Mempools

Mempools are fixed-size object pools. They are ideal for small objects with high allocation frequency and predictable sizes.

In include/spdk/env.h, the mempool API includes:

create/free: spdk_mempool_create(), spdk_mempool_create_ctor(), spdk_mempool_free()
get/put: spdk_mempool_get(), spdk_mempool_get_bulk(), spdk_mempool_put(), spdk_mempool_put_bulk()
introspection: spdk_mempool_count(), spdk_mempool_lookup(), spdk_mempool_obj_iter(), spdk_mempool_mem_iter()

Examples:

lib/thread/thread.c:_thread_lib_init() creates a message mempool.
lib/event/reactor.c:spdk_reactors_init() creates an event mempool.
NVMf RDMA and iSCSI modules create transport/session/task pools.

In the DPDK env implementation, SPDK mempool get/put is a thin wrapper over DPDK's mempool calls:

/* lib/env_dpdk/env.c */
void *
spdk_mempool_get(struct spdk_mempool *mp)
{
	void *ele;
	int rc;

	rc = rte_mempool_get((struct rte_mempool *)mp, &ele);
	if (rc != 0) {
		return NULL;
	}
	return ele;
}

void
spdk_mempool_put(struct spdk_mempool *mp, void *ele)
{
	rte_mempool_put((struct rte_mempool *)mp, ele);
}

The DPDK mempool guide describes a memory pool as a named allocator of fixed-size objects, typically backed by a ring and optionally accelerated by per-core caches. That is the same shape SPDK wants in hot paths: known object size, bounded object count, fast get/put, and an explicit NULL/error path when the pool is empty.

Operational meaning:

Mempools turn allocation pressure into explicit resource pressure. If a mempool is empty, the system should either queue, retry, apply backpressure, or fail cleanly.

Iobuf: Why It Exists

iobuf is a shared pool of data buffers with per-thread caches and wait queues. It exists because many SPDK transports and modules need temporary data buffers, but allocating from the heap in the I/O path is too slow and unpredictable.

The public iobuf API lives in include/spdk/thread.h, not in a separate iobuf.h in this tree.

Important types:

struct spdk_iobuf_opts: pool counts, buffer sizes, NUMA behavior
struct spdk_iobuf_channel: per-thread cache state
struct spdk_iobuf_entry: wait queue entry for async buffer acquisition

Important functions:

spdk_iobuf_set_opts()
spdk_iobuf_initialize()
spdk_iobuf_register_module()
spdk_iobuf_channel_init()
spdk_iobuf_get()
spdk_iobuf_put()
spdk_iobuf_entry_abort()
spdk_iobuf_channel_fini()
spdk_iobuf_finish()

Iobuf Initialization

lib/thread/iobuf.c:spdk_iobuf_initialize():

Rounds small and large buffer sizes up to the iobuf alignment.
Initializes iobuf nodes for each relevant NUMA ID.
Registers &g_iobuf as an io_device.
Marks iobuf initialized.

Because iobuf is registered as an io_device, it uses the io_channel mechanism from the previous chapter.

spdk_iobuf_finish() unregisters that io_device and eventually frees modules and node pools in iobuf_unregister_cb().

The initialization code shows three design choices at once: buffer sizes are rounded so each object remains aligned, a node pool is built per relevant NUMA ID, and the whole iobuf service is registered as an io_device so threads access it through channels.

/* lib/thread/iobuf.c */
int
spdk_iobuf_initialize(void)
{
	struct spdk_iobuf_opts *opts = &g_iobuf.opts;
	struct iobuf_node *node;
	int32_t i;
	int rc = 0;

	/* Round up to the nearest alignment so that each element remains aligned */
	opts->small_bufsize = SPDK_ALIGN_CEIL(opts->small_bufsize, IOBUF_ALIGNMENT);
	opts->large_bufsize = SPDK_ALIGN_CEIL(opts->large_bufsize, IOBUF_ALIGNMENT);

	IOBUF_FOREACH_NUMA_ID(i) {
		node = &g_iobuf.node[i];
		rc = iobuf_node_initialize(node, i);
		if (rc) {
			goto err;
		}
	}

	spdk_io_device_register(&g_iobuf, iobuf_channel_create_cb, iobuf_channel_destroy_cb,
				sizeof(struct iobuf_channel), "iobuf");
	g_iobuf_is_initialized = true;

	return 0;

The important ownership rule is that a module does not directly hold the central pools. It registers, opens an iobuf channel on the current SPDK thread, and then borrows buffers through that channel.

Iobuf Module Registration

Only registered iobuf modules can create iobuf channels. lib/thread/iobuf.c:spdk_iobuf_register_module() stores module names in g_iobuf.modules. spdk_iobuf_channel_init() searches for the module name before creating the channel.

This is a useful guardrail:

It lets stats be grouped by module.
It prevents accidental anonymous pool usage.
It makes wait queues module-aware.

Example:

lib/nvmf/transport.c builds an iobuf module name for transports and calls spdk_iobuf_register_module() when the transport uses iobuf.

Iobuf Channels And Per-Thread Caches

lib/thread/iobuf.c:spdk_iobuf_channel_init():

Verifies the module exists.
Gets a parent io_channel for &g_iobuf.
Finds a free channel slot in the parent channel context.
Sets ch->parent and ch->module.
Initializes small and large caches for each NUMA ID.
Populates caches from central pools.

The per-thread channel caches reduce contention on central pools. A hot thread can get and put buffers from its local cache most of the time.

Channel creation makes the module guardrail and the io_channel dependency concrete:

/* lib/thread/iobuf.c */
int
spdk_iobuf_channel_init(struct spdk_iobuf_channel *ch, const char *name,
			uint32_t small_cache_size, uint32_t large_cache_size)
{
	struct spdk_io_channel *ioch;
	struct iobuf_channel *iobuf_ch;
	struct iobuf_module *module;
	uint32_t i;
	int32_t numa_id;
	int rc;

	TAILQ_FOREACH(module, &g_iobuf.modules, tailq) {
		if (strcmp(name, module->name) == 0) {
			break;
		}
	}

	if (module == NULL) {
		SPDK_ERRLOG("Couldn't find iobuf module: '%s'\n", name);
		return -ENODEV;
	}

	ioch = spdk_get_io_channel(&g_iobuf);
	if (ioch == NULL) {
		SPDK_ERRLOG("Couldn't get iobuf IO channel\n");
		return -ENOMEM;
	}

The channel stores both the parent io_channel and the module pointer. Later, wait queue entries record that module, which lets teardown and stats reason about ownership.

Failure mode:

If cache population cannot dequeue enough buffers from the central pool, initialization returns -ENOMEM and logs that the user may need to increase small_pool_count or large_pool_count.

`spdk_iobuf_get()`

lib/thread/iobuf.c:spdk_iobuf_get() takes:

iobuf channel
requested length
optional wait entry
optional callback

It asserts the parent io_channel belongs to the current spdk_thread.

Then it chooses the small or large pool:

len <= small.bufsize: small pool
otherwise: large pool, with an assertion that len <= large.bufsize

Then:

- if an entry and callback are provided, queue the entry and callback as a pair - return NULL

If a local cached buffer exists, return it immediately.
Otherwise dequeue a batch from the central ring, cache all but one, and return one.
If no central buffers exist:

Important callback detail:

If a buffer is available immediately, the callback is not executed. The caller receives the buffer directly.

Important ownership detail:

If no buffer is available and spdk_iobuf_get() queues the entry, the wait entry belongs to the iobuf wait queue until the callback runs or until the caller aborts that exact queued request with spdk_iobuf_entry_abort(). Keep the struct spdk_iobuf_entry storage alive and do not reuse it for another request while it is queued. The abort API in include/spdk/thread.h:1273 through include/spdk/thread.h:1281 also requires the same length that was passed to the original spdk_iobuf_get().

Do not pass an entry without a valid callback. include/spdk/thread.h:1285 through include/spdk/thread.h:1298 documents that the callback is mandatory when an entry is provided. The implementation stores entry->cb_fn = cb_fn when queuing and spdk_iobuf_put() later calls entry->cb_fn(entry, buf).

The code below is the core of the NOMEM behavior. spdk_iobuf_get() is running on the thread that owns the channel. It tries the channel cache first, then the central ring, and only queues the entry if both are empty.

/* lib/thread/iobuf.c */
buf = (void *)STAILQ_FIRST(&pool->cache);
if (buf) {
	STAILQ_REMOVE_HEAD(&pool->cache, stailq);
	assert(pool->cache_count > 0);
	pool->cache_count--;
	pool->stats.cache++;
} else {
	struct spdk_iobuf_buffer *bufs[IOBUF_BATCH_SIZE];
	size_t sz, i;

	/* If we're going to dequeue, we may as well dequeue a batch. */
	sz = spdk_ring_dequeue(pool->pool, (void **)bufs, spdk_min(IOBUF_BATCH_SIZE,
			       spdk_max(pool->cache_size, 1)));
	if (sz == 0) {
		if (entry) {
			STAILQ_INSERT_TAIL(pool->queue, entry, stailq);
			entry->module = ch->module;
			entry->cb_fn = cb_fn;
			pool->stats.retry++;
		}

		return NULL;
	}

Notice that NULL does not carry enough meaning by itself. If the caller supplied an entry and callback, ownership of that entry moved to the wait queue and the callback is now the resume path. If the caller did not supply an entry, it simply failed to get a buffer synchronously.

`spdk_iobuf_put()`

lib/thread/iobuf.c:spdk_iobuf_put() returns a buffer.

It:

- returns to local cache or central pool depending on cache size

- removes the first waiter - calls the waiter's callback with the returned buffer

Chooses NUMA ID if iobuf NUMA is enabled.
Chooses small or large pool based on the same length rule.
If no waiters exist:
If waiters exist:

This is an important backpressure path. A waiting I/O can resume when another I/O returns a buffer.

That handoff is visible in spdk_iobuf_put(). With no waiters, the buffer returns to the local cache or central ring. With waiters, SPDK does not cache it; it gives the returned buffer directly to the oldest queued entry.

/* lib/thread/iobuf.c */
if (STAILQ_EMPTY(pool->queue)) {
	if (pool->cache_size == 0) {
		spdk_ring_enqueue(pool->pool, (void **)&buf, 1, NULL);
		return;
	}

	iobuf_buf = (struct spdk_iobuf_buffer *)buf;

	STAILQ_INSERT_HEAD(&pool->cache, iobuf_buf, stailq);
	pool->cache_count++;

	/* The cache size may exceed the configured amount. We always dequeue from the
	 * central pool in batches of known size, so wait until at least a batch
	 * has been returned to actually return the buffers to the central pool. */
	sz = spdk_min(IOBUF_BATCH_SIZE, pool->cache_size);
	if (pool->cache_count >= pool->cache_size + sz) {
		struct spdk_iobuf_buffer *bufs[IOBUF_BATCH_SIZE];
		size_t i;

		for (i = 0; i < sz; i++) {
			bufs[i] = STAILQ_FIRST(&pool->cache);
			STAILQ_REMOVE_HEAD(&pool->cache, stailq);
			assert(pool->cache_count > 0);
			pool->cache_count--;
		}

		spdk_ring_enqueue(pool->pool, (void **)bufs, sz, NULL);
	}
} else {
	entry = STAILQ_FIRST(pool->queue);
	STAILQ_REMOVE_HEAD(pool->queue, stailq);
	entry->cb_fn(entry, buf);

The callback runs on the same thread context that called spdk_iobuf_put(), so the resumed operation must be prepared for callback execution during buffer return. That is common in SPDK: progress frequently happens as part of a completion, message, or resource-return path.

Beginner rule:

The len passed to spdk_iobuf_put() must match the length class used by spdk_iobuf_get(). The public docs explicitly say it must be the exact same value.

NOMEM Is Often A Designed State

When spdk_iobuf_get() returns NULL with a queued entry, that is not necessarily fatal. It can mean "wait until a buffer is returned."

But NOMEM can also indicate bad sizing:

too few small buffers
too few large buffers
per-channel caches too large for central pool
I/O unit size larger than large buffer size
leak where buffers are not returned
wrong path using iobuf for unexpectedly large data

lib/nvmf/transport.c checks iobuf options against transport io_unit_size. It warns when requested shared buffers exceed available pool size.

Aborting Iobuf Waiters

Sometimes a request waiting for a buffer must be canceled because the connection, qpair, or operation is torn down.

lib/thread/iobuf.c:spdk_iobuf_entry_abort() walks NUMA caches and removes the entry from the appropriate wait queue.

lib/nvmf/transport.c uses spdk_iobuf_for_each_entry() and spdk_iobuf_entry_abort() to abort pending buffer requests for requests that should no longer continue.

The channel finalizer asserts that no entries from this module remain queued before it returns cached buffers. This turns the "abort waiters" rule into a debug-time invariant.

/* lib/thread/iobuf.c */
/* Make sure none of the wait queue entries are coming from this module */
STAILQ_FOREACH(entry, cache->small.queue, stailq) {
	assert(entry->module != ch->module);
}
STAILQ_FOREACH(entry, cache->large.queue, stailq) {
	assert(entry->module != ch->module);
}

/* Release cached buffers back to the pool */
while (!STAILQ_EMPTY(&cache->small.cache)) {
	buf = STAILQ_FIRST(&cache->small.cache);
	STAILQ_REMOVE_HEAD(&cache->small.cache, stailq);
	spdk_ring_enqueue(node->small_pool, (void **)&buf, 1, NULL);
	cache->small.cache_count--;
}

Edge case:

If teardown forgets to abort waiters, a later buffer return can call a callback for an operation that no longer has valid ownership.

Memory Domains

include/spdk/dma.h defines memory domains. They abstract memory that may belong to different DMA-capable domains.

Key functions:

spdk_memory_domain_create()
spdk_memory_domain_set_translation()
spdk_memory_domain_set_pull()
spdk_memory_domain_set_push()
spdk_memory_domain_set_data_transfer()
spdk_memory_domain_translate_data()
spdk_memory_domain_transfer_data()
spdk_memory_domain_get_system_domain()

Why this exists:

Some data may live in memory registered with an RDMA NIC, accelerator, GPU-like device, or another domain. Instead of always copying into system memory first, SPDK can ask domains how to translate, pull, push, or transfer data.

Beginner simplification:

Memory domains are SPDK's way to ask, "Can this memory be used from there, and if not, how do we move it?"

The implementation in lib/dma/dma.c is small but important. A memory domain is a registered object with optional callbacks for translate, pull, push, transfer, invalidate, and memzero. The built-in system domain is inserted at process startup.

/* lib/dma/dma.c */
struct spdk_memory_domain {
	enum spdk_dma_device_type type;
	spdk_memory_domain_pull_data_cb pull_cb;
	spdk_memory_domain_push_data_cb push_cb;
	spdk_memory_domain_transfer_data_cb transfer_cb;
	spdk_memory_domain_translate_memory_cb translate_cb;
	spdk_memory_domain_invalidate_data_cb invalidate_cb;
	spdk_memory_domain_memzero_cb memzero_cb;
	TAILQ_ENTRY(spdk_memory_domain) link;
	struct spdk_memory_domain_ctx *ctx;
	char *id;
	size_t user_ctx_size;
	uint8_t user_ctx[];
};

static struct spdk_memory_domain g_system_domain = {
	.type = SPDK_DMA_DEVICE_TYPE_DMA,
	.id = "system",
};

The public translate_data operation is deliberately not a copy. include/spdk/dma.h says it translates a description of memory from one domain into another and that no data is moved. If a domain cannot translate, the implementation returns -ENOTSUP, which lets the upper layer choose a fallback.

/* lib/dma/dma.c */
int
spdk_memory_domain_translate_data(struct spdk_memory_domain *src_domain, void *src_domain_ctx,
				  struct spdk_memory_domain *dst_domain,
				  struct spdk_memory_domain_translation_ctx *dst_domain_ctx,
				  void *addr, size_t len,
				  struct spdk_memory_domain_translation_result *result)
{
	assert(src_domain);
	assert(dst_domain);
	assert(result);

	if (spdk_unlikely(!src_domain->translate_cb)) {
		return -ENOTSUP;
	}

	return src_domain->translate_cb(src_domain, src_domain_ctx, dst_domain, dst_domain_ctx,
					addr, len, result);
}

Zero Copy

"Zero copy" means a data path avoids one or more CPU memory copies. It does not mean:

no DMA
no descriptors
no memory registration
no ownership rules
no fallback path
no metadata handling

SPDK has several zero-copy-adjacent concepts:

DMA-safe data buffers avoid copying into driver-owned kernel memory.
NVMe-oF may use transport buffers or bdev-provided buffers.
Socket implementations may use MSG_ZEROCOPY depending on thresholds; see include/spdk_internal/sock_module.h.
NVMf target options include using zero-copy operations if the underlying bdev supports them; see include/spdk/nvmf.h.
Memory domains can allow direct data movement between domains.

The practical question is not "is this zero-copy?" It is:

Who owns the buffer?
Is the buffer valid until completion?
Is it aligned and registered for the device or transport?
Can the next layer consume the same iovecs?
What fallback happens if zero-copy is unsupported?

The bdev layer is a good place to see a fallback. When a module asks bdev for a data buffer, bdev first accepts already-present aligned iovecs. If the caller's iovs are missing or not aligned for the bdev, it allocates an iobuf-backed buffer and completes later through a callback. This is about the data buffers associated with an I/O, not ordinary control-plane allocations.

/* lib/bdev/bdev.c */
void
spdk_bdev_io_get_buf(struct spdk_bdev_io *bdev_io, spdk_bdev_io_get_buf_cb cb, uint64_t len)
{
	struct spdk_bdev *bdev = bdev_io->bdev;
	uint64_t alignment;

	assert(cb != NULL);
	bdev_io->internal.get_buf_cb = cb;

	alignment = spdk_bdev_get_buf_align(bdev);

	if (_is_buf_allocated(bdev_io->u.bdev.iovs) &&
	    _are_iovs_aligned(bdev_io->u.bdev.iovs, bdev_io->u.bdev.iovcnt, alignment)) {
		/* Buffer already present and aligned */
		cb(spdk_bdev_io_get_io_channel(bdev_io), bdev_io, true);
		return;
	}

	bdev_io_get_buf(bdev_io, len);
}

The lower helper asks iobuf for enough space to cover payload, alignment slack, and separate metadata. Immediate success continues synchronously; NOMEM with a queued entry resumes in bdev_io_get_iobuf_cb().

/* lib/bdev/bdev.c */
max_len = bdev_io_get_max_buf_len(bdev_io, len);

if (spdk_unlikely(max_len > mgmt_ch->iobuf.cache[0].large.bufsize)) {
	SPDK_ERRLOG("Length %" PRIu64 " is larger than allowed\n", max_len);
	bdev_io_get_buf_complete(bdev_io, false);
	return;
}

bdev_io->internal.buf.len = len;
buf = spdk_iobuf_get(&mgmt_ch->iobuf, max_len, &bdev_io->internal.iobuf,
		     bdev_io_get_iobuf_cb);
if (buf != NULL) {
	_bdev_io_set_buf(bdev_io, buf, len);
}

When a bounce buffer is needed, bdev saves the original iovecs, replaces the I/O's visible data iovec with one managed buffer, and pulls or pushes data around the device operation. This is not an accidental copy; it is the correctness path for incompatible memory domains, metadata handling, or alignment.

/* lib/bdev/bdev.c */
static void
_bdev_io_pull_bounce_data_buf(struct spdk_bdev_io *bdev_io, void *buf, size_t len,
			      bdev_copy_bounce_buffer_cpl cpl_cb)
{
	struct spdk_bdev_shared_resource *shared_resource = bdev_io->internal.ch->shared_resource;

	assert(bdev_io->internal.f.has_bounce_buf == false);

	bdev_io->internal.data_transfer_cpl = cpl_cb;
	bdev_io->internal.f.has_bounce_buf = true;
	/* save original iovec */
	bdev_io->internal.bounce_buf.orig_iovs = bdev_io->u.bdev.iovs;
	bdev_io->internal.bounce_buf.orig_iovcnt = bdev_io->u.bdev.iovcnt;
	/* set bounce iov */
	bdev_io->u.bdev.iovs = &bdev_io->internal.bounce_buf.iov;
	bdev_io->u.bdev.iovcnt = 1;
	bdev_io->u.bdev.iovs[0].iov_base = buf;
	bdev_io->u.bdev.iovs[0].iov_len = len;

The extended bdev submit path decides whether that fallback is required. The comments in the source make the reason explicit: if the bdev cannot access a memory domain directly, or if metadata must be inserted/stripped for an unaware I/O, bdev cannot simply pass the original iovs through.

/* lib/bdev/bdev.c */
/* We need to allocate bounce buffer
 * - if bdev doesn't support memory domains,
 * -  if it does support them, but we need to execute an accel sequence and the data buffer is
 *    from accel memory domain (to avoid doing a push/pull from that domain), or
 * - if IO is not aware of metadata.
 */
static inline bool
bdev_io_needs_bounce_buffer(struct spdk_bdev_desc *desc, struct spdk_bdev_io *bdev_io)
{
	if (bdev_io_use_memory_domain(bdev_io)) {
		if (!bdev_io->bdev->memory_domains_supported ||
		    (bdev_io_needs_sequence_exec(bdev_io) &&
		     (bdev_io->internal.memory_domain == spdk_accel_get_memory_domain() ||
		      bdev_io_needs_metadata(desc, bdev_io)))) {
			return true;
		}

		return false;
	}

For the reader, this is the clearest practical definition of SPDK zero copy: the ideal path reuses the caller's iovecs all the way down, but the real path must prove those iovecs are aligned, domain-compatible, metadata-compatible, and alive until completion. Otherwise SPDK allocates a managed buffer and makes the copy visible in code.

Metadata And DIF/DIX

Storage buffers may include metadata or protection information. The performance app references DIF/DIX paths in app/spdk_nvme_perf/perf.c, and many public APIs distinguish data and metadata buffers.

Metadata complicates zero-copy:

metadata may be separate from data
protection information may need insert, strip, generate, check, or update
hardware and transport capabilities differ
"hide metadata" options can change what a host sees

The bdev helper bdev_io_get_max_buf_len() shows this as a sizing rule. For separate metadata, the buffer request includes num_blocks * md_len; for alignment, it adds the maximum padding needed to align the returned pointer.

/* lib/bdev/bdev.c */
static inline uint64_t
bdev_io_get_max_buf_len(struct spdk_bdev_io *bdev_io, uint64_t len)
{
	struct spdk_bdev *bdev = bdev_io->bdev;
	uint64_t md_len, alignment;

	md_len = spdk_bdev_is_md_separate(bdev) ? bdev_io->u.bdev.num_blocks * bdev->md_len : 0;

	/* 1 byte alignment needs 0 byte of extra space, 64 bytes alignment needs 63 bytes of extra space, etc. */
	alignment = spdk_bdev_get_buf_align(bdev) - 1;

	return len + alignment + md_len;
}

Beginner rule:

Do not assume a block is only user data bytes. Always inspect bdev block size, metadata size, and protection information settings before reasoning about buffer length.

Bounce Buffers

A bounce buffer is a temporary buffer used when the original buffer cannot be used directly.

Reasons:

source data memory is not DMA-safe
alignment does not meet device requirements
memory domain cannot be translated
metadata layout does not match the next layer
transport requires a contiguous or differently sized buffer

Bounce buffers are not "bad" by themselves. They are a correctness fallback. But unexpected bounce-buffer use can destroy performance, so it should be visible in source reading and metrics.

Edge Cases And Failure Modes

Passing non-NULL unused to SPDK allocation APIs: returns NULL.
Ordinary malloc() buffer passed to DMA path: may fail translation or device access.
iobuf request larger than large buffer size: assertion path.
iobuf put with mismatched length: wrong pool/cache behavior.
Module not registered with iobuf: spdk_iobuf_channel_init() returns -ENODEV.
Not enough pool entries to populate channel caches: -ENOMEM.
Forgetting to return iobuf buffers: pool starvation and wait queues grow.
Forgetting to abort waiters on teardown: callback can fire after owner is gone.
NUMA pool too small for selected topology: startup or channel init fails.
Zero-copy path unsupported by underlying bdev: must fall back or fail as designed.
Metadata/DIF path changes data length assumptions.

Misconceptions To Kill

"Zero copy means no buffers." Zero-copy is still buffer management.
"DMA memory is just fast memory." It is memory with properties needed for device access.
"Mempools are premature optimization." In SPDK hot paths, they are how failure and latency are controlled.
"NOMEM always means fatal." It may mean queue and retry.
"iobuf is global only." It has global pools plus per-thread channel caches.
"Metadata is rare, so ignore it." Metadata and protection info are central to many enterprise storage paths.

Diskengine Relevance

Diskengine may observe SPDK failures as generic I/O failures or slow operations. Memory pressure can be the hidden cause.

Useful classification:

Env memory failure: hugepages or DPDK memory unavailable.
Mempool exhaustion: fixed object pool too small or leak.
Iobuf pressure: data buffers exhausted; requests wait.
DMA translation failure: buffer or device domain mismatch.
Zero-copy fallback: correctness preserved but latency/CPU changes.
Metadata mismatch: request shape incompatible with bdev or export path.

When a diskengine reconciliation loop repeatedly retries an SPDK RPC or I/O operation, inspect whether SPDK is making progress or waiting on buffers.

Prose Diagram: Iobuf Get/Put

Think of iobuf as a warehouse with thread-local shelves:

A module registers as a warehouse customer.
Each SPDK thread opens a local shelf with spdk_iobuf_channel_init().
spdk_iobuf_get() first checks the local shelf.
If empty, it takes a box from the central warehouse and may stock extra boxes on the shelf.
If the warehouse is empty, the request writes its name on a waiting list.
spdk_iobuf_put() either puts the box back on the shelf or hands it directly to the first waiter.

The warehouse is shared, but shelf access is thread-owned.

Source Reading Exercise

Read iobuf allocation flow:

lib/thread/iobuf.c:spdk_iobuf_initialize()
lib/thread/iobuf.c:spdk_iobuf_register_module()
lib/thread/iobuf.c:spdk_iobuf_channel_init()
lib/thread/iobuf.c:spdk_iobuf_get()
lib/thread/iobuf.c:spdk_iobuf_put()
lib/thread/iobuf.c:spdk_iobuf_entry_abort()

Then connect it to a transport:

lib/nvmf/transport.c:nvmf_transport_use_iobuf()
lib/nvmf/transport.c:spdk_iobuf_register_module() call sites
lib/nvmf/transport.c:spdk_iobuf_channel_init() call sites
lib/nvmf/transport.c:spdk_iobuf_get() call sites
lib/nvmf/transport.c:nvmf_request_iobuf_get_cb()

Questions:

Which path returns a buffer immediately?
Which path queues an entry?
Where is the module recorded on the wait entry?
How does teardown abort pending entries?

Operational Lab

Source-only sizing lab:

Read struct spdk_iobuf_opts in include/spdk/thread.h.
Write down small pool count, large pool count, small buffer size, large buffer size, and NUMA behavior.
Find a transport or module that calls spdk_iobuf_channel_init().
Compare its small and large cache sizes to the global pool counts.
Explain what happens if every reactor creates a channel at once.

Runtime lab:

Start SPDK with a workload that uses NVMf or another iobuf consumer.
Query iobuf stats if the RPC is available in the built app.
Watch cache hits, main pool use, and retry counts.
Increase queue depth or I/O size and observe whether retries grow.

Self-Check

Why is spdk_dma_zmalloc() different from calloc()?
What are mempools good for?
Why does iobuf have per-thread channels?
What happens when spdk_iobuf_get() cannot allocate a buffer and an entry is provided?
Why must teardown abort iobuf waiters?
What does zero-copy not guarantee?
How can metadata change buffer reasoning?

References

Local source: include/spdk/env.h
Local source: lib/env_dpdk/env.c
Local source: lib/env_dpdk/memory.c
Local source: lib/dma/dma.c
Local source: include/spdk/dma.h
Local source: include/spdk/thread.h
Local source: lib/thread/iobuf.c
Local source: lib/bdev/bdev.c
Local source: lib/nvmf/transport.c
Local source: include/spdk/nvmf.h
Local source: include/spdk_internal/sock_module.h
Official SPDK docs: Direct Memory Access (DMA) From User Space, https://spdk.io/doc/memory.html
Official SPDK docs: System Configuration User Guide, https://spdk.io/doc/system_configuration.html
Official SPDK docs: An Overview of SPDK Applications, memory size and hugepage options, https://spdk.io/doc/app_overview.html
Official DPDK docs: Environment Abstraction Layer guide, https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html
Official DPDK docs: Memory Pool Library guide, https://doc.dpdk.org/guides/prog_guide/mempool_lib.html