SPDK From First Principles

SPDK deep learning path

Chapter 11: `io_device` And `io_channel`

By the end of this chapter, a beginner should be able to explain why SPDK has

Source: drafts/runtime/11-io-device-and-io-channel.md

Reader Promise

By the end of this chapter, a beginner should be able to explain why SPDK has io_device and io_channel, how channels provide per-thread resources, why spdk_get_io_channel() and spdk_put_io_channel() must happen on the owning thread, and how spdk_for_each_channel() safely visits every thread-local channel for a device.

This chapter is the bridge between the execution model and the bdev chapters. Bdevs, NVMe-oF poll groups, accelerators, iobuf, and many other SPDK components use io_channels to avoid shared locks in hot I/O paths.

The official SPDK concurrency guide describes the original pattern this way: lower layers often had devices with multiple queues that could be assigned to threads and then used without a lock. SPDK generalized the shared device as spdk_io_device and the thread-specific queue or state as spdk_io_channel. The same guide also stresses that SPDK's thread abstraction does not create system threads itself; an external event loop repeatedly polls each spdk_thread, and SPDK uses messages to run work on the right thread.

Primary references:

  • SPDK Message Passing and Concurrency: <https://spdk.io/doc/concurrency.html>
  • SPDK thread.h API reference: <https://spdk.io/doc/thread_8h.html>

Mental Model

An io_device is the shared identity of something that can have per-thread I/O state.

An io_channel is the per-spdk_thread state for that device.

The important word is "identity." An io_device is not a base class and it is not necessarily a hardware device. In SPDK, the registered io_device key is an opaque pointer. Its uniqueness comes from its address. A bdev module can use a controller object, NVMf can use a target object, and iobuf can use a singleton global. Once the pointer is registered, each SPDK thread can ask for its own channel attached to that identity.

Prose diagram:

io_device: NVMe bdev controller or NVMf target or iobuf singleton
  shared data:
    name
    create_channel callback
    destroy_channel callback
    registered/unregistered state

spdk_thread A
  io_channel for this device
    private context for A

spdk_thread B
  io_channel for this device
    private context for B

The point is not object orientation. The point is hot-path locality. Each SPDK thread gets its own channel context so it can submit I/O without taking a global lock for every operation. Shared state still exists, but the per-I/O fast path should mostly touch data owned by the current spdk_thread.

That division only works because SPDK is already message-driven. When code must touch another thread's channel, it sends a message to that thread. It does not borrow the channel pointer and mutate it from the wrong CPU.

Source Anchors

  • include/spdk/thread.h: spdk_io_device_register(), spdk_io_device_unregister(), spdk_get_io_channel(), spdk_put_io_channel(), spdk_io_channel_get_ctx(), spdk_io_channel_from_ctx(), spdk_io_channel_get_thread(), spdk_for_each_channel(), spdk_for_each_channel_continue()
  • lib/thread/thread.c: struct io_device, struct spdk_io_channel, spdk_io_device_register(), spdk_io_device_unregister(), spdk_get_io_channel(), spdk_put_io_channel(), put_io_channel(), thread_get_io_channel(), spdk_io_channel_ref(), spdk_io_channel_get_ctx(), spdk_for_each_channel(), _call_channel(), spdk_for_each_channel_continue(), _call_completion(), __pending_unregister()
  • lib/thread/iobuf.c: spdk_iobuf_initialize(), spdk_iobuf_channel_init(), spdk_iobuf_channel_fini(), spdk_iobuf_get_stats()
  • lib/nvmf/nvmf.c: nvmf_tgt_create_poll_group(), nvmf_tgt_destroy_poll_group(), spdk_io_device_register() use for the NVMf target
  • lib/nvmf/transport.c: spdk_for_each_channel() use for transport listener and poll group operations

Why Channels Exist

Imagine an NVMe controller with one shared object and many reactor threads. Each reactor needs its own queue pair or poll-group state. If every I/O used one global queue protected by a mutex, SPDK would lose much of its value.

Instead:

  • The controller or module registers an io_device.
  • Each spdk_thread asks for a channel when it needs to do I/O.
  • The create callback allocates or initializes per-thread resources.
  • The hot path uses the channel.
  • The destroy callback releases per-thread resources.

This keeps shared state small and moves hot I/O state into thread-local ownership. The model also gives SPDK a generic way to visit all per-thread state for a device when a global operation has to be translated into local work on each thread.

This is why the names can feel narrower than the implementation. A channel can represent a hardware submission queue, but it can also represent a poll group, a cache, a per-thread stats bucket, or a module-specific context object. The common property is not hardware. The common property is "owned by one spdk_thread for one registered device identity."

Registering An io_device

lib/thread/thread.c:spdk_io_device_register() registers a device pointer plus callbacks:

spdk_io_channel header

  • io_device: caller-owned identity pointer
  • create_cb: called when a thread creates its first channel for the device
  • destroy_cb: called when a thread releases its last channel for the device
  • ctx_size: bytes of per-channel context to allocate after the
  • name: debug name

The internal object created by registration is small. It stores the caller's opaque device pointer, the channel callbacks, the context size, unregister state, a device refcount, and a tree of threads that currently have a channel for this device.

/* lib/thread/thread.c */
struct io_device {
	void				*io_device;
	char				name[SPDK_MAX_DEVICE_NAME_LEN + 1];
	spdk_io_channel_create_cb	create_cb;
	spdk_io_channel_destroy_cb	destroy_cb;
	spdk_io_device_unregister_cb	unregister_cb;
	struct spdk_thread		*unregister_thread;
	uint32_t			ctx_size;
	uint32_t			for_each_count;
	RB_ENTRY(io_device)		node;

	uint32_t			refcnt;

	bool				pending_unregister;
	bool				unregistered;
	RB_HEAD(thread_link_tree, thread_link) threads;
};

The threads tree is the key to spdk_for_each_channel(). It is not a list of all SPDK threads. It is only the threads that currently have a channel for this device. That keeps channel iteration scoped to actual users.

Registration requires a current spdk_thread. Calling it from a non-SPDK thread logs an error and asserts. The function allocates an internal struct io_device, initializes its thread tree and refcount, and inserts it into the global io_device tree under g_devlist_mutex.

/* lib/thread/thread.c */
void
spdk_io_device_register(void *io_device, spdk_io_channel_create_cb create_cb,
			spdk_io_channel_destroy_cb destroy_cb, uint32_t ctx_size,
			const char *name)
{
	struct io_device *dev, *tmp;
	struct spdk_thread *thread;

	assert(io_device != NULL);
	assert(create_cb != NULL);
	assert(destroy_cb != NULL);

	thread = spdk_get_thread();
	if (!thread) {
		SPDK_ERRLOG("called from non-SPDK thread\n");
		assert(false);
		return;
	}

	dev = calloc(1, sizeof(struct io_device));
	if (dev == NULL) {
		SPDK_ERRLOG("could not allocate io_device\n");
		return;
	}

The function continues by copying the name, saving callbacks, initializing RB_INIT(&dev->threads), and inserting the device in g_io_devices. Duplicate registration of the same identity pointer is rejected by the red-black tree insert.

Beginner rule:

The io_device pointer is a key. It must remain valid until unregistration and all channels complete destruction. SPDK stores that pointer and passes it back to create and destroy callbacks; it does not own the pointed-to object.

Public API Contract

The public header is worth reading before the implementation because it states the user-visible lifetime rules directly.

/* include/spdk/thread.h */
/**
 * Release a reference to an I/O channel. This happens asynchronously.
 *
 * This must be called on the same thread that called spdk_get_io_channel()
 * for the specified I/O channel. If this releases the last reference to the
 * I/O channel, The destroy_cb function specified in spdk_io_device_register()
 * will be invoked to release any associated resources.
 *
 * \param ch I/O channel to release a reference.
 */
void spdk_put_io_channel(struct spdk_io_channel *ch);

/**
 * Take a reference to an existing I/O channel.
 *
 * This can be called on an existing io_channel that was previously returned from
 * spdk_get_io_channel(). This must be called on the same thread that called
 * spdk_get_io_channel() for the specified I/O channel. spdk_put_io_channel() must
 * be called to release the reference when it is no longer needed.
 */
struct spdk_io_channel *spdk_io_channel_ref(struct spdk_io_channel *ch);

Two details matter:

operations.

  • spdk_put_io_channel() is asynchronous.
  • spdk_put_io_channel() and spdk_io_channel_ref() are same-thread

Those rules are not suggestions. They are part of the ownership model. A channel pointer is a capability to use one thread's private context, so it has to be released on that same thread.

Getting A Channel

lib/thread/thread.c:spdk_get_io_channel():

  1. Finds the registered io_device.
  2. Gets the current spdk_thread.
  3. Rejects exited threads.
  4. Checks whether this thread already has a channel for the device.
  5. If yes, increments the channel refcount and returns it.
  6. If no, allocates struct spdk_io_channel + ctx_size.
  7. Inserts the channel into the thread's io_channel tree.
  8. Increments the device refcount.
  9. Adds the thread to the device's thread tree.
  10. Calls the device create callback.
  11. On create failure, unwinds the insertion and refcount.

The existing-channel case is the first subtle point. A thread has at most one channel for a given registered device, but that one channel can have multiple references. Repeated gets on the same thread return the same channel pointer with a larger refcount.

/* lib/thread/thread.c */
ch = thread_get_io_channel(thread, dev);
if (ch != NULL) {
	ch->ref++;

	SPDK_DEBUGLOG(thread, "Get io_channel %p for io_device %s (%p) on thread %s refcnt %u\n",
		      ch, dev->name, dev->io_device, thread->name, ch->ref);

	/*
	 * An I/O channel already exists for this device on this
	 *  thread, so return it.
	 */
	pthread_mutex_unlock(&g_devlist_mutex);
	spdk_trace_record(TRACE_THREAD_IOCH_GET, 0, 0,
			  (uint64_t)spdk_io_channel_get_ctx(ch), ch->ref);
	return ch;
}

That is why every successful get needs a matching put. If one subsystem gets a channel twice and only puts once, the destroy callback will not run, and the thread will keep a channel reference alive.

The new-channel case allocates the channel header and the caller's context in one block. The context pointer returned by spdk_io_channel_get_ctx(ch) is just the bytes immediately after the struct spdk_io_channel header. This is simple and fast: one allocation, stable address, no separate map lookup for the per-device context.

/* lib/thread/thread.c */
ch = calloc(1, sizeof(*ch) + dev->ctx_size);
if (ch == NULL) {
	SPDK_ERRLOG("could not calloc spdk_io_channel\n");
	pthread_mutex_unlock(&g_devlist_mutex);
	return NULL;
}

thr_link = calloc(1, sizeof(struct thread_link));
if (thr_link == NULL) {
	free(ch);
	SPDK_ERRLOG("could not calloc thread_link\n");
	pthread_mutex_unlock(&g_devlist_mutex);
	return NULL;
}

ch->dev = dev;
ch->destroy_cb = dev->destroy_cb;
ch->thread = thread;
ch->ref = 1;
ch->destroy_ref = 0;
RB_INSERT(io_channel_tree, &thread->io_channels, ch);

dev->refcnt++;

thr_link->thread = thread;
thr_link->id = thread->id;
if (RB_INSERT(thread_link_tree, &dev->threads, thr_link)) {
	assert(false);
}

After the channel is inserted and the device refcount is raised, SPDK drops the global device-list mutex and calls the device's create callback:

/* lib/thread/thread.c */
pthread_mutex_unlock(&g_devlist_mutex);

rc = dev->create_cb(io_device, (uint8_t *)ch + sizeof(*ch));
if (rc != 0) {
	pthread_mutex_lock(&g_devlist_mutex);
	RB_REMOVE(io_channel_tree, &ch->thread->io_channels, ch);
	dev->refcnt--;
	free(ch);
	RB_REMOVE(thread_link_tree, &dev->threads, thr_link);
	free(thr_link);
	SPDK_ERRLOG("could not create io_channel for io_device %s (%p): %s (rc=%d)\n",
		    dev->name, io_device, spdk_strerror(-rc), rc);
	if (dev->unregistered && dev->refcnt == 0) {
		do_remove_dev = true;
	}
	pthread_mutex_unlock(&g_devlist_mutex);

The callback is not invoked under g_devlist_mutex. That avoids making device specific initialization block unrelated io_device registration, lookup, and unregister work. The cost is that failure handling must unwind carefully. The implementation removes the channel from the thread tree, decrements the device refcount, removes the thread link, frees both allocations, and handles the corner case where the device was unregistered while the create callback was running.

The channel context is accessed with:

void *ctx = spdk_io_channel_get_ctx(ch);

ctx has the size supplied at registration time. The type is private to the module that registered the device.

Putting A Channel

lib/thread/thread.c:spdk_put_io_channel():

the same thread to run put_io_channel()

  • verifies there is a current SPDK thread
  • verifies the channel belongs to this thread
  • decrements the channel refcount
  • if the refcount reaches zero, increments destroy_ref and sends a message to
/* lib/thread/thread.c */
void
spdk_put_io_channel(struct spdk_io_channel *ch)
{
	struct spdk_thread *thread;

	spdk_trace_record(TRACE_THREAD_IOCH_PUT, 0, 0,
			  (uint64_t)spdk_io_channel_get_ctx(ch), ch->ref);

	thread = spdk_get_thread();
	if (!thread) {
		SPDK_ERRLOG("called from non-SPDK thread\n");
		assert(false);
		return;
	}

	if (ch->thread != thread) {
		wrong_thread(__func__, "ch", ch->thread, thread);
		return;
	}

	ch->ref--;

	if (ch->ref == 0) {
		ch->destroy_ref++;
		spdk_thread_send_msg(thread, put_io_channel, ch);
	}
}

Why deferred destruction?

Because code may call spdk_put_io_channel() while still unwinding a stack that used the channel. Deferring actual destruction to a later message makes the lifetime safer and lets an immediate re-get on the same thread race cleanly against destruction. If another reference appears before the deferred message runs, the message observes that and leaves the channel alive.

/* lib/thread/thread.c */
put_io_channel(void *arg)
{
	struct spdk_io_channel *ch = arg;
	bool do_remove_dev = true;
	struct spdk_thread *thread;
	struct thread_link *thr_link, *ptmp;

	thread = spdk_get_thread();
	if (!thread) {
		SPDK_ERRLOG("called from non-SPDK thread\n");
		assert(false);
		return;
	}

	assert(ch->thread == thread);

	ch->destroy_ref--;

	if (ch->ref > 0 || ch->destroy_ref > 0) {
		/*
		 * Another reference to the associated io_device was requested
		 *  after this message was sent but before it had a chance to
		 *  execute.
		 */
		return;
	}

Only after that check does SPDK remove the channel from the thread tree, remove the thread link from the device, call the destroy callback, decrement the device refcount, and possibly free an already-unregistered device.

/* lib/thread/thread.c */
pthread_mutex_lock(&g_devlist_mutex);
RB_REMOVE(io_channel_tree, &ch->thread->io_channels, ch);
RB_FOREACH_SAFE(thr_link, thread_link_tree, &ch->dev->threads, ptmp) {
	if (thr_link->thread == thread) {
		RB_REMOVE(thread_link_tree, &ch->dev->threads, thr_link);
		free(thr_link);
		break;
	}
}
pthread_mutex_unlock(&g_devlist_mutex);

/* Don't hold the devlist mutex while the destroy_cb is called. */
ch->destroy_cb(ch->dev->io_device, spdk_io_channel_get_ctx(ch));

pthread_mutex_lock(&g_devlist_mutex);
ch->dev->refcnt--;

The destroy callback runs on the owning thread, but not while the global device list mutex is held. That matters because destroy callbacks can be module specific and may need to tear down resources that have their own locks or message ordering.

Wrong-Thread Rules

The docs in include/spdk/thread.h say spdk_put_io_channel() must be called on the same thread that called spdk_get_io_channel(). The implementation enforces this through wrong_thread() in lib/thread/thread.c.

/* lib/thread/thread.c */
static void
wrong_thread(const char *func, const char *name, struct spdk_thread *thread,
	     struct spdk_thread *curthread)
{
	if (thread == NULL) {
		SPDK_ERRLOG("%s(%s) called with NULL thread\n", func, name);
		abort();
	}
	SPDK_ERRLOG("%s(%s) called from wrong thread %s:%" PRIu64 " (should be "
		    "%s:%" PRIu64 ")\n", func, name, curthread->name, curthread->id,
		    thread->name, thread->id);
	assert(false);
}

This rule surprises beginners because the channel pointer looks like an ordinary C pointer. It is not ordinary ownership. It is a thread-local capability.

Common wrong-thread bug:

Thread A gets channel
Thread A submits async operation
Completion runs on Thread B
Completion calls spdk_put_io_channel(channel_from_A)
wrong-thread assert

The fix is usually to send a message back to Thread A or design the operation so completion ownership is clear. Passing a channel pointer to another thread is not automatically wrong if the other thread only uses it as an opaque value to send a message back, but dereferencing it or putting it from that other thread breaks the model.

Unregistering An io_device

lib/thread/thread.c:spdk_io_device_unregister():

completion.

  1. Finds the device.
  2. Records the unregister callback and unregistering thread.
  3. If for_each_count > 0, marks pending unregister and returns.
  4. Marks the device unregistered.
  5. Removes it from the global tree so new lookups fail.
  6. If there are references, defers deletion.
  7. If no references remain, frees it or schedules unregister callback
/* lib/thread/thread.c */
void
spdk_io_device_unregister(void *io_device, spdk_io_device_unregister_cb unregister_cb)
{
	struct io_device *dev;
	uint32_t refcnt;
	struct spdk_thread *thread;

	thread = spdk_get_thread();
	if (!thread) {
		SPDK_ERRLOG("called from non-SPDK thread\n");
		assert(false);
		return;
	}

	pthread_mutex_lock(&g_devlist_mutex);
	dev = io_device_get(io_device);
	if (!dev) {
		SPDK_ERRLOG("io_device %p not found\n", io_device);
		assert(false);
		pthread_mutex_unlock(&g_devlist_mutex);
		return;
	}

The interesting part is the foreach interaction. If a channel iteration is active, unregister cannot remove the device from the tree immediately, because the iterator is still walking the device's thread list. Instead it records a pending unregister and returns.

/* lib/thread/thread.c */
if (dev->pending_unregister && dev->for_each_count > 0) {
	SPDK_ERRLOG("io_device %p already has a pending unregister\n", io_device);
	assert(false);
	pthread_mutex_unlock(&g_devlist_mutex);
	return;
}

dev->unregister_cb = unregister_cb;
dev->unregister_thread = thread;

if (dev->for_each_count > 0) {
	SPDK_WARNLOG("io_device %s (%p) has %u for_each calls outstanding\n",
		     dev->name, io_device, dev->for_each_count);
	dev->pending_unregister = true;
	pthread_mutex_unlock(&g_devlist_mutex);
	return;
}

dev->unregistered = true;
RB_REMOVE(io_device_tree, &g_io_devices, dev);
refcnt = dev->refcnt;

Important distinction:

Unregistering prevents new channel lookup once the device is removed from g_io_devices, but existing channels can keep the internal device alive until their refcounts drop. The actual unregistration can be deferred until all active channels are destroyed; the public API reference states that behavior directly.

Iterating Channels With spdk_for_each_channel()

Some operations must touch every per-thread channel for a device. Examples:

  • pause a transport on every poll group
  • remove a bdev from every channel
  • collect iobuf stats
  • disconnect qpairs across all NVMf poll groups

spdk_for_each_channel() is the primitive that turns "do this globally" into "send a message to each thread that currently has local state for this device." The official thread.h reference says this happens asynchronously; the callback can run after spdk_for_each_channel() returns, callbacks run serially, and each callback must call spdk_for_each_channel_continue() to advance the iteration.

The implementation creates an iterator, records the originating thread, finds the registered device, and sends _call_channel to the first thread in the device's threads tree.

/* lib/thread/thread.c */
void
spdk_for_each_channel(void *io_device, spdk_channel_msg fn, void *ctx,
		      spdk_channel_for_each_cpl cpl)
{
	struct spdk_io_channel_iter *i;
	struct thread_link *thr_link;

	i = calloc(1, sizeof(*i));
	if (!i) {
		SPDK_ERRLOG("Unable to allocate iterator\n");
		assert(false);
		return;
	}

	i->io_device = io_device;
	i->fn = fn;
	i->ctx = ctx;
	i->cpl = cpl;
	i->orig_thread = _get_thread();

	i->orig_thread->for_each_count++;

	pthread_mutex_lock(&g_devlist_mutex);
	i->dev = io_device_get(io_device);

If no channel exists, completion is sent back to the originating thread. If at least one channel exists, SPDK increments the device's for_each_count, sets cur_thread, and sends a message to that thread:

/* lib/thread/thread.c */
thr_link = RB_MIN(thread_link_tree, &i->dev->threads);
if (thr_link != NULL) {
	i->dev->for_each_count++;
	i->cur_thread = thr_link->thread;
	spdk_thread_send_msg(i->cur_thread, _call_channel, i);
	pthread_mutex_unlock(&g_devlist_mutex);
	return;
}

_call_channel() rechecks whether the channel still exists once the message executes. That recheck is necessary because the channel may have been put and destroyed after the iterator chose the thread but before that thread processed the message.

/* lib/thread/thread.c */
static void
_call_channel(void *ctx)
{
	struct spdk_io_channel_iter *i = ctx;

	/*
	 * It is possible that the channel was deleted before this
	 *  message had a chance to execute.  If so, skip calling
	 *  the fn() on this thread.
	 */
	pthread_mutex_lock(&g_devlist_mutex);
	i->ch = thread_get_io_channel(i->cur_thread, i->dev);
	pthread_mutex_unlock(&g_devlist_mutex);

	if (i->ch) {
		i->fn(i);
	} else {
		spdk_for_each_channel_continue(i, 0);
	}
}

The callback must eventually call spdk_for_each_channel_continue(i, status). That function either sends the iterator to the next thread or sends completion back to the original thread. A non-zero status stops the remaining channel visits.

/* lib/thread/thread.c */
void
spdk_for_each_channel_continue(struct spdk_io_channel_iter *i, int status)
{
	struct spdk_thread *thread;
	struct io_device *dev;

	assert(i->cur_thread == spdk_get_thread());

	i->status = status;

	pthread_mutex_lock(&g_devlist_mutex);
	dev = i->dev;
	if (status) {
		goto end;
	}

	thread = io_dev_get_next_thread(i->dev, i->cur_thread);
	if (thread != NULL) {
		i->cur_thread = thread;
		spdk_thread_send_msg(i->cur_thread, _call_channel, i);
		pthread_mutex_unlock(&g_devlist_mutex);
		return;
	}

Beginner rule:

If you use spdk_for_each_channel(), your per-channel callback owns progress. Forgetting spdk_for_each_channel_continue() hangs the whole iteration and can block unregister.

Pending Unregister Races

io_device unregister and channel iteration interact carefully.

If unregister happens while spdk_for_each_channel() is active, unregister sets pending_unregister and returns. When the last iteration completes, spdk_for_each_channel_continue() sends __pending_unregister to the unregistering thread.

/* lib/thread/thread.c */
end:
	dev->for_each_count--;
	i->ch = NULL;
	pthread_mutex_unlock(&g_devlist_mutex);

	spdk_thread_send_msg(i->orig_thread, _call_completion, i);

	pthread_mutex_lock(&g_devlist_mutex);
	if (dev->pending_unregister && dev->for_each_count == 0) {
		spdk_thread_send_msg(dev->unregister_thread, __pending_unregister, dev);
	}
	pthread_mutex_unlock(&g_devlist_mutex);
}

static void
__pending_unregister(void *arg)
{
	struct io_device *dev = arg;

	assert(dev->pending_unregister);
	assert(dev->for_each_count == 0);
	spdk_io_device_unregister(dev->io_device, dev->unregister_cb);
}

This prevents the device from disappearing while a multi-thread channel walk is in progress. It also explains why a stuck foreach can make device teardown look stuck: unregister is waiting behind for_each_count.

Edge case:

If a second unregister is attempted while one is pending and foreach work remains, the implementation treats it as an error.

Example: iobuf Uses io_device Internally

lib/thread/iobuf.c:spdk_iobuf_initialize() registers a singleton io_device using &g_iobuf as the device pointer. This is a good example because iobuf is not a hardware queue. It is a global buffer service that still needs per-thread cache state.

/* lib/thread/iobuf.c */
spdk_iobuf_initialize(void)
{
	struct spdk_iobuf_opts *opts = &g_iobuf.opts;
	struct iobuf_node *node;
	int32_t i;
	int rc = 0;

	/* Round up to the nearest alignment so that each element remains aligned */
	opts->small_bufsize = SPDK_ALIGN_CEIL(opts->small_bufsize, IOBUF_ALIGNMENT);
	opts->large_bufsize = SPDK_ALIGN_CEIL(opts->large_bufsize, IOBUF_ALIGNMENT);

	IOBUF_FOREACH_NUMA_ID(i) {
		node = &g_iobuf.node[i];
		rc = iobuf_node_initialize(node, i);
		if (rc) {
			goto err;
		}
	}

	spdk_io_device_register(&g_iobuf, iobuf_channel_create_cb, iobuf_channel_destroy_cb,
				sizeof(struct iobuf_channel), "iobuf");
	g_iobuf_is_initialized = true;

spdk_iobuf_channel_init() gets an io_channel for &g_iobuf and stores it as the public iobuf channel's parent. The parent channel is the per-thread iobuf context. The public spdk_iobuf_channel then has module-specific cache sizes and points back to that parent.

/* lib/thread/iobuf.c */
ioch = spdk_get_io_channel(&g_iobuf);
if (ioch == NULL) {
	SPDK_ERRLOG("Couldn't get iobuf IO channel\n");
	return -ENOMEM;
}

iobuf_ch = spdk_io_channel_get_ctx(ioch);

for (i = 0; i < IOBUF_MAX_CHANNELS; ++i) {
	if (iobuf_ch->channels[i] == NULL) {
		iobuf_ch->channels[i] = ch;
		break;
	}
}

if (i == IOBUF_MAX_CHANNELS) {
	SPDK_ERRLOG("Max number of iobuf channels (%" PRIu32 ") exceeded.\n", i);
	rc = -ENOMEM;
	goto error;
}

ch->parent = ioch;
ch->module = module;

On failure, the code calls spdk_iobuf_channel_fini(ch), which puts the parent channel after cleaning the per-module caches. This is the same ownership rule in module form: get the per-thread parent channel, attach local state, and put that parent on teardown.

iobuf also uses spdk_for_each_channel() to collect per-thread stats. The stats request allocates a context, initializes module entries, and asks SPDK to walk every iobuf channel. Each visited channel contributes its local counters and calls spdk_for_each_channel_continue().

/* lib/thread/iobuf.c */
int
spdk_iobuf_get_stats(spdk_iobuf_get_stats_cb cb_fn, void *cb_arg)
{
	struct iobuf_module *module;
	struct iobuf_get_stats_ctx *ctx;
	uint32_t i;

	ctx = calloc(1, sizeof(*ctx));
	if (ctx == NULL) {
		return -ENOMEM;
	}

	TAILQ_FOREACH(module, &g_iobuf.modules, tailq) {
		++ctx->num_modules;
	}

	ctx->cb_fn = cb_fn;
	ctx->cb_arg = cb_arg;

	spdk_for_each_channel(&g_iobuf, iobuf_get_channel_stats, ctx,
			      iobuf_get_channel_stats_done);
	return 0;
}

The inference to carry forward is simple: even services that look global often use io_channels internally so the hot path can update per-thread cache state without a shared lock.

Example: NVMf Target Poll Groups

lib/nvmf/nvmf.c registers the NVMf target as an io_device. Its create callback creates poll-group state for a thread. Transport operations then use spdk_for_each_channel() to add, remove, pause, resume, or inspect poll groups across threads.

/* lib/nvmf/nvmf.c */
spdk_io_device_register(tgt,
			nvmf_tgt_create_poll_group,
			nvmf_tgt_destroy_poll_group,
			sizeof(struct spdk_nvmf_poll_group),
			tgt->name);

tgt->state = NVMF_TGT_RUNNING;

When a thread first gets a channel for the target, the create callback receives the target as io_device and a zeroed struct spdk_nvmf_poll_group as ctx_buf. It initializes transport poll groups and subsystem poll-group state for that thread, then inserts the poll group into the target's shared list.

/* lib/nvmf/nvmf.c */
static int
nvmf_tgt_create_poll_group(void *io_device, void *ctx_buf)
{
	struct spdk_nvmf_tgt *tgt = io_device;
	struct spdk_nvmf_poll_group *group = ctx_buf;
	struct spdk_nvmf_transport *transport;
	struct spdk_thread *thread = spdk_get_thread();
	int rc;

	group->tgt = tgt;
	TAILQ_INIT(&group->tgroups);
	TAILQ_INIT(&group->qpairs);
	group->thread = thread;
	pthread_mutex_init(&group->mutex, NULL);

	TAILQ_FOREACH(transport, &tgt->transports, link) {
		rc = nvmf_poll_group_add_transport(group, transport);
		if (rc != 0) {
			nvmf_tgt_cleanup_poll_group(group);
			return rc;
		}
	}

The destroy callback reverses the shared-list insertion and cleans the per-thread poll group. It receives the same ctx_buf, so it does not need a global lookup to find the channel's private NVMf state.

/* lib/nvmf/nvmf.c */
static void
nvmf_tgt_destroy_poll_group(void *io_device, void *ctx_buf)
{
	struct spdk_nvmf_tgt *tgt = io_device;
	struct spdk_nvmf_poll_group *group = ctx_buf;

	SPDK_DTRACE_PROBE1_TICKS(nvmf_destroy_poll_group, spdk_thread_get_id(group->thread));

	pthread_mutex_lock(&tgt->mutex);
	TAILQ_REMOVE(&tgt->poll_groups, group, link);
	tgt->num_poll_groups--;
	pthread_mutex_unlock(&tgt->mutex);

	assert(!(tgt->state == NVMF_TGT_PAUSING || tgt->state == NVMF_TGT_RESUMING));
	nvmf_tgt_cleanup_poll_group(group);
}

NVMf pause is a compact example of foreach channel iteration. The target-level operation is "pause polling." The local operation is "on each poll-group thread, pause every transport poll group in this channel context."

/* lib/nvmf/nvmf.c */
static void
_nvmf_tgt_pause_polling(struct spdk_io_channel_iter *i)
{
	struct spdk_io_channel *ch = spdk_io_channel_iter_get_channel(i);
	struct spdk_nvmf_poll_group *group = spdk_io_channel_get_ctx(ch);
	struct spdk_nvmf_transport_poll_group *tgroup;

	TAILQ_FOREACH(tgroup, &group->tgroups, link) {
		nvmf_transport_poll_group_pause(tgroup);
	}

	spdk_for_each_channel_continue(i, 0);
}

This pattern recurs throughout SPDK:

module-global object
  registered as io_device
per-thread channel
  contains poll group, qpair, queue, cache, or stats state
foreach channel
  performs coordinated cross-thread operation

Edge Cases And Failure Modes

Registering from a non-SPDK thread is not a recoverable user error in the normal code path. The implementation logs and asserts because the registered device must be associated with SPDK's message-driven thread model.

Registering the same device pointer twice is rejected. The pointer is the key, so duplicate registration would make channel lookup ambiguous.

Getting an unknown or unregistered device returns NULL. After unregister removes the device from g_io_devices, new channel lookups fail even if old channels are still draining.

Getting a channel from no current spdk_thread returns NULL. Getting from an exited thread also returns NULL. A channel is not just a resource allocation; it is tied to a live SPDK thread.

If the create callback fails, spdk_get_io_channel() unwinds the partial channel insertion and returns NULL. Module create callbacks should leave their context either fully initialized on success or safely cleanupable on failure.

Putting from the wrong thread triggers wrong_thread(). The safe fix is to send a message to the owning thread and put the channel there.

Putting too many times is a serious ownership bug. The implementation assumes matching get/ref and put calls. Treat the channel refcount like a strict borrowed reference count, not like a best-effort cache handle.

Destroy callbacks should not block for long periods. They run on the owning SPDK thread, so blocking the callback blocks that thread's message and poller progress.

Forgetting spdk_for_each_channel_continue() causes the foreach operation to stop permanently. If unregister is waiting behind that foreach, teardown waits too.

Unregister during foreach is explicitly supported by deferral. It is not instant deletion. Deletion waits until the active foreach count reaches zero.

Thread exit with live channels is a teardown smell. The thread exit path has to wait for owned resources to drain, and logs can point back to channels that were not put.

Misconceptions To Kill

context for any registered device.

context for a registered device.

thread must put it.

live until put.

asynchronous walk.

spdk_for_each_channel_continue().

  • "io_channel is a hardware channel." Not always. It is SPDK per-thread
  • "The channel context is shared by all threads." No. Each thread gets its own
  • "I can put a channel anywhere because I have the pointer." No. The owner
  • "Unregister immediately calls destroy on every channel." Existing channels
  • "spdk_for_each_channel() is synchronous." No. It is a message-driven
  • "The foreach callback can just return when done." It must call

Diskengine Relevance

Diskengine-triggered operations often look global: delete a volume, pause exports, remove a namespace, disconnect clients. Inside SPDK, global operations frequently become spdk_for_each_channel() walks over per-thread state.

When debugging a stuck disk deletion or target teardown:

  • identify the io_device
  • list which threads have channels
  • find whether a foreach is outstanding
  • confirm each per-channel callback calls continue
  • check whether unregister is pending behind foreach
  • check whether any user still holds a channel ref

The useful debugging question is not only "who owns the target?" It is also "which SPDK threads still own per-thread state for that target, and is any message-driven iteration waiting for one of them?"

Prose Diagram: Channel Lifetime

Think of an io_channel as a checkout of one SPDK thread's private state:

message on Thread A.

completion can run and the internal device object can be freed.

  1. The device registers an identity pointer and callbacks.
  2. Thread A checks out its channel with spdk_get_io_channel().
  3. Thread A can check out the same channel again; the refcount increases.
  4. Thread A returns each checkout with spdk_put_io_channel().
  5. When the count reaches zero, SPDK schedules final return processing as a
  6. If no new reference appears first, the destroy callback runs on Thread A.
  7. If the device was unregistered and no channels remain, unregister

The model is deliberately plain: one shared identity, one local context per thread, refcounted local lifetime, message-driven cross-thread visits.

Source Reading Path

Read these files in this order:

This gives the public contract.

spdk_io_device_register(), spdk_get_io_channel(), spdk_put_io_channel(), and spdk_for_each_channel().

spdk_iobuf_channel_init(). This shows a non-hardware use of io_channels.

nvmf_tgt_create_poll_group(), and _nvmf_tgt_pause_polling(). This shows the same mechanism applied to target poll groups.

transport operations fanning out to existing poll-group channels.

  1. include/spdk/thread.h, starting at the io_device and io_channel APIs.
  2. lib/thread/thread.c, starting at struct io_device, then
  3. lib/thread/iobuf.c, starting at spdk_iobuf_initialize() and
  4. lib/nvmf/nvmf.c, starting at spdk_io_device_register(tgt, ...),
  5. lib/nvmf/transport.c, search for spdk_for_each_channel(). This shows