Chapter 10: Reactors, `spdk_thread`, Messages, Pollers

Reader Promise

By the end of this chapter, a beginner should be able to explain the difference between an OS thread, an SPDK reactor, and an spdk_thread; trace a message sent by spdk_thread_send_msg(); understand how pollers run; and diagnose wrong-thread assertions, blocked reactors, leaked pollers, and thread-exit hangs.

This is one of the most important chapters in the book. Many SPDK bugs are not algorithm bugs. They are ownership bugs: code runs on the wrong spdk_thread, blocks a reactor, keeps an io_channel too long, or forgets to unregister a poller.

Mental Model

Use this vocabulary precisely:

• OS thread: the kernel-scheduled execution context, usually pinned to a CPU core by DPDK/SPDK.
• Reactor: SPDK's per-core event loop object. A reactor owns a list of lightweight SPDK threads and event queues.
• spdk_thread: a lightweight cooperative context. It has message queues, pollers, io_channels, stats, and a cpumask.
• Message: a function pointer plus context enqueued to an spdk_thread.
• Poller: a callback that runs repeatedly on an spdk_thread, either every loop or on a timer.

The official SPDK docs describe the event framework as an asynchronous, polled-mode, shared-nothing framework. That is not just style. It is the reason the runtime is shaped around message passing instead of ordinary shared-state locking. The concurrency guide says the common SPDK pattern is to assign data to one thread; other threads send that owner a message when they need an operation performed there. In the local implementation, that "thread" is usually an spdk_thread, not a POSIX thread.

Prose diagram:

CPU core 3
  OS thread "reactor_3"
    reactor object for lcore 3
      spdk_thread "app_thread"
        message ring
        active pollers
        timed pollers
        io_channels
      spdk_thread "nvmf_tgt_poll_group_3"
        message ring
        active pollers
        io_channels

The reactor runs the OS thread. The reactor polls each spdk_thread. Each spdk_thread drains messages and runs pollers.

The distinction matters because there are two schedulers in play. Linux schedules the OS thread. SPDK schedules lightweight spdk_thread objects inside that OS thread by calling spdk_thread_poll(). A reactor may run more than one spdk_thread, and an spdk_thread may be rescheduled between reactors when allowed by its cpumask and binding state. So "current pthread", "current core", and "current SPDK owner" are related facts, not synonyms.

SPDK's public concurrency documentation makes one more point that should stay in your head while reading the source: the spdk_thread library does not spawn system threads by itself. A lower framework must repeatedly poll it. In SPDK applications using the event framework, reactors are that lower framework.

Source Anchors

• include/spdk_internal/event.h: struct spdk_reactor, spdk_reactors_init(), spdk_reactors_start(), spdk_reactors_stop()
• lib/event/reactor.c: spdk_reactors_init(), reactor_construct(), spdk_reactors_start(), reactor_run(), _reactor_run(), reactor_post_process_lw_thread(), spdk_reactors_stop()
• include/spdk/thread.h: spdk_thread_create(), spdk_thread_poll(), spdk_thread_send_msg(), spdk_for_each_thread(), spdk_poller_register(), spdk_poller_unregister(), spdk_thread_exit()
• lib/thread/thread.c: struct spdk_thread, spdk_thread_create(), spdk_set_thread(), spdk_get_thread(), spdk_thread_poll(), thread_poll(), msg_queue_run_batch(), spdk_thread_send_msg(), poller_register(), thread_execute_poller(), thread_execute_timed_poller(), spdk_poller_unregister(), spdk_for_each_thread(), spdk_thread_exit(), thread_exit()
• lib/event/app_rpc.c: rpc_framework_get_reactors(), _rpc_framework_get_reactors()

Reactor Initialization

lib/event/reactor.c:spdk_reactors_init() creates the event framework's reactor state.

It:

• creates g_spdk_event_mempool
• allocates the g_reactors array aligned to 64 bytes
• initializes the thread library with spdk_thread_lib_init_ext()
• constructs a reactor for each env core
• records the scheduling reactor
• sets reactor state to initialized

The thread library call is crucial. Reactors cannot run spdk_thread objects until the thread library exists, because spdk_thread uses message mempools, message rings, poller queues, and io_channel registries.

The reactor object is intentionally small and cache-line aligned. Its most important ownership fields are the lightweight thread list, event queue, accounting counters, and interrupt-mode state:

/* include/spdk_internal/event.h */
struct spdk_reactor {
	/* Lightweight threads running on this reactor */
	TAILQ_HEAD(, spdk_lw_thread)			threads;
	uint32_t					thread_count;

	/* Logical core number for this reactor. */
	uint32_t					lcore;

	uint64_t					tsc_last;

	struct spdk_ring				*events;
	int						events_fd;

	uint64_t					busy_tsc;
	uint64_t					idle_tsc;

	/* Each bit of cpuset indicates whether a reactor probably requires event notification */
	struct spdk_cpuset				notify_cpuset;
	/* Indicate whether this reactor currently runs in interrupt */
	bool						in_interrupt;
	struct spdk_fd_group				*fgrp;
	int						resched_fd;
	uint16_t					trace_id;
} __attribute__((aligned(SPDK_CACHE_LINE_SIZE)));

That excerpt is the runtime model in data-structure form. threads is where scheduled spdk_thread contexts live. events is the older event-framework queue for core-targeted events. busy_tsc and idle_tsc are why tools can report whether a reactor is doing useful work instead of only showing "100% CPU" in Linux top. in_interrupt and fgrp are the bridge to interrupt mode.

Initialization wires the event framework to the thread library before constructing per-core reactors:

/* lib/event/reactor.c */
g_reactor_count = spdk_env_get_last_core() + 1;
rc = posix_memalign((void **)&g_reactors, 64,
		    g_reactor_count * sizeof(struct spdk_reactor));
...
memset(g_reactors, 0, (g_reactor_count) * sizeof(struct spdk_reactor));

rc = spdk_thread_lib_init_ext(reactor_thread_op, reactor_thread_op_supported,
			      sizeof(struct spdk_lw_thread), msg_mempool_size);
if (rc != 0) {
	SPDK_ERRLOG("Initialize spdk thread lib failed\n");
	...
	return rc;
}

SPDK_ENV_FOREACH_CORE(i) {
	reactor_construct(&g_reactors[i], i);
}

The sizeof(struct spdk_lw_thread) argument is a useful clue. The thread library owns generic spdk_thread behavior, but the event framework asks for enough per-thread context to attach each spdk_thread to a reactor scheduling record.

Reactor Start

lib/event/reactor.c:spdk_reactors_start() sets the reactor state to running, launches a reactor OS thread on every selected core except the current core, and then runs the current core's reactor inline.

That last detail explains why spdk_app_start() blocks: the main OS thread becomes a reactor runner until shutdown.

The start path shows this directly:

/* lib/event/reactor.c */
current_core = spdk_env_get_current_core();
SPDK_ENV_FOREACH_CORE(i) {
	if (i != current_core) {
		reactor = spdk_reactor_get(i);
		...
		rc = spdk_env_thread_launch_pinned(reactor->lcore, reactor_run, reactor);
		if (rc < 0) {
			SPDK_ERRLOG("Unable to start reactor thread on core %u\n", reactor->lcore);
			assert(false);
			return;
		}
	}
	spdk_cpuset_set_cpu(&g_reactor_core_mask, i, true);
}

/* Start the main reactor */
reactor = spdk_reactor_get(current_core);
assert(reactor != NULL);
reactor_run(reactor);

spdk_env_thread_wait_all();

This is why a simple call stack can be surprising in a debugger. The application calls into the framework, the framework starts helper OS threads, and then the caller's own OS thread enters reactor_run() and stays there until shutdown.

lib/event/reactor.c:reactor_run() is the long-running loop. It:

• names the POSIX thread reactor_<lcore>
• registers trace ownership
• repeatedly runs either interrupt mode handling or _reactor_run()
• periodically performs scheduler work if enabled
• exits when reactor state changes
• drains and destroys remaining spdk_thread objects

What _reactor_run() Does

lib/event/reactor.c:_reactor_run() is the normal polling loop body.

It:

- gets the spdk_thread - calls spdk_thread_poll(thread, 0, reactor->tsc_last) - updates reactor busy or idle time based on return code - post-processes the lightweight thread

1. Runs a batch of reactor events.
2. If the reactor has no SPDK threads, accounts idle time and returns.
3. For each lightweight thread on the reactor:

The important point: a reactor does not call arbitrary module code directly. It calls spdk_thread_poll(), and the thread runs messages and pollers.

The normal polling body is short enough to read as a control-flow contract:

/* lib/event/reactor.c */
_reactor_run(struct spdk_reactor *reactor)
{
	struct spdk_thread	*thread;
	struct spdk_lw_thread	*lw_thread, *tmp;
	uint64_t		now;
	int			rc;

	event_queue_run_batch(reactor);

	if (spdk_unlikely(TAILQ_EMPTY(&reactor->threads))) {
		now = spdk_get_ticks();
		reactor->idle_tsc += now - reactor->tsc_last;
		reactor->tsc_last = now;
		return;
	}

	TAILQ_FOREACH_SAFE(lw_thread, &reactor->threads, link, tmp) {
		thread = spdk_thread_get_from_ctx(lw_thread);
		rc = spdk_thread_poll(thread, 0, reactor->tsc_last);

		now = spdk_thread_get_last_tsc(thread);
		if (rc == 0) {
			reactor->idle_tsc += now - reactor->tsc_last;
		} else if (rc > 0) {
			reactor->busy_tsc += now - reactor->tsc_last;
		}
		reactor->tsc_last = now;

		reactor_post_process_lw_thread(reactor, lw_thread);
	}
}

There are two queues being serviced here. event_queue_run_batch() handles event-framework events targeted at the reactor core. The TAILQ_FOREACH_SAFE loop handles spdk_thread objects scheduled on that reactor. The return value from spdk_thread_poll() becomes reactor busy/idle accounting, so a poller that reports busy when it did no useful work will distort metrics.

spdk_thread Structure

lib/thread/thread.c:struct spdk_thread contains:

• active pollers queue
• timed pollers tree
• paused pollers queue
• message ring
• local message cache
• critical message slot
• io_channel tree
• cpumask
• state
• lock count
• interrupt-mode state
• trace ID
• user context

This is why spdk_thread is more than "a callback queue." It is the unit of SPDK ownership for pollers, messages, and per-thread device resources.

The source layout makes that ownership concrete:

/* lib/thread/thread.c */
struct spdk_thread {
	uint64_t			tsc_last;
	struct spdk_thread_stats	stats;
	TAILQ_HEAD(active_pollers_head, spdk_poller)	active_pollers;
	RB_HEAD(timed_pollers_tree, spdk_poller)	timed_pollers;
	struct spdk_poller				*first_timed_poller;
	TAILQ_HEAD(paused_pollers_head, spdk_poller)	paused_pollers;
	struct spdk_ring		*messages;
	SLIST_HEAD(, spdk_msg)		msg_cache;
	size_t				msg_cache_count;
	spdk_msg_fn			critical_msg;
	uint64_t			id;
	uint64_t			next_poller_id;
	enum spdk_thread_state		state;
	int				pending_unregister_count;
	uint32_t			for_each_count;

	RB_HEAD(io_channel_tree, spdk_io_channel)	io_channels;
	TAILQ_ENTRY(spdk_thread)			tailq;

	char				name[SPDK_MAX_THREAD_NAME_LEN + 1];
	struct spdk_cpuset		cpumask;
	int32_t				lock_count;
	bool				is_bound;
	bool				in_interrupt;
	struct spdk_fd_group		*fgrp;
	uint16_t			trace_id;
	uint8_t				ctx[0];
};

Several bugs become easier to understand once you group those fields by lifetime. Messages and pollers are execution work. io_channels are per-thread access paths into registered devices. state, pending_unregister_count, and for_each_count are shutdown gates. lock_count is a cooperative-scheduling guard: SPDK expects locks not to remain held when messages or pollers finish.

Creating An spdk_thread

lib/thread/thread.c:spdk_thread_create():

• allocates cache-line-aligned memory
• copies or initializes the cpumask
• initializes io_channel and poller containers
• creates a message ring
• fills a local message cache from g_spdk_msg_mempool if possible
• assigns a name and trace ID
• assigns a monotonic thread ID
• inserts the thread into the global thread list
• calls the reactor thread-op hook so the event framework can schedule it
• marks the thread running
• records the first created thread as the app thread

The event framework created the app thread in lib/event/app.c:spdk_app_start(). Other modules create their own SPDK threads when they need separate lightweight contexts.

The constructor does not spawn an OS thread. It allocates an object, initializes the queues, creates the message ring, puts the object on the global thread list, and then calls the framework hook that knows how to schedule it:

/* lib/thread/thread.c */
RB_INIT(&thread->io_channels);
TAILQ_INIT(&thread->active_pollers);
RB_INIT(&thread->timed_pollers);
TAILQ_INIT(&thread->paused_pollers);
SLIST_INIT(&thread->msg_cache);
thread->msg_cache_count = 0;

thread->messages = spdk_ring_create(SPDK_RING_TYPE_MP_SC, 65536, SPDK_ENV_NUMA_ID_ANY);
if (!thread->messages) {
	SPDK_ERRLOG("Unable to allocate memory for message ring\n");
	free(thread);
	return NULL;
}
...
thread->id = g_thread_id++;
TAILQ_INSERT_TAIL(&g_threads, thread, tailq);
g_thread_count++;
...
if (g_new_thread_fn) {
	rc = g_new_thread_fn(thread);
} else if (g_thread_op_supported_fn && g_thread_op_supported_fn(SPDK_THREAD_OP_NEW)) {
	rc = g_thread_op_fn(thread, SPDK_THREAD_OP_NEW);
}
...
thread->state = SPDK_THREAD_STATE_RUNNING;

In an event-framework app, the SPDK_THREAD_OP_NEW hook is implemented by reactor code. That hook is where a pure spdk_thread becomes visible to reactor scheduling. This split is why SPDK libraries can be embedded into other async frameworks: the thread library supplies the work queues and ownership rules, while the embedding framework supplies actual polling.

Messages

spdk_thread_send_msg(thread, fn, ctx) is the standard cross-thread handoff.

lib/thread/thread.c:spdk_thread_send_msg():

1. Checks that the target thread is not exited.
2. Tries to take a message object from the sender's local cache.
3. Falls back to g_spdk_msg_mempool.
4. Stores fn and ctx.
5. Enqueues the message to the target thread's message ring.
6. Sends a notification if needed.

The function is asynchronous. It does not call fn. It only queues the work.

The message will run when the target thread is polled by its reactor and msg_queue_run_batch() drains messages inside thread_poll().

The sender side is a small allocation plus a ring enqueue:

/* lib/thread/thread.c */
spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx)
{
	struct spdk_thread *local_thread;
	struct spdk_msg *msg;
	int rc;

	assert(thread != NULL);

	if (spdk_unlikely(thread->state == SPDK_THREAD_STATE_EXITED)) {
		SPDK_ERRLOG("Thread %s is marked as exited.\n", thread->name);
		abort();
	}

	local_thread = _get_thread();
	msg = NULL;
	...
	if (msg == NULL) {
		msg = spdk_mempool_get(g_spdk_msg_mempool);
		if (!msg) {
			SPDK_ERRLOG("msg could not be allocated\n");
			abort();
		}
	}

	msg->fn = fn;
	msg->arg = ctx;

	rc = spdk_ring_enqueue(thread->messages, (void **)&msg, 1, NULL);
	if (rc != 1) {
		SPDK_ERRLOG("msg could not be enqueued\n");
		abort();
	}

	thread_send_msg_notification(thread);
	return 0;
}

There is no retry return code here. The public docs say errors are handled internally and are fatal, and the implementation matches that: sending to an exited thread, exhausting the message mempool, or failing to enqueue aborts the process. That is intentional pressure to keep ownership handoffs simple and reliable.

The receiver side runs the function pointer while the target spdk_thread is current:

/* lib/thread/thread.c */
count = spdk_ring_dequeue(thread->messages, messages, max_msgs);
if (count == 0) {
	return 0;
}

for (i = 0; i < count; i++) {
	struct spdk_msg *msg = messages[i];

	assert(msg != NULL);

	SPDK_DTRACE_PROBE2(msg_exec, msg->fn, msg->arg);

	msg->fn(msg->arg);

	SPIN_ASSERT(thread->lock_count == 0, SPIN_ERR_HOLD_DURING_SWITCH);

	if (thread->msg_cache_count < SPDK_MSG_MEMPOOL_CACHE_SIZE) {
		SLIST_INSERT_HEAD(&thread->msg_cache, msg, link);
		thread->msg_cache_count++;
	} else {
		spdk_mempool_put(g_spdk_msg_mempool, msg);
	}
}

The assertion after msg->fn() is part of the cooperative contract. A message callback may mutate owner-thread state without taking a global lock, but it must return to the reactor cleanly. Holding an SPDK spinlock across the end of a message is treated as a scheduling bug.

Beginner rule:

If you need code to run on a different spdk_thread, send a message. Do not call the function directly unless the function explicitly allows it.

Pollers

A poller is a callback registered on the current spdk_thread.

lib/thread/thread.c:poller_register() requires spdk_get_thread() to be non-NULL. It allocates a struct spdk_poller, names it, records the callback and argument, assigns a per-thread poller ID, converts the period from microseconds to ticks, initializes interrupt support if needed, and inserts it into either:

• active pollers, if period is zero
• timed pollers, if period is nonzero

Public wrappers:

• include/spdk/thread.h:spdk_poller_register()
• include/spdk/thread.h:spdk_poller_register_named()
• include/spdk/thread.h:SPDK_POLLER_REGISTER()

Poller return values matter:

• Return SPDK_POLLER_IDLE when no useful work was done.
• Return SPDK_POLLER_BUSY when useful work was done.
• Do not use negative values as an error, backoff, or unregister mechanism.

The public contract is in include/spdk/thread.h:27 through include/spdk/thread.h:34: pollers should return enum spdk_thread_poller_rc, which has only SPDK_POLLER_IDLE and SPDK_POLLER_BUSY. The callback typedef in include/spdk/thread.h:100 through include/spdk/thread.h:106 points back to that enum.

The current implementation counts only rc > 0 as busy and, in debug builds, logs a return value of -1 in lib/thread/thread.c:1012 through lib/thread/thread.c:1016 and lib/thread/thread.c:1073 through lib/thread/thread.c:1077. That debug log does not make negative returns a public control path. If a poller needs to stop, pause, or change scheduling, use the explicit poller APIs from the owner thread.

Registration binds a poller to the current spdk_thread. That means the thread at registration time becomes the owner for unregister, pause, resume, and execution:

/* lib/thread/thread.c */
poller_register(spdk_poller_fn fn,
		void *arg,
		uint64_t period_microseconds,
		const char *name)
{
	struct spdk_thread *thread;
	struct spdk_poller *poller;

	thread = spdk_get_thread();
	if (!thread) {
		assert(false);
		return NULL;
	}

	if (spdk_unlikely(thread->state == SPDK_THREAD_STATE_EXITED)) {
		SPDK_ERRLOG("thread %s is marked as exited\n", thread->name);
		return NULL;
	}

	poller = calloc(1, sizeof(*poller));
	...
	poller->fn = fn;
	poller->arg = arg;
	poller->thread = thread;
	poller->id = thread->next_poller_id++;
	poller->period_ticks = convert_us_to_ticks(period_microseconds);
	...
	thread_insert_poller(thread, poller);

The public API documents period_microseconds == 0 as "call as often as possible." The local implementation turns that into placement on active_pollers; nonzero periods go through the timed-poller tree. A timer is not a separate timer thread. It is a poller that becomes eligible when the owning spdk_thread is polled and the current tick has reached next_run_tick.

How A Poller Runs

Inside lib/thread/thread.c:thread_poll():

1. A critical message runs first if present.
2. A batch of regular messages is drained.
3. Active pollers are executed.
4. Post-poller handlers run if registered.
5. Timed pollers whose deadline has passed are executed.

Active pollers are round-robin by queue movement. Timed pollers live in an RB tree keyed by next run time.

Here is the main ordering:

/* lib/thread/thread.c */
thread_poll(struct spdk_thread *thread, uint32_t max_msgs, uint64_t now)
{
	uint32_t msg_count;
	struct spdk_poller *poller, *tmp;
	spdk_msg_fn critical_msg;
	int rc = 0;

	thread->tsc_last = now;

	critical_msg = thread->critical_msg;
	if (spdk_unlikely(critical_msg != NULL)) {
		critical_msg(NULL);
		thread->critical_msg = NULL;
		rc = 1;
	}

	msg_count = msg_queue_run_batch(thread, max_msgs);
	if (msg_count) {
		rc = 1;
	}

	TAILQ_FOREACH_REVERSE_SAFE(poller, &thread->active_pollers,
				   active_pollers_head, tailq, tmp) {
		int poller_rc;

		poller_rc = thread_execute_poller(thread, poller);
		if (poller_rc > rc) {
			rc = poller_rc;
		}
		if (thread->num_pp_handlers) {
			thread_run_pp_handlers(thread);
		}
	}

Messages run before ordinary active pollers. That ordering is why spdk_thread_send_msg() is commonly used to request a poller unregister, qpair disconnect, channel release, or state transition on the owner thread. The message gets a chance to flip state before the next poller pass.

thread_execute_poller() and thread_execute_timed_poller() both assert that thread->lock_count == 0 after the callback. This is the source of lock-count asserts when code holds an SPDK spinlock across a point where SPDK expects cooperative progress.

Execution and cleanup are state-machine driven:

/* lib/thread/thread.c */
poller->state = SPDK_POLLER_STATE_RUNNING;
rc = poller->fn(poller->arg);

SPIN_ASSERT(thread->lock_count == 0, SPIN_ERR_HOLD_DURING_SWITCH);

poller->run_count++;
if (rc > 0) {
	poller->busy_count++;
}

switch (poller->state) {
case SPDK_POLLER_STATE_UNREGISTERED:
	TAILQ_REMOVE(&thread->active_pollers, poller, tailq);
	free(poller);
	break;
case SPDK_POLLER_STATE_PAUSING:
	TAILQ_REMOVE(&thread->active_pollers, poller, tailq);
	TAILQ_INSERT_TAIL(&thread->paused_pollers, poller, tailq);
	poller->state = SPDK_POLLER_STATE_PAUSED;
	break;
case SPDK_POLLER_STATE_RUNNING:
	poller->state = SPDK_POLLER_STATE_WAITING;
	break;
default:
	break;
}

Returning busy increments statistics. Unregistration is different: it is represented by poller->state = SPDK_POLLER_STATE_UNREGISTERED, and cleanup happens when the owner thread polls the poller machinery again. This is why returning -1 does not unregister a poller.

The No-Blocking Rule

A reactor is a cooperative event loop. If a poller blocks, that OS thread stops polling every other spdk_thread assigned to that reactor.

The official event-framework documentation says event functions should not block because they are called directly from the destination core's event loop. The same reasoning applies to spdk_thread messages and pollers in this chapter. They are not worker-thread jobs that can sleep independently; they are pieces of the reactor loop. A 10 ms sleep in one callback is 10 ms during which the reactor does not drain other messages, does not progress other pollers, and does not update completion paths assigned to that OS thread.

Do not:

• sleep in a poller
• perform blocking filesystem I/O in a hot callback
• wait synchronously for an RPC response from the same framework
• hold locks across callbacks that may pump SPDK threads
• busy-loop inside a poller instead of returning and letting the reactor continue

Use:

• messages for ownership handoff
• pollers for repeated progress
• async callbacks for completion
• NOMEM or retry queues for resource pressure

Wrong-Thread Assertions

SPDK APIs often require that operations happen on the same spdk_thread that owns the object.

lib/thread/thread.c:wrong_thread() logs the function, object name, current thread, and expected thread, then asserts.

The diagnostic is deliberately explicit:

/* lib/thread/thread.c */
wrong_thread(const char *func, const char *name, struct spdk_thread *thread,
	     struct spdk_thread *curthread)
{
	if (thread == NULL) {
		SPDK_ERRLOG("%s(%s) called with NULL thread\n", func, name);
		abort();
	}
	SPDK_ERRLOG("%s(%s) called from wrong thread %s:%" PRIu64 " (should be "
		    "%s:%" PRIu64 ")\n", func, name, curthread->name, curthread->id,
		    thread->name, thread->id);
	assert(false);
}

For pollers, unregister enforces this owner rule before it changes state:

/* lib/thread/thread.c */
thread = spdk_get_thread();
if (!thread) {
	assert(false);
	return;
}

if (poller->thread != thread) {
	wrong_thread(__func__, poller->name, poller->thread, thread);
	return;
}

...

/* Simply set the state to unregistered. The poller will get cleaned up
 * in a subsequent call to spdk_thread_poll().
 */
poller->state = SPDK_POLLER_STATE_UNREGISTERED;

This is a good example of the difference between object identity and CPU identity. The code compares struct spdk_thread *, not lcore number. If two lightweight threads happen to run on the same reactor, unregistering a poller from the wrong one is still a bug.

Common causes:

• unregistering a poller from a different thread than the one that registered it
• putting an io_channel from the wrong thread
• calling module-specific functions on a callback thread rather than the resource owner thread
• mixing OS thread identity with spdk_thread identity

Misconception to kill:

"I am on the same CPU core, so I am on the right SPDK thread." Not necessarily. Ownership is spdk_thread, not just core.

Thread Exit

lib/thread/thread.c:spdk_thread_exit() marks a thread as exiting. It does not instantly free the thread.

lib/thread/thread.c:thread_exit() waits until:

• message ring is empty
• no spdk_for_each_thread() or spdk_for_each_channel() operations are outstanding
• active pollers are unregistered
• timed pollers are unregistered
• paused pollers are gone
• io_channels are released
• pending io_device unregisters are complete

Only then does the state become exited. lib/event/reactor.c:reactor_post_process_lw_thread() sees an exited and idle thread, removes it from the reactor, and destroys it.

If shutdown hangs, inspect the thread for remaining messages, pollers, io_channels, or outstanding foreach operations.

The local exit gate is one of the best debugging guides in the file:

/* lib/thread/thread.c */
if (spdk_ring_count(thread->messages) > 0) {
	SPDK_INFOLOG(thread, "thread %s still has messages\n", thread->name);
	return;
}

if (thread->for_each_count > 0) {
	SPDK_INFOLOG(thread, "thread %s is still executing %u for_each_channels/threads\n",
		     thread->name, thread->for_each_count);
	return;
}

TAILQ_FOREACH(poller, &thread->active_pollers, tailq) {
	if (poller->state != SPDK_POLLER_STATE_UNREGISTERED) {
		SPDK_INFOLOG(thread, "thread %s still has active poller %s\n",
			     thread->name, poller->name);
		return;
	}
}

RB_FOREACH(ch, io_channel_tree, &thread->io_channels) {
	SPDK_INFOLOG(thread, "thread %s still has channel for io_device %s\n",
		     thread->name, ch->dev->name);
	return;
}

spdk_thread_exit() itself requires the exiting thread to be current:

/* lib/thread/thread.c */
spdk_thread_exit(struct spdk_thread *thread)
{
	SPDK_DEBUGLOG(thread, "Exit thread %s\n", thread->name);

	assert(tls_thread == thread);

	if (thread->state >= SPDK_THREAD_STATE_EXITING) {
		SPDK_INFOLOG(thread,
			     "thread %s is already exiting\n",
			     thread->name);
		return 0;
	}

	thread->exit_timeout_tsc = spdk_get_ticks() + (spdk_get_ticks_hz() *
				   SPDK_THREAD_EXIT_TIMEOUT_SEC);
	thread->state = SPDK_THREAD_STATE_EXITING;

This is why shutdown code often sends a message to the target thread to make that thread unregister its own pollers, put its own channels, and call spdk_thread_exit() on itself. Calling exit directly from a different spdk_thread violates the same ownership model as poller unregister.

The reactor side performs final removal only when the thread is both exited and idle:

/* lib/event/reactor.c */
if (spdk_unlikely(spdk_thread_is_exited(thread) &&
		  spdk_thread_is_idle(thread))) {
	_reactor_remove_lw_thread(reactor, lw_thread);
	spdk_thread_destroy(thread);
	return true;
}

Interrupt Mode

SPDK's classic model is polling. This tree also supports interrupt mode. In reactor code, reactor_run() chooses reactor_interrupt_run() when reactor->in_interrupt is true. In thread code, spdk_thread_poll() services the thread fd group when the thread is in interrupt mode.

For beginners, the important distinction:

• Poll mode repeatedly calls pollers for low latency and high CPU use.
• Interrupt mode waits on file descriptors where supported, reducing CPU but adding complexity.

Do not assume every poller or device path has the same interrupt-mode behavior.

The interrupt-mode docs frame this tradeoff plainly: default poll mode keeps cores in a tight loop for low latency, while interrupt mode lets a core sleep until an event arrives on a file descriptor. The blocking sleep point is at the reactor fd-group level, not inside every spdk_thread_poll() call.

The local spdk_thread_poll() implementation reflects this split. In poll mode it runs messages and pollers. In thread interrupt mode it performs a non-blocking wait on the thread's fd group with timeout 0:

/* lib/thread/thread.c */
orig_thread = _get_thread();
tls_thread = thread;

if (now == 0) {
	now = spdk_get_ticks();
}

if (spdk_likely(!thread->in_interrupt)) {
	rc = thread_poll(thread, max_msgs, now);
	if (spdk_unlikely(thread->state == SPDK_THREAD_STATE_EXITING)) {
		thread_exit(thread, now);
	}
} else {
	/* Non-block wait on thread's fd_group */
	rc = spdk_fd_group_wait(thread->fgrp, 0);
}

thread_update_stats(thread, spdk_get_ticks(), now, rc);

tls_thread = orig_thread;

The reactor nests each thread interrupt fd group into the reactor fd group when interrupt mode is enabled:

/* lib/event/reactor.c */
grp = spdk_thread_get_interrupt_fd_group(thread);
spdk_fd_group_nest(target->fgrp, grp);
spdk_thread_send_msg(thread, _reactor_set_thread_interrupt_mode, target);

Then reactor_interrupt_run() can block on the reactor fd group:

/* lib/event/reactor.c */
int block_timeout = -1; /* _EPOLL_WAIT_FOREVER */

spdk_fd_group_wait(reactor->fgrp, block_timeout);

Notice that tls_thread is set around both spdk_thread_poll() paths. That is what makes spdk_get_thread() and same-thread assertions meaningful while a reactor is executing a lightweight thread. Interrupt mode changes where the reactor can sleep and how ready fd events are dispatched; it does not remove the owner-thread rules. It also is not a universal replacement for polling. Pollers and device paths must participate in interrupt mode, for example through spdk_poller_register_interrupt() and fd registration, before the reactor can sleep usefully.

Edge Cases And Failure Modes

• Message mempool exhaustion: spdk_thread_send_msg() aborts if it cannot allocate a message.
• Message ring enqueue failure: aborts.
• Target thread exited: sending a message aborts.
• Poller registered outside any spdk_thread: assert path.
• Poller unregistered from the wrong thread: wrong-thread assert.
• Poller callback blocks: reactor stalls.
• Poller callback returns busy forever: stats show busy even if no useful work happens.
• Thread exit with active pollers: exit waits and logs.
• Thread exit with io_channels: exit waits and logs.
• Reactor shutdown with non-app running threads: logs that spdk_thread_exit() was not called.

Misconceptions To Kill

• "spdk_thread is a pthread." It is not. It is a lightweight SPDK context run by a reactor.
• "Messages run immediately." They run later when the target thread polls.
• "Pollers are background threads." They are callbacks on an spdk_thread.
• "A timed poller runs exactly at its period." It runs when the thread is polled and its deadline has passed.
• "Blocking only hurts my poller." Blocking hurts the whole reactor OS thread.
• "Returning -1 from a poller unregisters it." Unregistration is explicit.

Diskengine Relevance

Diskengine integrations tend to cross boundaries: an external controller sends RPCs, SPDK translates them into bdev or transport work, and completions come back asynchronously. Bugs appear when a control path assumes synchronous behavior.

When reading diskengine-facing SPDK code, always annotate:

• callback owner thread
• resource owner thread
• whether a function sends a message
• whether a function registers a poller
• where completion is delivered

That habit prevents most wrong-thread misunderstandings.

Prose Diagram: Message Delivery

Imagine a message as a sealed envelope:

1. Sender writes function pointer and context into the envelope.
2. Sender drops it into the target thread's mailbox.
3. Reactor eventually visits that target thread.
4. spdk_thread_poll() opens a batch of envelopes.
5. Each function runs on the target thread.

The sender does not wait by the mailbox.

Source Reading Exercise

Read the loop from reactor to poller:

1. lib/event/reactor.c:spdk_reactors_start()
2. lib/event/reactor.c:reactor_run()
3. lib/event/reactor.c:_reactor_run()
4. lib/thread/thread.c:spdk_thread_poll()
5. lib/thread/thread.c:thread_poll()
6. lib/thread/thread.c:thread_execute_poller()
7. lib/thread/thread.c:thread_execute_timed_poller()

Then read the message path:

1. lib/thread/thread.c:spdk_thread_send_msg()
2. lib/thread/thread.c:msg_queue_run_batch()
3. lib/thread/thread.c:thread_poll()

Questions:

• Where does TLS spdk_thread get set?
• What happens before active pollers run?
• How does SPDK decide busy vs idle?
• What causes a thread to be destroyed?

Operational Lab

Use RPC and logs:

1. Start an SPDK target with a small reactor mask.
2. Call framework_get_reactors.
3. Identify reactors, their threads, busy ticks, idle ticks, and interrupt state.
4. Add or enable a component that registers a poller.
5. Call framework_get_reactors again and observe thread/poller changes.

Source-only variation:

• Pick one module that calls spdk_thread_send_msg() and trace why it needs to cross ownership boundaries.

Concrete source-only path:

• lib/nvmf/transport.c:nvmf_transport_poll_group_create_poller() registers an NVMe-oF transport poller on the current thread.
• lib/nvmf/nvmf.c:_nvmf_tgt_disconnect_qpairs() sends itself a message when some qpairs are still disconnecting, so teardown is retried asynchronously instead of blocking.
• lib/nvmf/ctrlr.c:nvmf_ctrlr_add_io_qpair() and nearby paths send messages to the subsystem, controller, or poll-group owner thread rather than directly mutating remote-owner state.

When you read one of those paths, mark four facts in the margin: current spdk_thread, target spdk_thread, object being protected by ownership, and the callback that eventually completes the operation.

Self-Check

1. What is the difference between an OS thread, a reactor, and an spdk_thread?
2. Why does spdk_thread_send_msg() not call the function directly?
3. Where are active pollers stored?
4. Where are timed pollers stored?
5. Why must pollers avoid blocking?
6. What conditions must be satisfied before an spdk_thread exits?
7. Why can being on the same CPU core still be the wrong SPDK thread?

References

• Local source: include/spdk_internal/event.h
• Local source: lib/event/reactor.c
• Local source: include/spdk/thread.h
• Local source: lib/thread/thread.c
• Local source: lib/event/app_rpc.c
• Local source: lib/nvmf/transport.c
• Local source: lib/nvmf/nvmf.c
• Local source: lib/nvmf/ctrlr.c
• Official SPDK docs: Event Framework
• Official SPDK docs: Message Passing and Concurrency
• Official SPDK docs: thread.h File Reference
• Official SPDK docs: Interrupt Mode
• Official SPDK docs: spdk_top