Deep Dive: Reading SPDK C Source

Chapter Goal

This chapter gives you a practical method for reading SPDK C source as a beginner. SPDK is not hard because each line is strange. It is hard because ownership, callbacks, threads, modules, and generated registration paths are spread across many files. The goal is to help you read with a map instead of chasing symbols randomly.

Beginner Mental Model

SPDK is a C codebase built around explicit objects and asynchronous control flow. Many functions do not "return the final result." They start work and arrange for a callback to run later. Many data structures are intrusive. The object contains the list link, reference state, and operation table. Many modules register themselves during program startup.

The reading model:

entry point
  |
  +-- decode inputs
  |
  +-- find or allocate object
  |
  +-- choose thread/channel/module
  |
  +-- submit async work
  |
  +-- callback updates state
  |
  +-- response or completion returns upward

If you read SPDK as purely synchronous C, it will feel confusing. Read it as an event system.

Source Anchors

• app/spdk_tgt/spdk_tgt.c: small target application entry point.
• lib/event/app.c: spdk_app_start: framework startup path.
• lib/init/subsystem.c: spdk_subsystem_init: subsystem initialization state machine.
• include/spdk_internal/init.h: subsystem structure and callbacks.
• include/spdk/thread.h: public thread, message, poller, and I/O channel API.
• include/spdk_internal/thread.h: internal thread helpers and stats accessors.
• lib/thread/thread.c: thread, message, poller, and I/O channel implementation.
• include/spdk/bdev.h: public bdev APIs.
• include/spdk/bdev_module.h: bdev module interface.
• lib/bdev/bdev.c: core bdev implementation.
• lib/bdev/bdev_rpc.c: bdev RPC handlers and JSON output examples.
• module/bdev/malloc/bdev_malloc.c: compact bdev module to read first.
• module/bdev/nvme/bdev_nvme.c: larger bdev module with async attach and paths.
• lib/nvme/nvme.c: spdk_nvme_probe: NVMe library probe entry.
• lib/nvmf/nvmf.c: NVMe-oF target object orchestration.
• lib/nvmf/transport.c: transport registration and poll group logic.
• include/spdk/queue.h: queue macros such as TAILQ and STAILQ.
• include/spdk/util.h: utility macros such as SPDK_CONTAINEROF and SPDK_COUNTOF.
• scripts/gdb_macros.py: examples of walking SPDK lists in a debugger.
• doc/concepts.md: official conceptual overview.
• doc/event.md: official event framework guide.
• doc/bdev_pg.md: official bdev programmer guide.

Start From The Outside

Before reading internals, identify the public entry. For an application, start at main. For an RPC, start at SPDK_RPC_REGISTER. For a library API, start in include/spdk/*.h. For a module, start at its registration structure.

Good first paths:

• app/spdk_tgt/spdk_tgt.c for a target app.
• module/bdev/malloc/bdev_malloc_rpc.c for a small RPC.
• module/bdev/malloc/bdev_malloc.c for a small bdev module.
• lib/bdev/bdev_rpc.c for JSON output and bdev query structure.
• test/unit/lib/event/reactor.c/reactor_ut.c for thread and reactor behavior in small tests.

Avoid starting in the biggest transport first. lib/nvmf/tcp.c, lib/nvmf/rdma.c, and module/bdev/nvme/bdev_nvme.c are important, but they assume you already know SPDK patterns.

Read With Four Questions

For every function, ask:

• Who owns this object?
• What thread is this code running on?
• Is this synchronous or callback-based?
• What state transition is being attempted?

These questions matter more than memorizing every struct field. SPDK correctness often depends on object lifetime, thread affinity, and callback ordering.

Follow Registration Macros

SPDK uses registration macros heavily. You need to recognize them as entry points.

Examples:

• SPDK_RPC_REGISTER registers a JSON-RPC method.
• SPDK_LOG_REGISTER_COMPONENT registers a log flag.
• SPDK_TRACE_REGISTER_FN registers tracepoint descriptions.
• bdev modules register operation tables and module structures.
• subsystems are declared with subsystem registration macros in event modules.

When a symbol seems unused, search for a registration macro. Constructor-style registration means normal textual call graphs can miss the edge.

Useful commands:

rg -n 'SPDK_RPC_REGISTER' lib module
rg -n 'SPDK_LOG_REGISTER_COMPONENT' lib module app
rg -n 'SPDK_TRACE_REGISTER_FN' lib module
rg -n 'write_config_json|dump_info_json|submit_request' lib module include

Intrusive Lists

SPDK uses BSD-style queues. The object contains its own link field. That means list traversal often looks unlike high-level language collections.

Common macros:

• TAILQ_HEAD
• TAILQ_ENTRY
• TAILQ_FOREACH
• TAILQ_FOREACH_SAFE
• STAILQ_HEAD
• STAILQ_ENTRY
• STAILQ_INSERT_TAIL
• RB_HEAD
• RB_ENTRY

Source anchors:

• include/spdk/queue.h for queue macros.
• lib/nvmf/transport.c for transport and qpair list traversal.
• lib/nvmf/ctrlr.c for controller outstanding request traversal.
• module/bdev/raid/bdev_raid.c for RAID bdev and base device lists.

Reading trick:

TAILQ_FOREACH(item, &owner->list, link)

Read this as:

for each item stored in owner->list using item->link

The link field is not data. It is the hook that lets the object live in that list.

SPDK_CONTAINEROF

SPDK_CONTAINEROF recovers an outer object pointer from a field pointer. It is central to object-oriented C.

Example pattern:

struct outer *o = SPDK_CONTAINEROF(member_ptr, struct outer, member);

Read it as:

member_ptr points inside a struct outer;
compute the address of the containing struct outer.

Source anchors:

• include/spdk/util.h: SPDK_CONTAINEROF.
• lib/nvmf/transport.c uses it for iobuf request recovery.
• module/bdev/crypto/vbdev_crypto.c uses it to recover virtual bdev objects.
• module/accel/mlx5/accel_mlx5.c uses it for task wrappers.

If SPDK_CONTAINEROF feels magical, draw the struct layout on paper. It is pointer arithmetic with type checking conventions.

Operation Tables

Many SPDK modules implement interfaces through function pointer tables. The table is the contract. The module fills the table. The core calls through it.

Bdev module example:

• public concepts: include/spdk/bdev.h.
• module contract: include/spdk/bdev_module.h.
• core caller: lib/bdev/bdev.c.
• small implementation: module/bdev/malloc/bdev_malloc.c.

Look for fields such as:

• submit_request
• io_type_supported
• get_io_channel
• dump_info_json
• write_config_json
• destruct

When you see a call through fn_table, stop and find the table initialization. That tells you which module-specific code runs.

Threads And Messages

SPDK threads are not OS threads in the usual beginner sense. An SPDK thread is an execution context scheduled on a reactor. Work crosses contexts through spdk_thread_send_msg. Pollers are callbacks run by threads. I/O channels are per-thread handles to per-device context.

Source anchors:

• include/spdk/thread.h.
• lib/thread/thread.c.
• lib/event/reactor.c.
• test/unit/lib/event/reactor.c/reactor_ut.c.
• test/unit/lib/thread/thread.c/thread_ut.c.

Important reading rule:

If code sends a message, control continues elsewhere later. The next line after spdk_thread_send_msg is not the completion of that work. Find the message callback. Then find what it calls.

Prose diagram:

thread A
  calls spdk_thread_send_msg(thread B, fn, ctx)
  returns to its own loop

thread B
  later runs fn(ctx)
  mutates state owned by B
  may send another message back to A

I/O Channels

An I/O channel is per-thread access to an I/O device. The same bdev can have one channel context per SPDK thread using it. This avoids locks in hot paths and keeps per-thread state close to the poller.

Source anchors:

• include/spdk/thread.h: spdk_get_io_channel.
• lib/thread/thread.c for channel allocation and reference behavior.
• module/bdev/raid/bdev_raid.c: raid_bdev_create_cb.
• module/bdev/crypto/vbdev_crypto.c: vbdev_crypto_get_io_channel.
• module/blob/bdev/blob_bdev.c: bdev_blob_create_channel.

Reading trick:

• find the get_io_channel function.
• find the channel struct.
• find create and destroy callbacks.
• find which fields are per-thread.
• find which fields point back to shared objects.

Async Completion Style

SPDK callbacks often carry a context object. The context object records what must happen next. A common style is:

allocate ctx
fill ctx with callback, arg, object pointers
submit async operation
return

later:
completion callback receives ctx
updates state
calls user's callback
frees ctx

Source anchors:

• lib/init/json_config.c uses context objects during config replay.
• lib/nvmf/nvmf.c uses context objects for target and poll group operations.
• module/bdev/nvme/bdev_nvme.c uses async probe and attach contexts.
• module/bdev/raid/bdev_raid.c uses process contexts for rebuild and replace work.

When reading callbacks, locate the allocation site. That tells you lifetime and intended owner.

JSON Decoder Tables

RPC handlers often define a request struct and a decoder array. The decoder array maps JSON keys to fields. This is one of the cleanest ways to read an RPC.

Reading recipe:

1. find rpc_* handler
2. find request struct
3. find decoder table
4. list required and optional fields
5. find semantic validation after decoding
6. find success response
7. find error responses

Source anchors:

• module/bdev/malloc/bdev_malloc_rpc.c.
• lib/bdev/bdev_rpc.c.
• lib/nvmf/nvmf_rpc.c.
• module/event/subsystems/nvmf/nvmf_rpc.c.

Do not guess parameter names from memory. The decoder table is the authority for that handler.

State Machines

SPDK often represents state as enums. State machines appear in controllers, subsystems, qpairs, RAID, and app initialization. Find the enum first. Then search all assignments. Then search all switch statements.

Useful commands:

rg -n 'enum .*state|state =' lib/nvmf module/bdev lib/event
rg -n 'case .*STATE|case .*_STATE_' lib module

Example places:

• lib/nvmf/subsystem.c for NVMe-oF subsystem state changes.
• lib/nvmf/ctrlr.c for controller and qpair state checks.
• module/bdev/raid/bdev_raid.c for RAID bdev states and background processes.
• lib/event/app.c for startup and shutdown flow.

State reading is slow but reliable. It prevents the beginner mistake of assuming every function can run in every state.

Error Handling

SPDK C often returns negative errno-style values. RPCs convert errors to JSON-RPC errors. I/O paths convert errors to completion statuses. Asynchronous callbacks often pass an int rc or a status enum.

Read errors backward:

observed error
  |
search exact string or status
  |
find branch
  |
identify condition
  |
find who supplied the bad value or state

Do not stop at the branch that logs. The logged branch is where the failure became visible. The root cause can be earlier.

Tests Are Reading Guides

Unit tests are often easier to read than production setup scripts. They construct a tiny world and prove one behavior.

Useful tests:

• test/unit/lib/rpc/rpc.c/rpc_ut.c for RPC method dispatch.
• test/unit/lib/jsonrpc/jsonrpc_server.c/jsonrpc_server_ut.c for parsing.
• test/unit/lib/event/reactor.c/reactor_ut.c for reactor stats and thread behavior.
• test/unit/lib/thread/thread.c/thread_ut.c for thread primitives.
• test/unit/lib/bdev/bdev.c/bdev_ut.c for bdev edge cases.
• test/unit/lib/bdev/nvme/bdev_nvme.c/bdev_nvme_ut.c for NVMe bdev behavior.
• test/unit/lib/nvmf/nvmf.c/nvmf_ut.c for target object behavior.

If production code is too large, find the unit test for the function. Mocks reveal the expected contract.

A First Source Walkthrough

Read bdev_malloc_create this way:

1. find SPDK_RPC_REGISTER in module/bdev/malloc/bdev_malloc_rpc.c
2. open rpc_bdev_malloc_create
3. inspect decoder table
4. find validation
5. find call into malloc bdev create function
6. open module/bdev/malloc/bdev_malloc.c
7. find the bdev function table
8. find submit_request
9. find read/write completion behavior
10. find write_config_json

This path teaches RPC, bdev module shape, JSON config output, and I/O completion without a huge transport.

A Larger Source Walkthrough

Read NVMe bdev attach this way:

1. find bdev_nvme_attach_controller registration
2. inspect params in module/bdev/nvme/bdev_nvme_rpc.c
3. find attach context allocation
4. follow into module/bdev/nvme/bdev_nvme.c
5. identify spdk_nvme_probe_async or probe path
6. open lib/nvme/nvme.c around spdk_nvme_probe
7. find probe callback and attach callback flow
8. return to bdev creation for namespaces
9. inspect write_config_json
10. inspect detach path for lifetime cleanup

This path is harder because it includes discovery, controllers, namespaces, and transport behavior. Use the smaller malloc path first.

Edge Cases

Function Appears Unused

It may be referenced by a function pointer table, registration macro, constructor, or test include. Search by symbol name and by nearby table field.

Callback Name Is Generic

Names like done, complete, cb, and finish are common. Find the context struct. The context struct reveals what operation is completing.

Same Object Has Public And Internal Structs

Headers under include/spdk expose public API. Headers under include/spdk_internal expose internal details. Do not assume internal fields are stable API.

A List Has Multiple Link Fields

An object can be in more than one list. Always note the exact link field used by TAILQ_FOREACH.

A Completion Runs On A Different Thread

If a callback sends a message, the final cleanup may happen elsewhere. Track thread ownership before deciding a race exists.

Tests Include .c Files

SPDK unit tests often include implementation .c files directly. This gives tests access to static functions. It is intentional in this codebase.

Misconceptions To Kill

• "C has no objects." SPDK uses structs plus function tables as object interfaces.
• "A return value means the operation is complete." Many operations complete later by callback.
• "A function pointer call is unknowable." Find the table initialization.
• "Lists hide data." Intrusive lists are explicit once you identify the link field.
• "Every SPDK thread is a pthread." SPDK threads are framework execution contexts.
• "Internal headers are public contracts." They are source-reading tools, not stable external API.
• "The biggest file is the best starting point." Start with a small module and transfer the pattern.
• "Search is enough." Search plus a state/lifetime map is better.

Lab: Draw A Callback Chain

Pick rpc_bdev_malloc_create. Draw the flow from RPC request to response. Mark synchronous calls with straight arrows. Mark callback boundaries with dashed arrows. Write down every object allocated or freed. Then repeat for an NVMe attach RPC and compare complexity.

Lab: Decode A Function Table

Open module/bdev/malloc/bdev_malloc.c. Find the bdev function table. List every function pointer assigned. For each function, write one sentence explaining when the bdev core calls it. Then open include/spdk/bdev_module.h and compare the table to the interface definition.

Lab: Follow A List

Open module/bdev/raid/bdev_raid.c. Find one TAILQ_FOREACH. Identify the list owner. Identify the item type. Identify the link field. Explain whether the loop is safe if items are removed. If not, find a nearby TAILQ_FOREACH_SAFE and compare.

Lab: Thread Ownership Map

Open lib/nvmf/nvmf.c. Search for spdk_thread_send_msg. Pick one call. Identify source thread, target thread, callback function, and context object. Write a one-page map of what state is safe to touch before and after the message.

Self-Check

1. Why is SPDK_RPC_REGISTER an entry point?
2. What does SPDK_CONTAINEROF recover?
3. Why does a function table matter?
4. What four questions should you ask for every function?
5. How do you read a TAILQ_FOREACH line?
6. Why can a return value fail to mean final completion?
7. What is an I/O channel?
8. Why are unit tests useful for source reading?

References

• doc/concepts.md for SPDK conceptual foundations.
• doc/event.md for event framework concepts.
• doc/bdev_pg.md for bdev module programming.
• doc/gdb_macros.md for debugger-oriented list walking.
• include/spdk/thread.h for thread and channel APIs.
• include/spdk/bdev_module.h for bdev module contracts.
• scripts/gdb_macros.py for practical struct traversal examples.
• test/unit/unittest.sh for the unit-test layout.