Chapter 35: Build, Test, Run, Contribute | SPDK From First Principles

Chapter Goal

This chapter teaches how to build SPDK, choose the right tests, and prepare a contribution. It is written for beginners who can read C but do not yet know SPDK's build and CI habits. The emphasis is pragmatic: build only what you need, test the risk you changed, and keep patches small.

Beginner Mental Model

SPDK is a C project with optional features. The build you get depends on configure flags and available system libraries. The tests range from tiny unit tests to hardware-heavy integration tests. The contribution process rewards small, source-grounded changes with focused tests.

configure
  |
  +-- select features and debug/sanitizer options
  |
make
  |
  +-- compile apps, libs, modules, tests
  |
focused tests
  |
  +-- prove your changed behavior
  |
broader checks
  |
  +-- style, unit group, integration when needed
  |
patch
  |
  +-- clear commit, clear scope, no unrelated churn

Do not run the entire world for every typo. Do not run only a smoke test for a shared lifetime change. Match test scope to risk.

Source Anchors

README.md: top-level project orientation.
configure: feature selection entry point.
mk/spdk.common.mk: common build variables and configure guard.
mk/spdk.unittest.mk: unit-test build include.
scripts/pkgdep.sh: dependency installation entry point.
scripts/setup.sh: hugepage and device setup.
scripts/check_format.sh: style and formatting checks.
test/unit/unittest.sh: unit-test driver.
autotest.sh: broad test runner.
test/common/autotest_common.sh: shared test helpers and run_test.
doc/getting_started.md: official build and first-run guide.
doc/system_configuration.md: official host setup guide.
doc/distributions.md: CI job naming and distro context.
doc/shfmt.md: shell formatting guide.
doc/gdb_macros.md: debugging helpers for failures.
scripts/core-collector.sh: coredump collection helper.

Build Prerequisites

SPDK needs compiler tools, libraries, and often host setup for hugepages and device access. The exact packages depend on distribution and selected features. Use the official dependency scripts where possible.

Common first steps:

scripts/pkgdep.sh
./configure
make -j

Before running dependency installation, inspect scripts/pkgdep.sh --help and the getting-started docs for your distro. Expect package installation to need root or sudo, and avoid running it blindly on production hosts.

For hardware-backed apps, also read:

doc/system_configuration.md.
scripts/setup.sh.
doc/memory.md.

Do not assume a build failure is a code bug. It can be a missing optional dependency, stale mk/config.mk, or a feature flag that requires a library.

Configure

./configure writes build configuration consumed by the makefiles. If mk/config.mk is missing, mk/spdk.common.mk asks you to run configure.

Common useful flags:

--enable-debug for debug builds and assertions.
--enable-asan for address sanitizer.
--enable-ubsan for undefined behavior sanitizer.
--with-rdma for RDMA features.
--with-uring for io_uring bdev support.
--with-vfio-user when vfio-user support is needed.
--with-usdt for USDT probes.

Use ./configure --help for the source of truth in your checkout. Feature names can change. Build options are temporally unstable across releases, so trust the local script.

Debug build mental model:

normal build:
  faster, closer to production, less diagnostic help

debug/sanitizer build:
  slower, catches more bugs, better for development

For a new contributor, debug or sanitizer builds are often worth the cost.

Make Targets

The common path is:

make -j

Unit test binaries are also built under the test tree. Some tests require optional features. Some apps only build if their dependencies are configured.

If a target fails:

read the first compiler error.
check whether generated config is current.
check whether the feature is enabled.
check whether dependencies are installed.
rebuild after a clean configure if flags changed significantly.

Avoid deleting build outputs blindly when a focused rebuild would explain the issue.

Host Setup

Many SPDK examples need hugepages. Hardware examples also need devices bound to suitable drivers. scripts/setup.sh is the helper. The official guide is doc/system_configuration.md.

Useful concepts:

hugepages back DMA-capable memory.
VFIO is commonly used for userspace PCI access.
permissions matter when running as non-root.
NUMA placement can affect performance.
containers need explicit hugepage and device access.

Source anchors:

scripts/setup.sh: configure_linux_hugepages.
scripts/setup.sh: configure_linux_pci.
scripts/setup.sh: usage.
doc/memory.md.
doc/containers.md.

Do not mix host setup debugging with code debugging. First prove the host can run an unmodified SPDK example.

Test Pyramid

Think of SPDK tests in layers:

unit tests
  fastest, mocked, source-level behavior

functional scripts
  run SPDK apps and RPCs, may need root or hugepages

integration tests
  use hardware, kernel clients, VMs, fabrics, fio

autotest/CI suites
  broad coverage, expensive, environment-dependent

A small RPC decoder change may only need a unit test and perhaps a script smoke test. A bdev lifetime change may need unit tests plus a functional create/delete/replay test. An NVMe-oF transport change may need target and host integration tests.

Unit Tests

The unit-test driver is test/unit/unittest.sh. It groups tests by subsystem. The make include is mk/spdk.unittest.mk. Many unit tests include implementation .c files directly. That allows testing static functions and mocked dependencies.

Useful anchors:

test/unit/lib/rpc/rpc.c/rpc_ut.c.
test/unit/lib/jsonrpc/jsonrpc_server.c/jsonrpc_server_ut.c.
test/unit/lib/event/reactor.c/reactor_ut.c.
test/unit/lib/thread/thread.c/thread_ut.c.
test/unit/lib/bdev/bdev.c/bdev_ut.c.
test/unit/lib/bdev/nvme/bdev_nvme.c/bdev_nvme_ut.c.
test/unit/lib/nvmf/nvmf.c/nvmf_ut.c.

Run focused unit tests first. Then run the relevant group. Then consider all unit tests if the change touches shared code.

Unit test reading model:

test setup
  |
  +-- mocks global dependencies
  |
  +-- constructs minimal objects
  |
  +-- calls function or callback
  |
  +-- asserts state, return code, cleanup

If a test is hard to write, your code may be too entangled.

Functional Tests

Functional tests usually start SPDK apps, send RPCs, and check behavior. They live under test/. They often source test/common/autotest_common.sh. The helper run_test gives named steps and common failure handling.

Examples:

test/nvme/nvme.sh.
test/nvmf/nvmf.sh.
test/vfio_user/vfio_user.sh.
test/setup/test-setup.sh.
test/dd/dd.sh.
test/accel/accel.sh.

Functional tests can require:

root.
hugepages.
free test devices.
kernel modules.
network setup.
optional libraries.

Do not treat an integration test skip as proof of success. Know whether the relevant path actually ran.

Fuzz And Sanitizers

SPDK includes fuzz-related scripts. Source anchors:

test/fuzz/autofuzz.sh.
test/fuzz/autofuzz_nvmf.sh.
test/fuzz/llvm.sh.

Sanitizer configure flags are useful for memory and undefined behavior bugs. ASAN is especially helpful for use-after-free and buffer overflows. UBSAN is useful for undefined C behavior.

Tradeoff:

sanitizers slow execution.
some timing-sensitive behavior changes.
reports are often more actionable than a later crash.

Use sanitizers for risky C changes, lifetime changes, and parser changes.

Style Checks

The main style script is scripts/check_format.sh. It covers multiple checks depending on tools installed. Source anchors:

scripts/check_format.sh: check_c_style.
scripts/check_format.sh: check_comment_style.
scripts/check_format.sh: check_include_style.
scripts/check_format.sh: check_bash_style.
doc/shfmt.md.

Run style checks before submitting. If a tool is missing locally, note it. Do not churn unrelated formatting in files you did not need to touch.

Style principles:

match local code.
keep functions focused.
avoid clever abstractions.
keep error paths clear.
include useful error messages.
avoid broad refactors inside bug fixes.

Documentation Build And References

SPDK docs live under doc/. Many docs are Doxygen-flavored markdown. doc/jsonrpc.md.jinja2 is generated from RPC metadata and comments. When changing RPC surfaces, check whether documentation or schema generation needs updating.

Useful docs:

doc/getting_started.md.
doc/applications.md.
doc/jsonrpc.md.jinja2.
doc/bdev.md.
doc/bdev_pg.md.
doc/nvmf.md.
doc/tracing.md.
doc/spdk_top.md.

Do not document behavior you did not test. Do not copy an example that only works on one machine without saying why.

Choosing Test Scope

Use this decision table in prose:

changed only docs:
  run markdown or doc-related checks if available; inspect rendered assumptions

changed one RPC decoder:
  run unit for that RPC area; run method manually if app-level behavior matters

changed JSON-RPC core:
  run jsonrpc and rpc unit tests; run a live RPC smoke test

changed bdev core:
  run bdev unit group; run one functional bdev workflow

changed a bdev module:
  run that module unit if present; run create, I/O, delete, config replay

changed thread/reactor:
  run event and thread unit tests; consider broader app smoke tests

changed NVMe-oF:
  run nvmf unit tests; run target/host functional tests matching transport

changed build scripts:
  run configure variants and check_format; inspect CI-relevant paths

When in doubt, test the smallest path that proves the branch, then one broader path that proves integration.

Contribution Shape

A good patch has:

one clear purpose.
small diff.
local style.
focused tests.
clear error handling.
no unrelated formatting.
no accidental generated output unless required.
commit message that explains why.

A weak patch has:

behavior change hidden in a refactor.
no test for the failing branch.
broad style churn.
invented naming inconsistent with the subsystem.
unclear ownership.
cleanup path omitted.

SPDK is a low-level storage project. Small and boring is usually better than broad and surprising.

Commit Message

A good commit message answers:

what changed?
why was it needed?
what user-visible behavior changes?
how was it tested?

Example shape:

bdev: reject invalid foo before creating bar

The foo RPC accepted X and failed later during Y, leaving Z state
confusing to clean up. Validate X before allocating bar so the error is
reported without partial state.

Tests: unit bdev foo invalid X

Keep the subject scoped. Use the subsystem prefix that matches nearby history.

Review Preparation

Before sending a patch, inspect:

git diff
git diff --stat
git status

Check:

only intended files changed.
generated files are expected.
no debug prints remain.
no local paths or machine names.
tests actually ran and passed.
docs match behavior.

If the worktree has unrelated changes from others, leave them alone. Do not revert work you did not create.

Common Build Failures

`mk/config.mk` Missing

Run ./configure. Source anchor: mk/spdk.common.mk.

Missing Dependency

Run or inspect scripts/pkgdep.sh. If enabling optional features, install their dependencies.

Feature Method Missing At Runtime

Check configure flags. Call rpc_get_methods. Search for SPDK_RPC_REGISTER.

Hugepage Failure

Read doc/system_configuration.md. Inspect scripts/setup.sh. Check app options in doc/applications.md.

Unit Test Binary Missing

Build tests. Check whether the relevant feature was configured. Check the Makefile under test/unit/....

Style Tool Missing

scripts/check_format.sh reports missing tools. Install the tool if needed or state what could not run.

Edge Cases

Tests Pass Without Exercising Feature

Optional features can skip. Read test output. Confirm the path you changed actually ran.

ASAN Changes Timing

ASAN can expose memory bugs and also alter timing. Use it for memory correctness, not performance conclusions.

Hardware Tests Are Not Portable

NVMe, RDMA, vfio-user, and vhost tests depend on environment. Document what was available.

Generated Config Diff Is Order-Sensitive

Config replay can depend on object order. Do not reorder generated output casually.

Public Header Changes Ripple

Changing include/spdk/*.h can affect API users. Prefer internal changes when public API is not required.

Dirty Worktree

Other agents or users may have unrelated edits. Do not clean or revert them. Keep your diff scoped.

Misconceptions To Kill

"One successful build proves the change." Build proves compilation, not behavior.
"Full autotest is always the right first test." Focused tests find failures faster.
"Unit tests are less real." They are often the best way to lock down edge cases.
"Integration tests replace unit tests." They catch different classes of bugs.
"Style fixes can be mixed in." Unrelated churn makes review harder.
"Debug builds are only for crashes." They help catch bad assumptions earlier.
"A skipped test is a pass for my feature." It may not have tested your feature.
"Contribution starts after coding." It starts with choosing a small, testable shape.

Lab: Build Matrix

Create a local build matrix on paper. Include:

default build.
debug build.
sanitizer build.
one feature-specific build relevant to your project.

For each, write what failure class it is best at catching. Then choose the minimum two builds for your next extension project.

Lab: Test Scope Selection

Pick a hypothetical change:

Add an optional JSON field to bdev_get_bdevs output.

Write:

files likely touched.
unit tests to run.
functional smoke test to run.
style check to run.
docs to inspect.

Now repeat for:

Change spdk_thread_send_msg behavior.

Notice how test scope expands.

Lab: Read A Unit Makefile

Open a unit test Makefile under test/unit/lib/bdev. Find SPDK_LIB_LIST. Find the include of mk/spdk.unittest.mk. Explain how the test declares its dependencies. Then open the matching _ut.c file and find the test registration.

Lab: Reproduce A Style Failure

Read scripts/check_format.sh. Find check_c_style. Find check_include_style. Find check_bash_style. Write what external tools each check may require. Explain what you would report if one tool is not installed locally.

Lab: Contribution Readiness Checklist

For your next patch, fill this out:

purpose:
files changed:
risk area:
focused test:
broader test:
style check:
docs impacted:
rollback impact:
known environment limits:

If any line is blank, resolve it before submission.

Self-Check

What file tells make to run configure first when config is missing?
What script is the main unit-test driver?
What script handles style checks?
Why should test scope match blast radius?
Why can optional features make tests skip?
What is a good first test after changing an RPC decoder?
Why are sanitizer builds useful for lifetime changes?
What should you inspect before committing?

References

doc/getting_started.md for official first build instructions.
doc/system_configuration.md for host setup.
doc/applications.md for application options.
doc/distributions.md for CI and distribution context.
doc/shfmt.md for shell formatting.
scripts/pkgdep.sh for dependencies.
scripts/setup.sh for hugepages and devices.
scripts/check_format.sh for style checks.
test/unit/unittest.sh for unit tests.
autotest.sh for broad testing.
scripts/core-collector.sh for crash collection.

Round 3 Source Walkthrough: `configure` Is The Build Contract

SPDK's configure script is the source of truth for build features in your checkout. Do not rely on memory from an older release, a blog post, or a previous branch.

From configure:

set -e

trap 'echo -e "\n\nConfiguration failed\n\n" >&2' ERR

rootdir=$(readlink -f $(dirname $0))
source "$rootdir/scripts/common.sh"

set -e means the script stops on unhandled errors. The trap prints a clear failure message. rootdir makes later paths independent of the current shell directory. scripts/common.sh brings in shared helper functions.

The help text exposes the feature surface. This excerpt is partial:

echo " --enable-debug            Configure for debug builds"
echo " --enable-asan             Enable address sanitizer"
echo " --enable-ubsan            Enable undefined behavior sanitizer"
echo " --disable-tests           Disable building of functional tests"
echo " --disable-unit-tests      Disable building of unit tests"
echo " --with-vfio-user[=DIR]    Build custom vfio-user transport for NVMf target and vfio-user target."
echo " --with-rdma[=PROVIDER]    Build RDMA transport for NVMf target and initiator."
echo " --with-usdt               Build with userspace DTrace probes enabled."

This is why a build problem can be a configuration problem. If you forgot --with-rdma, RDMA code and tests may not build or run the way you expect. If you forgot --with-vfio-user, vfio-user chapters are theory only for that binary. If you used --disable-unit-tests, the unit-test target absence is expected.

Practical rule: when reading source, always ask "is this code compiled into my binary?"

Round 3 Source Walkthrough: Unit Test Driver

From test/unit/unittest.sh:

set -xe

testdir=$(readlink -f $(dirname $0))
rootdir=$(readlink -f $(dirname $0)/../..)
source "$rootdir/test/common/autotest_common.sh"

cd "$rootdir"

set -x prints commands, which is valuable in CI logs. set -e stops on failure. The script resolves both test directory and repo root, then sources shared test helpers.

A unit group is just a function that runs binaries:

function unittest_bdev() {
	$valgrind $testdir/lib/bdev/bdev.c/bdev_ut
	$valgrind $testdir/lib/bdev/nvme/bdev_nvme.c/bdev_nvme_ut
	$valgrind $testdir/lib/bdev/raid/bdev_raid.c/bdev_raid_ut
}

This shows how to run focused tests. If you changed module/bdev/nvme, do not start with a full hardware suite. Start with bdev_nvme_ut if it covers your path. If you changed RAID math, start with RAID unit tests. If the change is shared bdev lifetime, broaden to bdev_ut and related virtual bdev tests.

Edge cases:

Unit tests may use mocks that do not behave like real hardware.
A unit test can pass while integration fails due to threading or environment.
A hardware test can fail due to environment even when code is correct.
Valgrind or sanitizers can make timing-sensitive tests slower.

Round 3 Source Walkthrough: Shared Test Harness

From test/common/autotest_common.sh:

set -e
shopt -s nullglob
shopt -s extglob
shopt -s inherit_errexit

if [ -z "${output_dir:-}" ]; then
	mkdir -p "$rootdir/../output"
	export output_dir="$rootdir/../output"
fi

The shared harness also stops on failure. It configures shell behavior and ensures an output directory exists. SPDK functional tests are often shell-heavy orchestration around binaries, RPCs, logs, and devices. If you are writing tests, copy harness patterns instead of writing ad-hoc shell.

The harness sources build config:

if [[ -e $rootdir/test/common/build_config.sh ]]; then
	source "$rootdir/test/common/build_config.sh"
elif [[ -e $rootdir/mk/config.mk ]]; then
	build_config=$(< "$rootdir/mk/config.mk")
	source <(echo "${build_config//\?=/=}")
else
	source "$rootdir/CONFIG"
fi

This is how tests learn what was configured. A test can skip or change behavior when a dependency is absent. When a test unexpectedly skips, read this build-config path before assuming CI ignored your code.

Build Profiles For Learning

Use different build profiles for different questions:

fast local compile
  ./configure
  make -j

source-reading/debug build
  ./configure --enable-debug
  make -j

memory bug hunt
  ./configure --enable-debug --enable-asan --enable-ubsan
  make -j

vfio-user learning
  ./configure --with-vfio-user
  make -j

RDMA/NVMe-oF learning
  ./configure --with-rdma
  make -j

Do not combine every flag by default. Sanitizers are slower and can change timing. Feature flags require dependencies. Keep the build matched to the question.

Choosing The Right Test For A Change

Change	First test	Broader test
RPC decoder only	RPC/unit test or focused script	JSON-RPC docs/example check
bdev submit path	relevant bdev unit	null/malloc functional I/O
virtual bdev wrapper	module unit if present	create/use/delete/replay script
NVMe-oF target logic	nvmf unit	`test/nvmf/nvmf.sh` when environment permits
vhost/vfio-user	unit/fuzz if present	vhost or vfio-user functional suite
config writer	replay generated config	app restart plus query
diskengine wrapper	Go unit/integration plus SPDK RPC mock	real SPDK smoke test

Do not under-test lifetime changes. If you changed create/delete, test create/delete. If you changed config output, test replay. If you changed error handling, test the error branch.

Contribution Workflow

For SPDK itself:

read local style
make smallest patch
build focused target
run focused test
run formatting checks
write clear commit message
include why, not just what

For diskengine's SPDK integration:

write down desired SPDK graph
mock or isolate RPC wrapper where possible
test idempotency
test partial failure
test retry after observed existing state
run one real SPDK smoke path before trusting integration

The two worlds have different failure modes. SPDK C changes often fail through lifetime, thread affinity, resource ownership, or ABI. diskengine integration changes often fail through orchestration order, stale state, idempotency, and incomplete rollback.

Final Review Questions Before A Patch

What exact source path did I change?
What thread owns the state I touched?
What happens on -ENOMEM?
What happens if the object is deleted while I/O is outstanding?
What happens if config is saved and replayed?
What happens if the RPC is called twice?
What happens if shutdown begins during the operation?
What test proves the success path?
What test proves the failure path?
What logs or stats help debug it in production?

If the answer is "I do not know" for thread ownership or cleanup, stop and read more source before expanding the patch.