Chapter 27: Config Save, Replay, And --wait-for-rpc

Reader Promise

By the end of this chapter you should understand why SPDK configuration is not just a static file. SPDK applications are built around subsystems, RPC methods, bdev examination, and asynchronous state changes. A saved config is a replay script: a list of control-plane operations that try to recreate runtime objects in the right order.

That distinction matters. A config replay can fail because a name already exists, because a base bdev has not appeared yet, because a subsystem is not initialized, because a method is runtime-only, because an operation is asynchronous, or because diskengine's database thinks a volume should exist while SPDK's current graph says it does not.

The Mental Model

desired state
  |
  | JSON-RPC methods
  v
SPDK runtime objects
  |
  | framework_get_config and subsystem writers
  v
saved JSON config
  |
  | replay at next startup
  v
runtime objects again, if dependencies exist

A saved config is not magic persistence for every bit of memory. It is a control-plane reconstruction recipe.

Why --wait-for-rpc Exists

SPDK applications can start in a mode where initialization pauses and waits for RPC commands. This lets an orchestrator connect, create or restore objects, then tell SPDK to continue startup. For a storage system, that is useful because the orchestrator may need to:

• Attach controllers.
• Create bdevs.
• Load lvstores.
• Expose subsystems.
• Set names and policies that are not known at compile time.
• Reconcile desired state from an external database.

The danger is that "SPDK process is running" does not mean "storage graph is ready". With --wait-for-rpc, readiness becomes a two-step concept: process alive, then control-plane initialization complete.

Config Save Is A Set Of Writers

SPDK subsystems and modules can contribute config output. The saved config usually describes objects through RPC-equivalent operations:

• Construct this bdev.
• Attach this NVMe controller.
• Create this transport.
• Create this NVMe-oF subsystem.
• Add this namespace.
• Create this lvol store or import one.

Source anchors:

• lib/init/subsystem.c: subsystem initialization and config writer dispatch.
• include/spdk_internal/init.h: subsystem structure and write_config_json hook.
• module/event/subsystems/<name>/<name>.c: concrete subsystem registration and config hooks.
• lib/rpc/rpc.c: JSON-RPC server machinery.
• module/bdev/nvme/bdev_nvme_rpc.c: NVMe bdev RPCs.
• module/bdev/lvol/vbdev_lvol_rpc.c: lvol RPCs.
• lib/bdev/bdev.c: bdev registration and lookup semantics.
• scripts/rpc.py: the operator-facing Python wrapper for RPC calls.

Replay Is Order-Sensitive

The replay order matters because objects depend on other objects:

attach physical controller
  -> physical namespace bdev appears
    -> create/import lvstore
      -> lvol bdevs appear
        -> export lvol over NVMe-oF/vhost/vfio-user

If you try to export an lvol before the lvol exists, replay fails. If you try to import an lvstore before its base bdev is examined, replay may need to wait or fail depending on the operation. If you create an object with a duplicate name, replay may fail even though the desired end state already exists.

The bdev Examine Barrier

SPDK has a built-in safeguard for one common replay race. The bdev config writer emits bdev_set_options, module and bdev creation entries, and then appends bdev_wait_for_examine as the last bdev config RPC. The source comment in lib/bdev/bdev.c:spdk_bdev_subsystem_config_json says this must be last so all bdevs finish examine.

Preserve that ordering in custom replay scripts. If auto examine is disabled by bdev_set_options, use explicit bdev_examine where needed and still wait before creating dependent lvol, RAID, or export state.

diskengine Restore Loops

diskengine adds another desired-state layer. Its database can say "volume X should exist and be exported", while SPDK says:

• Base NVMe controller is missing.
• lvstore import has not completed.
• lvol exists but is degraded.
• bdev is present but export subsystem is missing.
• export exists but listener is not reachable.
• previous replay partially succeeded.

The control plane should be idempotent where possible. That means repeated restore attempts should converge rather than create duplicate objects or oscillate between create/delete states.

Common Replay Failure Modes

• Duplicate names: replay tries to create Malloc0, Nvme0n1, or an lvol that already exists.
• Missing base bdev: virtual bdev creation depends on a base device that has not appeared.
• Late examine: bdev examine discovers metadata asynchronously, so a dependent operation may run too early.
• Missing examine barrier: generated bdev config ends with bdev_wait_for_examine; hand-written replay should not drop that wait.
• Runtime-only method during startup: some RPCs make sense only after subsystem init.
• Startup-only method during runtime: some operations are not safe after the application has started serving IO.
• Ignored partial failure: a script logs an error but continues, leaving a half-built graph.
• Ignored init errors: --json-ignore-init-errors makes app startup continue after invalid config entries, so later state may be partial even when the process reaches runtime.
• Non-idempotent delete/create: delete may be async or blocked by open descriptors, so immediate recreate can race.
• External system disagreement: diskengine DB, SPDK graph, and guest-visible exports disagree.

How To Read Replay Code

When reading a config or restore path, ask:

1. What object is the source of truth?
2. What RPC creates or mutates it?
3. What dependencies must already exist?
4. Is the operation synchronous or callback-driven?
5. What happens if the object already exists?
6. What happens if the object is missing but should eventually appear?
7. What exact error gets returned to the orchestrator?

This is the same async reasoning pattern used everywhere else in SPDK.

Source Reading Exercise

Trace one operation: NVMe bdev attach.

1. Find the RPC entry point in module/bdev/nvme/bdev_nvme_rpc.c.
2. Follow the call into the attach path in module/bdev/nvme/bdev_nvme.c.
3. Identify where controller discovery becomes namespace bdev registration.
4. Find what would be saved by config output.
5. Write down what a replay script must assume before it can use the new bdev.

Operational Exercise

Take a hypothetical failed restore:

diskengine wants volume vol-a
SPDK has lvstore lvs0
SPDK does not have lvol vol-a
NVMe-oF subsystem nqn.excloud:vol-a exists with no namespace
guest attach is retrying

Classify the failure:

• Is it bdev graph state?
• lvol metadata state?
• export state?
• diskengine desired-state mismatch?
• replay ordering?

Then write the safest next check. Do not start by deleting things. Start by observing names, open descriptors, and whether async operations are still in flight.

Misconceptions To Kill

• "Config is just a file." It is a replay of operations.
• "If replay failed, nothing changed." Many failures are partial.
• "If the SPDK process is up, storage is ready." Startup may be paused or still examining bdevs.
• "Retry always helps." Retrying non-idempotent operations can create duplicates or noisy error loops.
• "The DB is truth." The DB is desired state; SPDK runtime and device reality still have to converge.

References

• SPDK JSON-RPC guide: https://spdk.io/doc/jsonrpc.html
• SPDK applications overview: https://spdk.io/doc/app_overview.html
• SPDK block device guide: https://spdk.io/doc/bdev.html
• SPDK logical volumes: https://spdk.io/doc/logical_volumes.html

Self-Check

• Why is config replay order-sensitive?
• What does --wait-for-rpc change about readiness?
• Why can a replay failure be partial?
• What makes a restore operation idempotent?
• Why should diskengine reconcile instead of blindly recreate every missing object?

Source Walkthrough: What framework_get_config Really Does

The name framework_get_config sounds like SPDK is reading a file. It is not. It is asking a live subsystem to serialize the operations that would recreate its current configuration.

From lib/init/subsystem_rpc.c:

subsystem = subsystem_find(req.name);
if (!subsystem) {
	spdk_jsonrpc_send_error_response_fmt(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
					     "Subsystem '%s' not found", req.name);
	free(req.name);
	return;
}

w = spdk_jsonrpc_begin_result(request);
subsystem_config_json(w, subsystem);
spdk_jsonrpc_end_result(request, w);

Read this slowly.

subsystem_find(req.name) means the request is scoped to one subsystem name. If you ask for a subsystem that this app did not register, the error is not a storage error; it is a framework/config ownership error. The subsystem is the owner of the config writer.

spdk_jsonrpc_begin_result(request) starts the JSON-RPC response. SPDK is not building a shell script. It is writing structured JSON that clients can feed back as RPC calls.

subsystem_config_json(w, subsystem) is the important call. It delegates to the subsystem. That means bdev, nvmf, vhost, accel, sock, and other subsystems each decide what they can safely dump.

spdk_jsonrpc_end_result(request, w) sends the completed JSON response. If the subsystem emits stale, incomplete, or non-replayable state, the RPC machinery will still successfully send it. Replay correctness belongs to the subsystem writers and the object lifecycle, not to JSON-RPC transport itself.

The helper in lib/init/subsystem.c is tiny:

void
subsystem_config_json(struct spdk_json_write_ctx *w, struct spdk_subsystem *subsystem)
{
	if (subsystem && subsystem->write_config_json) {
		subsystem->write_config_json(w);
	} else {
		spdk_json_write_null(w);
	}
}

This is the whole mental model. A subsystem either has a write_config_json hook or it does not. If it has one, that hook emits the recreate recipe. If it does not, config output for that subsystem is null. There is no hidden global snapshot that knows how every object should be rebuilt.

The edge case is easy to miss: a subsystem can be alive and functional but still not dump every runtime fact. Config writers intentionally omit transient facts such as queue depth counters, current poller load, connection retry state, and in-flight I/O. A saved config is not a crash-consistent heap image. It is an explicit set of creation calls.

Source Walkthrough: Replay Enters Through spdk_subsystem_load_config

Replay is the reverse direction: take JSON, decode it into RPC calls, and run those calls during initialization.

From lib/init/json_config.c:

void
spdk_subsystem_load_config(void *json, ssize_t json_size, spdk_subsystem_init_fn cb_fn,
			   void *cb_arg, bool stop_on_error)
{
	assert(cb_fn);
	assert(spdk_thread_is_app_thread(NULL));

	json_config_prepare_ctx(cb_fn, cb_arg, stop_on_error, json, json_size, false);
}

The two assertions tell you where replay lives. cb_fn must exist because config loading is asynchronous: SPDK must call back when replay finishes or fails. spdk_thread_is_app_thread(NULL) means config replay starts on the app thread, not from an arbitrary worker thread. That matters because subsystem initialization and many RPC handlers assume framework-thread ownership.

json_config_prepare_ctx(...) prepares the replay context. The key parameter is stop_on_error. If it is true, the first failed method stops replay. If it is false, replay may continue after a failure. Continuing can be useful for diagnostics, but it can leave partial state. A partial replay is not a rollback. It is a process that now contains whatever earlier calls successfully created.

That is why the operational rule is: after replay failure, inspect state before retrying. Do not assume "failed replay" means "no resources were created".

Source Walkthrough: Bdev Config Dump Is Per Object

The bdev subsystem does not dump a mystical "bdev graph". It iterates registered bdevs and asks each bdev's function table how to emit its own config.

From lib/bdev/bdev.c:

TAILQ_FOREACH(bdev, &g_bdev_mgr.bdevs, internal.link) {
	if (bdev->fn_table->write_config_json) {
		bdev->fn_table->write_config_json(bdev, w);
	}

	bdev_qos_config_json(bdev, w);
}

The TAILQ_FOREACH is the live list. If an object is not registered at the moment of the dump, it will not appear. If an object is registered but its module has no config writer, it may not be replayable from the dump. The bdev->fn_table->write_config_json call means config behavior is module-specific: malloc bdevs, NVMe bdevs, lvol bdevs, RAID bdevs, and virtual bdevs each have their own writer logic.

bdev_qos_config_json(bdev, w) is separate. QoS is configuration layered on top of the bdev object. This is a common pattern in SPDK: the base object and its policies are often emitted by different helpers. If you add a feature and expect it to survive config replay, you must find the owner that dumps it.

Edge cases:

• If a bdev is being unregistered while config is dumped, the writer must obey bdev lifetime rules.
• If a virtual bdev depends on a base bdev, config order must preserve the base-before-child relationship.
• If a bdev represents hardware, replay assumes the hardware can be found again under compatible transport IDs.
• If a bdev name is generated rather than stable, replay may create a different graph than the one you expected.

--wait-for-rpc As A Deliberate Gap In Startup

--wait-for-rpc is best understood as a controlled pause between "framework can accept some RPCs" and "application is fully initialized".

Without the pause, startup tries to proceed using built-in config or supplied JSON. With the pause, an external orchestrator can connect and issue startup-phase methods. This is useful when the real desired state is not in a local JSON file but in a database, cluster manager, or diskengine controller.

The dangerous misunderstanding is to treat a listening RPC socket as readiness. During the pause, the process exists and the RPC endpoint may answer, but the storage graph may intentionally be incomplete. Some methods are valid in startup state; some are runtime only. rpc_get_methods is your friend because it tells you what the current process is willing to accept.

Practical startup states:

binary execs
  -> env/DPDK setup begins
  -> app framework starts
  -> RPC can be initialized
  -> wait-for-rpc pause may happen
  -> orchestrator creates startup objects
  -> orchestrator sends framework_start_init
  -> subsystems finish init
  -> runtime methods become available
  -> data path can serve real I/O

If diskengine starts creating exports before the underlying bdevs exist, it is not "almost ready"; it is violating dependency order. If diskengine waits forever for a runtime-only method while SPDK is still in startup state, it is not a network problem; it is a state-machine problem.

What A Replay Script Must Prove At Each Step

For every RPC in a replay file, ask four questions:

1. Is the method available in the current framework state?
2. Are all named dependencies already present?
3. Is the operation idempotent if the object already exists?
4. Does the operation finish synchronously or continue through a callback?

Example: an lvol export over NVMe-oF.

bdev_nvme_attach_controller or base local NVMe attach
  -> bdev appears
  -> bdev_lvol_create_lvstore or import existing lvstore
  -> lvol bdev appears
  -> nvmf_create_transport
  -> nvmf_create_subsystem
  -> nvmf_subsystem_add_ns
  -> nvmf_subsystem_add_listener

Each arrow is a dependency. If nvmf_subsystem_add_ns runs before the lvol bdev appears, the failure is not in NVMe-oF. The target cannot export a namespace backed by a bdev that does not exist. If nvmf_subsystem_add_listener runs twice with the same address, the desired final state may be correct but the create operation may still fail because the listener already exists.

That is the heart of idempotent orchestration: desired state and create command are not the same thing.

diskengine Reconciliation Rules

For diskengine, treat SPDK as an eventually reconciled runtime graph:

database desired state
  -> diskengine reconcile loop
  -> SPDK JSON-RPC operations
  -> SPDK runtime graph
  -> observed RPC state
  -> next reconcile decision

A safe reconcile loop reads before it writes. It should decide whether the object is missing, present and correct, present but different, present but deleting, or present but unusable. These states require different actions.

For example, if a volume export failed halfway, SPDK may contain:

• an lvol bdev,
• an NVMe-oF subsystem,
• no namespace,
• one listener,
• or a namespace attached to the wrong bdev.

Blindly rerunning the whole creation script can produce duplicate-name errors at the subsystem step and never reach the missing namespace step. The better approach is to converge each layer:

1. Prove base bdev.
2. Prove lvstore.
3. Prove lvol.
4. Prove subsystem.
5. Prove namespace mapping.
6. Prove listener.
7. Prove host visibility.

Edge Case: framework_get_config Can Lie By Omission

It is not malicious, but it can omit what you care about.

framework_get_config can omit:

• external clients currently connected,
• in-flight I/O,
• reconnect backoff state,
• temperature/media-health changes,
• host kernel driver bindings,
• hugepage reservation,
• RDMA NIC state,
• diskengine database state,
• cluster membership,
• user permissions and socket ownership.

Those are outside the saved SPDK object recipe. A config that replays perfectly can still fail to serve a VM if the NIC is down, the controller was rebound to the kernel, or diskengine exposes a different NQN.

Edge Case: Replay After Crash

After a crash, three truths may disagree:

SPDK saved config        what the process last knew how to recreate
on-disk metadata         what blobstore/lvol/SSD metadata actually contains
diskengine database      what the control plane wants to exist

If the saved config says "create lvol X" but on-disk metadata already contains lvol X, the correct operation may be import/discover, not create. If diskengine says export X but SPDK has no base bdev because the NVMe controller failed to attach, the correct operation is to repair attachment, not to delete the database row.

The safest post-crash sequence is:

1. Start SPDK with the least destructive config that discovers base devices.
2. Query bdevs and lvstores.
3. Compare discovered objects to diskengine desired state.
4. Create only missing non-destructive wrapper/export objects.
5. Delay destructive cleanup until references, guests, and metadata are understood.

Edge Case: Method Phase Mismatch

SPDK RPC methods are registered with a state such as startup or runtime. If a method is not available, do not immediately assume the code is missing. Ask whether you are in the wrong phase.

Symptoms:

• method exists in source via SPDK_RPC_REGISTER,
• method appears in docs,
• rpc_get_methods does not list it for the current phase,
• calling it returns an unknown-method style error.

Debugging path:

search SPDK_RPC_REGISTER("method")
  -> identify SPDK_RPC_STARTUP or SPDK_RPC_RUNTIME
  -> call rpc_get_methods
  -> check whether app is paused under wait-for-rpc
  -> send framework_start_init only when startup config is complete

This edge case matters when diskengine is both an orchestrator and a health monitor. Startup logic and runtime healing logic must not use exactly the same RPC sequence without checking phase.

Source Reading Checklist For Adding Replay Support

When you add a new SPDK feature and want it to survive save/replay, read in this order:

1. The RPC that creates the object.
2. The object struct that stores the durable configuration.
3. The subsystem or bdev config writer.
4. The generated JSON shape from framework_get_config.
5. A clean-process replay of that JSON.
6. The delete path and shutdown path.

Ask:

• Which parameters are required to recreate the object?
• Which parameters are derived and should not be dumped?
• Which parameters are secrets or host-specific and should be handled carefully?
• What order is required relative to base objects?
• What error appears if the object already exists?
• What happens if replay creates the object but a later RPC fails?

If you cannot answer those, the feature is not replay-ready yet.