Chapter 33: Debugging Playbooks

Reader Promise

This chapter turns the failure taxonomy into action. The goal is not to memorize every SPDK failure. The goal is to stop debugging randomly. Each playbook starts with a symptom, identifies the likely layer, names the first safe observations, and then points to the SPDK source families that explain the behavior.

The rule: observe before mutating. Do not delete bdevs, kill controllers, or replay config until you know which layer is inconsistent.

Playbook Format

Volume Missing

Source anchors:

• lib/bdev/bdev.c
• module/bdev/nvme/bdev_nvme.c
• module/bdev/lvol/vbdev_lvol.c
• lib/lvol/lvol.c
• lib/blob/blobstore.c

Mutation boundary: do not recreate an lvol until you know whether metadata import is still pending or the lvol exists under a different name.

RAID Stuck Configuring

First classify whether the RAID bdev lacks base bdevs, has metadata disagreement, or is waiting on rebuild/online transition.

Checks:

• List bdevs and confirm every base name.
• Confirm base block sizes and lengths.
• Check RAID state and base membership.
• Check whether a base was removed and later re-added with a different identity.

Source anchors:

• module/bdev/raid/bdev_raid.c
• module/bdev/raid/raid1.c
• module/bdev/raid/raid0.c
• lib/bdev/bdev.c

Mutation boundary: do not force base replacement until you know whether the current array state is degraded-but-recoverable or inconsistent.

Controller Reconnect Loop

Symptoms:

• Repeated attach/reconnect logs.
• IO stalls then resumes.
• NVMe-oF path oscillates.
• Multipath never settles on an active path.

Checks:

• Identify transport: PCIe, RDMA, TCP, or vfio-user.
• Check controller timeout and reconnect options.
• Check qpair failure reason if available.
• Confirm whether the issue is one controller, one path, or every path.

Source anchors:

• lib/nvme/nvme_ctrlr.c
• lib/nvme/nvme_qpair.c
• module/bdev/nvme/bdev_nvme.c:bdev_nvme_reconnect_ctrlr
• module/bdev/nvme/bdev_nvme.c:bdev_nvme_failover_ctrlr
• module/bdev/nvme/bdev_nvme.c:bdev_nvme_reset_ctrlr

Mutation boundary: do not detach a controller backing live bdevs until descriptors, exports, and diskengine desired state are accounted for.

Guest IO Hang

The key question is where the IO stopped:

guest driver
  -> vhost/vfio-user queue
    -> SPDK transport request
      -> bdev submit
        -> lower bdev/lvol/RAID/NVMe
          -> completion callback
            -> CQ/used-ring notification

Checks:

• Did the guest submit a descriptor, SQE, or doorbell?
• Did SPDK receive the request?
• Did bdev stats increase?
• Did the lower bdev complete?
• Did SPDK write completion state back to the guest-visible queue?
• Did the guest receive an interrupt or poll the completion?

Source anchors:

• lib/vhost/vhost_blk.c
• lib/nvmf/vfio_user.c
• lib/nvmf/ctrlr_bdev.c
• lib/bdev/bdev.c
• module/bdev/raid/bdev_raid.c
• module/bdev/lvol/vbdev_lvol.c

Mutation boundary: do not reset the guest-facing device until you know whether the bdev completed. A completed bdev with a waiting guest points to export/notification, not storage media.

Config Replay Failure

Symptoms:

• SPDK starts but expected objects are missing.
• Replay logs show duplicate-name or missing-base errors.
• diskengine retries restore and produces repeated RPC failures.

Checks:

• Determine which RPC failed first.
• Check whether prior RPCs partially succeeded.
• Compare saved config, current SPDK graph, and diskengine DB.
• Identify whether bdev examine is still in progress.

Source anchors:

• lib/rpc/rpc.c
• lib/init/json_config.c
• lib/init/subsystem.c
• module/bdev/nvme/bdev_nvme_rpc.c
• module/bdev/lvol/vbdev_lvol_rpc.c
• lib/nvmf/nvmf_rpc.c

Mutation boundary: do not rerun a non-idempotent create loop until duplicate-name and partial-success state are understood.

High Latency

Classify the latency:

• Device media latency.
• NVMe qpair timeout/retry.
• Reactor starvation.
• bdev queueing/QoS.
• lvol/blobstore metadata operation.
• RAID rebuild or degraded path.
• Transport congestion.
• Guest notification delay.

First safe checks:

• bdev stats.
• reactor/thread/poller stats.
• NVMe health/log pages.
• transport-specific counters.
• diskengine reconciliation logs.

Source anchors:

• lib/thread/thread.c
• lib/event/reactor.c
• lib/bdev/bdev.c
• module/bdev/nvme/bdev_nvme.c
• lib/nvme/nvme_ctrlr_cmd.c

Mutation boundary: do not tune queue depth or disable polling until you know whether latency is queueing, media, CPU starvation, or transport.

ENOMEM And NOMEM

SPDK often distinguishes ordinary allocation failure from the bdev NOMEM retry path. Do not treat every memory-looking error as fatal.

Checks:

• Which allocation failed?
• Is the IO queued for retry?
• Is an iobuf wait path involved?
• Is there a no-memory poller?
• Are large requests being split?

Source anchors:

• lib/bdev/bdev.c:_bdev_io_handle_no_mem
• nearby bdev NOMEM retry helpers in lib/bdev/bdev.c
• lib/thread/thread.c
• lib/thread/iobuf.c
• lib/env_dpdk/env.c

Mutation boundary: do not fail user-visible IO until you know whether SPDK expects to retry internally.

Self-Check

• Why should a guest IO hang be split into "bdev completed" and "guest notified" branches?
• What makes config replay dangerous to retry blindly?
• Why can RAID configuring be a base-device problem rather than a RAID algorithm problem?
• Why is high latency a taxonomy problem before it is a tuning problem?
• What is the first safe action in every playbook?

Playbook: App Will Not Start

Symptom:

spdk_tgt exits during startup, no useful RPC response exists

Mental model:

Startup failures happen before normal runtime observation is available. You are debugging command line, build options, DPDK/env setup, host setup, or config replay.

First safe checks:

./build/bin/spdk_tgt --help
scripts/setup.sh status
grep -i huge /proc/meminfo
ls -l /dev/vfio

Source path:

• configure
• lib/event/app.c
• lib/env_dpdk/init.c
• scripts/setup.sh
• doc/system_configuration.md

Likely causes:

• no hugepages,
• wrong user permissions,
• PCI device still bound to kernel,
• feature not compiled in,
• invalid command-line option,
• config replay failed before RPC became usable.

Edge cases:

• A missing optional dependency can remove a module or RPC you assumed existed.
• A stale mk/config.mk can preserve old configure choices.
• A container can show hugepages on the host but not mount them inside the container.
• Starting two SPDK apps with overlapping CPU/device ownership can produce confusing errors.

Fix/rollback:

Start a minimal app with no hardware-backed devices. Then add one feature at a time: hugepages, VFIO, NVMe, NVMf, vhost. If the minimal app fails, do not debug diskengine yet.

Playbook: RPC Socket Exists But RPC Fails

Symptom:

scripts/rpc.py -s /var/tmp/spdk.sock bdev_get_bdevs
connects sometimes, fails sometimes, or method is unknown

Mental model:

There are three separate failures:

client cannot connect
  != method does not exist
  != method exists but params/operation fail

Source excerpt from lib/init/subsystem_rpc.c:

subsystem = subsystem_find(req.name);
if (!subsystem) {
	spdk_jsonrpc_send_error_response_fmt(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
					     "Subsystem '%s' not found", req.name);
	free(req.name);
	return;
}

This branch is not a socket failure. The client reached the server, JSON-RPC dispatch reached the framework handler, and the handler rejected the requested subsystem name.

Commands:

scripts/rpc.py -s /exact/socket rpc_get_methods
scripts/rpc.py -s /exact/socket framework_get_config -n bdev
scripts/rpc.py -s /exact/socket framework_get_config -n nvmf

Likely causes:

• wrong socket path,
• stale socket,
• wrong process,
• method exists but is rejected with SPDK_JSONRPC_ERROR_INVALID_STATE,
• module not linked into this binary,
• params valid JSON but semantically invalid.

Edge cases:

• --wait-for-rpc can make startup methods available while runtime methods are not.
• rpc_get_methods shows registered methods, while rpc_get_methods '{"current": true}' filters to methods legal in the current phase.
• Old rpc.py against a newer server can call renamed or removed methods.
• Multiple SPDK processes can use different sockets; always pass -s explicitly.

Fix/rollback:

Do not delete state. First prove which process owns the socket and which methods are currently registered.

Playbook: NVMe Attach Fails

Symptom:

bdev_nvme_attach_controller fails
no Nvme0n1 bdev appears
diskengine keeps retrying attach

Mental model:

Attach has two halves: control-plane RPC accepts a transport ID, then the NVMe library connects/discovers/enumerates namespaces. Failure can be params, transport, permissions, fabric, controller, or namespace.

First safe checks:

scripts/rpc.py bdev_nvme_get_controllers
scripts/rpc.py bdev_get_bdevs
scripts/rpc.py bdev_nvme_get_io_paths

For PCIe:

lspci -nn
scripts/setup.sh status
ls -l /dev/vfio

For RDMA/TCP:

scripts/rpc.py nvmf_get_subsystems
scripts/rpc.py nvmf_get_transports

Likely source family:

• module/bdev/nvme/bdev_nvme_rpc.c
• module/bdev/nvme/bdev_nvme.c
• lib/nvme/nvme.c
• lib/nvme/nvme_pcie.c
• lib/nvme/nvme_rdma.c

Edge cases:

• Controller object exists but no namespaces were exposed.
• PCI controller is visible but bound to kernel nvme.
• NVMf target listener exists but host NQN is not allowed.
• Multipath can leave controller present but no usable active path.
• Reconnect may hide intermittent path failure until I/O starts.

Fix/rollback:

Detach only after checking descriptors and consumers. If the controller backs live bdevs or RAID bases, removal can propagate guest-visible errors.

Playbook: NVMf Export Fails

Symptom:

nvmf_subsystem_add_ns or nvmf_subsystem_add_listener fails
initiator cannot connect
storage node lvol exists

Mental model:

An NVMf export is not one object. It is transport, subsystem, namespace, listener, and host/access policy. Each layer can be correct while another is missing.

Source excerpt from the target write path in lib/nvmf/ctrlr_bdev.c:

if (spdk_unlikely(!nvmf_bdev_ctrlr_lba_in_range(bdev_num_blocks, start_lba, num_blocks))) {
	SPDK_ERRLOG("end of media\n");
	rsp->status.sct = SPDK_NVME_SCT_GENERIC;
	rsp->status.sc = SPDK_NVME_SC_LBA_OUT_OF_RANGE;
	return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
}

This branch happens after connect succeeds, but it proves a broader point: NVMf target errors are often protocol statuses generated above the physical SSD. An NVMf failure does not automatically mean media failure.

Commands:

scripts/rpc.py nvmf_get_transports
scripts/rpc.py nvmf_get_subsystems
scripts/rpc.py bdev_get_bdevs
scripts/rpc.py framework_get_config -n nvmf

Likely causes:

• transport not created,
• subsystem duplicate NQN,
• namespace bdev missing,
• listener address conflict,
• host NQN not allowed,
• wrong serial/model assumptions,
• subsystem paused or not active.

Edge cases:

• A listener can exist on one address while diskengine advertises another.
• A namespace can point to an old bdev after a failed rebuild/recreate sequence.
• A storage-node export can be healthy while the baremetal initiator path is broken.

Fix/rollback:

Prefer converge-in-place: add missing listener or namespace if safe. Do not delete the whole subsystem until you understand connected hosts and namespace mappings.

Playbook: vhost Path Fails

Symptom:

VM disk is missing, QEMU cannot connect to socket, or guest I/O errors immediately

Mental model:

vhost has a Unix socket/control-plane side and a data-plane ring side. A socket file alone does not prove the bdev backend exists or that the guest queue can complete I/O.

Source excerpt from lib/vhost/vhost_blk.c:

rc = spdk_bdev_writev(bvdev->bdev_desc, ch,
		      &task->iovs[1], iovcnt, req.sector * 512,
		      payload_len, blk_request_complete_cb, task);

This is the moment a guest write enters the bdev stack. If this call is never reached, debug descriptors/session/socket. If it is reached and returns -ENOMEM, debug resource queues. If it succeeds but callback never returns, debug lower bdev completion.

Commands:

scripts/rpc.py vhost_get_controllers
scripts/rpc.py bdev_get_bdevs
scripts/rpc.py bdev_get_iostat -b <backing_bdev>
scripts/rpc.py thread_get_pollers

Likely causes:

• QEMU path points at wrong socket,
• socket permissions wrong,
• vhost controller exists but backing bdev missing,
• bdev is readonly,
• guest sends malformed or unaligned request,
• lower RAID/NVMe path fails.

Edge cases:

• Guest sees an I/O error but SPDK root cause is lower-layer NVMf disconnect.
• bdev completed but guest did not get used-ring notification.
• socket file remains after crashed process.

Fix/rollback:

Detach VM/QEMU cleanly before deleting vhost controllers. Deleting a guest-facing device while QEMU still believes it exists can make recovery noisier.

Playbook: High Latency

Symptom:

fio inside VM is slow
latency spikes every few minutes
no obvious hard I/O errors

Mental model:

Latency is a path property. You must locate the queue or slow completion point.

Commands:

scripts/rpc.py bdev_get_iostat
scripts/rpc.py thread_get_stats
scripts/rpc.py thread_get_pollers
scripts/rpc.py framework_get_reactors
scripts/rpc.py bdev_nvme_get_transport_statistics
scripts/rpc.py nvmf_get_stats

Source families:

• lib/event/reactor.c
• lib/thread/thread.c
• lib/bdev/bdev.c
• module/bdev/nvme/bdev_nvme.c
• lib/nvmf/transport.c
• module/bdev/raid/*
• module/bdev/lvol/*

Likely causes:

• reactor overloaded,
• CPU isolated incorrectly,
• NUMA mismatch,
• RAID rebuild,
• NVMe-oF reconnect/backoff,
• iobuf pressure,
• SSD thermal throttling,
• guest queue depth too low or too high,
• diskengine reconciliation causing churn.

Edge cases:

• A degraded RAID can be correct but slower.
• A stats sample can lie if taken across a topology change.
• One busy poller can hurt unrelated bdevs on the same thread.
• A transport reconnect loop can look like random device latency.

Fix/rollback:

Change one thing at a time. Record before/after stats with identical workload parameters.

Playbook: Config Replay Fails

Symptom:

saved config does not recreate expected objects
duplicate-name errors appear
later RPCs fail because dependencies are missing

Mental model:

Replay is a sequence of object-creation RPCs. It is not transactional.

Source excerpt from lib/init/json_config.c:

spdk_subsystem_load_config(void *json, ssize_t json_size, spdk_subsystem_init_fn cb_fn,
			   void *cb_arg, bool stop_on_error)
{
	assert(cb_fn);
	assert(spdk_thread_is_app_thread(NULL));

	json_config_prepare_ctx(cb_fn, cb_arg, stop_on_error, json, json_size, false);
}

The callback proves replay is asynchronous. stop_on_error controls whether replay continues after a failure. Neither mode rolls back already-created objects.

Commands:

scripts/rpc.py framework_get_config -n bdev
scripts/rpc.py framework_get_config -n nvmf
scripts/rpc.py bdev_get_bdevs
scripts/rpc.py rpc_get_methods

Likely causes:

• wrong order,
• startup/runtime phase mismatch,
• duplicate resources from partial prior replay,
• hardware names changed,
• lvol metadata already exists but create path was used,
• generated config omitted an external dependency.

Edge cases:

• The first replay error may not be the one diskengine logs most often.
• Retrying a create path can hide the missing dependent object behind duplicate errors.
• A config from another host may contain impossible PCI addresses or transport IDs.

Fix/rollback:

Diff desired config, current runtime graph, and on-disk metadata. Reconcile layer by layer instead of rerunning the whole script blindly.

References

• SPDK JSON-RPC: https://spdk.io/doc/jsonrpc.html
• SPDK bdev guide: https://spdk.io/doc/bdev.html
• SPDK NVMe-oF guide: https://spdk.io/doc/nvmf.html
• SPDK system configuration: https://spdk.io/doc/system_configuration.html
• Local source: lib/init/subsystem_rpc.c
• Local source: lib/init/json_config.c
• Local source: lib/vhost/vhost_blk.c
• Local source: lib/nvmf/ctrlr_bdev.c
• Local source: module/bdev/nvme/bdev_nvme.c
• Local source: module/bdev/raid/*