SPDK From First Principles

SPDK deep learning path

Chapter 26: JSON-RPC And Configuration

This chapter teaches the SPDK control plane from the point of view of a new operator.

Source: drafts/control-debug/26-json-rpc-and-configuration.md

Chapter Goal

This chapter teaches the SPDK control plane from the point of view of a new operator. By the end, you should know what an RPC request is, where it enters SPDK, how methods are registered, why some calls are startup-only, and how JSON configuration is saved and replayed. The chapter is source-grounded, so every important concept points to a local SPDK file or function.

Beginner Mental Model

SPDK applications are long-running storage engines. The data path moves I/O. The control plane changes the engine while it is running. JSON-RPC is the control-plane language. An RPC request says: call this named method, with these parameters, and return either a result or an error. The common client is scripts/rpc.py. The common server endpoint is a Unix domain socket such as /var/tmp/spdk.sock. The common target application is app/spdk_tgt/spdk_tgt.c. The core idea is simple:

operator or orchestrator
        |
        | JSON object over a socket
        v
SPDK JSON-RPC server
        |
        | method lookup
        v
registered C handler
        |
        | validates params and mutates state
        v
subsystem, bdev, transport, or app framework

Do not confuse JSON-RPC with the I/O path. Creating a malloc bdev by RPC is control-plane work. Submitting a read to that bdev is data-path work. Both may interact, but they have different timing and safety rules.

Source Anchors

  • include/spdk/rpc.h: public registration macros and RPC state constants.
  • include/spdk/rpc.h: SPDK_RPC_REGISTER: macro used by many modules to register methods.
  • lib/rpc/rpc.c: spdk_rpc_register_method: adds a method to the internal method list.
  • lib/rpc/rpc.c: jsonrpc_handler: dispatches a parsed JSON-RPC method to a registered handler.
  • lib/rpc/rpc.c: rpc_rpc_get_methods: implements rpc_get_methods.
  • lib/init/rpc.c: spdk_rpc_initialize: initializes the framework RPC server.
  • lib/jsonrpc/jsonrpc_server.c: jsonrpc_parse_request: parses raw JSON-RPC input.
  • lib/jsonrpc/jsonrpc_server_tcp.c: spdk_jsonrpc_server_listen: listens on Unix or TCP sockets.
  • lib/event/app.c: spdk_app_start: starts a normal SPDK application.
  • lib/event/app.c: rpc_framework_start_init: starts delayed initialization when --wait-for-rpc is used.
  • lib/event/app.c: rpc_framework_wait_init: lets a client wait for initialization completion.
  • lib/init/subsystem.c: spdk_subsystem_init: initializes registered subsystems in dependency order.
  • lib/init/json_config.c: spdk_subsystem_load_config: replays JSON config through RPC handlers.
  • lib/init/subsystem_rpc.c: rpc_framework_get_config: implements framework_get_config.
  • module/bdev/malloc/bdev_malloc_rpc.c: rpc_bdev_malloc_create: a small concrete bdev RPC.
  • lib/bdev/bdev_rpc.c: rpc_bdev_get_bdevs: a common runtime query RPC.
  • doc/jsonrpc.md.jinja2: generated official JSON-RPC reference source.
  • doc/applications.md: official application command-line and configuration guide.

What A JSON-RPC Request Looks Like

SPDK follows the JSON-RPC 2.0 shape. The payload is usually one JSON object. It can be sent by scripts/rpc.py, by a service manager, or by a custom client.

{
  "jsonrpc": "2.0",
  "method": "bdev_malloc_create",
  "params": {
    "name": "Malloc0",
    "num_blocks": 1024,
    "block_size": 4096
  },
  "id": 1
}

The method name is just a string until SPDK looks it up. The params are method-specific. The id lets the client match a response to a request. A notification has no id, but most operational tools use ids.

The server does not know what num_blocks means. Only the target method handler knows. That is why each RPC handler has a decoder table. For example, bdev RPC handlers use spdk_json_decode_object with arrays of spdk_json_object_decoder. When decoding fails, the handler sends a JSON-RPC error rather than half-applying the requested change.

The usual command line hides this JSON shape, but it does not change the protocol. When you run:

scripts/rpc.py bdev_malloc_create -b Malloc0 4 4096

the Python client creates the same request object, connects to the configured socket, sends UTF-8 JSON, waits for one response, and raises an exception if the response contains an error member. Stock rpc.py and stock SPDK applications both default to /var/tmp/spdk.sock, but they are independent processes. Pass -s on the client side and -r or --rpc-socket on the app side when multiple SPDK apps or custom sockets are involved.

# scripts/rpc.py
def create_parser():
    parser = argparse.ArgumentParser(
        description='SPDK RPC command line interface', usage='%(prog)s [options]')
    parser.add_argument('-s', dest='server_addr',
                        help='RPC domain socket path or IP address', default='/var/tmp/spdk.sock')
    parser.add_argument('-p', dest='port',
                        help='RPC port number (if server_addr is IP address)',
                        default=5260, type=int)
    parser.add_argument('-t', dest='timeout',
                        help='Timeout as a floating point number expressed in seconds waiting for response. Default: 60.0',
                        default=None, type=float)

The matching C-side default is SPDK_DEFAULT_RPC_ADDR in include/spdk/init.h, used by the app option defaults and printed in lib/event/app.c help text.

The lower-level client path is intentionally generic. It does not need a Python method for every RPC name because unknown attributes become calls by name. That is why a Python helper can lag behind C registration and client.call("method_name", params) can still be useful for testing.

# python/spdk/rpc/client.py
def add_request(self, method, params):
    self._request_id += 1
    req = {
        'jsonrpc': '2.0',
        'method': method,
        'id': self._request_id,
    }

    if params:
        req['params'] = copy.deepcopy(params)

    self._logger.debug("append request:\n%s\n", json.dumps(req))
    self._reqs.append(req)
    return self._request_id

def call(self, method, params=None):
    self._logger.debug("call('%s')" % method)
    params = {} if params is None else params
    if self.timeout <= 0:
        raise JSONRPCException("Timeout value is invalid: %s\n" % self.timeout)
    req_id = self.send(method, params)

The practical result is that there are three names in play: the CLI command name, the JSON-RPC method string, and the C registration string. For in-tree helpers they normally match, but the server only cares about the method string sent in the JSON object.

Registration: How Methods Appear

Many SPDK RPC methods are not listed in one central table. They are registered by C files as the program is loaded. The macro SPDK_RPC_REGISTER in include/spdk/rpc.h creates a constructor-like registration hook. When the binary starts, the hook calls spdk_rpc_register_method. That function records the method name, handler function, and state mask.

The pattern looks like this:

SPDK_RPC_REGISTER("bdev_malloc_create", rpc_bdev_malloc_create, SPDK_RPC_RUNTIME)

Read it as:

  • name: bdev_malloc_create
  • handler: rpc_bdev_malloc_create
  • valid phase: runtime

This is why adding a new RPC normally means editing a module-specific *_rpc.c file. The bdev malloc example lives in module/bdev/malloc/bdev_malloc_rpc.c. NVMe bdev RPCs live in module/bdev/nvme/bdev_nvme_rpc.c. NVMe-oF target RPCs live in lib/nvmf/nvmf_rpc.c and module/event/subsystems/nvmf/nvmf_rpc.c.

The public macro is small, but it explains the whole registration model:

/* include/spdk/rpc.h */
#define SPDK_RPC_STARTUP	0x1
#define SPDK_RPC_RUNTIME	0x2

/* Give SPDK_RPC_REGISTER a higher execution priority than
 * SPDK_RPC_REGISTER_ALIAS_DEPRECATED to ensure all of the RPCs are registered
 * before we try registering any aliases.
 */
#define SPDK_RPC_REGISTER(method, func, state_mask) \
static void __attribute__((constructor(1000))) rpc_register_##func(void) \
{ \
	spdk_rpc_register_method(method, func, state_mask); \
}

This is a constructor, so the registration function runs before normal application startup reaches subsystem initialization. The method exists only if the object file containing this constructor is linked into the process. That is the source-level reason an RPC can appear in the documentation but be absent from a particular binary.

spdk_rpc_register_method() stores methods in a global list and rejects duplicates. There is no extra ownership transfer for the handler function; the handler is a function pointer compiled into the process. The method name is duplicated because the registration layer owns the stored name.

/* lib/rpc/rpc.c */
void
spdk_rpc_register_method(const char *method, spdk_rpc_method_handler func, uint32_t state_mask)
{
	struct spdk_rpc_method *m;

	m = _get_rpc_method_raw(method);
	if (m != NULL) {
		SPDK_ERRLOG("duplicate RPC %s registered...\n", method);
		g_rpcs_correct = false;
		return;
	}

	m = calloc(1, sizeof(struct spdk_rpc_method));
	assert(m != NULL);

	m->name = strdup(method);
	assert(m->name != NULL);

	m->func = func;
	m->state_mask = state_mask;

	SLIST_INSERT_HEAD(&g_rpc_methods, m, slist);
}

Dispatch: From Socket To Handler

The JSON-RPC server accepts bytes from a socket. jsonrpc_parse_request turns the bytes into JSON values. Then the generic RPC layer calls jsonrpc_handler. jsonrpc_handler checks whether the method exists and whether it is allowed in the current framework state. If the method is allowed, the handler receives:

  • the request object, used to send the response.
  • the params JSON value, or null if no params were supplied.

The handler then owns three jobs:

  • validate params.
  • call the subsystem API.
  • send exactly one response or error.

The handler should not trust the client. Bad JSON, wrong types, missing fields, invalid names, duplicate objects, and impossible sizes are normal inputs. Source anchor: test/unit/lib/jsonrpc/jsonrpc_server.c/jsonrpc_server_ut.c tests valid, invalid, and partial parse cases.

At the socket layer, SPDK first checks whether a complete JSON value has arrived. If parsing reports SPDK_JSON_PARSE_INCOMPLETE, the connection keeps buffering instead of calling a handler with partial input. Only after a complete value is found does the server allocate a request object, copy the request bytes, allocate response storage, and continue parsing into JSON values.

/* lib/jsonrpc/jsonrpc_server.c */
ssize_t
jsonrpc_parse_request(struct spdk_jsonrpc_server_conn *conn, const void *json, size_t size)
{
	struct spdk_jsonrpc_request *request;
	ssize_t rc;
	size_t len;
	void *end = NULL;

	rc = spdk_json_parse((void *)json, size, NULL, 0, &end, 0);
	if (rc == SPDK_JSON_PARSE_INCOMPLETE) {
		return 0;
	}

	request = calloc(1, sizeof(*request));
	if (request == NULL) {
		SPDK_DEBUGLOG(rpc, "Out of memory allocating request\n");
		return -1;
	}

	pthread_spin_lock(&conn->queue_lock);
	conn->outstanding_requests++;
	STAILQ_INSERT_TAIL(&conn->outstanding_queue, request, link);
	pthread_spin_unlock(&conn->queue_lock);

	request->conn = conn;

The generic RPC layer then does the method lookup and state check before the module handler runs. The handler receives the original request object because that object owns response delivery. If the handler starts asynchronous work, it must keep the request pointer until the completion path sends the response.

/* lib/rpc/rpc.c */
static void
jsonrpc_handler(struct spdk_jsonrpc_request *request,
		const struct spdk_json_val *method,
		const struct spdk_json_val *params)
{
	struct spdk_rpc_method *m;

	assert(method != NULL);

	m = _get_rpc_method(method);
	if (m == NULL) {
		spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_METHOD_NOT_FOUND, "Method not found");
		return;
	}

	if (m->is_alias_of != NULL) {
		if (m->is_deprecated && !m->deprecation_warning_printed) {
			SPDK_WARNLOG("RPC method %s is deprecated.  Use %s instead.\n", m->name, m->is_alias_of->name);
			m->deprecation_warning_printed = true;
		}
		m = m->is_alias_of;
	}

	if ((m->state_mask & g_rpc_state) == g_rpc_state) {
		m->func(request, params);
	} else {
		if (g_rpc_state == SPDK_RPC_STARTUP) {
			spdk_jsonrpc_send_error_response_fmt(request,
							     SPDK_JSONRPC_ERROR_INVALID_STATE,
							     "Method may only be called after "
							     "framework is initialized "
							     "using framework_start_init RPC.");
		} else {
			spdk_jsonrpc_send_error_response_fmt(request,
							     SPDK_JSONRPC_ERROR_INVALID_STATE,
							     "Method may only be called before "
							     "framework is initialized. "
							     "Use --wait-for-rpc command line "
							     "parameter and then issue this RPC "
							     "before the framework_start_init RPC.");
		}
	}
}

That last branch is where an otherwise valid method becomes an invalid-state error. It is deliberately outside each individual handler, so most handlers can focus on parameter decoding and domain work instead of duplicating phase checks.

Startup RPCs And Runtime RPCs

SPDK has phases. Some choices are only safe before the subsystem starts. Some changes are safe after the system is running. This is encoded in state masks such as SPDK_RPC_STARTUP and SPDK_RPC_RUNTIME.

Startup RPC examples:

  • bdev_set_options in lib/bdev/bdev_rpc.c.
  • iobuf_set_options in module/event/subsystems/iobuf/iobuf_rpc.c.
  • nvmf_set_config in module/event/subsystems/nvmf/nvmf_rpc.c.
  • sock_impl_set_options in lib/sock/sock_rpc.c.

Runtime RPC examples:

  • bdev_get_bdevs in lib/bdev/bdev_rpc.c.
  • bdev_malloc_create in module/bdev/malloc/bdev_malloc_rpc.c.
  • thread_get_stats in lib/event/app_rpc.c.
  • nvmf_get_subsystems in lib/nvmf/nvmf_rpc.c.

The distinction matters because startup options often size pools, choose implementations, or set global behavior. Changing them after the relevant subsystem starts would be ambiguous or unsafe. If an RPC fails with a state error, the method name may be correct and the JSON may be valid. The failure can still be correct because the timing is wrong.

The state mask is a bitmask because a method can be valid in more than one phase. rpc_get_methods uses the same state relationship as the dispatcher when the caller asks for current methods only. This is why rpc_get_methods is a better diagnostic than reading documentation alone: it answers for this process, in this state, with this linked set of modules.

/* lib/rpc/rpc.c */
static void
rpc_rpc_get_methods(struct spdk_jsonrpc_request *request, const struct spdk_json_val *params)
{
	struct rpc_get_methods req = {};
	struct spdk_json_write_ctx *w;
	struct spdk_rpc_method *m;

	if (params != NULL) {
		if (spdk_json_decode_object(params, rpc_rpc_get_methods_decoders,
					    SPDK_COUNTOF(rpc_rpc_get_methods_decoders), &req)) {
			SPDK_ERRLOG("spdk_json_decode_object failed\n");
			spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
							 "Invalid parameters");
			return;
		}
	}

	w = spdk_jsonrpc_begin_result(request);
	spdk_json_write_array_begin(w);
	SLIST_FOREACH(m, &g_rpc_methods, slist) {
		if (!rpc_is_allowed(m->name)) {
			continue;
		}
		if (m->is_alias_of != NULL && !req.include_aliases) {
			continue;
		}
		if (req.current && ((m->state_mask & g_rpc_state) != g_rpc_state)) {
			continue;
		}

--wait-for-rpc

Normal startup parses config and initializes subsystems before the app enters steady state. --wait-for-rpc changes that. With --wait-for-rpc, the app starts the RPC server early and waits for an explicit framework_start_init RPC. This allows an external orchestrator to send startup RPCs before subsystem initialization.

without --wait-for-rpc:
  process starts
  load config if provided
  init subsystems
  runtime begins

with --wait-for-rpc:
  process starts
  RPC server opens
  orchestrator sends startup RPCs
  orchestrator sends framework_start_init
  init subsystems
  runtime begins

Source anchors:

  • include/spdk/event.h: spdk_app_start documents delayed initialization behavior.
  • lib/event/app.c: rpc_framework_start_init starts initialization from RPC.
  • lib/event/app.c: rpc_framework_wait_init reports initialization completion.

The main misconception is that --wait-for-rpc means the application is fully ready. It does not. It means the RPC server is ready while the application is intentionally not fully initialized. Only startup-safe calls should be sent before framework_start_init.

The public event header describes delayed subsystem initialization in terms of the framework-start RPC path. That matters because an embedding application calling spdk_app_start() sees the same behavior as the stock spdk_tgt binary, even though the user-facing JSON-RPC method name and the C handler name are not identical.

/* include/spdk/event.h */
 * If opts->delay_subsystem_init is set
 * (e.g. through --wait-for-rpc flag in spdk_app_parse_args())
 * this function will only start a limited RPC server accepting
 * only a few RPC commands - mostly related to pre-initialization.
 * With this option, the framework won't be started and start_fn
 * won't be called until the user sends an `rpc_framework_start_init`
 * RPC command, which marks the pre-initialization complete and
 * allows start_fn to be finally called.

The header text is describing the C-side handler path. The C handler is named rpc_framework_start_init, while the JSON-RPC method operators actually call is framework_start_init, as shown by the SPDK_RPC_REGISTER("framework_start_init", rpc_framework_start_init, SPDK_RPC_STARTUP) registration.

The implementation pauses the RPC server while subsystem initialization runs, then sends the boolean response from the completion path. framework_wait_init is different: it either returns immediately when SPDK is already runtime, or registers a poller that waits for the runtime state.

/* lib/event/app.c */
static void
rpc_framework_start_init(struct spdk_jsonrpc_request *request,
			 const struct spdk_json_val *params)
{
	if (params != NULL) {
		spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
						 "framework_start_init requires no parameters");
		return;
	}

	spdk_rpc_server_pause(g_spdk_app.rpc_addr);
	spdk_subsystem_init(rpc_framework_start_init_cpl, request);
}
SPDK_RPC_REGISTER("framework_start_init", rpc_framework_start_init, SPDK_RPC_STARTUP)

static void
rpc_framework_wait_init(struct spdk_jsonrpc_request *request,
			const struct spdk_json_val *params)
{
	struct subsystem_init_poller_ctx *ctx;

	if (spdk_rpc_get_state() == SPDK_RPC_RUNTIME) {
		spdk_jsonrpc_send_bool_response(request, true);
	} else {

Configuration Files Are RPC Sequences

SPDK JSON configuration is best understood as a sequence of RPC calls. A config file does not bypass the RPC layer. It is loaded by spdk_subsystem_load_config, which replays methods through the same handler model.

That means a config file has the same constraints as live RPC:

  • method names must exist.
  • params must decode.
  • startup-only methods must run during startup.
  • object creation order matters.
  • references must point to objects that already exist or can be discovered later.

A simplified config shape looks like this:

{
  "subsystems": [
    {
      "subsystem": "bdev",
      "config": [
        {
          "method": "bdev_malloc_create",
          "params": {
            "name": "Malloc0",
            "num_blocks": 1024,
            "block_size": 4096
          }
        }
      ]
    }
  ]
}

The exact generated output varies by subsystem. Always inspect real output from framework_get_config rather than assuming a hand-written format is canonical.

The loader decodes each config entry as a method name plus optional params object. Before sending it, it asks the RPC registry for the method state mask. This explains three common replay behaviors: unknown methods can be skipped when the subsystem is not linked, methods from the wrong phase are skipped until the matching pass, and dual startup/runtime methods are not run twice.

/* lib/init/json_config.c */
struct config_entry {
	char *method;
	struct spdk_json_val *params;
};

static struct spdk_json_object_decoder jsonrpc_cmd_decoders[] = {
	{"method", offsetof(struct config_entry, method), spdk_json_decode_string},
	{"params", offsetof(struct config_entry, params), cap_object, true}
};

static void
app_json_config_load_subsystem_config_entry(void *_ctx)
{
	struct load_json_config_ctx *ctx = _ctx;
	struct spdk_jsonrpc_client_request *rpc_request;
	struct spdk_json_write_ctx *w;
	struct config_entry cfg = {};
	struct spdk_json_val *params_end;
	size_t params_len = 0;
	uint32_t state_mask = 0, cur_state_mask, startup_runtime = SPDK_RPC_STARTUP | SPDK_RPC_RUNTIME;
	int rc;

	if (ctx->config_it == NULL) {
		SPDK_DEBUG_APP_CFG("Subsystem '%.*s': configuration done.\n", ctx->subsystem_name->len,
				   (char *)ctx->subsystem_name->start);

When the entry is allowed for the current pass, SPDK constructs a JSON-RPC client request and copies the raw params JSON into it. The comment in the source is important: the config loader does not pre-validate the params against the target method. The handler still owns validation.

/* lib/init/json_config.c */
	rc = spdk_rpc_get_method_state_mask(cfg.method, &state_mask);
	if (rc == -ENOENT) {
		if (!ctx->stop_on_error) {
			ctx->config_it = spdk_json_next(ctx->config_it);
			spdk_thread_send_msg(spdk_thread_get_app_thread(), app_json_config_load_subsystem_config_entry,
					     ctx);
		} else if (!spdk_subsystem_exists(ctx->subsystem_name_str)) {
			SPDK_NOTICELOG("Skipping method '%s' because its subsystem '%s' "
				       "is not linked into this application.\n",
				       cfg.method, ctx->subsystem_name_str);
			ctx->config_it = spdk_json_next(ctx->config_it);
			spdk_thread_send_msg(spdk_thread_get_app_thread(), app_json_config_load_subsystem_config_entry,
					     ctx);
		} else {
			SPDK_ERRLOG("Method '%s' was not found\n", cfg.method);
			app_json_config_load_done(ctx, rc);
		}
		goto out;
	}
	cur_state_mask = spdk_rpc_get_state();
	if ((state_mask & cur_state_mask) != cur_state_mask) {
		SPDK_DEBUG_APP_CFG("Method '%s' not allowed -> skipping\n", cfg.method);
		ctx->config_it = spdk_json_next(ctx->config_it);
/* lib/init/json_config.c */
	w = spdk_jsonrpc_begin_request(rpc_request, ctx->rpc_request_id, NULL);
	if (!w) {
		spdk_jsonrpc_client_free_request(rpc_request);
		app_json_config_load_done(ctx, -ENOMEM);
		goto out;
	}

	spdk_json_write_named_string(w, "method", cfg.method);

	if (cfg.params) {
		/* No need to parse "params". Just dump the whole content of "params"
		 * directly into the request and let the remote side verify it. */
		spdk_json_write_name(w, "params");
		spdk_json_write_val_raw(w, cfg.params->start, params_len);
	}

	spdk_jsonrpc_end_request(rpc_request, w);

framework_get_config

framework_get_config asks each subsystem to write configuration JSON for the state it owns. The implementation starts in lib/init/subsystem_rpc.c: rpc_framework_get_config. Subsystems provide writer callbacks through structures declared around include/spdk_internal/init.h. Concrete writers appear across the tree.

Useful source anchors:

  • lib/nvmf/nvmf.c: spdk_nvmf_tgt_write_config_json.
  • module/event/subsystems/nvmf/nvmf_tgt.c: nvmf_subsystem_write_config_json.
  • module/bdev/raid/bdev_raid.c: raid_bdev_write_config_json.
  • module/bdev/crypto/vbdev_crypto.c: vbdev_crypto_config_json.
  • module/bdev/nvme/bdev_mdns_client.c: bdev_nvme_mdns_discovery_config_json.

The output is not a database snapshot. It is a best-effort replay recipe. If an object cannot be represented as RPCs, it may not appear the way you expect. If runtime state is intentionally transient, it may not be included.

The framework-level RPC is thin. It decodes the subsystem name, finds the subsystem, and calls subsystem_config_json. The important ownership boundary is that the framework does not know how to serialize every subsystem. Each subsystem owns its writer.

/* lib/init/subsystem_rpc.c */
static void
rpc_framework_get_config(struct spdk_jsonrpc_request *request,
			 const struct spdk_json_val *params)
{
	struct rpc_framework_get_config_ctx req = {};
	struct spdk_json_write_ctx *w;
	struct spdk_subsystem *subsystem;

	if (spdk_json_decode_object(params, rpc_framework_get_config_decoders,
				    SPDK_COUNTOF(rpc_framework_get_config_decoders), &req)) {
		spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS, "Invalid arguments");
		return;
	}

	subsystem = subsystem_find(req.name);
	if (!subsystem) {
		spdk_jsonrpc_send_error_response_fmt(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
						     "Subsystem '%s' not found", req.name);
		free(req.name);
		return;
	}

	free(req.name);

	w = spdk_jsonrpc_begin_result(request);
	subsystem_config_json(w, subsystem);
	spdk_jsonrpc_end_result(request, w);
}

The subsystem structure makes that contract explicit:

/* include/spdk_internal/init.h */
struct spdk_subsystem {
	const char *name;
	void (*init)(void);
	void (*fini)(void);

	/**
	 * Write JSON configuration handler.
	 *
	 * \param w JSON write context
	 */
	void (*write_config_json)(struct spdk_json_write_ctx *w);
	TAILQ_ENTRY(spdk_subsystem) tailq;
};

The final call is intentionally small:

/* lib/init/subsystem.c */
void
subsystem_config_json(struct spdk_json_write_ctx *w, struct spdk_subsystem *subsystem)
{
	if (subsystem && subsystem->write_config_json) {
		subsystem->write_config_json(w);
	} else {
		spdk_json_write_null(w);
	}
}

This design lets bdev, NVMf, vhost, and other subsystem owners decide what a replayable configuration means for their objects. For example, a bdev module can emit create RPCs for devices it owns, while a stats module should not emit counters as configuration.

CLI wrappers sit above this RPC:

  • framework_get_config is the per-subsystem RPC.
  • save_subsystem_config is a CLI helper for one subsystem.
  • save_config is a CLI helper for live configuration across subsystems.

Adding A Small RPC

A new RPC needs two sides if you want it to be pleasant to operate: a C handler registered with SPDK_RPC_REGISTER, and usually a Python CLI wrapper under python/spdk/rpc and scripts/rpc.py's imported CLI modules. The server side is authoritative. The Python wrapper is convenience and validation for humans.

The malloc bdev create RPC is a compact model. Its decoder table says which JSON params exist, which ones are required, and which C fields receive decoded values. The true flag at the end of a decoder entry means optional.

/* module/bdev/malloc/bdev_malloc_rpc.c */
static const struct spdk_json_object_decoder rpc_bdev_malloc_create_decoders[] = {
	{"name", offsetof(struct malloc_bdev_opts, name), spdk_json_decode_string, true},
	{"uuid", offsetof(struct malloc_bdev_opts, uuid), spdk_json_decode_uuid, true},
	{"num_blocks", offsetof(struct malloc_bdev_opts, num_blocks), spdk_json_decode_uint64},
	{"block_size", offsetof(struct malloc_bdev_opts, block_size), spdk_json_decode_uint32},
	{"physical_block_size", offsetof(struct malloc_bdev_opts, physical_block_size), spdk_json_decode_uint32, true},
	{"optimal_io_boundary", offsetof(struct malloc_bdev_opts, optimal_io_boundary), spdk_json_decode_uint32, true},
	{"md_size", offsetof(struct malloc_bdev_opts, md_size), spdk_json_decode_uint32, true},
	{"md_interleave", offsetof(struct malloc_bdev_opts, md_interleave), spdk_json_decode_bool, true},
	{"dif_type", offsetof(struct malloc_bdev_opts, dif_type), spdk_json_decode_int32, true},
	{"dif_is_head_of_md", offsetof(struct malloc_bdev_opts, dif_is_head_of_md), spdk_json_decode_bool, true},
	{"dif_pi_format", offsetof(struct malloc_bdev_opts, dif_pi_format), spdk_json_decode_uint32, true},
	{"numa_id", offsetof(struct malloc_bdev_opts, numa_id), spdk_json_decode_int32, true},
};

The handler is ordinary C glue: initialize defaults, decode params, call the module API, translate the module result to a JSON-RPC response, and free decoded owned strings. Notice that decode failure is treated as a request failure before create_malloc_disk() is called. That keeps the method from applying a partial mutation.

/* module/bdev/malloc/bdev_malloc_rpc.c */
static void
rpc_bdev_malloc_create(struct spdk_jsonrpc_request *request,
		       const struct spdk_json_val *params)
{
	struct malloc_bdev_opts req = {NULL};
	struct spdk_json_write_ctx *w;
	struct spdk_bdev *bdev;
	int rc = 0;

	req.numa_id = SPDK_ENV_NUMA_ID_ANY;

	if (spdk_json_decode_object(params, rpc_bdev_malloc_create_decoders,
				    SPDK_COUNTOF(rpc_bdev_malloc_create_decoders),
				    &req)) {
		SPDK_DEBUGLOG(bdev_malloc, "spdk_json_decode_object failed\n");
		spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR,
						 "spdk_json_decode_object failed");
		goto cleanup;
	}

	rc = create_malloc_disk(&bdev, &req);
	if (rc) {
		spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc));
		goto cleanup;
	}
/* module/bdev/malloc/bdev_malloc_rpc.c */
	free_rpc_construct_malloc(&req);

	w = spdk_jsonrpc_begin_result(request);
	spdk_json_write_string(w, spdk_bdev_get_name(bdev));
	spdk_jsonrpc_end_result(request, w);
	return;

cleanup:
	free_rpc_construct_malloc(&req);
}
SPDK_RPC_REGISTER("bdev_malloc_create", rpc_bdev_malloc_create, SPDK_RPC_RUNTIME)

For a new RPC, keep the first version boring: put it in the module's existing *_rpc.c file, use a typed decoder table, reject unknown or invalid params early, call one narrow module API, return one clear result, and register it in the narrowest valid phase. If the operation is asynchronous, follow the delete-style pattern: pass the request pointer as callback context and send the response exactly once from the completion callback. If the operation changes global subsystem sizing, make it SPDK_RPC_STARTUP. If it creates or queries runtime objects, make it SPDK_RPC_RUNTIME. If it truly works in both phases, expect config replay to avoid running it twice.

Config Replay Timeline

1. binary starts
2. app framework parses app options
3. RPC methods have been registered by constructors
4. JSON config is parsed if provided
5. STARTUP replay pass runs startup-only and startup/runtime methods
6. subsystem initialization begins
7. subsystem init callbacks create base framework state
8. RPC state becomes RUNTIME
9. RUNTIME replay pass runs runtime-only methods
10. startup/runtime methods already run in STARTUP are skipped in RUNTIME
11. app start function and runtime service begin
12. external clients mutate or inspect runtime state

The replay model explains many startup errors. If a config tries to create a RAID bdev before its base bdevs exist, replay can fail. If a config uses a startup-only option after runtime begins, replay is too late. If an RPC method was not compiled in because a feature was disabled, replay cannot find it. lib/event/app.c calls spdk_subsystem_load_config once while the RPC state is SPDK_RPC_STARTUP and again after setting SPDK_RPC_RUNTIME. lib/init/json_config.c skips methods not allowed in the current pass and avoids running startup/runtime methods twice. The --json-ignore-init-errors option flips json_config_ignore_errors, causing the loader to continue past invalid config entries; treat a successful process start under that option as potentially partial replay.

Method Discovery

Use rpc_get_methods to ask the server what it currently supports. Source anchor: lib/rpc/rpc.c: rpc_rpc_get_methods. The method can optionally filter by current state. That is useful when debugging why a call is refused.

Recommended beginner workflow:

1. start app with the features you expect
2. call rpc_get_methods
3. check whether the method exists
4. check whether it is allowed in the current state
5. then debug params

This avoids a common mistake: spending time on JSON syntax when the method was never registered.

RPC Error Classes

An SPDK RPC failure usually belongs to one of these classes:

  • transport error: could not connect to the socket.
  • parse error: JSON is malformed or incomplete.
  • method error: method name is unknown.
  • state error: method exists but is not allowed now.
  • params error: JSON is valid but does not match the decoder.
  • semantic error: params decode, but requested state is invalid.
  • asynchronous error: operation started but later completion reports failure.

Treat the error text as a clue, not a complete diagnosis. Many handlers include targeted messages through spdk_jsonrpc_send_error_response or formatted variants. Search the method handler for the string to find the exact branch.

Edge Cases

Socket Exists But The Server Is Gone

A stale Unix socket path can remain after an abnormal process exit. The client may report connection failure even though the path exists. Check the process, not just the file.

The Method Exists In Documentation But Not In Your Binary

SPDK features can depend on configure options and linked libraries. If a module is not built in, its registration macro never runs. rpc_get_methods is more reliable than memory.

Startup Method Sent Too Late

The JSON shape can be perfect and still fail. Look for SPDK_RPC_STARTUP registrations. If the app has already completed init, the call belongs in a config file or before framework_start_init.

Runtime Method Sent Too Early

With --wait-for-rpc, runtime state may not exist. Do not create runtime objects before subsystem init unless the method explicitly supports startup.

Save Config Does Not Preserve Everything

framework_get_config serializes configuration, not every runtime counter or transient queue. Stats, active I/O, poller run counts, and temporary reconnect state are not durable configuration.

Replay Order Is Real

Config is not declarative magic. It is closer to a script. If object B depends on object A, A must appear first or be discoverable by a later examine step.

Misconceptions To Kill

  • "JSON-RPC is slow, so it must affect every I/O." The control plane is separate from the hot I/O path.
  • "If the docs list a method, my binary has it." Build options and linked modules decide what is registered.
  • "Startup RPC means run at process start only." It means valid before subsystem initialization completes.
  • "Runtime RPC means safe at any instant." The handler still must validate object state and concurrency.
  • "A config file is a dump of memory." It is a replayable RPC recipe.
  • "Unknown method means typo." It can also mean the module was not compiled or not linked.
  • "A successful create RPC means the whole stack is healthy." It means the handler accepted and completed that operation.
  • "All RPCs are synchronous inside." Some handlers initiate work and respond from a completion callback.

Lab: Trace One RPC Handler

Pick bdev_malloc_create. Find its registration in module/bdev/malloc/bdev_malloc_rpc.c. Find the handler function. Find the decoder table. Write down each parameter accepted by the decoder. Find the call that creates the malloc bdev. Find where the handler sends success. Find where it sends an error. Now run the same reading exercise for bdev_get_bdevs in lib/bdev/bdev_rpc.c. Compare a mutating RPC with a query RPC.

Lab: Classify RPC Phase

Use rg to list registrations:

rg -n 'SPDK_RPC_REGISTER' lib module

For ten methods, classify them as startup, runtime, or both. For each startup method, write one sentence explaining why late mutation could be unsafe. For each runtime method, write one sentence explaining what state must already exist.

Lab: Build A Minimal Config Replay

Start with a malloc bdev config. Add one bdev_malloc_create. Load it into spdk_tgt. Call bdev_get_bdevs. Call framework_get_config. Compare your input config to SPDK's output. Note differences in ordering, omitted defaults, and generated fields.

Source Reading Path

Read these files in this order when debugging a real RPC problem:

  1. scripts/rpc.py and python/spdk/rpc/client.py to see the exact socket path, timeout, request id, params object, and error text the client produced.
  2. doc/jsonrpc.md.jinja2 or https://spdk.io/doc/jsonrpc.html to confirm the documented method name, params, errors, and whether the docs mention startup-only behavior.
  3. include/spdk/rpc.h to confirm the registration macro and state constants.
  4. The module's *_rpc.c file to find SPDK_RPC_REGISTER, the decoder table, the handler, and the response path.
  5. lib/rpc/rpc.c to confirm method lookup, alias handling, state gating, and rpc_get_methods filtering.
  6. lib/init/json_config.c when the same call works live but fails from a config file.
  7. lib/init/subsystem_rpc.c plus the subsystem's write_config_json callback when saved config is missing an object.

This order starts at the user's command and moves inward. It usually finds spelling, socket, build, and phase problems before you need to reason about the deeper subsystem.

Debug Checklist

  • Can the client connect to the socket?
  • Does rpc_get_methods list the method?
  • Is the method allowed in the current state?
  • Does the JSON parse?
  • Does the params object match the decoder?
  • Does the handler require a named object that already exists?
  • Is the feature compiled into this binary?
  • Is the failure synchronous or completed from a callback?
  • Does framework_get_config produce a replay that includes the object?
  • Does replay succeed from a clean process?

Self-Check

  1. What local file contains SPDK_RPC_REGISTER?
  2. What function dispatches parsed RPC methods to registered handlers?
  3. Why can a startup RPC fail after the app is running?
  4. What does --wait-for-rpc delay?
  5. Why is framework_get_config not the same as a memory dump?
  6. How would you prove that a method is missing because of build options?
  7. Why does replay order matter?
  8. What is the difference between a params error and a semantic error?

References

  • doc/jsonrpc.md.jinja2 for the generated official RPC reference.
  • doc/applications.md for SPDK application options and JSON configuration behavior.
  • SPDK official JSON-RPC documentation: https://spdk.io/doc/jsonrpc.html.
  • SPDK official application overview and deferred initialization documentation: https://spdk.io/doc/app_overview.html.
  • doc/getting_started.md for first-run setup context.
  • doc/bdev.md for block-device RPC examples.
  • doc/nvmf.md for NVMe-oF target RPC examples.
  • test/unit/lib/rpc/rpc.c/rpc_ut.c for RPC registration and method-list unit tests.
  • test/unit/lib/jsonrpc/jsonrpc_server.c/jsonrpc_server_ut.c for JSON-RPC parsing tests.