Chapter 29: diskengine Storage Node Mode | SPDK From First Principles

Chapter Goal

This chapter explains diskengine storage-node mode as a set of reconciliation loops around SPDK. By the end, you should be able to look at a storage-node failure and ask the right question:

Is the desired object present in the database, present in SPDK, present in the NVMe-oF target, and connected to the right lower-level object?

The chapter is written for a beginner who can read C and Go but has not yet built SPDK integrations. It focuses on how diskengine uses SPDK, why the code is shaped as polling loops, and where you would extend the C/Go boundary safely.

Storage-node mode covers this object chain:

physical NVMe SSD
  -> SPDK NVMe bdev
  -> SPDK lvstore
  -> SPDK lvol bdev
  -> SPDK NVMe-oF subsystem namespace
  -> remote initiator connects and does I/O

The Go process does not sit on the data path. It asks SPDK to build the data path, checks that the path still exists, and repairs missing pieces when it can.

Beginner Mental Model

Storage-node mode owns local SSD capacity. Compute nodes or bare-metal nodes do not send reads and writes to diskengine's Go handlers. They connect to an NVMe-oF export served by SPDK. Once the connection is established, data I/O is handled by SPDK transports, SPDK bdevs, and the NVMe device path.

That separation is the key idea:

The database records desired state and accounting state.
SPDK holds the live storage object graph.
Linux sysfs exposes devices only while they are kernel-bound.
The diskengine storage-node loops repeatedly compare those worlds and make one small change at a time.

The usual lifecycle is:

Find a kernel-visible NVMe device under sysfs.
Bind its PCI function to vfio-pci so SPDK can own it from userspace.
Attach it to SPDK with bdev_nvme_attach_controller.
Ask SPDK to examine the bdev for existing metadata.
Create or import an lvstore.
Create lvol bdevs inside the lvstore.
Create or reuse an NVMe-oF subsystem.
Add listeners and namespaces so initiators can connect.
Recheck after restarts and partial failures.

This is eventually consistent. A row may say CREATING, UP, RESIZING, or DELETING before SPDK has fully converged. The loops make progress by retrying idempotent operations and by treating SPDK as the source of truth for live objects.

Important Names

A few names appear throughout the chapter:

physical_disks: database rows for local SSDs, with serial, PCI address, node id, RDMA placement, health, and state.
NvmeDisk<disk_id>: deterministic SPDK NVMe controller name used by disk initialization.
lvs<disk_id>: deterministic lvstore name used when formatting a new disk.
lvstore UUID: SPDK's durable identifier for the logical volume store on the base bdev.
lvol UUID: SPDK's identifier for a logical volume bdev.
NQN: NVMe Qualified Name. In storage-node mode it names the NVMe-oF subsystem to which a remote initiator connects.
listener: an address on which an NVMe-oF subsystem accepts connections, such as RDMA IP plus port 4420.
namespace: the object that attaches one bdev to one NVMe-oF subsystem.

Do not blur subsystem and namespace. The subsystem is the target-side controller identity. The namespace is the exported storage object inside that subsystem.

Entry Point And Loop Ownership

Storage-node mode starts in diskengine's mode-specific entry path and lands in:

/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/storagenode.go

The important part is not only that it starts many goroutines. The important part is the startup ordering: first prove the SPDK RPC socket exists, then run a verification pass, then start independent loops.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/storagenode.go
func Start() {
	ctx := context.Background()
	cancelCtx, cancel = context.WithCancel(ctx)

	if err := ensureSockExists(); err != nil {
		logger.Error.Fatalf("SPDK RPC sock check failed: %v", err)
	}

	// Run a one-time verifyState pass before starting loops.
	if client, err := spdkclient.CreateClientWithJsonCodec("unix", config.Value.SPDK_RPC_SOCK); err != nil {
		logger.Error.Printf("verifyState (startup): failed to create spdk client: %v", err)
	} else {
		if err := verifyState(ctx, client); err != nil {
			logger.Warn.Printf("verifyState (startup): %v", err)
		}
		client.Close()
	}

context.WithCancel gives every loop a shared shutdown signal. ensureSockExists fails fast if SPDK is not reachable at the configured Unix socket. The startup verifyState pass is deliberately before the loops: it discovers whether SPDK already has bdevs, lvstores, controllers, or namespace exports after a process restart.

The loop startup then follows one repeated pattern:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/storagenode.go
	wg.Add(1)
	go func() {
		defer wg.Done()
		diskInitLoop(ctx)
	}()

	wg.Add(1)
	go func() {
		defer wg.Done()
		diskDiscoverLoop(ctx)
	}()

	wg.Add(1)
	go func() {
		defer wg.Done()
		nvmeofExportLoop(ctx)
	}()

Each loop is independent and polling-based. That is simple and robust, but it means ordering is not guaranteed by goroutine start order. For example, provisioning may observe a CREATING lvol before export reconciliation has refreshed subsystem state. The code must therefore be retry-safe.

Stop cancels and waits:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/storagenode.go
func Stop() {
	if cancel != nil {
		logger.Info.Println("Signaling storage node to stop processing.")
		cancel()
		// Wait for all background goroutines to exit cleanly
		wg.Wait()
	}
}

The data path remains SPDK's responsibility. Stopping diskengine stops reconciliation and scraping, not necessarily every SPDK object that was already created.

JSON-RPC Is The C/Go Boundary

diskengine does not call SPDK C functions directly. It sends JSON-RPC methods to the SPDK process. The local wrapper layer is intentionally thin:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go
func (c *Client) BdevNvmeAttachController(params BdevNvmeAttachControllerParams) ([]string, error) {
	resp, err := c.Call("bdev_nvme_attach_controller", params)
	if err != nil {
		return nil, fmt.Errorf("BdevNvmeAttachController call failed: %w", err)
	}
	var names []string
	data, err := json.Marshal(resp.Result)
	if err != nil {
		return nil, fmt.Errorf("BdevNvmeAttachController: marshal failed: %w", err)
	}
	if err := json.Unmarshal(data, &names); err != nil {
		return nil, fmt.Errorf("BdevNvmeAttachController: unmarshal failed: %w", err)
	}
	return names, nil
}

c.Call is the real boundary. The string "bdev_nvme_attach_controller" must match SPDK's registered RPC method. The wrapper then converts the untyped JSON result back into a Go type. For NVMe attach, SPDK returns an array of bdev names, so the wrapper marshals resp.Result back to JSON and unmarshals into []string.

For simpler RPCs the wrapper only checks the call result:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go
func (c *Client) BdevLvolCreateLvstore(params BdevLvolCreateLvstoreParams) (string, error) {
	resp, err := c.Call("bdev_lvol_create_lvstore", params)
	if err != nil {
		return "", fmt.Errorf("BdevLvolCreateLvstore call failed: %w", err)
	}
	uuid, ok := resp.Result.(string)
	if !ok {
		return "", fmt.Errorf("BdevLvolCreateLvstore: unexpected response type: %T", resp.Result)
	}
	return uuid, nil
}

func (c *Client) NvmfSubsystemAddNs(params NvmfSubsystemAddNsParams) error {
	_, err := c.Call("nvmf_subsystem_add_ns", params)
	if err != nil {
		return fmt.Errorf("NvmfSubsystemAddNs call failed: %w", err)
	}
	return nil
}

This is the place to start if you extend diskengine's C/Go integration. Add a typed params struct, add a wrapper that calls the exact SPDK RPC name, and then decide whether the caller needs a typed return value or only success/failure.

Disk Discovery

Disk discovery only sees devices while the Linux kernel still owns them. The loop scans sysfs and inserts or updates physical_disks rows.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_discover.go
// enumerateSysDisks crudely scans /sys/block/nvme*/device to build serial & pci addr list.
// NOTE: NVMe devices bound to vfio-pci are no longer exposed under /sys/block, so
// this only enumerates kernel-bound NVMe controllers (pre-vfio bind).
func enumerateSysDisks() ([]sysDisk, error) {
	paths, err := filepath.Glob("/sys/block/nvme*n*/device")
	if err != nil {
		return nil, err
	}
	var result []sysDisk
	for _, p := range paths {
		serialBytes, _ := os.ReadFile(filepath.Join(p, "serial"))
		serial := strings.TrimSpace(string(serialBytes))

The comment is an operational warning. After a disk is bound to vfio-pci, it can disappear from /sys/block. That is not automatically a disk failure. It means SPDK owns the PCI device and exposes it through SPDK's object model instead of the kernel block layer.

The discovery step then compares sysfs to database ownership:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_discover.go
if !found {
	// New disk - create row in NEW state
	logger.Info.Printf("discoverStep: new disk serial %s detected", sd.Serial)
	hasEnvPlacement := envRdmaIPs != ""
	rdmaIP := "0.0.0.0"
	rdmaPort := 4420
	if config.Value.RDMA_PORT != 0 {
		rdmaPort = config.Value.RDMA_PORT
	}
	if hasEnvPlacement {
		rdmaIP = envRdmaIPs
	}
	nd := types.PhysicalDisk{
		BareMetalID: nodeIDInt,
		PCIAddr: sd.PCIAddr,
		SizeBytes: func() int64 {
			if sd.SizeBytes > 0 {
				return sd.SizeBytes
			}
			return 1 // fallback
		}(),
		RackID:   int64(config.Value.RACK_ID),
		RDMAIP:   rdmaIP,
		RDMAPort: rdmaPort,
		Serial:   sd.Serial,
		Health:   types.DISK_HEALTH_OK,
		State:    types.DISK_STATE_NEW,
	}

The row is created in NEW state. Initialization is separate. That split lets discovery stay simple: it records what Linux can see and leaves SPDK ownership work to diskInitLoop.

Disk Initialization

Initialization turns a NEW disk row into an SPDK-managed disk. It binds the PCI device to VFIO, attaches the NVMe controller, examines metadata, and either creates or imports an lvstore.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_init.go
func initialiseDisk(ctx context.Context, client *spdkclient.Client, disk types.PhysicalDisk) error {
	logger.Info.Printf("initialiseDisk: start disk %d serial %s pci %s", disk.ID, disk.Serial, disk.PCIAddr)
	// 1. Bind to vfio-pci if not already
	if err := bindToVfio(disk.PCIAddr); err != nil {
		return fmt.Errorf("bind vfio: %w", err)
	}
	logger.Info.Printf("initialiseDisk: vfio bound for disk %d", disk.ID)

	// 2. Attach (or re-attach) NVMe controller
	attachParams := spdkclient.BdevNvmeAttachControllerParams{
		// Use deterministic controller name so re-attaches succeed idempotently.
		Name:   fmt.Sprintf("NvmeDisk%d", disk.ID),
		Trtype: "pcie",
		Traddr: disk.PCIAddr,
	}
	names, err := client.BdevNvmeAttachController(attachParams)

bindToVfio is the transition from Linux block-device ownership to SPDK ownership. Name: fmt.Sprintf("NvmeDisk%d", disk.ID) is important because a deterministic controller name allows restart recovery. If diskengine used random names, it could create duplicates or fail to match DB rows to SPDK controllers.

The next block handles idempotency:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_init.go
if err != nil {
	if strings.Contains(strings.ToLower(err.Error()), "already exists") {
		bdevName, findErr := findNvmeBdevName(client, attachParams.Name)
		if findErr != nil {
			return fmt.Errorf("nvme attach already exists, but failed to find bdev: %w", findErr)
		}
		logger.Info.Printf("initialiseDisk: nvme controller %s already exists; using bdev %s", attachParams.Name, bdevName)
		names = []string{bdevName}
	} else {
		return fmt.Errorf("nvme attach: %w", err)
	}
}
if len(names) == 0 {
	return fmt.Errorf("nvme attach returned no bdev names")
}
bdevName := names[0]

An "already exists" error is not automatically fatal. It can mean SPDK already has the controller from a previous pass or restart. The code turns that error into recovery by finding the existing bdev.

After attach, diskengine forces SPDK metadata discovery:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_init.go
// Always call bdev_examine to ensure lvstore is loaded from on-disk metadata.
// This is critical for restart recovery - even if controller "already exists",
// the lvstore may not have been examined yet.
spdkLvs, err := examineAndFindLvstore(client, false, bdevName, 30*time.Second)
if err != nil {
	return err
}
expectedLvsName := fmt.Sprintf("lvs%d", disk.ID)

dbLvsUUID, dbExists, err := repository.GetLvstoreUUIDForDisk(ctx, disk.Serial)
if err != nil {
	return err
}

bdev_examine tells SPDK to inspect a bdev for metadata owned by modules such as lvol. examineAndFindLvstore then polls bdev_lvol_get_lvstores for up to 30 seconds. That timeout exists because SPDK's examine path is asynchronous inside the SPDK process.

The lvstore reconciliation has three useful cases:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_init.go
if spdkLvs == nil {
	if !dbExists {
		logger.Info.Printf("initialiseDisk: no lvstore in SPDK/DB for disk %s; creating", disk.Serial)
		clusterSz := uint32(4 * 1024 * 1024) // 4 MiB
		lvsUUID, err := client.BdevLvolCreateLvstore(spdkclient.BdevLvolCreateLvstoreParams{
			BdevName:  names[0],
			LvsName:   expectedLvsName,
			ClusterSz: &clusterSz,
		})
		if err != nil {
			return fmt.Errorf("create lvstore: %w", err)
		}

No lvstore in SPDK and no lvstore in DB means this is a blank disk from diskengine's point of view, so it formats a new lvstore with a 4 MiB cluster size.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_init.go
	if !dbExists {
		logger.Info.Printf("initialiseDisk: lvstore exists in SPDK but missing in DB for disk %s; importing", disk.Serial)
		totalBytes := int64(spdkLvs.ClusterSize * spdkLvs.TotalDataClusters)
		freeBytes := int64(spdkLvs.ClusterSize * spdkLvs.FreeClusters)
		if err := repository.InsertLvstore(ctx, disk.Serial, spdkLvs.UUID, int64(spdkLvs.ClusterSize), totalBytes, freeBytes); err != nil {
			err := fmt.Errorf("initialiseDisk: failed to insert existing lvstore from SPDK for disk %s: %w", disk.Serial, err)
			logger.Error.Println(err)
			return err
		}
		logger.Info.Printf("initialiseDisk: imported existing lvstore %s for disk %s into DB", spdkLvs.UUID, disk.Serial)
		return nil
	}

	if spdkLvs.UUID != dbLvsUUID {
		err := fmt.Errorf("initialiseDisk: lvstore UUID mismatch for disk %s (SPDK %s vs DB %s)", disk.Serial, spdkLvs.UUID, dbLvsUUID)
		logger.Error.Println(err)
		return err
	}

If SPDK finds an lvstore that the DB does not know about, diskengine imports it. If both exist but UUIDs differ, diskengine stops instead of papering over the mismatch. A UUID mismatch means the DB thinks this serial owns one durable allocation space while SPDK found another.

What SPDK Does For NVMe Attach

The diskengine wrapper calls bdev_nvme_attach_controller. SPDK registers that RPC in C:

// module/bdev/nvme/bdev_nvme_rpc.c
static void
rpc_bdev_nvme_attach_controller_done(void *cb_ctx, size_t bdev_count, int rc)
{
	struct rpc_bdev_nvme_attach_controller_ctx *ctx = cb_ctx;
	struct spdk_jsonrpc_request *request = ctx->request;

	if (rc < 0) {
		spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc));
		free_rpc_bdev_nvme_attach_controller_ctx(ctx);
		return;
	}

	ctx->bdev_count = bdev_count;
	spdk_bdev_wait_for_examine(rpc_bdev_nvme_attach_controller_examined, ctx);
}

The attach completion callback does not immediately return names. It first calls spdk_bdev_wait_for_examine. This matches diskengine's conservative behavior: after attach, it asks for examine and polls for lvstores, because SPDK object visibility is not just a synchronous function return.

The attach handler also contains duplicate-path logic:

// module/bdev/nvme/bdev_nvme_rpc.c
ctrlr = nvme_ctrlr_get_by_name(ctx->req.name);

if (ctrlr) {
	if (ctx->req.multipath == BDEV_NVME_MP_MODE_DISABLE) {
		spdk_jsonrpc_send_error_response_fmt(request, -EALREADY,
						     "A controller named %s already exists and multipath is disabled",
						     ctx->req.name);
		goto cleanup;
	}

	assert(ctx->req.multipath == BDEV_NVME_MP_MODE_FAILOVER ||
	       ctx->req.multipath == BDEV_NVME_MP_MODE_MULTIPATH);

For storage-node mode, the attach uses local PCIe and deterministic names. If SPDK says the controller name already exists, diskengine treats that as a possible restart/idempotency case and searches for the existing bdev. The SPDK code also shows that multipath defaults and duplicate path rules are SPDK policy, not diskengine policy.

Lvstore And Lvol Concepts

An lvstore is SPDK's blobstore-backed allocation pool on top of a base bdev. Think of it as the durable allocator and metadata layer for a disk. An lvol is a logical volume inside that allocator. SPDK exposes each lvol as another bdev.

diskengine creates lvstores on physical SSD bdevs and creates lvols for volumes or snapshots. That is why the chain is:

NvmeDisk7n1 -> lvstore uuid -> lvol uuid -> namespace in subsystem NQN

SPDK's lvstore create RPC decodes the request, chooses a clear method, and later calls the lvol-store constructor:

// module/bdev/lvol/vbdev_lvol_rpc.c
rpc_bdev_lvol_create_lvstore(struct spdk_jsonrpc_request *request,
			     const struct spdk_json_val *params)
{
	struct rpc_bdev_lvol_create_lvstore req = {};
	int rc = 0;
	enum lvs_clear_method clear_method;

	if (spdk_json_decode_object(params, rpc_bdev_lvol_create_lvstore_decoders,
				    SPDK_COUNTOF(rpc_bdev_lvol_create_lvstore_decoders),
				    &req)) {
		spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR,
						 "spdk_json_decode_object failed");
		goto cleanup;
	}

The same function then passes the base bdev name, lvstore name, cluster size, clear method, and callback into SPDK's lvol-store constructor:

// module/bdev/lvol/vbdev_lvol_rpc.c
	rc = vbdev_lvs_create_ext(req.bdev_name, req.lvs_name, req.cluster_sz, clear_method,
				  req.num_md_pages_per_cluster_ratio, req.md_page_size,
				  rpc_lvol_store_construct_cb, request);
	if (rc < 0) {
		spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc));
		goto cleanup;
	}

The important parameters from diskengine are bdev_name, lvs_name, and cluster_sz. SPDK owns the actual metadata format and returns the lvstore UUID through the callback.

Lvol creation is similar:

// module/bdev/lvol/vbdev_lvol_rpc.c
rpc_bdev_lvol_create_cb(void *cb_arg, struct spdk_lvol *lvol, int lvolerrno)
{
	struct spdk_json_write_ctx *w;
	struct spdk_jsonrpc_request *request = cb_arg;

	if (lvolerrno != 0) {
		goto invalid;
	}

	w = spdk_jsonrpc_begin_result(request);
	spdk_json_write_string(w, lvol->unique_id);
	spdk_jsonrpc_end_result(request, w);
	return;

invalid:
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
					 spdk_strerror(-lvolerrno));
}

The result diskengine stores is lvol->unique_id. That ID becomes the bdev name diskengine later attaches as an NVMe-oF namespace.

Provisioning An Lvol

Provisioning handles database lvol rows in CREATING state. The source tells you the required order:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go
func provisionLvol(ctx context.Context, spdkClient *spdkclient.Client, lvol repository.LvolToProcess) error {
	logger.Info.Printf("Provisioning lvol %s", lvol.LvolUUID)

	if lvol.NQN == "" || lvol.RDMAIP == "" || lvol.RDMAPort == 0 {
		return fmt.Errorf("missing NVMe-oF placement info (nqn/ip/port) for disk serial %s", lvol.DiskSerial)
	}

	if err := ensureNvmeofReady(spdkClient, lvol.NQN, lvol.RDMAIP, lvol.RDMAPort, lvol.DiskSerial); err != nil {
		return fmt.Errorf("ensure nvmeof ready failed: %w", err)
	}

	thinProvisioning := false
	clearMethod := "write_zeroes"
	lvolName := fmt.Sprintf("%d", lvol.LvolID)
	req := spdkclient.BdevLvolCreateParams{
		LvstoreUUID:   &lvol.LvstoreUUID,
		LvolName:      lvolName,
		SizeInMib:     uint64(lvol.CapacityBytes / (1024 * 1024)),
		ClearMethod:   &clearMethod,
		ThinProvision: &thinProvisioning,
	}

The placement check prevents creating a local lvol that cannot be exported. ensureNvmeofReady is called before lvol creation so that namespace attach has a subsystem and listener to target. thinProvisioning := false means diskengine asks SPDK for thick lvol allocation here. clearMethod := "write_zeroes" asks SPDK to clear allocated blocks with write zeroes.

Creation and duplicate recovery are next:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go
createdLvolUUID, err := spdkClient.BdevLvolCreate(req)
if err != nil {
	if isAlreadyExistsErr(err) {
		logger.Warn.Printf("provisionLvol: lvol %s already exists in SPDK; attempting recovery", lvol.LvolUUID)
		existingUUID, findErr := findExistingLvolUUID(spdkClient, lvol.LvstoreUUID, lvolName)
		if findErr != nil {
			return fmt.Errorf("lvol already exists but could not find UUID: %w", findErr)
		}
		logger.Info.Printf("provisionLvol: recovered existing lvol UUID %s for lvol %d", existingUUID, lvol.LvolID)
		createdLvolUUID = existingUUID
	} else {
		return err
	}
}

This block handles the crash window where SPDK created the lvol but diskengine did not finalize the DB transaction. The next tick may attempt creation again. Instead of failing permanently, it searches SPDK for the existing lvol and continues.

Finally, the lvol bdev becomes a namespace:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go
if err := spdkClient.NvmfSubsystemAddNs(spdkclient.NvmfSubsystemAddNsParams{
	NQN: lvol.NQN,
	Namespace: spdkclient.NvmfNamespaceParams{
		BdevName: createdLvolUUID,
	},
}); err != nil {
	if isAlreadyExistsErr(err) {
		attached, checkErr := isNamespaceAttached(spdkClient, lvol.NQN, createdLvolUUID)
		if checkErr != nil {
			return fmt.Errorf("namespace attach returned already-exists but verification failed: %w", checkErr)
		}
		if !attached {
			return fmt.Errorf("namespace already exists but bdev %s not attached to subsystem %s", createdLvolUUID, lvol.NQN)
		}
	} else {
		logger.Error.Printf("Lvol %s was created in SPDK but not attached - this will cause state drift", createdLvolUUID)
		return fmt.Errorf("failed to attach lvol to subsystem: %w", err)
	}
}

This is the most important partial failure in storage-node mode. The lvol can exist even when namespace attach fails. In that case SPDK has allocated storage, but the remote initiator cannot connect to it. The code logs state drift and returns an error so a later loop or operator can reconcile.

SPDK's nvmf_subsystem_add_ns returns the assigned NSID on success; internally spdk_nvmf_subsystem_add_ns_ext() returns 0 when it cannot assign a namespace. The SPDK JSON-RPC layer converts that zero into an RPC error. Current diskengine's wrapper only needs success or failure, so it ignores the numeric NSID after a successful call and relies on the one-namespace-per-subsystem convention used by the initiator side.

Only after namespace attach does diskengine finalize the DB:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go
if err := repository.FinalizeProvisioningForLvol(ctx, lvol, createdLvolUUID); err != nil {
	logger.Error.Printf("Failed to finalize provisioning for lvol %s: %v", createdLvolUUID, err)
	return fmt.Errorf("failed to finalize provisioning: %w", err)
}

The ordering matters. If the DB were finalized before namespace attach, clients could be told a volume is available while SPDK cannot serve it.

NVMe-oF Target Readiness

ensureNvmeofReady is a compact version of export reconciliation. It verifies the transport, subsystem, and listener needed for one placement.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/utils.go
func ensureNvmeofReady(client *spdkclient.Client, nqn string, ip string, port int, diskSerial string) error {
	ips := splitCSV(ip)
	if len(ips) == 0 {
		ips = []string{ip}
	}
	transports, err := client.NvmfGetTransports(spdkclient.NvmfGetTransportsParams{})
	if err != nil {
		return fmt.Errorf("nvmf_get_transports: %w", err)
	}
	hasRDMA := false
	for _, t := range transports {
		if strings.EqualFold(t.Trtype, "RDMA") {
			hasRDMA = true
			break
		}
	}

It starts by reading actual SPDK state. splitCSV allows multiple RDMA IPs. hasRDMA is a process-wide target property: the RDMA transport needs to exist once before RDMA listeners can work.

If RDMA is missing, diskengine creates it with fixed options:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/utils.go
if !hasRDMA {
	ioUnit := 16384
	numShared := 1024
	maxQDepth := 256
	zcopy := true
	maxIoQpairsPerCtrlr := 128
	noWrBatching := false
	maxSrqDepth := 8192
	acceptorBacklog := 256
	if err := client.NvmfCreateTransport(spdkclient.NvmfCreateTransportParams{
		Trtype:              "RDMA",
		IoUnitSize:          &ioUnit,
		NumSharedBuffers:    &numShared,
		MaxQueueDepth:       &maxQDepth,
		Zcopy:               &zcopy,
		MaxIoQpairsPerCtrlr: &maxIoQpairsPerCtrlr,
		NoWrBatching:        &noWrBatching,
		MaxSrqDepth:         &maxSrqDepth,
		AcceptorBacklog:     &acceptorBacklog,
	}); err != nil {
		return fmt.Errorf("nvmf_create_transport RDMA: %w", err)
	}
}

These values are diskengine policy. SPDK accepts many transport options; diskengine chooses a fixed RDMA configuration here. If you extend transport support, this is one place where policy would need to become configurable.

Subsystem creation is also explicit:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/utils.go
if target == nil {
	allowAny := true
	serial := diskSerial
	model := "diskengine"
	if err := client.NvmfCreateSubsystem(spdkclient.NvmfCreateSubsystemParams{
		NQN:          nqn,
		AllowAnyHost: &allowAny,
		SerialNumber: &serial,
		ModelNumber:  &model,
	}); err != nil {
		return fmt.Errorf("nvmf_create_subsystem %s: %w", nqn, err)
	}

	subs, err = client.NvmfGetSubsystems(spdkclient.NvmfGetSubsystemsParams{})
	if err != nil {
		return fmt.Errorf("nvmf_get_subsystems (post-create): %w", err)
	}

AllowAnyHost: true is a security-relevant choice. It means diskengine is not restricting host NQNs at this layer. The later listener handler in SPDK rejects secure-channel setup when allow_any_host is set, so this mode should be treated as open within the trusted storage network unless additional controls exist outside this code.

Listener creation is per address:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/utils.go
wantPort := fmt.Sprintf("%d", port)
for _, addr := range ips {
	haveListener := false
	for _, la := range target.ListenAddresses {
		if strings.EqualFold(la.Trtype, "RDMA") && la.Traddr == addr && la.Trsvcid == wantPort {
			haveListener = true
			break
		}
	}
	if !haveListener {
		adrfam := "IPv4"
		if err := client.NvmfSubsystemAddListener(spdkclient.NvmfSubsystemAddListenerParams{
			NQN: nqn,
			ListenAddress: spdkclient.ListenAddress{
				Trtype:  "RDMA",
				AdrFam:  adrfam,
				Traddr:  addr,
				Trsvcid: wantPort,
			},
		}); err != nil {
			return fmt.Errorf("nvmf_subsystem_add_listener %s %s:%s: %w", nqn, addr, wantPort, err)
		}
	}
}

The idempotency check is exact: same transport type, same address, same service id. If the desired IP changes, this code adds a new listener but does not remove the old one.

NVMe-oF Export Reconciliation

Provisioning handles a new CREATING lvol. Export reconciliation handles existing UP or RESIZING lvols and AMI snapshots. It is deliberately non-destructive: it adds missing exports but does not remove extra ones.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go
func reconcileExports(ctx context.Context, client *spdkclient.Client, envRdmaIPs string, rdmaPort int) error {
	if err := reconcileDiskPlacementFromEnv(ctx, envRdmaIPs, rdmaPort); err != nil {
		logger.Error.Printf("reconcileExports: env placement reconcile failed: %v", err)
	}
	placements, err := repository.GetReadyLvolPlacements(ctx, config.Value.BAREMETAL_ID)
	if err != nil {
		return err
	}
	if len(placements) == 0 {
		logger.Info.Println("reconcileExports: no lvol placements to reconcile")
	}

	transports, err := client.NvmfGetTransports(spdkclient.NvmfGetTransportsParams{})
	if err != nil {
		return err
	}

It starts by repairing missing RDMA placement from environment configuration, then reads database placements. After that it snapshots SPDK state so each tick can make decisions from a consistent local view.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go
subsystems, err := client.NvmfGetSubsystems(spdkclient.NvmfGetSubsystemsParams{})
if err != nil {
	return err
}
subIdx := make(map[string]int)
for i := range subsystems {
	subIdx[subsystems[i].NQN] = i
}

bdevs, err := client.BdevGetBdevs(spdkclient.BdevGetBdevsParams{})
if err != nil {
	return err
}
bdevSet := make(map[string]struct{})
for _, b := range bdevs {
	bdevSet[b.UUID] = struct{}{}
}

subIdx lets the loop answer "does this NQN already exist?" quickly. bdevSet prevents exporting a namespace for a bdev that SPDK cannot currently see.

Namespace reconciliation is where the lvol becomes reachable:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go
if _, ok := bdevSet[p.UUID]; !ok {
	logger.Warn.Printf("reconcileExports: bdev %s not present yet; skipping export", p.UUID)
	continue
}
hasNs := false
if sys.Namespaces != nil {
	for _, ns := range *sys.Namespaces {
		if ns.Name == p.UUID {
			hasNs = true
			break
		}
	}
}
if !hasNs {
	if err := client.NvmfSubsystemAddNs(spdkclient.NvmfSubsystemAddNsParams{
		NQN: p.NQN,
		Namespace: spdkclient.NvmfNamespaceParams{
			BdevName: p.UUID,
			UUID:     &p.UUID,
		},
	}); err != nil {
		logger.Error.Printf("reconcileExports: add ns %s to %s failed: %v", p.UUID, p.NQN, err)
		continue
	}
}

The bdev check handles async ordering and restart state. A DB row may be UP while SPDK has not loaded the lvstore yet. In that case the loop skips export instead of creating a namespace with a nonexistent backing bdev. The next tick can retry after disk initialization or examine has loaded the lvol.

What SPDK Does For NVMf Objects

SPDK's nvmf_create_transport handler rejects duplicate transports:

// lib/nvmf/nvmf_rpc.c
if (spdk_nvmf_tgt_get_transport(tgt, ctx->trtype)) {
	SPDK_ERRLOG("Transport type '%s' already exists\n", ctx->trtype);
	spdk_jsonrpc_send_error_response_fmt(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR,
					     "Transport type '%s' already exists", ctx->trtype);
	nvmf_rpc_create_transport_ctx_free(ctx);
	return;
}

ctx->opts.transport_specific = params;
ctx->request = request;

rc = spdk_nvmf_transport_create_async(ctx->trtype, &ctx->opts, nvmf_rpc_create_transport_done, ctx);

diskengine avoids this by calling nvmf_get_transports first and creating RDMA only if it is absent. If two loops race, SPDK can still return a duplicate error; the next tick should see the transport.

Subsystem creation starts by allocating an NVMe subsystem for an NQN:

// lib/nvmf/nvmf_rpc.c
subsystem = spdk_nvmf_subsystem_create(tgt, req->nqn, SPDK_NVMF_SUBTYPE_NVME,
				       req->max_namespaces);
if (!subsystem) {
	spdk_jsonrpc_send_error_response_fmt(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR,
					     "Unable to create subsystem %s", req->nqn);
	goto cleanup;
}

Later in the same handler, after serial/model and controller-id validation, SPDK applies access policy:

// lib/nvmf/nvmf_rpc.c
spdk_nvmf_subsystem_set_allow_any_host(subsystem, req->allow_any_host);
spdk_nvmf_subsystem_set_ana_reporting(subsystem, req->ana_reporting);

And after the remaining namespace limit options, SPDK starts the subsystem asynchronously:

// lib/nvmf/nvmf_rpc.c
rc = spdk_nvmf_subsystem_start(subsystem,
			       rpc_nvmf_subsystem_started,
			       request);
if (rc) {
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR,
					 "Failed to start subsystem");
}

The NQN must be unique in the target. A duplicate NQN will fail creation in SPDK. diskengine usually checks nvmf_get_subsystems first, but duplicate create races are still possible if multiple actors change the target.

Listener add pauses the subsystem:

// lib/nvmf/nvmf_rpc.c
subsystem = spdk_nvmf_tgt_find_subsystem(tgt, ctx->nqn);
if (!subsystem) {
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS, "Invalid parameters");
	nvmf_rpc_listener_ctx_free(ctx);
	return;
}

if (rpc_listen_address_to_trid(&ctx->address, &ctx->trid)) {
	spdk_jsonrpc_send_error_response(ctx->request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
					 "Invalid parameters");
	nvmf_rpc_listener_ctx_free(ctx);
	return;
}

rc = spdk_nvmf_subsystem_pause(subsystem, 0, nvmf_rpc_listen_paused, ctx);

Adding a listener is not just appending to a Go slice. SPDK pauses the subsystem, applies the listener change, and resumes through callbacks. That is why listener conflicts or invalid addresses must be treated as operational failures, not as harmless metadata mismatches.

Namespace add also pauses the subsystem:

// lib/nvmf/nvmf_rpc.c
subsystem = spdk_nvmf_tgt_find_subsystem(tgt, ctx->nqn);
if (!subsystem) {
	SPDK_ERRLOG("Unable to find subsystem with NQN %s\n", ctx->nqn);
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS, "Invalid parameters");
	nvmf_rpc_ns_ctx_free(ctx);
	return;
}

rc = spdk_nvmf_subsystem_pause(subsystem, ctx->ns_params.nsid, nvmf_rpc_ns_paused, ctx);
if (rc != 0) {
	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR, "Internal error");
	nvmf_rpc_ns_ctx_free(ctx);
}

If namespace attach fails after lvol creation, the storage object still exists. diskengine's provisioning code recognizes this as state drift. Export reconciliation can repair missing namespaces when the bdev exists and placement rows are ready.

Verify State

verifyState is not the main reconciler. It is the auditor. It runs once at startup and then from the provisioning loop. It compares DB rows to live SPDK state and reports drift.

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/verifystate.go
bdevInfos, err := client.BdevGetBdevs(spdkclient.BdevGetBdevsParams{})
if err != nil {
	logger.Error.Println("Failed to get block devices from SPDK:", err)
	return fmt.Errorf("verifyState: bdev_get_bdevs: %w", err)
}

bdevSet := map[string]struct{}{}
for _, info := range bdevInfos {
	if info.ProductName != "Logical Volume" {
		continue
	}
	bdevSet[info.UUID] = struct{}{}
}

This code intentionally filters to logical volume bdevs. Physical NVMe bdevs are not volume exports. For volume/snapshot drift, the important objects are lvol bdev UUIDs.

It also reconciles disk state with controller presence:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/verifystate.go
for _, d := range dbDisks {
	expectedCtrl := fmt.Sprintf("NvmeDisk%d", d.ID)
	_, present := ctrlSet[expectedCtrl]
	switch {
	case present && d.State == types.DISK_STATE_ERROR:
		if err := repository.UpdateDiskState(ctx, d.ID, types.DISK_STATE_ERROR, types.DISK_STATE_UP); err != nil {
			logger.Warn.Printf("verifyState: failed to set disk %d UP: %v", d.ID, err)
		}
	case !present && d.State == types.DISK_STATE_UP:
		if err := repository.UpdateDiskState(ctx, d.ID, types.DISK_STATE_UP, types.DISK_STATE_NEW); err != nil {
			logger.Warn.Printf("verifyState: failed to set disk %d NEW: %v", d.ID, err)
		}
	}
}

If the expected controller exists, an ERROR disk can be moved back to UP. If the controller is missing while the DB says UP, the disk is moved to NEW so initialization can reattach it. This is a recovery mechanism after SPDK restart or controller loss.

The auditor also detects orphans:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/verifystate.go
for bdev := range bdevSet {
	if _, present := dbSet[bdev]; !present {
		msg := fmt.Sprintf("orphan bdev %s present in SPDK but not in DB", bdev)
		logger.Error.Printf("STATE DRIFT: %s (likely from failed provisioning)", msg)
		drifts = append(drifts, msg)
	}
}

if len(subsMap) > 0 {
	for nqn, nsSet := range subsMap {
		for bdev := range nsSet {
			if _, present := dbSet[bdev]; !present {
				msg := fmt.Sprintf("subsystem %s exposes bdev %s not present in DB", nqn, bdev)
				logger.Error.Printf("STATE DRIFT: %s (orphan namespace export)", msg)
				drifts = append(drifts, msg)
			}
		}
	}
}

This is how diskengine notices stale SPDK state after a crash or failed transaction. The code reports but does not blindly delete. That is conservative: deleting a namespace or bdev without proving ownership could destroy a legitimate export.

Health, Resize, Snapshots, Delete, And Metrics

The same pattern appears in the secondary loops:

Health calls bdev_nvme_get_controller_health_info for the deterministic controller name and updates disk health.
Resize compares DB capacity to SPDK bdev capacity, calls bdev_lvol_resize, then zeroes the newly exposed region through an NBD export.
Snapshot creation calls bdev_lvol_snapshot, handles duplicate snapshot names, and records the SPDK snapshot lvol UUID.
Snapshot and lvol deletion call bdev_lvol_delete and only finalize "not found" if the parent lvstore is loaded.
Metrics scrape SPDK bdev I/O stats and export them out of band.

The deletion safety check is worth reading:

// /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/lvol_delete.go
err := client.BdevLvolDelete(params)
if err != nil {
	msg := strings.ToLower(err.Error())
	if strings.Contains(msg, "not found") || strings.Contains(msg, "no such") || strings.Contains(msg, "does not exist") {
		if l.LvstoreUUID != "" {
			if _, loaded := loadedLvstores[l.LvstoreUUID]; !loaded {
				logger.Warn.Printf("lvolDelete: lvol %s not found but lvstore %s not loaded yet; deferring deletion", l.LvolUUID, l.LvstoreUUID)
				continue
			}
		}
		logger.Info.Printf("lvolDelete: lvol %s confirmed deleted (lvstore loaded)", l.LvolUUID)
	} else {
		logger.Error.Printf("lvolDelete: delete lvol %s failed: %v", l.LvolUUID, err)
		continue
	}
}

This prevents a subtle data-loss bug. If SPDK says "not found" while the lvstore is not loaded, the lvol might still exist on disk and reappear after examine. diskengine defers deletion instead of finalizing prematurely.

Storage Node Reconciler Diagram

flowchart LR DB[(database desired state)] RPC[SPDK JSON-RPC socket] SPDK[(live SPDK object graph)] DB --> D1[disk discovery] D1 --> DB DB --> D2[disk init] D2 --> RPC DB --> P[provisioning] P --> RPC DB --> E[NVMe-oF export reconcile] E --> RPC DB --> R[resize] R --> RPC DB --> S[snapshot create/delete] S --> RPC DB --> X[lvol delete] X --> RPC DB --> H[health and metrics] H --> RPC RPC --> SPDK SPDK --> RPC RPC --> DB SPDK --> B1[NVMe bdev] B1 --> B2[lvstore] B2 --> B3[lvol bdev] B3 --> B4[NVMe-oF namespace]

Every loop reads DB state, reads SPDK state, makes a small change, and retries on the next tick if SPDK is not ready.

Edge Cases And Failure Modes

Duplicate NQN or subsystem:

diskengine checks nvmf_get_subsystems before creating a subsystem. SPDK still owns uniqueness. If another actor creates the same NQN between the check and create, nvmf_create_subsystem can fail. The correct recovery is to re-read subsystem state and verify whether the existing subsystem has the expected listeners and namespaces.

Duplicate listener:

diskengine treats a listener as present only when transport type, address, and service id all match. A listener on the wrong IP or wrong port is not equivalent. The export loop adds the missing listener but does not remove stale listeners, so operators must inspect nvmf_get_subsystems after placement changes.

Namespace reuse:

A namespace attach can fail because the namespace or bdev is already attached. Provisioning handles "already exists" by verifying that the expected bdev is attached to the expected subsystem. It does not assume every duplicate error is good. If the duplicate namespace belongs to a different bdev, that is a real conflict.

Partial export failure:

The lvol may be created before nvmf_subsystem_add_ns fails. That leaves an allocated lvol bdev with no export. verifyState may report an orphan bdev or missing namespace, and reconcileExports may repair it if the DB row reaches a ready placement state.

Stale JSON-RPC state after crash:

SPDK can still have bdevs, lvstores, subsystems, listeners, or namespaces after diskengine restarts. Startup verifyState, deterministic controller names, duplicate recovery, and bdev_examine are all there to converge instead of recreating everything.

Async initialization and ordering:

SPDK attach, examine, listener add, namespace add, and transport create involve asynchronous callbacks internally. diskengine must not assume a successful RPC makes every derived object immediately visible. Polling bdev_lvol_get_lvstores, refreshing subsystem lists after create, and retrying export are practical responses to that model.

Reconnects and multipath assumptions:

Storage-node export code creates RDMA listeners and namespaces. It does not implement initiator reconnect policy. SPDK's NVMe attach RPC supports multipath options, and SPDK's NVMe-oF target supports ANA-related features, but this storage-node path does not appear to configure host-specific multipath policy. Treat multipath behavior as an initiator and SPDK target configuration concern unless diskengine grows explicit policy.

Security and authentication gaps:

The subsystem create path sets AllowAnyHost: true. The inspected source does not add host allow lists, DH-HMAC-CHAP keys, TLS PSKs, or per-host authorization in storage-node mode. This may be acceptable on a trusted isolated fabric, but it is a real security assumption.

Teardown:

The deletion loop deletes lvol bdevs and finalizes DB state. The export reconciler shown here only adds missing exports; it does not remove stale listeners or namespaces as a general cleanup mechanism. If a namespace remains after a DB row is gone, verifyState reports drift. A future teardown extension should remove namespace first, then delete lvol, then update DB only after SPDK confirms both.

Lvstore exists in SPDK but not DB:

Disk initialization imports the lvstore. This is self-healing when the disk metadata is real and the DB lost or never had the row.

Lvstore exists in DB but not SPDK:

Initialization treats this as retryable after examine, because SPDK may not have loaded metadata yet. If it never appears, the operator must decide whether the disk was wiped, swapped, or misidentified.

Device disappears from sysfs after VFIO bind:

Expected. Once SPDK owns the PCI function, use SPDK RPCs to inspect it. Do not rely on /sys/block after initialization.

No IOMMU or VFIO failure:

Without VFIO binding, SPDK cannot own the physical PCIe controller in this mode. Disk initialization cannot create the base bdev and therefore cannot create an lvstore.

Snapshot delete while clones depend on it:

SPDK lvol/blobstore can reject unsafe delete operations when dependencies exist. The loop logs and retries instead of forcing DB deletion.

Misconceptions To Kill

"Storage-node mode is a data proxy."

No. It prepares SPDK exports. VM or bare-metal I/O goes through NVMe-oF and SPDK bdev paths, not Go request handlers.

"The DB is always the truth."

No. The DB is desired state plus bookkeeping. SPDK is the live storage state. The code constantly compares them.

"An lvol is exported automatically when created."

No. bdev_lvol_create creates a bdev. nvmf_subsystem_add_ns exports that bdev through an NVMe-oF subsystem.

"A successful create means there is no retry work left."

No. A crash can happen after SPDK create and before DB finalization. The code must handle duplicates and continue.

"Discovery should still see initialized disks under /sys/block."

No. VFIO-bound devices are not ordinary kernel block devices.

Lab: Provision One Lvol On Paper

Write a sequence for a new volume replica:

Disk is already UP with an lvstore UUID.
lvols row enters CREATING.
provisioningLoop observes it.
ensureNvmeofReady verifies or creates RDMA transport, subsystem, and listener.
BdevLvolCreate sends bdev_lvol_create.
SPDK returns the lvol UUID.
NvmfSubsystemAddNs sends nvmf_subsystem_add_ns.
FinalizeProvisioningForLvol stores SPDK lvol UUID and NQN in the DB.
verifyState checks DB lvol UUID against bdev_get_bdevs and namespace state against nvmf_get_subsystems.

For each step, name whether the object being changed is in the DB, Linux, or SPDK.

Operational Debug Exercise

Symptom: a bare-metal node cannot connect to a newly created volume.

On the storage node, check:

Is the lvol UP in DB?
Does bdev_get_bdevs show the lvol UUID?
Does nvmf_get_subsystems show the expected NQN?
Does the subsystem have a listener on the expected RDMA IP and port?
Does the subsystem have a namespace with the lvol UUID?
Did nvmeofExportLoop log add-listener or add-ns errors?
Did verifyState report orphan bdevs or orphan namespace exports?
Did SPDK restart without replaying saved bdev/nvmf config?

Source Reading Path

Read these in order when extending storage-node mode:

/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/storagenode.go
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient/wrappers.go
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/disk_init.go
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/utils.go
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/provisionlvol.go
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/nvmeofexport.go
/home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode/verifystate.go
module/bdev/nvme/bdev_nvme_rpc.c
module/bdev/lvol/vbdev_lvol_rpc.c
lib/nvmf/nvmf_rpc.c
doc/jsonrpc.md.jinja2
doc/nvmf.md
doc/lvol.md

Self-Check

Why does storage-node mode bind disks to VFIO?
What is the difference between an lvstore and an lvol?
Which loop ensures NVMe-oF exports exist for already-created lvols?
Why can lvol creation and namespace attachment partially succeed?
Why is verifyState run before loops start?
Why is "not found" during delete not enough to finalize deletion?
What security assumption is implied by AllowAnyHost: true?
Why does diskengine use deterministic names like NvmeDisk<id> and lvs<id>?

References

Local diskengine storage-node docs: /home/lolwierd/Projects/excloud/diskengine/diskengine/docs/storagenode.md
Local diskengine storage-node source: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/storagenode
Local diskengine SPDK client wrappers: /home/lolwierd/Projects/excloud/diskengine/diskengine/internal/spdkclient
Local SPDK NVMe bdev RPC source: module/bdev/nvme/bdev_nvme_rpc.c
Local SPDK lvol RPC source: module/bdev/lvol/vbdev_lvol_rpc.c
Local SPDK NVMf RPC source: lib/nvmf/nvmf_rpc.c
Local SPDK bdev examine RPC source: lib/bdev/bdev_rpc.c
Local SPDK JSON-RPC reference source: doc/jsonrpc.md.jinja2
Local SPDK lvol documentation: doc/lvol.md
Local SPDK NVMe-oF documentation: doc/nvmf.md