Chapter 6: PCIe, MMIO, DMA, IOMMU, VFIO, And Hugepages

Chapter Goal

SPDK is fast because it puts userspace code close to hardware. That means SPDK applications must understand what the kernel normally hides: PCIe discovery, BAR mapping, MMIO registers, DMA-capable memory, IOVA addresses, IOMMU permissions, VFIO ownership, and hugepage-backed memory.

By the end of this chapter, you should be able to explain why SPDK setup binds devices to vfio-pci, why hugepages matter, what an IOVA is, why an IOMMU group can block device assignment, and how a user process can safely ring an NVMe doorbell without using the kernel NVMe driver.

Beginner Mental Model

A PCIe NVMe SSD is not a file. It is a device on a bus:

CPU process
  normal virtual memory
  hugepage-backed DMA buffers
  mapped PCI BAR virtual address

kernel
  VFIO owns PCI device
  IOMMU maps allowed DMA ranges

PCIe device
  BAR registers for MMIO
  DMA engine reads commands / writes completions / reads or writes data

SPDK's userspace driver needs two kinds of access:

Register access: map the device BAR and write MMIO registers such as NVMe doorbells.
DMA access: give the device addresses for command queues, completion queues, and data buffers.

The kernel is still involved. It enforces permissions, sets up IOMMU mappings, exposes VFIO file descriptors, and reserves hugepages. SPDK bypasses the kernel storage stack; it does not bypass hardware protection.

PCIe Device Identity: BDF And BARs

PCIe devices are identified by bus-device-function addresses, often written as:

0000:5e:00.0
domain:bus:device.function

The device exposes configuration space and Base Address Registers (BARs). A BAR describes a region of device memory or I/O space that the host can map. NVMe controllers expose registers through a memory BAR. When SPDK maps that BAR, ordinary-looking pointer writes become MMIO transactions on PCIe.

SPDK's environment API declares PCI helpers in include/spdk/env.h. The BAR mapping API is spdk_pci_device_map_bar() at include/spdk/env.h:898, and the DPDK-backed implementation is in lib/env_dpdk/pci.c:740.

The public API deliberately returns two addresses for a BAR: a process virtual address that C code can dereference for MMIO, and a physical address value that matters to lower-level mapping code. The caller should not infer that the returned pointer is ordinary memory. It is a mapped device aperture.

/* include/spdk/env.h */
/**
 * \param dev PCI device.
 * \param bar BAR number.
 * \param mapped_addr A variable to store the virtual address of the mapping.
 * \param phys_addr A variable to store the physical address of the mapping.
 * \param size A variable to store the size of the bar (in bytes).
 *
 * \return 0 on success.
 */
int spdk_pci_device_map_bar(struct spdk_pci_device *dev, uint32_t bar,
			    void **mapped_addr, uint64_t *phys_addr, uint64_t *size);

The object ownership is split. The PCI device object owns the bus-specific operations, the SPDK env layer owns the cross-platform API, and the process gets a mapping only while the PCI device remains attached. Detach or failure paths must unmap the BAR; otherwise the process can keep a stale address into a device that may have been reset or assigned elsewhere.

MMIO Is Not RAM

MMIO means memory-mapped I/O. The CPU has a virtual address that points to a device register window, not DRAM. Stores to that address are side effects on a device.

For NVMe, doorbells are MMIO registers. SPDK models them in include/spdk/nvme_spec.h:611 as submission queue tail and completion queue head doorbells. SPDK writes them with spdk_mmio_write_4() in lib/nvme/nvme_pcie_internal.h:272 and lib/nvme/nvme_pcie_internal.h:295.

Important differences from RAM:

MMIO is usually uncached or specially ordered.
Writes may be posted and require barriers in driver logic.
Reading MMIO can be expensive and may have side effects.
You cannot treat a device register pointer as normal shared memory.

Misconception to kill: "If I have a pointer, it is memory." In userspace drivers, some pointers are portals into hardware.

The doorbell path shows why ordering matters. SPDK has already written one or more submission queue entries into host memory. Before it notifies the controller by writing the SQ tail doorbell, it issues a write memory barrier. That barrier is not decoration: the controller must not observe the MMIO doorbell before the command entries it will DMA-read are globally visible.

/* lib/nvme/nvme_pcie_internal.h */
if (spdk_likely(need_mmio)) {
	spdk_wmb();
	pqpair->stat->sq_mmio_doorbell_updates++;
	g_thread_mmio_ctrlr = pctrlr;
	spdk_mmio_write_4(pqpair->sq_tdbl, pqpair->sq_tail);
	g_thread_mmio_ctrlr = NULL;
}

Completion queue doorbells are similar, but the direction is different. The driver is telling the controller which completion entries it has consumed, so the controller can reuse that queue space.

/* lib/nvme/nvme_pcie_internal.h */
if (spdk_likely(need_mmio)) {
	pqpair->stat->cq_mmio_doorbell_updates++;
	g_thread_mmio_ctrlr = pctrlr;
	spdk_mmio_write_4(pqpair->cq_hdbl, pqpair->cq_head);
	g_thread_mmio_ctrlr = NULL;
}

These writes run on the SPDK polling thread that owns the qpair. There is no kernel NVMe request object on this fast path. The SPDK thread updates queue state in memory, uses barriers to order those updates, and then performs the MMIO write that tells the device to act.

DMA: The Device Touches Host Memory

Direct Memory Access lets the device read and write host memory without the CPU copying every byte.

For NVMe:

The controller DMA-reads SQ entries.
The controller DMA-writes CQ entries.
The controller DMA-reads host buffers for writes.
The controller DMA-writes host buffers for reads.

That requires addresses the device can use. CPU virtual addresses are not automatically valid PCIe DMA addresses. SPDK and DPDK translate or map memory into IOVA space.

SPDK's bdev required_alignment field (include/spdk/bdev_module.h:513) matters partly because DMA engines can have alignment restrictions. SPDK's env memory APIs include memzones that are IOVA-contiguous by default unless SPDK_MEMZONE_NO_IOVA_CONTIG is used (include/spdk/env.h:255).

The env API exposes this as a contract. A caller that reserves a memzone is not just asking for bytes; it is asking for memory with DMA-friendly properties that can be shared by name across SPDK processes. Unless the caller opts out, the zone is IOVA-contiguous, which matters for hardware structures that accept a base address plus length instead of a scatter-gather list.

/* include/spdk/env.h */
/**
 * Reserve a named, process shared memory zone with the given size, numa_id
 * and flags. Unless `SPDK_MEMZONE_NO_IOVA_CONTIG` flag is provided, the returned
 * memory will be IOVA contiguous.
 *
 * \param name Name to set for this memory zone.
 * \param len Length in bytes.
 * \param numa_id NUMA node ID to allocate memory on, or SPDK_ENV_NUMA_ID_ANY
 * for any NUMA node.
 * \param flags Flags to set for this memory zone.
 *
 * \return a pointer to the allocated memory address on success, or NULL on failure.
 */
void *spdk_memzone_reserve(const char *name, size_t len, int numa_id, unsigned flags);

SPDK's DMA documentation makes the same point from the operating-system side: normal virtual memory can move, page out, or lose stable physical backing, whereas SPDK relies on DPDK-backed pinned memory and recommends vfio-pci with an enabled IOMMU for a future-proof DMA model. In practical code review, a data buffer that reaches a PCIe device should be traced back to SPDK/DPDK DMA-safe allocation or an explicit registration path.

Physical Address, Virtual Address, IOVA

Three address spaces matter:

VA: CPU virtual address in the process.
PA: host physical address.
IOVA: I/O virtual address used by a device for DMA.

With no IOMMU, devices often DMA to physical addresses. With an IOMMU, devices DMA to IOVAs, and the IOMMU translates those IOVAs to physical memory while enforcing permissions.

DPDK exposes IOVA modes. SPDK passes an explicit --iova-mode to DPDK when configured (lib/env_dpdk/init.c:523). SPDK also forces iova-mode=pa in some no-IOMMU or limited-IOMMU cases (lib/env_dpdk/init.c:530 through lib/env_dpdk/init.c:547).

Beginner trap: IOVA may equal VA, may equal PA, or may equal neither depending on mode and platform. Code that assumes one globally will eventually fail.

The startup code shows that IOVA mode is not an academic option. SPDK lets the application request a mode, but if it detects VFIO no-IOMMU mode or an x86 environment where the IOMMU cannot cover the virtual address space, it forces PA mode instead. That choice changes what address values later get programmed into device-visible structures.

/* lib/env_dpdk/init.c */
if (opts->iova_mode) {
	/* iova-mode=pa is incompatible with no_huge */
	args = push_arg(args, &argcount, _sprintf_alloc("--iova-mode=%s", opts->iova_mode));
	if (args == NULL) {
		return -1;
	}
} else {
	/* When using vfio with enable_unsafe_noiommu_mode=Y, we need iova-mode=pa,
	 * but DPDK guesses it should be iova-mode=va. Add a check and force
	 * iova-mode=pa here. */
	if (!no_huge && rte_vfio_noiommu_is_enabled()) {
		args = push_arg(args, &argcount, _sprintf_alloc("--iova-mode=pa"));
		if (args == NULL) {
			return -1;
		}
	}

The DPDK EAL guide describes Linux IOVA detection as a heuristic based on bus requirements, physical address availability, and whether IOMMU groups are present. That is why this chapter avoids saying "IOVA is physical address" or "IOVA is virtual address" as a universal rule. The right answer is an application startup property plus a platform property.

IOMMU Protection

An IOMMU is an MMU for device DMA. It lets the kernel say: this device may DMA only to these IOVA ranges with these permissions. Without it, a buggy or malicious device can overwrite arbitrary physical memory.

Linux VFIO documentation describes VFIO as an IOMMU/device-agnostic framework for exposing direct device access to userspace in an IOMMU-protected environment. That is the security model SPDK relies on when using vfio-pci.

SPDK checks whether it is using IOMMU-backed DMA through spdk_iommu_is_enabled() (include/spdk/env.h:702), implemented in DPDK memory code around lib/env_dpdk/memory.c:1064.

The public API is intentionally boolean: most SPDK code should not care about the details of the kernel IOMMU backend. It needs to know whether DMA is being protected and translated through an IOMMU.

/* include/spdk/env.h */
/**
 * Reports whether the SPDK application is using the IOMMU for DMA
 *
 * \return True if we are using the IOMMU, false otherwise.
 */
bool spdk_iommu_is_enabled(void);

The DPDK env implementation makes the important distinction: VFIO may be available, but unsafe no-IOMMU mode is not treated as "IOMMU enabled" by SPDK.

/* lib/env_dpdk/memory.c */
bool
spdk_iommu_is_enabled(void)
{
#if VFIO_ENABLED
	return g_vfio.enabled && !g_vfio.noiommu_enabled;
#else
	return false;
#endif
}

Linux's newer IOMMUFD documentation describes the same abstraction in more general terms: userspace can manage I/O address spaces and I/O page tables, and an IOAS maps userspace memory into IOVA ranges. SPDK's current VFIO type1 path uses the older VFIO container interface, but the underlying concept is the same: the kernel owns the translation and protection boundary; userspace requests allowed mappings.

VFIO: Userspace Device Ownership

VFIO is the kernel framework that lets a userspace process safely own a device. A kernel driver such as nvme normally owns an NVMe SSD. For SPDK to drive it directly, that kernel driver must release it and vfio-pci must bind to it.

SPDK's scripts/setup.sh is the practical entry point. The help text says it allocates hugepages and binds NVMe, I/OAT, VMD, and Virtio devices (scripts/setup.sh:33). It selects vfio-pci when available (scripts/setup.sh:403) and warns about IOMMU group constraints (scripts/setup.sh:202 through scripts/setup.sh:213).

DPDK's Linux driver guide recommends vfio-pci for DPDK-bound devices and explains that most devices must be unbound from their kernel driver and bound to vfio-pci before the application runs.

Misconception to kill: "Binding to VFIO turns off the kernel." The kernel still mediates access. It just stops acting as the storage driver for that device.

The setup script's driver selection is conservative. If the user explicitly requests a driver, that wins. Otherwise, an enabled IOMMU steers the script to vfio-pci, and the script tries to load vfio_iommu_type1 because that module provides the VFIO type1 IOMMU backend used by the mapping code below.

# scripts/setup.sh
elif is_iommu_enabled; then
	driver_name=vfio-pci
	# Just in case, attempt to load VFIO_IOMMU_TYPE1 module into the kernel - this
	# should be done automatically by modprobe since this particular module should
	# be a part of vfio-pci dependencies, however, on some distros, it seems that
	# it's not the case. See #1689.
	if modinfo vfio_iommu_type1 > /dev/null; then
		modprobe vfio_iommu_type1
	fi
elif ! check_for_driver uio_pci_generic || modinfo uio_pci_generic > /dev/null 2>&1; then
	driver_name=uio_pci_generic
elif [[ -e $igb_uio_fallback ]]; then
	driver_path="$igb_uio_fallback"
	driver_name="igb_uio"
	echo "WARNING: uio_pci_generic not detected - using $driver_name"

This explains a common setup surprise: the selected driver is a function of both policy and hardware state. On an IOMMU-enabled host, SPDK setup naturally tries to put supported devices under VFIO because VFIO is the kernel interface that can expose device MMIO and DMA to userspace while still enforcing the IOMMU boundary.

IOMMU Groups

An IOMMU group is the smallest isolation unit the kernel can safely assign. If two PCIe functions cannot be isolated from each other, they appear in the same group. VFIO assignment normally requires the whole group to be safe: all devices in the group must be bound to VFIO-compatible drivers or unbound.

This is why setup can fail even when the target NVMe device looks correct. A bridge, multifunction device, or platform topology can put another active device in the same group.

Operational checklist:

readlink /sys/bus/pci/devices/0000:5e:00.0/iommu_group
ls -l /sys/bus/pci/devices/0000:5e:00.0/iommu_group/devices

If another device in the group is still bound to a kernel driver, VFIO may reject the group. DPDK's Linux driver guide calls out this limitation for devices behind bridges and multifunction groupings.

SPDK setup warns before the application gets that far. It looks up all devices in the same group and, when using a VFIO driver, reports any peer that is not bound to the same driver. UNBIND_ENTIRE_IOMMU_GROUP=yes exists, but the name is intentionally blunt because it can detach devices the operator did not mean to hand to SPDK.

# scripts/setup.sh
if ((${#iommug[@]} > 1)) && [[ $driver_name == vfio* ]]; then
	pci_dev_echo "$bdf" "WARNING: detected multiple devices (${#iommug[@]}) under the same IOMMU group!"
	for _bdf in "${iommug[@]}"; do
		[[ $_bdf == "$bdf" ]] && continue
		_driver=$(readlink -f "/sys/bus/pci/devices/$_bdf/driver") && _driver=${_driver##*/}
		if [[ $_driver == "$driver_name" ]]; then
			continue
		fi
		# See what DPDK considers to be a "viable" iommu group: dpdk/lib/eal/linux/eal_vfio.c -> rte_vfio_setup_device()
		pci_dev_echo "$bdf" "WARNING: ${_bdf##*/} not bound to $driver_name (${_driver:-no driver})"
		pci_dev_echo "$bdf" "WARNING All devices in the IOMMU group must be bound to the same driver or unbound"

DPDK later asks the kernel for the group status. The VFIO_GROUP_FLAGS_VIABLE bit is the hard gate: if the group is not viable, DPDK refuses to set up the device.

/* dpdk/lib/eal/linux/eal_vfio.c */
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
if (ret) {
	EAL_LOG(ERR, "%s cannot get VFIO group status, "
		"error %i (%s)", dev_addr, errno, strerror(errno));
	close(vfio_group_fd);
	rte_vfio_clear_group(vfio_group_fd);
	return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
	EAL_LOG(ERR, "%s VFIO group is not viable! "
		"Not all devices in IOMMU group bound to VFIO or unbound",
		dev_addr);
	close(vfio_group_fd);
	rte_vfio_clear_group(vfio_group_fd);
	return -1;
}

Linux VFIO documentation gives the reason for this rule: a group is the unit of ownership used by VFIO because isolation is not always possible at individual function granularity. The group can be enlarged by bridges, multifunction devices, missing ACS isolation, and platform topology.

Hugepages

Hugepages are large pages, commonly 2 MiB or 1 GiB, reserved outside ordinary pageable memory. SPDK and DPDK use them because DMA wants pinned, physically manageable memory and because large pages reduce translation overhead.

SPDK setup exposes:

HUGEMEM for hugepage memory size (scripts/setup.sh:54).
NRHUGE for number of pages (scripts/setup.sh:61).
HUGENODE for NUMA node selection (scripts/setup.sh:62).
HUGEPGSZ for page size (scripts/setup.sh:67).
SKIP_HUGE, CLEAR_HUGE, and persistence options.

The setup script mounts hugetlbfs if needed (scripts/setup.sh:594 through scripts/setup.sh:600) and configures hugepages (scripts/setup.sh:603).

SPDK's DPDK env code can disable hugepages with --no-huge, but lib/env_dpdk/init.c:403 through lib/env_dpdk/init.c:418 shows restrictions: disabling hugepages needs explicit memory sizing and is incompatible with some hugepage options and PA IOVA assumptions.

DPDK's EAL documentation describes hugepage-backed memory allocation, and the Linux getting-started guide notes hugepages must be reserved as root before running applications as non-root.

The setup help is worth reading because it is an operational API. These variables are how the administrator decides how much pinned hugepage memory exists, where it exists, and whether setup should touch it at all.

# scripts/setup.sh
echo "HUGEMEM           Size of hugepage memory to allocate (in MB). 2048 by default."
echo "                  For NUMA systems, the hugepages will be distributed on node0 by"
echo "                  default."
echo "NRHUGE            Number of hugepages to allocate. This variable overwrites HUGEMEM."
echo "HUGENODE          Specific NUMA node to allocate hugepages on. Multiple nodes can be"
echo "                  separated with comas. By default, NRHUGE will be applied on each node."
echo "HUGEPGSZ          Size of the hugepages to use in kB. If not set, kernel's default"
echo "                  setting is used."
echo "SHRINK_HUGE       If set to 'yes', hugepages allocation won't be skipped in case"
echo "                  number of requested hugepages is lower from what's already"
echo "                  allocated."
echo "CLEAR_HUGE        If set to 'yes', the attempt to remove hugepages from all nodes will"
echo "                  be made prior to allocation".
echo "SKIP_HUGE         If set to 'yes', the attempt to allocate hugepages will be skipped."

On Linux, setup first makes sure there is a hugetlbfs mount, then writes the requested page counts. For a non-root VFIO user, it also changes ownership of the hugepage mount and later checks the user's memlock limit.

# scripts/setup.sh
function configure_linux() {
	configure_linux_pci
	hugetlbfs_mounts=$(linux_hugetlbfs_mounts)

	if [ -z "$hugetlbfs_mounts" ]; then
		hugetlbfs_mounts=/mnt/huge
		echo "Mounting hugetlbfs at $hugetlbfs_mounts"
		mkdir -p "$hugetlbfs_mounts"
		mount -t hugetlbfs nodev "$hugetlbfs_mounts"
	fi

	configure_linux_hugepages

	if [ "$driver_name" = "vfio-pci" ]; then

The allocation helper writes the kernel nr_hugepages file and then reads it back. That read-back matters because runtime hugepage allocation can fail on a fragmented machine.

# scripts/setup.sh
allocated_hugepages=$(< "$hp_int")

if ((NRHUGE <= allocated_hugepages)) && [[ $SHRINK_HUGE != yes ]]; then
	echo "INFO: Requested $NRHUGE hugepages but $allocated_hugepages already allocated ${2:+on node$2}"
	return 0
fi

echo $((NRHUGE < 0 ? 0 : NRHUGE)) > "$hp_int"

allocated_hugepages=$(< "$hp_int")
if ((allocated_hugepages < NRHUGE)); then
	cat <<- ERROR

		## ERROR: requested $NRHUGE hugepages but $allocated_hugepages could be allocated ${2:+on node$2}.
		## Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.
	ERROR
	return 1
fi

--no-huge is therefore an escape hatch with constraints, not the default path. SPDK refuses combinations that would be internally contradictory: no hugepages plus hugepage-specific options, no hugepages without an explicit memory size, or no hugepages with PA IOVA mode.

/* lib/env_dpdk/init.c */
if (no_huge) {
	if (opts->hugepage_single_segments || opts->unlink_hugepage || opts->hugedir) {
		fprintf(stderr, "--no-huge invalid with other hugepage options\n");
		free_args(args, argcount);
		return -1;
	}

	if (opts->mem_size < 0) {
		fprintf(stderr,
			"Disabling hugepages requires specifying how much memory "
			"will be allocated using -s parameter\n");
		free_args(args, argcount);
		return -1;
	}

	/* iova-mode=pa is incompatible with no_huge */
	if (opts->iova_mode &&
	    (strcmp(opts->iova_mode, "pa") == 0)) {
		fprintf(stderr, "iova-mode=pa is incompatible with specified "
			"no-huge parameter\n");
		free_args(args, argcount);
		return -1;
	}

DMA Mapping In SPDK Source

When VFIO is enabled, SPDK maps memory into the IOMMU. The core helper _vfio_iommu_map_dma() in lib/env_dpdk/memory.c:1084 fills a vfio_iommu_type1_dma_map with:

read/write flags,
virtual address,
IOVA,
size.

It then calls ioctl(g_vfio.fd, VFIO_IOMMU_MAP_DMA, ...) at lib/env_dpdk/memory.c:1119, unless mapping is deferred because no SPDK-managed VFIO device has been attached yet.

BAR mapping has a similar IOMMU detail. spdk_pci_device_map_bar() maps the BAR and, if IOMMU is enabled, maps the BAR into IOMMU space too (lib/env_dpdk/pci.c:751 through lib/env_dpdk/pci.c:773). In VA mode, SPDK uses the mapped virtual address as IOVA; in PA mode, it uses the physical address.

This is the concrete bridge between abstract diagrams and real source: SPDK does not merely "get a pointer." It arranges permissions so a device and process can safely exchange DMA and MMIO.

Here is the SPDK side of the VFIO DMA map path. The vaddr field is the userspace address of the memory range, iova is the address the device will use on the bus, and size is the span to authorize. The flags permit both device reads and device writes.

/* lib/env_dpdk/memory.c */
dma_map->map.argsz = sizeof(dma_map->map);
dma_map->map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
dma_map->map.vaddr = vaddr;
dma_map->map.iova = iova;
dma_map->map.size = size;

if (g_vfio.device_ref == 0) {
	/* VFIO requires at least one device (IOMMU group) to be added to
	 * a VFIO container before it is possible to perform any IOMMU
	 * operations on that container. This memory will be mapped once
	 * the first device (IOMMU group) is hotplugged.
	 */
	goto out_insert;
}

ret = ioctl(g_vfio.fd, VFIO_IOMMU_MAP_DMA, &dma_map->map);
if (ret) {
	/* There are cases the vfio container doesn't have IOMMU group, it's safe for this case */
	SPDK_NOTICELOG("Cannot set up DMA mapping, error %d, ignored\n", errno);
}

The deferred path is easy to miss. SPDK can discover or allocate memory before the first SPDK-managed VFIO device has joined the container. Rather than failing early, it records the mapping and applies it when a device arrives. That keeps memory ownership in SPDK while respecting VFIO's rule that IOMMU operations need a container with at least one attached group.

DPDK has a parallel VFIO type1 path for memory it manages. The fields are the same because they are the kernel ABI for VFIO_IOMMU_MAP_DMA.

/* dpdk/lib/eal/linux/eal_vfio.c */
memset(&dma_map, 0, sizeof(dma_map));
dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
dma_map.vaddr = vaddr;
dma_map.size = len;
dma_map.iova = iova;
dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
		VFIO_DMA_MAP_FLAG_WRITE;

ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);

BAR mapping has a related but distinct IOMMU step. A BAR is device memory, not host RAM, but SPDK still maps the BAR into IOMMU space when an IOMMU is enabled. In VA IOVA mode, SPDK uses the mapped virtual address as the IOVA. In PA mode, it uses the physical address reported by the underlying mapping operation.

/* lib/env_dpdk/pci.c */
rc = dev->map_bar(dev, bar, mapped_addr, phys_addr, size);
if (rc) {
	return rc;
}

#if VFIO_ENABLED
/* Automatically map the BAR to the IOMMU */
if (!spdk_iommu_is_enabled()) {
	return 0;
}

if (rte_eal_iova_mode() == RTE_IOVA_VA) {
	/* We'll use the virtual address as the iova to match DPDK. */
	rc = vtophys_iommu_map_dma_bar((uint64_t)(*mapped_addr), (uint64_t) * mapped_addr, *size);
	if (rc) {
		dev->unmap_bar(dev, bar, *mapped_addr);
		return -EFAULT;
	}

	*phys_addr = (uint64_t)(*mapped_addr);
} else {
	/* We'll use the physical address as the iova to match DPDK. */
	rc = vtophys_iommu_map_dma_bar((uint64_t)(*mapped_addr), *phys_addr, *size);

The callback path is therefore: setup binds the device to VFIO, DPDK opens the VFIO group and container, SPDK env maps BARs and memory through DPDK/VFIO, and the NVMe driver later writes doorbells and posts queue memory addresses that the controller can legally DMA to or from.

Kernel-Bound Versus SPDK-Bound Devices

Kernel-bound NVMe:

application -> syscall/io_uring/libaio -> kernel block layer -> kernel nvme driver -> device

SPDK-bound NVMe:

SPDK app -> SPDK NVMe driver -> VFIO/MMIO/DMA -> device

The second path avoids syscalls and kernel block scheduling on the I/O fast path, but it changes ownership:

The kernel no longer exposes /dev/nvmeXnY for normal use.
Filesystems mounted from that device must be unmounted first.
The SPDK process must reserve and own DMA-capable memory.
Operational tooling must use SPDK RPCs or NVMe passthrough paths instead of normal block tools.

Misconception to kill: "SPDK can use a mounted kernel NVMe disk directly." For local PCIe SPDK NVMe, the device is normally rebound away from the kernel NVMe driver. Sharing a mounted block device with SPDK direct hardware access would be data corruption territory.

NUMA

PCIe devices attach near a CPU socket. Memory also belongs to NUMA nodes. If the SPDK polling core is on socket 0, the NVMe device is behind socket 1, and hugepages are allocated on socket 0, every DMA and CPU access may cross inter-socket links.

Symptoms:

Lower bandwidth than expected.
Higher tail latency.
CPU cycles spent waiting on remote memory.
Performance changes when core masks or hugepage allocation changes.

SPDK setup has HUGENODE; DPDK and SPDK expose NUMA-aware allocation. The beginner operational rule is: align device, polling core, and hugepage memory when possible.

Page Faults And Pinned Memory

DMA cannot wait for the kernel to page memory in from swap. Device DMA buffers must be resident and mapped. Hugepages are reserved and pinned in a way that fits DPDK/SPDK operation. Normal malloc() memory may be unsuitable because:

it can be paged,
it may not have stable physical mappings,
it may not be IOVA-contiguous,
it may not be registered with VFIO/IOMMU.

SPDK has APIs for DMA-safe allocation; later chapters will cover iobuf, mempools, and memory domains.

Failure Modes

No IOMMU enabled: VFIO binding fails or DPDK requires unsafe no-IOMMU mode.
Wrong driver: device remains bound to nvme, so SPDK cannot claim it through VFIO.
Active filesystem: rebinding a live kernel device risks data loss.
IOMMU group not viable: another device in the group is still kernel-bound.
Hugepages missing: EAL initialization fails or SPDK cannot allocate DMA buffers.
Hugepages on wrong NUMA node: performance collapses rather than failing loudly.
RLIMIT_MEMLOCK too low: non-root process cannot lock enough memory.
DMA entry limit reached: many small mappings, especially with --no-huge, exhaust VFIO map entries.
IOVA mode mismatch: device receives addresses it cannot translate.
BAR mapping failure: process cannot access MMIO registers.
Page fault in data path: using non-DMA-safe memory creates failures or bounce-buffer overhead.

These failures tend to look unrelated at first because they occur at different layers. A missing hugepage may fail during EAL initialization before SPDK has even probed an NVMe controller. An invalid IOMMU group may appear as a VFIO attach failure even though lspci shows the target SSD. An IOVA mode mismatch may let setup succeed but cause later DMA failures because the controller is given addresses the platform cannot translate. Debug them in dependency order: kernel boot/IOMMU state, driver binding, group viability, hugepage reservation, memlock limits, EAL IOVA mode, then SPDK device probe.

The most dangerous failure is not a clean startup error. It is accidental shared ownership: a mounted filesystem, the kernel NVMe driver, and an SPDK userspace driver cannot safely coordinate direct writes to the same PCIe SSD. That is why setup ignores active mountpoints and why production runbooks should make device ownership explicit before binding changes.

Operational Lab

Do this as a dry run on a development machine; do not rebind production devices.

Pick a PCI device BDF from lspci.
Inspect its current driver:

lspci -k -s 0000:5e:00.0

Inspect its IOMMU group:

readlink /sys/bus/pci/devices/0000:5e:00.0/iommu_group
ls -l /sys/bus/pci/devices/0000:5e:00.0/iommu_group/devices

Inspect hugepage state:

grep -i huge /proc/meminfo
find /sys/devices/system/node -path '*hugepages*' -name nr_hugepages -print -exec cat {} \;

Explain whether the device could be safely rebound to VFIO and what other devices would be affected.

Source Reading Exercise

Read:

scripts/setup.sh:33 through scripts/setup.sh:104.
scripts/setup.sh:194 through scripts/setup.sh:213.
scripts/setup.sh:403 through scripts/setup.sh:418.
scripts/setup.sh:500 through scripts/setup.sh:607.
lib/env_dpdk/pci.c:740 through lib/env_dpdk/pci.c:775.
lib/env_dpdk/memory.c:1084 through lib/env_dpdk/memory.c:1127.
lib/env_dpdk/init.c:523 through lib/env_dpdk/init.c:547.
lib/env_dpdk/init.c:403 through lib/env_dpdk/init.c:428.
lib/nvme/nvme_pcie_internal.h:260 through lib/nvme/nvme_pcie_internal.h:296.
dpdk/lib/eal/linux/eal_vfio.c:790 through dpdk/lib/eal/linux/eal_vfio.c:825.
dpdk/lib/eal/linux/eal_vfio.c:1428 through dpdk/lib/eal/linux/eal_vfio.c:1445.

Answer:

Which script variables control hugepage allocation?
Where does setup warn about IOMMU groups?
Which driver does setup prefer?
What fields are passed to VFIO_IOMMU_MAP_DMA?
When does SPDK choose VA as IOVA for BAR mapping?
Why does the NVMe submission doorbell path issue a write memory barrier first?
What does DPDK mean by a viable VFIO group?

Self-Check

What is a PCI BDF?
What is a BAR?
Why is MMIO different from normal memory?
What does DMA let the device do?
Why is an IOMMU important for userspace drivers?
What does VFIO provide?
Why do hugepages matter for SPDK?
Why can an IOMMU group stop device binding?
What is the difference between VA, PA, and IOVA?

References

Local source: scripts/setup.sh.
Local source: include/spdk/env.h.
Local source: lib/env_dpdk/pci.c.
Local source: lib/env_dpdk/memory.c.
Local source: lib/env_dpdk/init.c.
Local source: lib/nvme/nvme_pcie_internal.h.
Local source: dpdk/lib/eal/linux/eal_vfio.c.
SPDK System Configuration User Guide: https://spdk.io/doc/system_configuration.html
SPDK Direct Memory Access From User Space: https://spdk.io/doc/memory.html
DPDK EAL programmer's guide: https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html
DPDK Linux system requirements, hugepage setup: https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html
DPDK Linux drivers guide, VFIO and IOMMU groups: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
Linux kernel VFIO documentation: https://docs.kernel.org/driver-api/vfio.html
Linux kernel IOMMUFD userspace API documentation: https://docs.kernel.org/userspace-api/iommufd.html
NVM Express specifications landing page: https://nvmexpress.org/specifications/