Connecting from NVMe-oF initiators

Before you can use the benefits of the Ceph NVMe-oF gateway, you must install and configure an NVMe-oF gateway by using command-line interface.

Connecting from Red Hat Enterprise Linux

Verify that the target is reachable from the initiator.
```
nvme discover -t tcp -a GATEWAY_IP -s 8009
```
GATEWAY_IP can be fetched from GET /hosts API.

Connect to the NVMe-oF target.

nvme connect-all -traddr=GATEWAY_IP --transport=tcp -l 1800

Verify that the initiator is set up correctly for RHEL

List the NVMe-oF block devices.

nvme list

See the following example.

Node                    Generic           SN                   Model                   Namespace Usage                      Format           FW Rev

---------------------   ----------------  -------------------  ----------------------- --------- -------------------------- ---------------- --------

/home/nvme01_node01     /home/ng1n1       SPDK00000000000001   SPDK bdev Controller    1          10.74  GB /  10.74  GB      4 KiB +  0 B   23.01

...

Create a filesystem on the desired target, found in step 1.

mkfs NVME_NODE_PATH

See the following example.

$ mkfs /home/nvme01_node01
mke2fs 1.46.5 (20-Dec-2023)
Discarding device blocks: done
Creating filesystem with 2560 4k blocks and 2560 inodes

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

Mount the NVMe node on the NVMe-oF directory.

mkdir /mnt/nvmeof

mount NVME_NODE_PATH /mnt/nvmeof

See the following example.

mount /home/nvme01_node01 /mnt/nvmeof

Using sudo commands, list mounted NVMe-oF files.
```
ls /mnt/nvmeof
```
For example,
```
$ ls /mnt/nvmeof
lost+found
```

Create a text file within the "mnt/nvmeof" directory.

sudo bash -c "echo Hello NVMe-oF > /mnt/nvmeof/hello.txt"

Verify that the text file can now be reached.
```
$ cat /mnt/nvmeof/hello.txt
Hello NVMe-oF
```

Connecting from VMWare ESXi

List the NVMe-oF adapter.

esxcli nvme adapter list

See the following example.

[root@host01:~] esxcli nvme adapter list

Adapter  Adapter Qualified Name           Transport Type  Driver     Associated Devices
-------  -------------------------------- --------------  ---------  ------------------
vmhba64  aqn:nvmetcp:ac-1f-6b-0a-18-74-T  TCP             nvmetcp    vmnic0

Discover any NVMe-oF-gateway subsystems.

esxcli nvme fabrics discover -a NVME_TCP_ADAPTER -i GATEWAY_IP -p 8009

See the following example.

[root@host01:~] esxcli nvme fabrics discover -a vmhba64 -i 10.0.211.196 -p 8009

Transport Type Address Family Subsystem Type Controller ID Admin Queue Max Size Transport Address Transport Service ID Subsystem NQN              Connected

-------------- -------------- -------------- ------------- -------------------- ----------------- -------------------- -------------------------- ---------

TCP            IPv4           NVM            65535         128                   10.0.211.196     8009                 nqn.2016-06.io.spdk:cnode1  false

Connect to NVMe-oF gateway subsystem.

esxcli nvme fabrics connect -a NVME_TCP_ADAPTER -i GATEWAY_IP -p 4420 -s SUBSYSTEM_NQN

See the following example.

[root@host01:~] esxcli nvme fabrics connect -a vmhba64 -i 10.0.211.196 -p 4420 -s nqn.2016-06.io.spdk:cnode1

List NVMe/TCP controller list.

esxcli nvme controller list

See the following example.

[root@host01:~] esxcli nvme controller list

Name                                                                                        Controller Number  Adapter  Transport Type  Is Online
------------------------------------------------------------------------------------------  -----------------  -------  --------------  ---------
nqn.2016-06.io.spdk:cnode1#vmhba64#10.0.211.196:8009                                                      301  vmhba64  TCP                  true

List NVMe-oF namespaces in the storage_identifiers.

esxcli nvme namespace list

See the following example.

[root@host01:~] esxcli nvme namespace list

Name                                Controller Number  Namespace ID  Block Size  Capacity in MB
------------------                  -----------------  ------------  ----------  --------------

eui.0100000001000000e4d25c00001ae214               256             1         512          953869

eui.01abc123def456g7e4d25c00001ae214               301             1         512             500

eui.02abc123def456g7e4d25c00001ae215               301             2         512             500

eui.03abc123def456g7e4d25c00001ae216               301             3         512             500

Verify that the initiator is set up correctly for VMware ESXi

From the vSphere Client, go to the ESXi host.
On the Storage page, go to the Devices tab.
Verify that the NVMe/TCP disks are listed in the table.