The following are benchmarks from our testings of our iSCSI SSD storage.
67,300 read IOP/s on a VM on iSCSI

(Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)
Per VM and scales to 1,000,000 IOP/s total

root@dev-samm:/mnt/pmt1 128 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=read
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
bs: 1 (f=1): [R] [55.6% done] [262.1M/0K /s] [67.3K/0  iops] [eta 00m:04s]
38,500 random 4k write IOP/s on a VM on iSCSI

(Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)
Per VM and scales to 700,000 IOP/s total

root@dev-samm:/mnt/pmt1 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
bs: 1 (f=1): [w] [26.3% done] [0K/150.2M /s] [0 /38.5K iops] [eta 00m:14s]
Raw device latency on storage units

Intel DC3600 1.2T PCIe NVMe

root@s1-san6:/proc  # ioping /dev/nvme0n1p1
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=1 time=104 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=2 time=83 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=3 time=51 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=4 time=71 us

SanDisk SDSSDXPS960G SATA

root@pm-san5:/proc  # ioping /dev/sdc
4.0 KiB from /dev/sdc (device 894.3 GiB): request=1 time=4.2 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=2 time=4.1 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=3 time=4.1 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=4 time=4.1 ms

Micron_M600_MTFDDAK1T0MBF SATA

root@pm-san5:/proc  # ioping /dev/sdf
4.0 KiB from /dev/sdf (device 953.9 GiB): request=1 time=157 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=2 time=190 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=3 time=65 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=4 time=181 us
```shell
## Latency on the a VM

- (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)

```shell
root@dev-samm:/mnt 127 # ioping pmt1/
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=1 time=0.6 ms
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=2 time=0.7 ms
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=3 time=0.7 ms

--- pmt1/ (ext4 /dev/xvdb1) ioping statistics ---
3 requests completed in 2159.1 ms, 1508 iops, 5.9 mb/s
min/avg/max/mdev = 0.6/0.7/0.7/0.1 ms
root@dev-samm:/mnt  # ioping pmt2/
4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=1 time=0.6 ms
4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=2 time=0.8 ms

--- pmt2/ (ext4 /dev/xvdc1) ioping statistics ---
2 requests completed in 1658.4 ms, 1470 iops, 5.7 mb/s
min/avg/max/mdev = 0.6/0.7/0.8/0.1 ms
root@dev-samm:/mnt  # ioping pmt3/
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=1 time=0.6 ms
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=2 time=0.9 ms
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=3 time=0.9 ms

Sam McLeod

Benchmark results for an iSCSI SSD storage stack running through a full virtualization chain (Disk → LVM → MDADM → DRBD → iSCSI target → Network → XenServer → VM). Read performance reached 67,300 IOPS per VM (scaling to 1M total), while random 4K writes hit 38,500 IOPS per VM (scaling to 700K total). Raw device latency comparisons are provided for Intel DC3600 NVMe (~51–104 µs), SanDisk SATA SSD (~4.1 ms), and Micron M600 SATA SSD (~65–190 µs). End-to-end VM latency through the full iSCSI stack measured around 0.6–0.9 ms using ioping.

iSCSI Benchmarking

38,500 random 4k write IOP/s on a VM on iSCSI #