Drive io compared
Drive IO Compared
Something that is really hot right now is using Ceph in a homelab. Proxmox VE has Ceph options built in. I see a lot of people comment online with a few warnings:
- Ceph doesn't like Consumer SSDs.
- Ceph needs faster network, say 10Gbps.
However, what I don't see often is any data on IO operations. I don't have the hardware to do 10Gbps, or a 3 node cluster of ceph, or enterprise SSDs. Below are my findings of IO data that I mostly found around the internet, and some I ran myself.
ZFS Raw vdisk on Sata SSD on Proxmox
These results were taken from a Windows 11 virtual machine using CrystalDiskMark.
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| SEQ1M Q8T1 | 1175.17 | 946.10 | |
| SEQ1M Q1T1 | 1144.75 | 1054.35 | |
| RND4K Q32T1 | 38.77 | 34.37 | |
| RND4k Q1T1 | 28.61 | 25.18 |
Bare metal NVMe drives
I found someone online that documented a lot of IO data. There website is here.
They have a bunch of different tests, check out their website for more info. Below is the data from them.
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| SEQ1M Q8T1 | 3563.62 | 2491.86 | |
| SEQ1M Q1T1 | 2420.27 | 2493.54 | |
| RND4K Q32T1 | 576.30 | 487.26 | |
| RND4k Q1T1 | 51.35 | 187.03 |
Passing through NVMe drive to a virtual machine.
According to the data from the site (link above), this is a drive being passed through to the OS, not a virtual disk (vmdk,qcow2, etc).
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| SEQ1M Q8T1 | 1257.19 | 992.73 | |
| SEQ1M Q1T1 | 1220.09 | 1224.80 | |
| RND4K Q32T1 | 30.61 | 28.36 | |
| RND4k Q1T1 | 24.61 | 23.36 |
Passing through NVMe controller
This data is from the same site as above.
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| SEQ1M Q8T1 | 3575.33 | 2463.65 | |
| SEQ1M Q1T1 | 2003.00 | 2351.24 | |
| RND4K Q32T1 | 56.80 | 51.03 | |
| RND4k Q1T1 | 26.62 | 44.57 |
Interesting to see that the results are much better when passing through the NVMe controller. I wonder if that would be the same in Proxmox; it looks like the data was taken from unraid.
40Gbps on Truenas Scale over iscsi on NVMe drives
That same site from above has this post that has IO data for 40Gbps network on Truenas Scale using iscsi to connect to a server with NVMe drives.
Here is more data on the setup and below is the data.
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| SEQ1M Q8T1 | 4682.43 | 1930.93 | |
| SEQ1M Q1T1 | 784.58 | 469.84 | |
| RND4K Q32T1 | 95773.32 | 57353.15 | |
| RND4k Q1T1 | 332.37 | 551.89 |
It's interesting to compare this to the bare metal single NVMe drive above since in every metric it is faster. Latency is something that wasn't measured which is important when you are dealing with network attached storage.
Ceph
Here is really want I want to get more data on. The site above again has great data on this here is the post about their ceph cluster. There is lots of data on there I'm just going to condense The final test results.
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| seq | 1591 | 6523 | |
| random | 1109 | 6523 |
lets compare it to the raw vdisk in proxmox that is on a zfs sata drive:
| Workload | Read (MB/s) | Write (MB/s) | Notes |
|---|---|---|---|
| SEQ1M Q8T1 | 1175.17 | 946.10 | |
| RND4K Q32T1 | 38.77 | 34.37 |
As you can see the data here shows that the ceph cluster had better IOPs than the local sata vdisk. However the author of the blog post says:
Compared to running a normal ZFS pool, the performance is pretty bad. However, for the workloads I am running, the level of performance is perfectly adequate.
I would love to know more about this configuration and see the data from it. I think the latency data would be important as well.
Improvements with the data
There are some improvements with the data collected that I'd love to see:
- There is a mix of hypervisors involved. It would be nice to see all the data on the same hypervisor.
- There is a mix of tools to test, CrystalDiskMark vs fio mainly. I would prefer all the data to be from fio.
- All the CrystalDiskMark data is missing latency, which can be very important for network attached storage.
Future Tests
Here are some future tests I'd love to see:
- zfs raw vdisk on an NVMe drive.
- CephFS vs RBD
- More data on latency for all the CrystalDiskMark tests.