Poking at the Linux block layer on a cloud VM
Read a good post on the Linux block layer today. It’s all diagrams and theory, no terminal, so I sshed into a cloud VM to see if the stuff it talks about actually shows up. Mostly it does.
the scheduler is just a file
Every disk has an I/O scheduler and it’s literally a file you can cat:
$ cat /sys/block/vda/queue/scheduler
[none] mq-deadline bfq kyber
Active one’s in brackets. Here it’s none, i.e. no scheduling at all. Which
makes sense once you think about it: this is a virtio cloud disk, there’s no
spinning platter to reduce seeks on, so reordering requests would just burn CPU
for nothing. On fast disks, doing nothing is the right call.
every cpu gets its own queue
This was the main point of the post. The old block layer had one queue behind one lock, so every core fought over it. blk-mq shards it: at submit time each CPU drops work in its own software queue (no shared lock), and those feed a set of hardware queues that the device actually drains.
The hardware queues are the ones you can see in sysfs, under mq/:
$ for d in sda vda vdb; do echo -n "$d: "; ls /sys/block/$d/mq/ | wc -l; done
sda: 8
vda: 8
vdb: 8
8 each, and the box has 8 vCPUs. Even vdb, the 1MB cloud-init drive, has 8 —
they follow CPUs, not disks. And each one maps to a single CPU:
$ cat /sys/block/vda/mq/*/cpu_list
0
1
2
3
4
5
6
7
So virtio gave me one hardware queue per CPU, which means the per-CPU split goes all the way down to the device — nothing shared anywhere. That’s the good case. A plain SATA disk would show just one hardware queue with all 8 CPUs feeding into it (the per-CPU software queues are still there, they just funnel into that single one).
a peek inside a queue
Couple more things while I was poking around (this bit’s in debugfs, needs sudo).
First, all the hardware queues are default type — blk-mq can split out separate
read and poll queues, but virtio doesn’t bother, so one type handles everything
here.
Second, each queue has a pool of tags:
$ sudo cat /sys/kernel/debug/block/vda/hctx0/tags
nr_tags=256
...
busy=1
A tag is just a slot for one in-flight request — grab one when you submit, hand
it back when it completes. nr_tags=256 is the real cap on how many requests can
be outstanding on that queue at once; run out and the next one waits. With 8
queues at 256 each, that’s up to ~2000 in flight before anything blocks.
the part that surprised me
While I was in there I ran fio to check if the scheduler actually changes anything. For reads, nope:
| Scheduler | IOPS @ depth 1 | IOPS @ depth 64 |
|---|---|---|
| none | 1,643 | 56,400 |
| mq-deadline | 1,626 | 56,488 |
| bfq | 1,592 | 54,896 |
| kyber | 1,605 | 55,969 |
All basically the same across schedulers. What did matter was queue depth — same disk, but going from 1 to 64 requests in flight took it from ~1.6k to ~56k IOPS. ~34x, just from keeping the queue full.
Writes were the surprise though:
| Scheduler | Write IOPS | p99 latency |
|---|---|---|
| none | 26,497 | 6.1 ms |
| mq-deadline | 40,439 | 4.0 ms |
| bfq | 32,868 | 5.8 ms |
| kyber | 33,794 | 5.7 ms |
Everyone says use none on cloud disks. But here mq-deadline did ~53% more write
IOPS than none, with a lower tail latency on top. Its batching clearly earns its
keep on this volume’s writes. So, actually run the numbers I guess.
anyway
Theory held up — none on the fast disk, one queue per CPU. And I picked up
something the post didn’t mention: mq-deadline can quietly beat none on writes,
which I wasn’t expecting.