[OpenIndiana-discuss] No more interrupt vectors for 17th NVMe

[andy at spasu.net]

Hello All

Apologies for posting a Solaris question - latest OpenIndiana refuses to
install on this HW, panicking on install USB stick boot. But I presume that
interrupt vector shortage problems must be similar on Illumos and Solaris.
Suggestions on switching to Illumos highly appreciated.

Solaris 11.4.42.111.0 so far

Supermicro H12SSL MoBo with 16-core Epyc, 5 pcs. PCIe16v4/NVMe cards with 4
NVMe slots on each, totally 20 NVMe slots on these PCIe cards and 2 on
MBoard. 

The idea was to populate all 22 NVMe slots to build a low-power fileserver
with native ZFS and lightweight containers (non-kernel zones). However,
Solaris didin't let me use more than 14 slots, complaining "No interrupt
vector: pciex...". I looked at ::interrupts in mdb -k, and indeed, for level
5, four vectors were taken by ahci driver, and 28 by nvme - 2 vectors per
populated slot. Edited /kernel/drv/ahci.conf, explicitly telling to use
level 6 (where only bge and nge drivers sat), and so freed up 4 more vectors
in level 5, which let me use 2 more slots, 16 NVMe's total. Still not
exactly what I wanted. It's been quite a while since I touched the kernel...
Is it possible to spread nvme onto 2 interrupt levels? What's the correct
syntax in /kernel/drv/nvme.conf?

interrupt-priorities = 5,6;

Doesn't seem to have any effect - adding 17th NVMe stick results in the same
complain at booting

Thanks

Andrew