[OpenIndiana-discuss] No more interrupt vectors for 17th NVMe

[Till Wegmueller]

Hi,

I think you better look into this on either the illumos-developer list 
or Open a Bug on illumos.org/issues This seems like a new and 
incompatible hardware. Of course it can be made compatible, but that 
will need illumos developer work. Kernel devs are not on this list usually.

-Till

On 09.04.2024 10:06, andy at spasu.net wrote:
> Hello All
> 
> Apologies for posting a Solaris question - latest OpenIndiana refuses to
> install on this HW, panicking on install USB stick boot. But I presume that
> interrupt vector shortage problems must be similar on Illumos and Solaris.
> Suggestions on switching to Illumos highly appreciated.
> 
> Solaris 11.4.42.111.0 so far
> 
> Supermicro H12SSL MoBo with 16-core Epyc, 5 pcs. PCIe16v4/NVMe cards with 4
> NVMe slots on each, totally 20 NVMe slots on these PCIe cards and 2 on
> MBoard.
> 
> The idea was to populate all 22 NVMe slots to build a low-power fileserver
> with native ZFS and lightweight containers (non-kernel zones). However,
> Solaris didin't let me use more than 14 slots, complaining "No interrupt
> vector: pciex...". I looked at ::interrupts in mdb -k, and indeed, for level
> 5, four vectors were taken by ahci driver, and 28 by nvme - 2 vectors per
> populated slot. Edited /kernel/drv/ahci.conf, explicitly telling to use
> level 6 (where only bge and nge drivers sat), and so freed up 4 more vectors
> in level 5, which let me use 2 more slots, 16 NVMe's total. Still not
> exactly what I wanted. It's been quite a while since I touched the kernel...
> Is it possible to spread nvme onto 2 interrupt levels? What's the correct
> syntax in /kernel/drv/nvme.conf?
> 
> interrupt-priorities = 5,6;
> 
> Doesn't seem to have any effect - adding 17th NVMe stick results in the same
> complain at booting
> 
> Thanks
> 
> Andrew
> 
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> https://openindiana.org/mailman/listinfo/openindiana-discuss