We have an issue with a Supermicro server model A+ Server 2022TG-HTRF (2U Twin), AMD Opteron 6278 2.4GHz 16Cores processors and Mellanox Infiniband QDR (model MCX353A-QCB).

We are trying to deploy a cloud environment with OpenNebula, so we want to virtualize the infiniband device using sr-iov capabilities.

But we've found error messages when booting:

$ dmesg
May 9 08:52:28 samd3 kernel: [ 5.805295] mlx4_core: Mellanox ConnectX core driver v1.1 (May 7 2013)
May 9 08:52:28 samd3 kernel: [ 5.805304] mlx4_core: Initializing 0000:03:00.0
May 9 08:52:28 samd3 kernel: [ 5.805355] mlx4_core 0000:03:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
May 9 08:52:28 samd3 kernel: [ 5.805412] mlx4_core 0000:03:00.0: setting latency timer to 64
May 9 08:52:28 samd3 kernel: [ 5.805481] mlx4_core 0000:03:00.0: Enabling SR-IOV with 8 VFs
May 9 08:52:28 samd3 kernel: [ 5.805572] mlx4_core 0000:03:00.0: not enough MMIO resources for SR-IOV
May 9 08:52:28 samd3 kernel: [ 5.805579] mlx4_core 0000:03:00.0: Failed to enable SR-IOV, continuing without SR-IOV (err = -12).

The infiniband device has SR-IOV support, so we concluded we have to enable SR-IOV support in BIOS, but our current version (2.X) doesn't have this support.

$ sudo lspci -v
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies Device 0017
...
Capabilities: [108] Single Root I/O Virtualization (SR-IOV)

We installed a new BIOS version, 3.0, downloaded from the Supermicro support web (http://www.supermicro.com/support/bios/?MoboType=2 -> motherboard H8DGT-HF) but it hasn't the SR-IOV option in the PCIe - PnP BIOS menu. After that, a Supermicro spanish partner send us a BIOS 3.0A revision with the SR-IOV option but it didn't work for us. After enabling this option the system didn't boot, and we had to unplug the infiniband hca, and disable SR-IOV.