Set the following BIOS settings to get max throughput on Mellanox InfiniBand HDR (or 200Gb/s) or InfiniBand HDR100 or EDR gen4 PCI


NPS Configuration

Advanced->ACPI Settings->NPS2


CPU configuration

Advanced->CPU Configuration->SMT Control->Disabled
Advanced->CPU Configuration->L1 Stream HW Prefetcher->Enable
Advanced->CPU Configuration->L2 Stream HW Prefetcher->Enable


BIOS Performance Mode

Advanced->NB Configuration->SMEE->Disabled
Advanced->NB Configuration->Determinism Control->Manual
Advanced->NB Configuration->Determinism Slider->Performance


Preferred IO Device

Advanced->NB Configuration->Preferred IO->Manual
Advanced->NB Configuration->Preferred IO Bus->41


Preferred IO Bus value comes from the Mellanox Card PCIe BUS number from the following command and 41 is the IO bus number for the Mellanox card

root@super2:~# lspci |grep Mel
41:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
41:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]