I have installed a K80 to 7048GR-TR. I found the GPUDirect P2P(peer-to-peer) between the two GPU GK210 inside K80 is not getting expected performance. Is there a fix for this problem?
A: The ACS(access control services) need to be disabled for proper P2P bandwidth. The latest BIOS will disabled ACS of the PLX PCI-e switch inside the K80 to avoid the problem(Please contact tech support for BIOS later than BIOS 1.0c 2015/2/25). However, user is still encouraged to confirm ACSCtl is disabled in Linux due to the operating system can still enable ACS.
Pleaes run lspci -vvv and make sure you get "ACSCtl: SrcValid-" instead of "ACSCtl: SrcValid+" for PLX PCI-e switch.
If you get "ACSCtl: SrcValid+" for the PCI bridge: PLX Technology, run "setpci -s bus#:slot#.func# f2a.w=0000" to disable ACSCtl on the PLX switch.
e.g.
#setpci -s 03:08.0 f2a.w=0000 *assume 03:08.0 is the PLX pcie switch*
run lspci -vvv to confirm ACSCtrl is disabled on PLX switch.
#lspci -vvv | grep ACSCtrl
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
the ACSCtl setting will be reset in next boot, so please schedule the system to do the same setting at every boot.
Print
Share