SYS-2027GR-TRF: Has 4 x Nvidia Tesla K80-- GPU #4 in Perf mode P8 (but all other GPUs in P0) and has double-bit ECC unrecoverable errors and power and memory usage much lower than other GPUs - Random application crashes. Segmentation faults. Only happens on this server, other nodes run fine.
Double-bit ECC unrecoverable errors likely due to bad GPU. You may try to swap the GPU#4 that has double bit error with another working GPU to see if the problem follows the GPU#4 or not.