Diagnose and resolve high latency on Infiniband setup on Windows Server 2016

I have installed Mellanox ConnectX-3 cards in 2 Windows Servers. I have a running IB network with opensm as subnet manager and direct connection between the two servers. Throughput is "fine" at 2500MB/s when tested using ntttp. However, latency is the problem. I have ~.3ms ping times (same as my 1GB ethernet). and when running diskspd against a network share I get 10X latency and 1/10th the IOPS I get when accessing the disk on the host computer directly.