Wow. That’s bizarre.
The top-end speed I’ve measured was about 8 or 9 MegaBYTES per second on a single file.
The fact that you get increasing speed with increasing “threads” is telling. It almost sounds like there’s a TCP driver issue; but the fact that you’ve tried so many clean builds with the same results would tend to nullify that, and the fact that you’ve moved from the NIC to the LOM and getting the same results would tend to toss out a kernel driver issue.
But just for giggles, make sure that the Kernel Module being loaded with the NIC versus the LOM are actually different.
SAMBA’s default configs should give you AMPLE performance; no limitations there.
Go look at your Network Stats. Make sure they’re absolutely CLEAN as a whistle:
[root@mars ~]# ethtool -S eth0
Everything below and including the line “collisions” would be considered an error. If they’re non-zero numbers, there’s something going on.
If a packet gets tossed due to ANY kind of error, it can slow down TCP performance DRAMATICALLY.
Also, use “netstat -s”
31 active connections openings
8 passive connection openings
9 failed connection attempts
1 connection resets received
2 connections established
15007 segments received
13371 segments send out
54 segments retransmited
0 bad segments received. 11 resets sent
My numbers are pretty low here because I had to reboot my Linux box last night and I haven’t really done much on it since then; but the interesting number here is the “Segments Retransmitted.”
A high percentage of Segments Retransmitted versus Segments Send Out is a bad sign It doesn’t have to be ZERO (though that would be ideal) but for every retransmit, that indicates an error or a timeout which slows down the thread.