The drives, you divide by spindle count, I think. I would recommend at least two different Storage Pools.
I haven't seen the NICs become a chokepoint for virtual machines, but you are right - there is no way to send and recieve traffic on all four simultaneously
3 Gb SAS is standard, and like the NICs, usually isn't a bottleneck.
I have the same "question" and this is the reponse from INTEL Support:
"The connection bus from SAS disk to Storage Control Module (SCM) is not 3Gbps. It is a line from each of SCMs to each of SAS Expander and then from Expander to Backplane.
The bottle neck is the connection from SCM to each Compute Module. It is limited to 3Gbps, so request from one Compute Module to the drive will be limited by those 3Gbps. Even using external Storage you will be limited with those 3Gbps to each Compute Module.
Anyway it is impossible to reach 500MB/s for now. Necessary changes are under development but there are no timelines of fix for now.
On the other hand we have SSD disks validated because those disks have several advantages vs HDD: SSD drives are more fail-safe. No vibration issues can occur, so PFA state will occur rarely on SSD drives. SSD drive anyway will be faster than HDD one even in current configuration. Actually you will not always reach even those 3Gbps due to some other factors and those 3Gbps will not be a bottle neck all the time."