Indeed, 32GB/s is what I compute for the theoretical throughput of the memory interface.  However, when you have a high-performance computing application that is doing data streaming from memory, there should be very little else happening in the OS -- certainly not enough to cause a 50% drop (32GB/s to the measured 17-18GB/s or so).

Yes, the benchmarks show a small drop with lower bus frequency, but even 1333MHz doesn't get 32GB/s.  See here:

http://www.advancedclustering.com/company-blog/stream-benchmarking.html

I was told by a few people that it can have to do with a number of issues inside the CPU itself, i.e. that the CPU can't generate requests fast enough, or that the memory can't keep the memory bus full even when it is streaming due to various latencies.  We concluded that the second is probably not true, and that various DIMM banks are synchronized when streaming to hide the latencies, so the bus is always full.  Some other possibilities are that reading and writing at the same time were an issue, but tests of only reading and only writing also give similar throughput to the other benchmarks -- hence don't explain the problem.  Finally, dual-socket benchmarks give 37GB/s (vs theoretical of 2x32 = 64GB/s) at 1333MHz, and this part could be because of cache coherence -- one CPU having to look in the cache of the other before it goes to memory.  Nehalem seems not to have a probe filter ("snoop filter") that can hide this latency, like AMD Istanbul does.  Which may explain the slowdown.  However, even a single-socket Intel CPU gets 18GB/s-ish.. so it's still not explained.

I was hoping that someone who understands Nehalem and/or AMD architecture could explain exactly what's going on -- and whether it's the CPU/memory controller, or the DRAM modules, that are limiting the bandwidth, and why.

Milos