Re: RAID 5 Extremly slow write speed!?

webadmin@intel.com — Wed, 27 Jun 2012 18:53:28 GMT

Hi,

1) if you have a UPS backup, you should -> Computer Management->Device Manager->Disk Drives. Bring up the property page for your raid disk and select Policies. Make sure that both Enable write-caching is on and also select "Turn off Windows write-cache buffer flushing on the device". Without buffer flushing you should see performance ~200-250MB/sec. With buffer flushing you are probably writing slower than a single drive. Windows will flush everything when you do a shutdown. Also, see the Windows Sysinternals package for a command to "sync" a device.

The reason for the UPS is that you need a means to protect your disk data (including raid metadata) from being lost. If your system has other reliability issues that cause system crashes, you should let Windows flush its write-cache buffer. Also, you might not need an expensive UPS, but be aware that cheap UPS's can be the flakiest device that you can plugin. There's nothing more disconcerting to me than to see a UPS quit even when there is line power.

2) 110MB/sec is not very good either. You might check the other machine's buffer flush settings.

3) You are likely losing some performance because your raid disk drives aren't all the same model. Even though the consumer description may say 7200RPM, 16MB disk cache and SATA-2, there are different access times and effective transfer rates from model to model and from vendor to vendor. You will likely find that one vendor's drives perform better/worse in this setting.

Oh yes, don't use the so-called "green" drives for raid-5 and make sure your setup BIOS is configured to power up drives with a ~2 sec delay. You'll wind up watching a disk get reconstructed when a drive drops out of the raid configuration.

Art

Re: x540 10G direct connection

webadmin@intel.com — Mon, 25 Jun 2012 09:12:55 GMT

Hi,

I directly connect 3 Windows 7/64 boxes using T520-T2. Each box's port 1 is connected to another box's port 2. Takes just 3 cat 6 wires. More work is involved configuring Usually there's some other network to the outside world that has a gateway, and only one gateway per system is allowed (a TCP/IP "requirement"). Using windows network manager you configure each port but don't specify a gateway for these devices. You'll write a couple of routes specifying which port is talking to each of the other two systems. You may already know you can't use the same subnet for two different NICs on the same node.

Below, 10.0.x.y: x is the port number on this node, and y is the node number. Thus, are two subnets.

Node 1 port 1-- 10.0.1.1 -> Node 2 port 2 -- 10.0.2.2

Node 2 port 1-- 10.0.2.1 -> Node 3 port 2 -- 10.0.2.3

Node 3 port 1-- 10.0.3.1 -> Node 1 port 2 -- 10.0.2.1

You have to add these IP addresses to \Windows\System32\drivers\etc\hosts, and this file will be different on each node.

Using admin privileges add to each node the permanent routes from both NICs to the to NICs that are on the other two nodes.

When you have done the above, it's just easier to boot all the nodes especially for the sake of the routes.

Once done, you can use conventional Windows network apps on these connections. You might start just by doing a ping of the other two nodes from each node.

My own application interest includes runing MPI using the x520-T2 network, and Intel's MPI and MPICHv2 both work. Within one specific applicationI have witnessed both NICs running in parallel with inbound traffic exceeding 30 gbps and outbound over 10 gbps. I have increased several of the settings for each of the NICs to provide for more send/receive buffers and use jumbo frame size = 9014. I haven't been able to get access to running the largest frame size.

BTW my nodes are dual boot: the other platform is CentOS and the performance of the same app is comparable using Intel's MPI. So far, I'm unable to get OpenMPI to work on this point-point network.

This past week I tried a different view of the subnets by assigning a subnet to each of the wires. Because of the OpenMPI problem, I wanted to try something else. The Linux version and Intel MPI work, but OpenMPI's behavior is unchanged. I'll be testing the Windows 7 side of the nodes this week.

For the subnet/wire approach, for 10.0.x.y, x= wire number and y = node number. The wires remain plugged into the same NIC ports as before.

Wire 1 Box 1-- 10.0.1.1 -> Box 2-- 10.0.1.2

Wire 2 Box 2-- 10.0.2.2 -> Box 3-- 10.0.2.3

Wire 3 Box 3-- 10.0.3.3 -> Box 1-- 10.0.3.1

Changes to \Windows\System32\drivers\etc\hosts and routes have to be updated to support this configuration.

In the 3 node configuration, each node has a left and right node, and each node is only 1 "hop" away. While one can extend this ring further, the hop count would grow. If I were to add a 4th node, I'd look into doing a hypercube connection topology.

Art

Intel Communities: Message List

Re: RAID 5 Extremly slow write speed!?

Re: x540 10G direct connection