7 Replies
Last post:
Oct 21, 2009 10:40 AM by
pmcfee
Average User Rating
(0 ratings)
Yes.. .dozens of times:)
****
Background:
I am a Senior Solutions Architect who helps our Engineers at our integration facility to find resources to help when new product is having issues, or where complex designs require a more solutions based set of technical resources. I only supliment the already great and talented resources who debug and work every day with the IBM Modular product line (Intel). Any time a new product is released, I ask to be informed so I can come get some 'hands on' the new systems so I can keep my knowledge of the changing technologies sharp.
***
The change to the Nehalem processor marks a major change for anyone who has to deal with integration. I had been getting feedback from out facility of more then "the usual issues" of a new product so I made time to go over and take an example and document it. This may not reflect all issues and as any new product is released, is sure to be worked out. I have to say up-front that the issues are NOT in any way sestemic of a core technology issue or even critical in nature. IBM and Intel have for years done very well at avoiding and midigating those level issues. What is reflected below are things that, IMAO, need to be reviewed to see if there are ways to make improvement. These oppinions reflect the integration and setup aspect of the system and also do not capture long term runtime experiance (always looking for demo gear to prove that out for our labs 
****************
System: HS22 Model 7870
Product ID: 7870C3U
SN: (not listed to protect the inocent)
System Specs:
1Socket (quad core)
RAM
PC3-10600 PN:43x5061 (qty 6)
Areas of Improvement:
1) System Integration times: These are how long it takes us from time of unboxing to where we can test, update and ship a box. This "T Time" has more than doubled. This impacts how quickly we can meet customer demands for shipping. Most of the issues stem from boot times. From the time the hardware finished being installed and power is activated to the point where a technical person can interact with the system for loading (or updating). This has more than doubled and has the greatest impact on how long it takes to debug and root cause a hardware issue.
Blade placed into chassis and powered: 0.00
Chassis Acceptance of Blade: 2:12
First screen: 34sec
BIOS Options: 52sec
Legacy boot Option (select media for loading): 2:24
(Total Time to get to be able to select boot target to load OS: 3:16)
2) System Boot Error from factory and (I believe the cause of long boot times after initial flash) "PXE-99: Unexpected Network Error". I did not have time to further root cause this with any kind of sniffer on the Blade Switch.
3) RAM Debug Issues: With the new processors binding to RAM, their is a "difficult" process to debug bad RAM and to replace it. Due to the long boot cycles, if the system is NOT being stable, RAM is the first (and most usually) the culperate. Errors on RAM with the new form factor are definatly higher. The comments I have relate to a few things about how this impacts the Nehalem systems. BIOS does not always indicate slot and so 16DIMMs x 3:16min per boot cycle make root cause of DIMM very laborious. When a DIMM is noted as being bad, it can be easily replaced BUT the system locks out the slot. This requires that the battery is pulled from the system board for a 10sec or so to clear the error. This also has impacts on EFI / BIOS settings and configurations that will cause issues for customers. This also impacts systems with the "minimum" DIMM configuration sets where they will see an impact in system core speed and boot capabilities due to the RAM failure. (I did not personally see this but errors are indicated in the BIOS as to the slot and I saw no indicators on the planar board for indication of bad DIMM, such as light path, I need to confirm this, and document how a customer can check if their system "changes" are due to bad DIMM)
2) IPMI Issues: Many customers us IPMI to manage and monitor system components. The Blade Managment Module does have the ability to affect the IPMI definition of the HS22 systems, nor can you make the changes (view only at this time) in the BIOS.This is even more of an issue when the systems are laid down with VMWare Hypervisor which does not have an OS level agent to report system events back and so relies purly upon IPMI for reporting and statistics. This is actually such an issue that it is in debate if customers will accept shipment of product as it is not "properly configured". {Ex: BSMP IP Address Range 192.168.70.200 host IP was set but can not change to reflect our customer's IP range set of 10.x.x.x)
3) Missing Serial Numbers: We still on occation receive units which have no Serial number. When you insert them into the chasis, the managment module shows blank data for the model and serial number. The only fix at this time due to the IPMI issue listed above is to declare the unit 'bad" and replace the entire system. This imediatly throughs an entire day of productivity out, at best... assuming we have spares in stock. Though I did not see this process, they had this same issue on the x3650M2 Nehalem systems but there is a fix of using a flash utility from IBM. The issue with the blade units is being worked on by IBM but no ETA.
4) Logic and understanding of EFI and its relation to "Legacy boot" option. It is likly a learning issue, but in the last 10min I had that I could work on this topic I tried to learn how to make some of what I would classify as basic changes to the system to impact boot times and boot targets (such as SAN boot, iSCSI targets, setting BOOT for ONLY Fibre HBA and then impact the boot option for the ONE time that the initial OS would be loaded). I was not able to lay this out and need to play with it. The Engineer did explain that there seems to be some level of reliance on the "Legacy boot" option that has to be used and worked with that has not been fully worked out.
Good things: I like to make sure to also take note of things the Engineers say "that is great!"
1) RAID Capabilities in BIOS saves enormous amounts of time and can allow use to quickly setup what is almost always basic RAID 1 for hardware integration and testing
2) System performance is VERY impressive. I have not had any HPC builds to do any "kicking of the tires" on my own but the system performance for the little time I have had my hands on them (post BOOT) have very fast load and application times for VMs (which is much of what we design.
3) Legacy boot option options are still very helpful.
I hope this helps. Look me up if you have any additional questions or points of clarifications you are needing.
