1. IT departments will be embroiled in IT transformation from now on.

2. Private Clouds proliferate

3. IT shops are surprised at the number of corporate applications in the public cloud

4. MicroServers will be a bigger part of the market than 10% by EOY 13

5. Mission Critical will continue the steady migration to virtualized infrastructure

Prediction #1:  IT departments will be embroiled in transformation because they will now have competition.  Their customers will go to the public web if they can’t get support from the corporate shop.  This transformation is already happening.  One shop I’ve talked with is striving to provide their business unit customers with rapid deployment of virtual servers.  The old way of forcing these business units to wait up to six months or more for a server are rapidly ending.

When a business unit is told they’ll have to wait, the business opportunity that they are trying to tackle might just slip away.  Instead of letting that happen they’ll go to a public cloud provider and get a server in minutes.   One article recently referred to this as ‘Rogue Clouds’.  (See prediction #3 above!)

Of course IT shops don’t want to lose control but the proliferation of alternatives for their customers is turning all this around.  IT shops have to transform.   IT shops have to transform to remain relevant.

To do this they are choosing to move to a homogenous environment of low cost 2 socket servers or even MicroServers (see prediction 4 above).  By having a standardized environment on which to build a highly virtualized compute farm or even private cloud (See prediction #2 above) the IT shop can provide flexibility and high availability to their customers.   Again, a shop I’ve talked to is using VMware’s vSphere to provide high availability to their customers.  They say that they have better availability and uptime at a much lower cost than they were getting on their AIX environment.   Suddenly Mission Critical in the private cloud starts to make sense.  (See prediction #5 above)

Let’s build out that AIX example.  Say the AIX server is hosting 100 virtual machines in combinations of LPARS and virtual software.  Let’s say that there is an unanticipated fault that brings down the entire machine.  Unless these virtual machines are clustered to a virtual machine on another server, all 100 have just gone down.  To bring the machine back up, the structure of the virtual machines has to be recreated followed by restarting restored images of the 100 servers.  That’s going to take a while.   One choice to avoid this is to purchase or lease a second AIX server to cluster as a passive high availability host.   The impact to the corporation could be hours of outage or costs in the 7 to 8 figures for the clustered solution.

Another choice to port the applications to a virtual server in a private cloud where high availability can be selected as a configuration option while instantiating the server.   If the host server happens to fail then the server is instantiated on another identical server immediately; no need to await configuration of the server.   The net impact to the corporation is possibly a second or so of outage.  The cost of the 2 socket server isn’t very high and replacing it in the rack is a choice as having spare servers isn’t an expensive proposition.   And now with server disaggregation (yes, Intel is a collaborator) the task of configuring server farms is becoming simpler than assembling a model using toy blocks.

The role of IT is rapidly changing.  The role of the IT staff is changing.  The role of the server is changing.  The architecture of different brands of servers is losing any distinction.  The change is occurring at an ever rapid rate.  The only thing we in IT can be assured of is that the ground under our feet is shifting and flexibility is a guiding principle for our careers.

Follow me on Twitter @WallySP for more!

If you are using legacy UNIX infrastructure to host those applications…

What is a Mission Critical application?

Everybody has their own definition, regardless of how the definition is framed it boils down to the applications that are required to run the business; if the application is not running then the mission of the business is threatened or even shut down.  Consequently mission critical applications are hosted on the most robust and redundant hardware platforms available.

Thirty-five years ago it was unthinkable to host such an application on anything but a mainframe.  Midrange servers existed but only the VAX VMS Cluster could sport the high availability requirements to rival the mainframe.   Mostly these servers, if used in business, were used for departmental operations.   Mini computers were relegated to the periphery of the business.

Then SMP (Symmetric Multi-Processing) servers began showing up in the 90’s allowing for large UNIX servers and the configuration of servers into high availability architectures.  Processors used for these servers ranged from proprietary RISC processors like SPARC or POWER to industry standards processors like 386 or 486 processors and eventually the early Pentium processors from Intel.  Lower costs compared to the mainframe led IT departments to host Mission Critical applications on these UNIX servers. The last ten years have seen another shift.  These large UNIX servers are increasingly being supplanted by lower cost servers sporting high performance and reliable Intel Xeon processors.   The chart below from Intel’s promotion of the Itanium Processor dramatically shows this shift.

But the shift in IT spending isn’t the only message from this chart.  The chart shows the amount of IT spend for hosting Mission Critical applications but also shows that the total amount spent for hosting Mission Critical applications has GONE DOWN over the last decade!  In 2001 IT was spending $58.136B for hosting Mission Critical applications.   In 2011 the spend for Mission Critical applications was $56.5B.  That’s about $1.6B less in 2011 than in 2001!  In addition, it shows that about 26% of the amount spent, that spent for RISC Servers, is for supporting only about 3% of the hardware in the Data Center.

How can this be?  The answer is that IT Directors are increasingly turning to Intel Xeon based servers for hosting their Mission Critical applications because as we all know there are an ever increasing number of Mission Critical applications in the business.
How is it possible that the number and scope of Mission Critical applications has skyrocketed in size but the amount of money spent by IT to run these applications has gone down?

The answer’s lies in Moore’s Law.

Basically Moore’s Law says that the numbers of transistors on a processor will double every 18 to 24 months.   This means that the capability of a processor to process data increases significantly every 18 to 24 months.   The low cost Intel Xeon processor has been adding functionality along with speed, and has been capable of running just about any Mission Critical application in IT’s portfolio. So the capability and performance of the Intel processors increases with Moore’s Law.  Instead of doubling the size of a processor every 18 to 24 months, Intel is relentlessly driven to shrink the manufacturing process of the features of the processor.   Shrinking feature size is another way to achieve high manufacturing volume while keeping the processor size similar to its predecessor all the while doubling the numbers of transistors.  By manufacturing a high volume of processors Intel is able to sell these processors at a reasonable price, near the price point of the previous generation.

Shrinking the manufacturing process of the processor features each generation requires new machinery in a whole new processor fabrication facility.  To construct a new fab today is running from $3 to $5 Billion each and it is estimated that the costs of a 300mm fab will rise to $10 Billion soon.   The chart below shows that the number of companies with their own fabs will shrink in the next generation.

So what does this have to do with your paying too much for your Mission Critical compute infrastructure?  It’s pretty simple.  Intel makes hundreds of millions of chips each generation.  Fab costs, the costs of ramping up a new process technology, and R&D costs are amortized across hundreds of millions of processors allowing Intel to keep the costs for current generation processors and succeeding generations down.

All developers of processors face these issues in the development and manufacture of the processors they design.   For companies making only hundreds of thousands or even just a few million processors, the fixed costs of a fab, the variable costs of ramping up a new process, and the variable costs of R&D have to be amortized across just those processors.  With fewer processors to spread the costs across, something has got to give.  Either the manufacturing process has to stay the same as the previous generation or the price has to go up to absorb the costs.

Instead Intel has been able to provide increased performance and features generation after generation while keeping the price to the consumer at a reasonable level.  Most IT managers are aware of this.  They are faced with needing to meet ever tightening Service Level Agreements while their budgets are being kept flat or even being reduced.  These managers are able to meet these demands by putting their Mission Critical application on servers built on the reliable and powerful Intel family of Xeon processors.

You have 60 UNIX servers from one vendor running a consumer facing application; the finest money could buy 4 years ago. They have performed adequately and the vendor has provided good support but now the bill is due as the vendor has been increasing the costs of support.   Upgrades are expensive forklift changes with no promise of reduced costs.  That’s the hard place.  The rock is that the increasing costs of keeping this consumer facing application going is leading marketing to impose a fee for use on the consumers.  Consumers are up in arms and setting up petitions on change.org.  The news channels are featuring this new fee as an example of the squeeze on consumers.

This is the big squeeze that many enterprises are finding themselves in.  How do I reduce the Total Cost of Ownership (TCO) of an application while at the same time how can I provide the new services that consumers have come to expect to be free?  And if you don’t do it the competition will.

In fact from the ZDNet report on IT Budget priorities in 2012 this is the divide between IT budget winners and IT budget losers:

… these results paint a picture of a two-speed IT market in which some organizations are pushing ahead aggressively with transformative projects based on new technology or new delivery methods while others are bunkering down and looking inside for cost cutting opportunities. The danger for the latter group is they may find it increasingly hard to compete against leaner, more agile, more modern and more automated competitors.

You can get ahead in this game.  ‘New technology and new delivery methods’ are another way of saying that these budget winners are moving to servers based on low cost Intel Xeon Processors.  These processors are bringing the qualities you have found in the UNIX Processors into the low cost Intel Xeon family of high performance processors.

You’ll have to make sacrifices.  It will mean some work and those expensive servers will have to find a new home.  Maybe there is a ‘Rescue’ organization in your community that will take them off your hands and find them a new good home.

The hardest part will be going from sole source vendors to having a wide range of choices of vendors.   Gosh, you can choose from IBM, Dell, HP, Bull, Cisco, Oracle, SGI, SuperMicro, Lenovo just to name a few.  You also have a choice of rock solid enterprise operating systems from Linux and Linux variants such as Oracle, Red Hat, SUSE, to non-Linux operating systems such as  Windows, to Solaris.   Now you can be sure to get the lowest cost possible and you no longer are locked into one vendor.

Now you have options but you need a plan to get there.  With this many servers hosting one application the effort can seem daunting.

Break the effort down.  What are the easy parts to migrate?  For instance, products that are COTS (Common Off The Shelf) software that have a version for UNIX and a version for Linux and there isn’t much extensive customizations involved can be low hanging fruit, so to speak.  Databases that can be migrated using a tool from the database vendor can be next.  For instance Oracle Streams for Oracle databases or Replication server for Sybase are examples of these tools.  In this manner a lot of the application can be easily converted.

So now the user facing layer has been changed.  The backend layer consisting of databases has been changed.  Now the middleware layer can be complex and include logic servers with lots of custom code or with a big ERP package the middleware can be all COTS software.   This middleware can be a big challenge.   But now you have experience in migration and can approach it as a veteran.

One word of caution here; I keep hearing stories of missing something while migrating from veterans of in this IT modernization effort.  Usually missed are the undocumented or poorly documented data feeds that were added in a hurry to or from some application.  You can try to avoid this by running a rehearsal migration before the production one and have the application tested as if in UAT or with a test harness.

What am I talking about here?  See my white paper on migration methodologies on our website. 

Let me end this with a story about how using the Intel Xeon Processor family significantly helped a University meet their data processing requirements.  The University of Colorado had to address their rapidly rising compute costs.  They decided to migrate their mission-critical IT environment from legacy UNIX servers running RISC processors to Intel® Xeon® processors. The results were staggering. Their data center footprint dropped to 1/20th (5%) its former size. Power consumption dropped to 1/10th (10%) of its pre-migration levels. And their ERP performance jumped by 400% to 600%. In Year 1 alone, they saved $600,000.   While your results may vary you can read the white paper on this achievement on the Intel.com web site.

Do you find yourself caught in the squeeze I discuss here?  What are some of the steps you have taken?

In the first post on this topic on server modernization, I started us off on moving an Oracle Database the Old Way "Quickly" and have gotten us to this point.

Now we will set up all of the scripts for the application and the network links. This is the tedious part - there are many individual scripts to export/import the data in the tables. Organize these into a giant script that will run them all at once. Below is a sample of the beginning of the script. Make one for each separate group of tables you are moving. (You can see how old this is from the ORACLE_HOME.)

#!/bin/sh
######################################################################
# set -x
#  FILENAME: expimp.sh
#  USAGE: expimp.sh 'user to export' 'SID to export from' 'SID to export to'
#
#  for example:
# nohup expimp_resh.sh struct &
# Will export/import the database structure from one system to the other
#  and use a dummy label for the USER
#
######################################################################
RET=0
E_ORAHOME="Legacy server ORACLE_HOME"
I_ORAHOME="Target server ORACLE_HOME"
PATH=`pwd`
TABLE_NAME=$1
RESH="/usr/bin/resh target-1"
SRC_MKNOD="/etc/mknod"
DST_MKNOD="/etc/mknod"
DATE=`date`
echo $DATE
ORATAB=/etc/oratab
EXP="/home/oracle/product/7.3.4/bin/exp"
IMP="/home/oracle/product/7.3.4/bin/imp"
EXPFILE="/export/home/wpereira/exp${TABLE_NAME}.par"
IMPFILE="$PATH/imp${TABLE_NAME}.par"
EXPLOG="/export/home/wpereira/exp${TABLE_NAME}.log"
IMPLOG="$PATH/imp${TABLE_NAME}.log"
SRC_USER="system/’password’"
DST_USER="system/’password’"
EXP_PIPE="/tmp/eximpipe.$$"
IMP_PIPE="/tmp/eximpipe.$$"
${RESH}  "
    $SRC_MKNOD ${EXP_PIPE} p
    export ORACLE_SID=ORCL
    export TWO_TASK
    export ORACLE_HOME=${E_ORAHOME}
    $EXP userid=${SRC_USER} file=${EXP_PIPE} log=${EXPLOG}  parfile=${EXPFILE} & " &
#
#
ORACLE_SID="ORCL"; export ORACLE_SID
ORACLE_HOME=$I_ORAHOME; export ORACLE_HOME
$DST_MKNOD ${IMP_PIPE} p
${RESH} "cat ${EXP_PIPE}" > ${IMP_PIPE} &
# run the import
$IMP userid=${DST_USER} file=${IMP_PIPE} log=${IMPLOG} parfile=${IMPFILE} &
exit 0

Be sure to set up a high speed link between the legacy server and the new server. This link will allow for remote execution of the scripts.

Once the database has been created, you need to create all the objects in the database for the application.  Similar to allocating space for tablespaces on disks, Oracle has to allocate space for the objects like tables while the rows are being loaded. This space allocation takes time, so the better approach is to pre-allocate the space prior to the actual movement of the data which means that the application is shut down.

In this step you are also creating the VIEWS, PROCEDURES, PACKAGES, TRIGGERS, SYNONYMS, etc. of the application in the database. (You’ll have to disable the TRIGGERS before the data loads.)

Now that you have the export/import scripts set up, invoke them in one super script. You probably won’t be able to do the entire database in parallel. During the rehearsal, you’ll find where to put in the ‘wait’ statements.

Once all the ROWS have been imported you can invoke the set of scripts to create the INDEX's. When these are finished, run the script to ENABLE the CONSTRAINTS which are actually generating INDEX's. The next major step is to ENABLE the TRIGGERS. This is also done by script.

The audit of the database is among the last steps. Match the output of DBA_OBJECTS in both databases. You also need to get the number of ROWs in the legacy database and compare that to the number of ROWs imported into the target database. This is where the team comes in handy; each member is doing a different audit task.

Now open the application to use by the testers. Because you don’t want to add any test data to the production database this has to be done carefully.

If everything passes muster, then the target database can be opened for production. For the paranoid, the legacy database can be opened for parallel production in the event that the firm wants to move back to the old system.

Finally, document the entire effort so that the configuration is clear for others to understand. And go get some rest.

This process doesn’t take advantage of Oracle Streams or Oracle’s GoldenGate. But I believe there is a use for this methodology even today. What do you think? Do you agree or disagree?  Do you have migrations that will require this older approach because no other will work? Do you have any suggestions to improve this process? What have you seen when doing this with Datapump? Do you recommend any changes to this methodology when using 10.2 or 11.2 as the target database?

It’s time for the final migration.  You’ve completed a Proof of Concept and a rehearsal for the process but now you have to complete the process and move the actual production application while it remains in production. Regardless of the migration methodology you’ve chosen, if you don’t get a lump in your stomach at the start, you’ve got ice water running in your veins. I’ve heard this process compared to changing all the tires on an 18-wheeler while it is still running down the road.

All of the previous steps taken lead up to this event.  The initial analysis defined the scope of the migration.  The proof of concept determined what the scope really is, and defined all of the components involved in the migration.  Also, the proof of concept likely ported and proofed the components that could be rewritten or modified without impact on the production system.  The rehearsal provided a timeline for the migration, along with an estimate of the production outage that may need to occur.

With all this data, the application owners and those charged with the migration need to sit down and plan this final step.  Cold feet can’t stop this intrepid team. Resumes prepared and recruiters notified, the team pushes on.

The final step plan starts with targeting the possible outage at the most reasonable time. This is often a weekend or a late evening, depending upon how long the outage would last.  Often, holiday weekends are chosen  (I once did a migration over the fourth of July holiday).  Include in your plan for best case and worst case scenarios for the outage duration.

Start the migration by cleaning up the target server.  Set the BIOS to the optimal settings you determined during the POC.  Load the operating system and set the parameters. If you’re installing Linux, add any required packages and, of course, add sysstat.  Install any application systems that are required by the application you run. Tune these for the application and platform.

Begin the migration by setting up the application system.  If you are moving a database, you set up the database.  If you are hosting a custom application, install the source code and compile the executable.  Set up the network links for the final conversion.

As soon as all the preliminary tasks are done, begin the migration.  For a migration of a custom coded application, this could as simple as pointing the other applications dependent on it to the new host.  If a binary data store like a database is involved, then the migration can be a lot more complex.  But you know what to do -- you’ve found out the best method in the POC and practiced it in the migration rehearsal.

Once the data is moved over and the application is ready to run, if possible run the test harness against the new system.  Audit the database to be sure all the tables, indexes, procedures, views, referential integrity constraints, database links, synonyms, and other database objects are all there. Finally, the last step is opening it back up to production. Then, go get some sleep!

After one migration I went back to the hotel and slept hard. I’d had maybe 4 hours rest in 72 hours, and I was pretty bleary.  When I went in Monday morning, there were cars in the parking lot.

"Hurrah! It worked! I thought. The company wasn’t going to let any workers into the plant if the migration had failed, but here they were working.   Subsequent testing by the user team assembled by the company to proof the process showed a few glitches on Monday (and for a few more days) but those were quickly tackled by our QA team. The lesson here is that once you’re finished with the steps, you aren’t completely done. You still have to monitor the situation to ensure all is well.

However, the migration is now finished.  You’ve got an all new platform for the application.  The old one can be put out to pasture (but it’s probably not a good idea to use it to make a fishing reef, given all the chemicals in a computer system!)

I was at a conference recently and while I demonstrated the Xeon E7 processor RAS (Reliability, Availability, Serviceability) features, I discussed RISC to IA migration with the attendees. Occasionally I got blank stares."Oh, we’re doing server migration," was a common response, along with "we’re migrating our UNIX servers to Linux."

Perhaps we here at Intel get ourselves tripped up over our usage of terms. Many of us live in the world of processor architectures.  This leads us to use the terms RISC to IA, which refer to the different processor architectures.  Moving up the stack to the hardware layer, some refer to Server Migration or Platform Migration.  Let’s take another step up the stack to the operating systems. Here, what we discuss is referred to as UNIX to Linux migration.  This also includes moving applications from proprietary servers to the cloud and even data center consolidation where large proprietary servers are replaced by Virtual Servers running on Intel based hardware.

If you’re doing any of the above, then you need to read these blogs for hints and suggestions.  You should also visit the server migration website for helpful tips.  The website is public and is filled with white papers, case studies and How-Tos. We add to the site all the time, so keep visiting us!

The term proprietary hardware usually refers to servers that are marketed uniquely by one vendor.  For example, POWER 7, sold uniquely by IBM, or SPARC sold by Oracle. Both fall into this definition.  While these processors have many good features, they have a serious limitation.  Usually, once you buy into the server line, you are limited to the same processor based servers for future upgrades. These upgrades can involve significant physical changes in the data center.  Often, the term ‘forklift upgrade’ is used as this type of upgrade entails the replacement of an entire rack of hardware.

The term ‘Open Servers’ is used for hardware that utilizes the x86 architecture.  This hardware is characterized by two processor vendors and numerous choices for server platform manufacturers, as well as form factors.

Another term used here is commodity server.  This usually means something cheap but low cost Xeon processors are anything but ‘cheap’ in the pejorative sense.   Here, commodity is related to availability from a wide range of vendors in a variety of form factors.

Another term used at a very technical level is Endianness. This refers to the byte order of mutli-byte data. In a crude example, say the number 1758 is represented by two bytes. On a Big Endian machine, like SPARC or POWER, the first byte has 17 and the second byte has 58. On a Little Endian machine, like the Intel Xeon, the first byte has 58 and the second byte has 17. There are a lot of reasons for each architecture but suffice it to say that multi-byte data has to be converted before it can be used by a machine with a different architecture.  While some programs are written to be ‘endian neutral,’ most databases are not, and custom built programs aren’t either.

This endian difference is the reason we have to go through this migration protocol. If you search for endian difference on Intel.com, you can find white papers and even a video addressing how to code for this.

I want to end on where the term ‘endian’  came from (I worked with Danny Cohen 10 years after he wrote this paper). Almost 300 years ago, Jonathon Swift wrote Gulliver’s Travels, a satire on the politics of the day.  He had the characters of the world he created go to war over which end of an egg to break, the big end or the little end, when taking the shell off of a hardboiled egg.  To me, the bottom line is that we have to be careful how seriously we take ourselves. While endian differences created considerable work for us, there should never be an argument about who is right or wrong.

You’ve done the Proof of Concept (PoC) and now everyone clamors to put the system into production. Some see the application running after the PoC and ask you to link it in to the network.  Yet, you know that you put this system together, and the finishing touches were done with chewing gum and bailing wire.

Looking at the production system, everyone can see that the source application has been receiving data since you started the PoC.  How is that new data going to get into the application?  The bottom line is that it really can’t. The referential integrity issues alone would stop you and of course, you can’t go and apply the applications log files (unless they are in character form).

What are your options here?  You could export all of the data that has changed during the period, but how do you distinguish or even find the data that has changed?  Not to mention, when you apply the changed data, new data still comes in. What are you going to do with that?

The solution is a migration of the actual production system while it runs.  There are various techniques to do this but all require you to assure the boss that everything will be OK.  You know now that you can do the migration (the POC proved that) but how will it be done on the production system. Most importantly, how do you minimize downtime? How long is the downtime going to be?

So, now is the time to take those lessons learned in the PoC and apply them to the migration process.  What does this mean?  In the PoC process you probably moved a part of the system, and then moved another part.  You discovered you need to adjust the tuning of the server’s operating system and then the DBA told you that you forgot a link.  This was followed by problems in the execution of the application that required analysis and correction.  (See? bailing wire and chewing gum.)

Luckily, you documented all of the changes and determined the proper implementation sequence (or at least you think it is the proper implementation sequence). You also discovered that you can parallelize a lot of the steps and reduce the down time of the application.  Meanwhile, you have management asking when you can have the application migrated to the new platform.  For instance, at one site we were moving a retail program, RETEK, and Thanksgiving was fast approaching.  They needed the extra horsepower to handle the Christmas rush, and they had to input all of the Christmas marketing programs into the system.

To answer these questions and address the concerns, execute a rehearsal of what you plan to do for the actual migration.  Plan is the operative word here.

Take what you learned from the PoC and organize it into a coherent plan with the events sequenced correctly.  Plan on running as many tasks as you can in parallel.  Plan on doing as many tasks as you can before you have to shut down the production system for the cutover.

Back in the day when export/import was all that was available for the migration of an Oracle database, I would build the target database, create all of the production tablespaces as small I could, and create the users.  Then, I could ‘ADD DATA FILE’ in parallel until each tablespace was big enough for the data. After that, I would create all of the tables with enough extents to hold all the data (we don’t want Oracle to use time having to extend the tables while the production system is down). With the tables, I could create and compile all of the Packages and Procedures.  I then turned off all referential integrity constraints and Triggers. Only then was the production system put into single user mode for the export of the data.  All we needed to export were the rows of data; no indices.  Once the data was inserted, we fired off the index create statements, and ran them in parallel.  Once the indices were completed, the referential integrity constraints were ‘ENABLED’, thus setting off another round of index building.

Now that you've moved the application, you'll need to run your the application in your test harness for regression testing.  The testing will 'prove' that the migration is complete.

A final check of the database, and it opened up.    Here is an illustration of the process I just described.

Today, you can use GoldenGate or Streams for Oracle database migration or you can use Replication Server for Sybase. GoldenGate allows you to run the production and the new database in parallel, and even roll the migration back to the original database in the event that something goes wrong.

Wouldn’t it be better if that ‘something that goes wrong’ occurred when you executed the rehearsal of the migration of the Q/A database?   A little patience will result in greater likelihood of complete success.

Here are the rehearsal steps in a flow chart:

Now that you’ve done the rehearsal, documented every step, and developed scripts to automate every process that can be automated, you are ready to plan the actual production migration and answer the question, ‘When is the application going to be ready on the cheaper platform?’

	In my previous blog I described the methodology to run a Proof of Concept when migrating a production database that underlies a mission critical application.  This blog covers the analysis of the results of the Proof of Concept.  In other words, you’ve done the PoC, now what?
Flickr Image: Pat Guiney

The Proof of Concept has three primary objectives:

• To ‘prove’ that the application can run in the new environment
• To see how the application performs in the new environment
• To specify the steps necessary to conduct the actual migration.

The proof that the application can run in the new environment seems pretty straight forward.  Sure, if it runs after being ported, it runs.  Check that one off the list.  However, when we start the PoC, there’s no guarantee that it actually will run.  We use the UAT procedures - be it a regression test harness or a select team of users - to bang at the application and ensure that we have ported and hooked up everything.

As mentioned before, these tests are run frequently and usually after ‘flashing back’ the application to the initial starting point.   Once this is done, all of the components and set up steps need to be carefully documented.  These steps will be repeated in the subsequent phases of the entire migration.

The steps taken for the proof of concept can be seen as covering a continuum.  In some cases, the steps will need to be repeated in each phase, such as porting the data.  Other steps, once done for the proof of concept, like rewriting shell scripts or editing the C++ code will not need to be repeated.  Once these ports are done to the satisfaction of the customer, then that is all that needs to be done and these can be put aside for use in each of the subsequent phases.   Other steps fall in between.  These steps are setting environmental parameters that need to be set for this application.  For each subsequent phase, we need to reset the parameters but we don’t need to determine how to set them.  Documenting these settings is all that needs to be done.

So now that we know that the application will run, we then look at how it runs in the environment.   This is where environmental tuning parameters are adjusted.  This is where code may need to be rewritten.

Some proof of concepts start off with the application performing on the new target significantly worse than in the original environment.    An application can perform differently when hosted on a different platform.  For instance, an application that was CPU bound on the old platform could start having I/O queues or memory usage issues.    If you’ve been measuring application performance by looking at queues on the original host you’ll likely miss this.  The only queues you really should be looking at are the queues (CPU, I/O, and Network) on the PoC hardware which simulates the eventual target hardware.

I once got an application running that was ported from a mainframe into a relational database.  Performance was terrible even though on paper the new environment would significantly outperform the old host.  We looked at tuning parameters and we were optimal.  We looked at performance reports from the operating system and it was OK.  We looked at the performance reports from the database and sure enough, CPUs were barely running but the I/O to disk was pegged.  The customer was looking to us and the application vendor for a fix.  I looked at the code and found that the application vendor had written VSAM style data access methods in SQL!  In other words, instead of using relational set theory to winnow the data, his application read each row in sequence through the entire table.  The PoC stopped right there, and the customer kicked the application vendor out.

Like in the story, we need to observe carefully how the application is performing.  We can use the output from the testing harness to give us data on how long each tested task took and compare that to the SLA’s for the application.  We can look at the tools built into the operating systems, like Perfmon for Windows or the ‘stat’ tools, like vmstat, iostat, etc., in Linux.   (Let’s not neglect the application as the primary source of performance problems but we’ll discuss that later.)

The data from the tools needs to be analyzed.  The performance times measured by the test harness are pretty obvious as it usually reports response time.  If the task took more than a second to complete but requires sub-second response time, then we know there is a problem that needs to be looked into.  Is the application just running poorly?  Is the hardware inadequate?  Is the operating system or underlying software in need of tuning?  Perhaps the application architecture needs adjusting.  Our performance monitoring tools will provide us clues.  We’re looking for bottlenecks.  Bottlenecks can be found where data travels, like I/O or in the case of a database, in logging the database changes.  Look for queues in network and storage I/O and queues for the CPUs at the operating system level.   At the database level, we are looking where ‘waits’ are occurring.  For instance, in Oracle we are first looking at waits in the single threaded operation of writing to the log files as well as anything affecting the logging process.

Now we can more precisely determine the capacity requirements of the target platform.   Here we can project with greater confidence the characteristics of the platform we will be porting to.  For an application requiring bare metal this is critical; for an application that can be hosted in a Virtual Machine this process defines the initial set up requirements.  Remember this new platform will need to handle the projected growth and handle the unexpected black swan.

Now the PoC is over.  The results have been converted to foils to be presented to management for planning the conversion.  The documentation we wrote is now the recipe for the next step, the migration rehearsal.