One of the most dramatic steps we took to becoming a Value-add revenue generating partner was to completely throw out our existing network. Wow! As I write that it sounds a little “harsh”. Let me explain.
Our network ties together over 80 locations in Central Indiana. A year or so before we embarked on our five year plan, we had implemented an MPLS network, most of the 80 locations were connected with a T1.
Our Corporate Headquarters housed our servers, storage and the head-end of our network. Out of that same building, not only did we have corporate operations, but we also ran two charter high schools.
While this was a solid network providing four 9’s of uptime, we were limited. Already our Loss Prevention department wanted to view hi-def video from the stores, but over a 1.5 Mbps pipe viewing those video streams would bring the network to its knees. We certainly couldn’t let them view one store’s video from another store. So...they were locked out from doing so.
We were already streaming music to the stores over the T1; however, we wanted to be able to push a video stream to our TV walls (the area in which TVs for sale were displayed). We were sure we sell more TVs if shoppers could see them in action. Again, we were stymied by the network bandwidth, so we put a DVD player in each store and mailed them a DVD every month.
Forget doing a sales floor digital signage. No way was that going through a 1.5 Mbps needle.
Finally, even though we were on an MPLS network, we were essentially a hub and spoke network, with all of the traffic coming back to corporate before going out to the internet or to another location. If we were going to achieve our vision, if we were going to achieve our company’s vision, we had to find a better way.
Our first step was to sit down with our telecom provider, TWTelecom (now Level3) and put our cards on the table. We could put in bigger pipes but even the T1s were several hundred dollars a month. How could we afford to put 10 or even 20 Mbps pipes in? Believe it or not, the plan we came up with would dramatically reduce their Monthly Recurring Revenue (MRR), but we all agreed it was the right thing to meet our needs. So we embarked on Phase I of our Network Redesign.
Over the course of the next two years, we converted all of our retail locations to business-class cable. I can hear you now, “but what about Quality of Service, you don’t get that with cable!??!” True. But to ensure we had the resiliency we needed, we put Verizon 4G in each store as our failover. In a very real sense, we multiplied our bandwidth over 10 fold, and added a back-up circuit both for 80% LESS a month than the cost of a T1. For our larger sites (E.g. schools, nursing, etc.), we put in larger point-to-point ethernet pipes (when possible fiber-based) with business-class cable as the failover.
The project was drawn out over two years for a variety of reasons. Primarily, we wanted to time the retirement of the existing connection with its renewal. However, we also ran into issues with getting Right of Entry from some of the landlords where our stores resided. We also had one store that was adjacent to a major highway construction project which made it impossible for us to get cable service to the location. When we had finished moving all the other locations, we punted and put in a 10 Mbps point-to-point in that location. In 2017, when the highway is complete we will revisit that store.
As we were nearing the end of Phase I, we kicked of Phase II with the help of our networking partner, Sinewave. Phase II, was to move the head-end of our network out of corporate and into a state of the art, fully compliant, fully certified data center. We chose Lifeline Data Centers. Specifically, we chose their eastside location. The data center was built in the old Eastgate Consumer Mall. This mall, long abandon, had been built as an emergency bomb shelter during the cold war. During the Super Bowl that was held in Indianapolis, Homeland Security used the data center for their command center.
The initial step was to swing all of our connections to Lifeline and land on Sinewave’s stack. This would enable us to reuse some of our existing gear to build out our new fully-redundant stack. Once all of the traffic was successfully running through Lifeline, we swung it to run through our own gear.
As you might imagine, there were some complications with the migration. However, we were quickly able to resolve (with a LOT of help from Sinewave) the issues as they cropped up. Actual downtime to the stores or our schools was minimal. Most of the issues we encountered caused excessive latency in some locations. Once we identified the offending traffic and changed its routing, response times returned to pre-migration levels.
Once the network redesign was complete, we were able to help develop our new prototype retail store, complete with two 90”+ flat panels streaming hi-def content, TV wall content that can now be centrally managed, SIP trunking to the stores to improve the phone systems, and a “Best-Buy-esque” queuing system to speed checkout.
All of this left us with one final step on our journey. If we truly wanted our headquarters to be “just another spoke on the wheel”, we had to do something with all those servers. Next month, Amplify Your Value: Just Another Spoke on the Wheel Part Deux tells the story of moving our production environment to the cloud.
The series, “Amplify Your Value” explores our five year plan to move from an ad hoc reactionary IT department to a Value-add revenue generating partner. #AmplifyYourValue
Author’s note: In the interest of full transparency. To paraphrase the old Remington Shaver commercial from the 70’s, “I like it so much, I joined the company”. In October of this year, I left Goodwill to join Bluelock as the EVP of Product and Service Development. My vision is to help other companies experience the impact Goodwill has felt through this partnership.
We could not have made this journey without the support of several partners, including, but not limited to: Bluelock, Level 3 (TWTelecom), Lifeline Data Centers, Netfor, and CDW. (mentions of partner companies should be considered my personal endorsement based on our experience and on our projects and should NOT be considered an endorsement by my company or its affiliates).
Jeffrey Ton is the Executive Vice President of Product and Service Development. He is responsible for driving the company’s product strategy and service vision and strategy. Jeff focuses on the evolving IT landscape and the changing needs of our customers, together with the Bluelock team, ensures our products and services meet our client's needs and drives value in their organizations now and in the future
Find him on LinkedIn.
Follow him on Twitter (@jtonindy)
Check out more of his posts on Intel's IT Peer Network
Read more from Jeff on Rivers of Thought