Live blog: Moving an iSeries cross-city: Moving Production to the Hosted Data Center Floor

This post continues our live blog detailing dual iSeries Power 6 moves where we are moving a production server to a vendor hosted  data center and moving our backup Capacity Backup Unit iSeries (CBU) back to the corporate data center.

At this point, we’ve moved our CBU off the floor at the vendor hosted data center, and the production machine has arrived at the hosted data center. Both machines are now residing together in the vendor’s Staging Room.

In order to use the vendor’s electrical circuits, we need to swap out the production machine’s current Power Distribution Units (PDUs) and replace them with two new redundant PDUs. This is a fairly easy job and it gives us an excuse to rebalance the cabinet’s power cables and insure that all the component cables are securely plugged in before redeploying (avoiding the issue we had when we rolled the CBU into the Staging Room and a loose power cable caused us to lose connection to some of the CBU disk drives).

Also, before we can roll the production machine out to the data center floor, we have to secure its’ cabinet. Since the CBU cabinet was secured when it was deployed in the vendor’s data center and it doesn’t need those enhancements when it moves back to the corporate data center, we can cannibalize those components to secure the production machine. The CBU’s locked front and back doors as well as one of its locking side panels, are transferred to the production iSeries, and it’s ready to roll out.

On the data center floor

It takes three of us to move the cabinet up the 45-degree incline that leads to the data center. As we manuever the cabinet towards our row, I move to the side to help keep it rolling straight. The cabinet keeps veering to the left and I have to be careful not to get pinched between our 1000+ rolling cabinet and other cabinets and cages set up in the data center. We move the machine in place and then do the following to bring it up.

Our redundant PDUs have separate circuit breakers that can be turned off or turned on separately. Terry, our IBM CE, turns off the circuit breakers as we plug the PDUs into the vendors electrical circuits. Then he brings up just the PDU located on the left-hand side of the cabinet, and powers on all the iSeries components that have redundant power. This allows him to see if the cabinet can run just on the left-hand PDU if the PDU on the right-hand side of the cabinet fails. Terry is simulating a right-hand PDU failure to insure the machine won’t lose components if that PDU is lost.

Terry then powers off the left-hand PDU and powers up the right-hand PDU and performs the same drill. By doing this, we confirm that all the redundant power cords are plugged into two different power sources so the loss of one circuit or PDU won’t take down any of the components.

The next step is to confirm the disk drives have arrived without damage. Terry does this by powering on the System i machine and then starting both machine partitions in manual mode. As the machine comes up, he goes into each partition’s Dedicated Service Tools (DST) function and checks that all disk drives for each partition are reporting in without damage and that the RAID sets are still intact. They are. Terry also uses DST to insure everything else is in order and that there are no alarms.

At this point, we’re almost ready to bring up the machine. Our master iSeries technician Scott and our network group label and rerun the network cables to the cabinet’s Ethernet cards. They also reconnect a Plain Old Telephone Service (POTS) line to an internal modem that we use for system monitoring.

With the machine checked for alarms and connected to electrical power and our network, we pull the plug and restart the partitions.

Unleash the Applications Team

Since we’re moving our production machine into a new environment, we’re not taking any chances that our ERP applications or their supporting apps won’t work. We alert selected members of our Applications team to remote in and test whether our applications are working correctly. Since the disk drives arrived unharmed and there ar no alarms on the system, we’re relatively confident the data and applications are okay.

But since we physically moved the machine’s location, what we don’t know if whether all the programs requiring network connectivity will still work.

The Apps team checks out FTP, credit card processing, EDI, Web site connectivity, faxing, email, and roughly a dozen other applications. For testing, we use some of the same tests we use for our High Availability Exercises. Some of the applications have problems but those seem to be related to known issues and quirks with restarting the system after a 10-hour shutdown. With a few minor adjustments, everything starts OK.

The final thing the Apps team does is to review any scheduled applications from Help/Systems ROBOT job scheduler, looking for jobs that were supposed to run while the machine was down, but didn’t. They restart those apps, as needed.

Ready to go

At this point, we’ve moved the production system to its new home and certified that the machine arrived unharmed, the applications still work correctly, and the machine can connect up to whatever companion servers it needs to contact. The production machine is back up again in production mode, and we declare victory on this move.

The last and final step is to move our CBU machine back from the vendor’s Staging Room to the corporate data center. That move is scheduled for Tuesday. Blog to you about it, then.

Live from outside Chicago…Joe