Live blog: First Power i Move – Moving a System i CBU off the Vendor’s Staging Floor

At outlined in prior posts, we’re in the middle of moving two Power 6 machines between an outsourced data center and company headquarters. The backup CBU machine currently resides at the hosted outsourced data center, while the production machine resides at corporate headquarters. As an aide to other managers, I’m blogging these moves live as they are happening.

In three steps, we will be flip-flopping the machines at these locations so that after the move, the production machine will go to the outsourced data center and the CBU backup machine will move to the company data center. To do that, we are performing the following three moves:

1. Move the CBU off the floor at our outsourced data center and into a staging room. This clears the open space in the floor to receive the production machine when it relocates from the corporate data center. The CBU backup machine will still be replicating production data after it goes into the staging room.

2. Move the production machine to the outsourced data center and deploy it on the vendor’s DC floor.

3. Move the CBU back to the corporate data center and deploy it back on to the network.

On Thursday, we moved the CBU off the vendor DC floor to the staging room. To do that, we did the following:

• Stopped replication with the production machine
• Undid all the cables that connected it to the larger network infrastructure we placed on the floor
• Contacted our vendor to disconnect the power. For this data center, the vendor always disconnects and reconnects power under the raised floor.
• Rolled the machine from the floor into the Staging Room. Only risk here would be if the machine tips over. This wasn’t an issue on the level data center floor. But once you got off the floor, we had to maneuver the machine down a 45-degree incline ramp to get to the Staging Room. Two of us had to stand in front of the 1200 pound cabinet to insure we didn’t have a runaway. It was okay as long as we didn’t use a running head-start.
• Connected the machine from our network back to the Staging Room. We had earlier requested that the vendor run a network connection from Staging back to our cabinet. We found the connection in the room hooked up to a 16-port switch.
• Restarted replication so the CBU backup could catch up with production data changes that occurred while it was turned off.

Once in the Staging Room, the plan was to a) Hook up the Staging Room network connection to our cabinet, which was accomplished by our Network group; and b) Reconnect all the Ethernet cards to the Staging Room and back to the Corporate Network. This went relatively smooth but we ran into two hitches which delayed turning the machine back on and scared the bejesus out of us.

 1. When we removed the network cables from the machine’s ethernet ports, we forgot to label which ports need new network cables. There are more ports than cables on this machine and before we could power it back up, we had to identify all the ports that needed cables, consuming precious time. Lesson to be learned: Always label your cables before you unplug anything.

2. The machine has redundant PDUs and redundant power supplies for all components. In the Staging Room, we could only plug-in one of the redundant PDUs. When we did, three drawers of disk racks did not come up when we repowered the System i box. Not a fun feeling.

It turned out that one of the redundant power cables on the System i had jiggled loose from the machine and wasn’t powering up. This was okay when there were two redundant power supplies running on the machine but it become obvious when only one PDU was running. We examined all the power cables, plugged them back in, reset the disk controller card, and the disk were found again. Lesson to be learned: Check to make sure your power cables are firmly installed before a move–even if you’re only moving the machine 200 feet. Also, don’t assume your redundant cables are hooked up to your PDUs correctly. Double-check before moving a machine.

At this point, we are 1/3rd of the way through our moving process. We’ll pick up any Saturday night when we move our production machine to the outsourced data center.

Live from outside Chicago…Joe