These are the four cache batteries that power the database cache on a Power 7 720 machine
Over the weekend, I worked with IBM to change the disk cache batteries on one of our Power 7 720 CBU machines running IBM i 6.1.
When the cache batteries on a Power 7 box get within 90 days of their expected end of life, the i operating system starts issuing alerts for battery replacement. You then have (roughly) 90 days to work with IBM to get the batteries replaced.
If you don’t replace the batteries before they run down, your disk drive performance will significantly degrade because the batteries are powering your disk drive cache.
The trick is to schedule cache battery replacement during a slow processing period because the replacement batteries are not charged when they are installed. IBM delivers uncharged batteries for safety reasons and regulatory requirements (mostly fear of explosion).
This means that your IBM i disk cache will not be powered up when replacing old cache batteries. You’ll essentially be running without disk caching for several hours until the new batteries are charged while in the machine. Once the batteries are charged, disk caching will resume and you’ll see your system speed return to normal. In our case, that took 2-3 hours (YMMV).
I first wrote about the charging issue in 2012 in an IT Jungle article. Since then, I’ve discovered that in certain situations you may not be able to change your batteries concurrently.
For our Power 720 machine, we received four batteries. Two batteries were installed in the back of the CEC and two were installed inside the CEC. The two that were installed at the back of the CEC could be installed concurrently. But to install the batteries inside the CEC required us to open the CEC cover. For a 720 box, this now requires a system shutdown. Check with IBM or your CE to see if you need to power down your box for battery replacement.
This is the cache battery that has to be installed in the CEC, which may force you to bring down the machine during a battery replacement
Read this article to see why you may need to power down your system to open the CEC cover.
So expect two sets of processing delays when changing cache batteries. One if you have to power down your box for installing batteries in the CEC. And a second delay while the batteries themselves charge after installation.
Related Posts
The Twitter @-Sign Trap–Starting a Tweet with an @ is a Reply
IT Jungle: New Tricks for Old IBM i Backups
Using Google Search to Search Only One Web Site
Just in Time for Common–Joe’s New Business Card
IBM i QuickTip: When You Can’t FTP from a Windows Desktop to an IBM i
Community Post: List of IBM i System Monitoring Products
If the cache battery is not replaced, and runs out of power, then there will be three occurrences of processing delays. With a dead battery, the machine moves slower do to the cache function not happening. Another caveat: should your hardware maintenance lapse, and the cache battery needs replacing, on an old 9406-520 the bill from IBM will be about $1,700. The battery costs about $110. Thanks, Joe!
Thanks for the info Perry. Just as I was reading your comment, I got another hardware error from a second machine we purchased around the same time as the one we just replaced. Sure enough, I’ve got another machine that needs cache battery replacement. The nice thing about these warning is they usually occur when there’s about 3 months of life left in the battery. But I usually don’t let them wait much longer than a month after the first warning goes off before replacement, just to be safe.
You are welcome, Joe. I cannot begin to guess how many IT departments there are in which checking thoroughly the messages in QSYSOPR, and deeper monitoring of SST error logs for hardware/software errors, is almost never done. I have gone to a client to check out a tape drive problem only to discover that there was a failed disk drive on the machine. We know what would have happened had another drive failed. They were oblivious. I guess this is why we get the “big bucks”, right Joe? Hah!
In my experience, if one cache battery is calling for replacement, it’s highly likely that one or more others will be “expiring” at about the same time. It’s a good idea to check the status of all cache batteries for the partition/server – and possibly other servers as well, as Joe mentions in his comment – so you can replace all at the same time. A good CE will suggest and encourage this. Why make multiple service visits (not to mention schedule multiple downtime windows) if you can do it in one?
Actually, the batteries aren’t “failing” when you get the message. When a controller is started for the first time, or after a battery is replaced, the cache battery “days of use” counter is reset for something over 1000 days, basically three years. That’s the recommended life of the rechargeable battery. At 90 days before expiration, you get a warning, and when they expire they turn caching off as a safety precaution as, in IBMs opinion, the battery should no longer be trusted to hold a charge.
And in my experience with rechargeable devices, that three years is as long as you want to go, holding a charge after that starts to get iffy.
I certainly don’t recommend it, but if you stop caching and turn it back on again without changing the battery, the counter gets reset for another three years.