DB2 Basics: Patching DB2

Like any software, DB2 requires frequent patching. A database should be one of the most secure parts of any enterprise, and keeping it secure means keeping up with the fixes that are delivered in fix packs.

Fix Packs

DB2 delivers many things through fixpacks, including:

• Security Fixes
• Bug Fixes
• New Functionality – though IBM goes back and forth on this

IBM delivered Native Encryption in Fix Pack 5 of DB2 10.5. This was a pretty new feature to be delivered in a fix pack. However, I hear rumors that they’re going to stop delivering new features in fix packs. They seem to go through cycles of either delivering significant functionality via fix pack, then a few years later, reserving all functionality for version-level releases.

How Current is it Wise to be?

In the past, there have been several situations where IBM has released a fix pack, only to withdraw it 24 or 48 hours later, to be replaced with a new version of the fixpack (differentiated with an ‘a’ or ‘b’ after the fix pack number). I think they once even got to ‘c’. This means that at some time after releasing the fix pack, they found a problem so significant that they felt it too dangerous to just let it be. I have also waited with bated breath for some fix packs when a fix was to be included that was not previously available.

While I would never advocate moving to a new full version so soon, I am comfortable recommending any new fix pack that has been out a week or longer. Fix packs far more frequently solve problems than they cause problems. There is no need to wait a quarter or to always be down by one in my experience.

Justifying a Fix Pack

If you have ever worked for a company with an “If it ain’t broke, don’t fix it” mentality, you know how hard it can be to get even the most minor change approved. In many cases, that mentality may also apply to patching software. That may mean you have to have a very specific reason to apply a fix pack. I find that the best justification often comes from the page that lists all security and HIPER (High Impact/PERvasive) APARS. There are usually scary enough APARs listed for any fix pack to justify applying the fix pack.

A Few Important Terms

PMR

A PMR is a Problem Management Record. It is the case that IBM opens up when you call in for support. Except for the free edition, DB2 Express-C, all editions of DB2 come with support – this is not an additional charge you have to pay on top of licensing. Like with most software, there is an annual fee to maintain licensing compliance, and this fee includes support as well.

APAR

An APAR is an Authorized Program Analysis Record. IBM creates these when they run across bugs or problems in the DB2 code. Sometimes an APAR is the result of a PMR that a customer opened with IBM, and other times it comes from internal IBM sources. APARs are grouped together into fix packs. DB2 does not offer individual patches for individual APARs in most situations.

Special Build

Sometimes if you are stuck at a particular fix pack for a good reason and really need the fix for a particular APAR, IBM will offer you a “special build” (why is this not an acronym, when everything else is?!?). A special build means that IBM is building a particular fix on top of a lower fix pack just for you. This means the code may be a little less tested, but it is usually still code that was already developed and tested by IBM long before you opened a PMR. I remember the bad old days when a special build was just a few files that came with instructions for manual replacement. These days, applying a special build is about like applying a fixpack.

Vendor Certification

Some vendors certify on a specific fixpack of DB2. If you are working with a vended database, please check to make sure your vendor supports the fixpack before applying it.

Testing

Fix packs should always be installed in a non-produciton enviornment first, and tested as thoroughly as possible there – including load testing, if possible.

High-Level Overview of Fix Pack Installation

Installing a fix pack is a time-worn and time-tested procedure. I haven’t seen a fixpack go really wrong in years, but I have seen enough in my career to take some preventative steps. Here are the general parts of the process:

1. Ensure you have not just the code you are upgrading TO, but also the code you are upgrading FROM on the server
2. Take a full database backup (offline if possible, online if not)
3. Backup all configuration information and the database structure
4. Stop DB2 completely
5. Install the fixpack using the installFixPack command
6. Update all DB2 instances using the db2iupdt command
7. Start DB2
8. Connect to each local and remote database and bind the cli and ubind lists and db2schema.bnd
9. Verify all databases
10. Collect all configuration information again in case changes are later noticed
11. After you are sure the the fixpack will not be backed out, run the db2updvNNN command for the correct version to make any changes needed to the catalog data structures (this may be days or weeks later, or may be in the same outage window)

For any fix pack, you should read the readme provided with the fixpack to see if there are any new actions required.

High-Level Overview of Fix Pack Installation with HADR

With HADR, you can often apply a fix pack with 10 minutes or less of perceived outage. The steps are similar, but with actions performed on two different servers. For these steps, we start with the primary database residing on server1.

1. Ensure you have not just the code you are upgrading TO, but also the code you are upgrading FROM on both servers
2. (server1)Take a full online database backup
3. (server1 and server2)Backup all configuration information. On the primary(server 1), also the database structure
4. (server2)Verify that HADR is fully in sync and then stop HADR on server 2 only
5. (server2)Stop DB2 completely
6. (server2)Install the fixpack using the installFixPack command
7. (server2)Update all DB2 instances using the db2iupdt command
8. (server2)Start DB2
9. (server2)Start HADR and wait for the servers to be fully in sync
10. (server2)Issue the takeover command to make server2 the primary – without forcing it – Potential perceived outage during the takeover
11. (server2)Connect to each local and remote database and bind the cli and ubind lists and db2schema.bnd
12. (server2)Verify all databases
13. (server1)Stop DB2 completely
14. (server1)Install the fixpack using the installFixPack command
15. (server1)Update all DB2 instances using the db2iupdt command
16. (server1)Start DB2
17. (server1)Start HADR and wait for the servers to be fully in sync
18. <IF DESIRED>(server1)Issue the takeover command to make server1 the primary – without forcing it – Potential perceived outage during the takeover
19. (primary server)Collect all configuration information again in case changes are later noticed
20. (primary server)After you are sure the the fixpack will not be backed out, run the db2updvNNN command for the correct version to make any changes needed to the catalog data structures (this may be days or weeks later, or may be in the same outage window)

The biggest risk when doing a fix pack on HADR in my experience is that you may not have tested application connectivity to the standby previously – if this is the case, your applications may not function while the database is running on server2.

Back Out

Every good change comes with a back out plan, and fix packs should be no exception. My very first weekend on pager duty at IBM as a DB2 DBA with three months of experience, straight out of college, I had to back out a fix pack on a teammate’s account. The teammate was also three months out of college and had not documented a back out plan. I managed to back it out, but it was very painful. This was DB2 version 7, most likely, and looking back, I think I did not stop DB2 before getting the AIX SA to use smit to remove the applied code. Yes, that big of a mistake I made because I had NO clue. I was trying my best with three months of on-the-job training. I eventually got the thing backed out, but it took the better part of a day. The reason the back out was needed in that case? It was a vended database, and the vendor had not certified the fix pack in question. Because of early experiences like this, I have meticulous back out plans. Generally a fix pack back out looks like this:

1. Take a full database backup (offline if possible, online if not) – sometimes where you were is better than where you get to, even if it seemed pretty bad to start with
2. Backup all configuration information and the database structure – again, sometimes where you were is better than where you get to, even if it seemed pretty bad to start with
3. Stop DB2 completely
4. Install the original, old fix pack using the installFixPack command with the `-f level` option
5. If the above does not work, uninstall and reinstall db2 entirely using the db2_deinstall and db2_install commands – be prepared to fully reconfigure
6. Update all DB2 instances using the db2iupt command
7. Start DB2
8. Connect to each local and remote database and bind the cli and ubind lists and db2schema.bnd
9. Verify all databases
10. Collect all configuration information again in case changes are later noticed

Does that look familiar? Backing out a fix pack is almost the same procedure as applying a new fix pack, assuming you have not run the db2updvNNN command. If you have already run that command, and it made any changes, you will not be able to back out that fix pack.

Different Editions

The only difference I am aware of for any edition is that you cannot apply a fix pack to DB2 Express-C.
These instructions do not cover PureScale environments.

• Author Details

Ember Crooks

https://datageek.blog/

ember.crooks@gmail.com

Ember is always curious and thrives on change. She has built internationally recognized expertise in IBM Db2, and is now pivoting to focus on learning MySQL. Ember shares both posts about her core skill set and her journey learning MySQL. Ember lives in Denver and work from home