I can’t say that I work with federation on a daily basis. I have, however, set up federation for an Oracle data source from top to bottom, and support at least one system on a daily basis that uses federation to access Oracle, Informix, and DB2 for AS/400. This database also happens to be a BLU database, and as such, we want to stay really current on the fix pack. Currently with BLU being so new, there can be some drastic differences with just a fix pack.
The Problem
After applying Fix Pack 6 of 10.5, querying any Informix nickname received this error:
SQL1416N Wrapper library "libdb2informixU.a" is not compatible with the release of DB2 installed at the federated server. The wrapper library is compatible with the following DB2 releases "10.5.4.0". SQLSTATE=560CN
All of the other nicknames (Oracle and AS400) worked just fine. I called IBM DB2 support fairly quickly on this issue. After attempting to reinstall the fix pack about 4 times. Luckily this particular data was some that the client only uses about quarterly, so there was little urgency behind solving the problem. Between waiting on IBM DB2 support and waiting on the client to schedule the multiple outage windows needed, the PMR was opened on January 25th, and the problem was not finally solved until March 12th.
Timeline
Things I tried before calling IBM support:
- Re-installing the fix pack (several times)
- Running djxlinkInformix and djxlink from /opt/IBM/db2/V10.5/bin, as root, with DB2 down
- Downloading the fix pack specific to federation, running db2_install -p rcon, to attempt to install Informix federation libraries from scratch.
Support’s fist suggestion included following the shut-down instructions from the IBM DB2 Knowledge Center and trying fix pack 7. While I think I’m pretty decent at stopping DB2 for fix packs and upgrades, I humored support and followed the shutdown instructions in order, and the client was happy to go with the most recent fix pack. When I had originally recommended fix pack 6, 7 was not out yet, and by this time it had been out for enough time that I was comfortable going with it. I didn’t think going with the most recent fix pack was likely to help, but I also didn’t think it would hurt anything. I tend to err on the side of accepting IBM support’s recommendations unless I think they will actually be harmful or if it’s a system-down situation and I think they’re a complete waste of time. These initial suggestions did not solve the problem. On one of the attempts at installing fixpack 7, I took an installation trace using:
./installFixPack -l <log-file-name> -t <trace-file-name>
This generated some additional data, which I sent off to IBM support.
Support also asked for the following information:
$ cd /opt/IBM/db2/V10.5/install $ ./db2ls -q -b /opt/IBM/db2/V10.5 Install Path : /opt/IBM/db2/V10.5 Feature Response File ID Level Fix Pack Feature Description --------------------------------------------------------------------------------------------------------------------- BASE_CLIENT 10.5.0.7 7 Base client support JAVA_SUPPORT 10.5.0.7 7 Java support SQL_PROCEDURES 10.5.0.7 7 SQL procedures BASE_DB2_ENGINE 10.5.0.7 7 Base server support CONNECT_SUPPORT 10.5.0.7 7 Connect support DB2_DATA_SOURCE_SUPPORT 10.5.0.7 7 DB2 data source support JDK 10.5.0.7 7 IBM Software Development Kit (SDK) for Java(TM) LDAP_EXPLOITATION 10.5.0.7 7 DB2 LDAP support INSTANCE_SETUP_SUPPORT 10.5.0.7 7 DB2 Instance Setup wizard ACS 10.5.0.7 7 Integrated Flash Copy Support COMMUNICATION_SUPPORT_TCPIP 10.5.0.7 7 Communication support - TCP/IP DB2_UPDATE_SERVICE 10.5.0.7 7 DB2 Update Service REPL_CLIENT 10.5.0.7 7 Replication tools DB2_SAMPLE_DATABASE 10.5.0.7 7 Sample database source ODBC_DATA_SOURCE_SUPPORT 10.5.0.7 7 ODBC data source support TERADATA_DATA_SOURCE_SUPPORT 10.5.0.7 7 Teradata data source support JDBC_DATA_SOURCE_SUPPORT 10.5.0.7 7 JDBC data source support ORACLE_DATA_SOURCE_SUPPORT 10.5.0.7 7 Oracle data source support SQL_SERVER_DATA_SOURCE_SUPPORT 10.5.0.7 7 SQL Server data source support CUSTOM_DATA_SOURCE_SUPPORT 10.5.0.7 7 Custom data source support FIRST_STEPS 10.5.0.7 7 First Steps GUARDIUM_INST_MNGR_CLIENT 10.5.0.7 7 Guardium Installation Manager Client SYBASE_DATA_SOURCE_SUPPORT 10.5.0.7 7 Sybase data source support
After this, support determined that INFORMIX_DATA_SOURCE_SUPPORT was not installed. This is odd, as we had been using it successfully for months. The fix for this was to manually untar and copy some files:
/bin/gunzip -c /db2home/db2inst1/fixpack7/universal/db2/aix/install/../../../db2/aix/FILES/INFORMIX_DATA_SOURCE_SUPPORT_10.5.0.7_aix64_ppc.tar.gz | (cd /opt/IBM/db2/V10.5; /bin/tar xvf -)
This required another outage as it had to be done while this instance was down, and then updates for the instance needed to be run (as root), specifically:
cd /opt/IBM/db2/V10.5/bin ./djxlinkInformix cd /opt/IBM/db2/V10.5/instance ./db2iupdt -k db2inst1
This is what finally worked. I’m told that there’s a .metadata file where the installed DB2 componnents are listed. In this file, INFORMIX_DATA_SOURCE_SUPPORT was NOT listed. This meant that DB2 did not know to update it when a fix pack was installed. This seems to be chalked up to some sort of silent failure either when DB2 was first installed, when fix pack 4 was installed, or when someone added Informix support. None of these actions were performed by me. The error was limited to only the .metadata file, though, because we have been successfully using Informix support all year.
I can also now see INFORMIX_DATA_SOURCE_SUPPORT in the db2ls output, which I hope means this won’t occur the next time I go to apply a fix pack:
$ ./db2ls -q -b /opt/IBM/db2/V10.5
Install Path : /opt/IBM/db2/V10.5
Feature Response File ID Level Fix Pack Feature Description
---------------------------------------------------------------------------------------------------------------------
BASE_CLIENT 10.5.0.7 7 Base client support
JAVA_SUPPORT 10.5.0.7 7 Java support
SQL_PROCEDURES 10.5.0.7 7 SQL procedures
BASE_DB2_ENGINE 10.5.0.7 7 Base server support
CONNECT_SUPPORT 10.5.0.7 7 Connect support
DB2_DATA_SOURCE_SUPPORT 10.5.0.7 7 DB2 data source support
JDK 10.5.0.7 7 IBM Software Development Kit (SDK) for Java(TM)
LDAP_EXPLOITATION 10.5.0.7 7 DB2 LDAP support
INSTANCE_SETUP_SUPPORT 10.5.0.7 7 DB2 Instance Setup wizard
ACS 10.5.0.7 7 Integrated Flash Copy Support
COMMUNICATION_SUPPORT_TCPIP 10.5.0.7 7 Communication support - TCP/IP
DB2_UPDATE_SERVICE 10.5.0.7 7 DB2 Update Service
REPL_CLIENT 10.5.0.7 7 Replication tools
DB2_SAMPLE_DATABASE 10.5.0.7 7 Sample database source
INFORMIX_DATA_SOURCE_SUPPORT 10.5.0.7 7 Informix data source support
ODBC_DATA_SOURCE_SUPPORT 10.5.0.7 7 ODBC data source support
TERADATA_DATA_SOURCE_SUPPORT 10.5.0.7 7 Teradata data source support
JDBC_DATA_SOURCE_SUPPORT 10.5.0.7 7 JDBC data source support
ORACLE_DATA_SOURCE_SUPPORT 10.5.0.7 7 Oracle data source support
SQL_SERVER_DATA_SOURCE_SUPPORT 10.5.0.7 7 SQL Server data source support
CUSTOM_DATA_SOURCE_SUPPORT 10.5.0.7 7 Custom data source support
FIRST_STEPS 10.5.0.7 7 First Steps
GUARDIUM_INST_MNGR_CLIENT 10.5.0.7 7 Guardium Installation Manager Client
SYBASE_DATA_SOURCE_SUPPORT 10.5.0.7 7 Sybase data source support
IBM support tells me that simply running db2_install over again would have fixed this issue, too.
Summary
It took six weeks to get from problem to resolution. Part of that is that it was not critical to the client to have this functionality working, since this client only uses the Informix nicknames about once per quarter. The severity with IBM Support was somewhat lessened because of this. Also, at least two complete weeks of that was waiting to get an outage window that I needed. So the main question this whole thing raises for me is why a very obvious error was not generated either at the time of initial install of fix pack 4 or later with fix pack 6 that something was not right. This is the first time since the bad old days when special builds came as a few files to be carefully copied that I’ve had to mess with manually copying a file to the DB2 install directory. Somewhere along the line there was a silent failure here, and no one can tell me where or why. Hopefully if someone else runs into a similar issue, this blog entry will help them. On the positive side, I now have a new skill in my toolbox.
This is also a reminder to test all functionality including federation after any fix pack. Even if there are no error messages and if core functionality seems to work fine, there can be areas like this where something is wrong.
