Strange Problem Installing DB2 11.1, Fix/Mod Pack 1 on Ubuntu

I ran into an interesting problem installing 11.1, fix/mod pack 1 on Ubuntu, and thought I would share. Please read to the bottom before trying anything, as this entry goes through things I tried that did not work before I got to what did work.

The Problem

I needed to rebuild a sandbox VM, and use Ubuntu for my sandboxes most of the time. I installed DB2 from fix/mod pack 1 of DB2 11.1. Running my favorite install method (db2_install), I received this error message, after step 48:

Segmentation fault (core dumped)
/db2home/db2inst1/universal/db2/linuxamd64/install/db2setup: 1: /db2home/db2inst1/universal/db2/linuxamd64/install/db2setup: /usr/local/bin/db2ls: not found

I even rebuilt the VM to see if I had made a mistake somewhere. I ran db2prereqcheck again to ensure I had all the prerequisite packages.

The next thing I tried was installing using db2setup. As much as I hate working with a GUI, IBM has been pushing it for a while now. db2setup received this error:

./db2setup: 1: ./db2setup: /usr/local/bin/db2ls: not found

When I tried to create an instance on either of these installations, it failed with an error:

# ./db2icrt -u db2fenc1 db2inst1
DBI1446I  The db2icrt command is running.


DB2 installation is being initialized.

 Total number of tasks to be performed: 4 
Total estimated time for all tasks to be performed: 309 second(s) 

Task #1 start
Description: Setting default global profile registry variables 
Estimated time 1 second(s) 
Task #1 end 

Task #2 start
Description: Initializing instance list 
Estimated time 5 second(s) 
Task #2 end 

Task #3 start
Description: Configuring DB2 instances 
Estimated time 300 second(s) 
Task #3 end 

Task #4 start
Description: Updating global profile registry 
Estimated time 3 second(s) 
Task #4 end 

A minor error occurred during the execution.

For more information see the DB2 installation log at "/tmp/db2icrt.log.16398".
DBI1264E  This program failed. Errors encountered during execution were
      written to the installation log file. Program name:
      db2icrt. Log file name: /tmp/db2icrt.log.16398.

Explanation: 

This message is returned when some processes and operations have failed.
Detailed information about the error was written to the log file.

User response: 

Contact IBM support to get assistance in resolving this issue. Keep the
log file intact as this file is an important reference for IBM support.

Looking at the log file for the db2icrt, I saw this:

# cat /tmp/db2icrt.log.16398

DB2 Setup log file started at:  Wed Jan 04 06:45:39 PM 2017 PST
============================================================

Operating system information: Linux 4.4.0-31-generic.#50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 
                                        
                                        
New instances:                          
    Instance name:                          db2inst1
        Instance user information:          
            User name:                      db2inst1
        Fenced user information:            
            User name:                      db2fenc1
                                        
                                        
                                        
                                        

Setting default global profile registry variables :.......Success 
The maximum number of logical nodes was not specified for the instance
"db2inst1". Default parameters will be used.

Initializing instance list :.......Success 
ERROR: ERROR:  The embedded runtime path in /opt/ibm/db2/V11.1/bin/db2 is incorrect:
        /DoNotCreateThisPath_marker1_for_chglibpath:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/INST/lib:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/INST/lib/gskit:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/engn/lib:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/common/lib/Linux_AMD64:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/das/lib/Linux_AMD64:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/bldsupp/lib:/view/db2_v111m1fp1_linuxamd64_s1612051900/vbs/system/fm/lib/Linux_AMD64:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/fillspacexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:/marker2_for_chglibpath:/opt/ibm/db2/V11.1/lib64:/opt/ibm/db2/V11.1/lib64/gskit

ACTION: Please run /opt/ibm/db2/V11.1/install/db2chgpath and rerun the previous command.
        Contact IBM Support if the problem persists.
ERROR: An error occurred while configuring the instance "db2inst1".

Configuring DB2 instances :.......Failure 
Updating global profile registry :.......Success 

Following the instructions, I ran db2chgpath as root:

# /opt/ibm/db2/V11.1/install/db2chgpath
The output will be saved in /tmp/db2chgpath.log.16598
Starting the update of runtime path ...
Wed Jan  4 18:46:36 PST 2017

Changing the embedded runtime path for DB2 deliverables installed
under /opt/ibm/db2/V11.1 from /opt/ibm/db2/V11.1/ to /opt/ibm/db2/V11.1/
---> Please wait

Output was saved in the log file /tmp/db2chgpath.log.16598
Wed Jan  4 18:47:14 PST 2017
Program db2chgpath completed successfully.

After that, I was able to successfully create a db2 instance.

Before starting DB2, I had to find the default license file from the install code and apply it:

db2licm -a /db2home/db2inst1/universal/db2/license/db2ese.lic

Because everything was so odd with this, I then ran db2val to validate my DB2 installation. It failed on validating the instance I just started, saying:

DBI1342E  Instance db2inst1 does not exist in the current DB2
      copy.

Explanation: 

The specified instance does not exist in the current DB2 copy and will
not be validated.

When installing 11.1 from the GA code, I see 57 steps, and the above failures happened after only number 48, so there are clearly a significant number of things missing. It is clear that simply powering through and trying to address the errors one by one is probably not a good idea. The last issue could probably have been addressed by manually setting the one or more parameters at some level in the db2 registry. db2ilist from the code on /opt returned the instance name, while db2ilist from sqllib did not. But at some point the layers of errors point to the larger issue that needs to be addressed instead.

I get exactly the same error if I first install the 11.1 base code and then try to apply FixPack 11, which is particularly perplexing:

Segmentation fault (core dumped)
./installFixPack: 1: ./installFixPack: /usr/local/bin/db2ls: not found

The Fix

Without IBM listing it as a prerequisite, rpm is required. It can be installed using this syntax:

apt-get install rpm

After adding this to my list of pre-install steps, the install from the fix/mod pack 1 code is successful.

The Cause

An APAR has been opened for this – IT18668. I’ll add the link for this when it is published.

It sounds to me like this is part of the TSAMP install requirements. The funny thing is that TSAMP is expressly not supported on Ubuntu. It sounds like IBM will have it fixed in a future fix/mod pack. I don’t think Ubuntu is used for DB2 in production scenarios very often, so the install process is likely less thoroughly tested, but this is just my conjecture. Most of the DB2 on Linux that I see is RedHat.

This is one of the most severe install defects I have seen introduced in a fix pack. While it doesn’t affect normal operations of a running database, if you were actually using Ubuntu in production, you might be surprised by this during an update outage and have to back out or take a longer outage to resolve it. I did not observe this problem in the 11.1 GA code. IBM tells us that DB2 is now more stable than ever because changes to DB2 are used in their cloud environment(s) before they are released to the public. However, that cloud environment is deployed on specific hardware and uses specific OSes. Less frequently used OSes like Ubuntu may not have their code as thoroughly tested. Again, this is just my guess – I have no inside knowledge on this topic.

It is also interesting to note that this is not the first DB2 Ubuntu install issue where the workaround is installing rpm. DB2 10.1 GA had an install issue that was solved by installing rpm. Thanks to @Muralichava for pointing that out.

Thanks, IBM!

IBM was wonderfully helpful and amazing while I was working on this issue. Since I was just installing a random sandbox and not a real, licensed, environment, I didn’t have the right to open a PMR on it. I tweeted about the issue right before logging off for the night around 9 PM, and just a half hour into my work day the next day, they were asking for more info. Before noon, they had a solution. Big thanks to Jessica Rockwood (@jrockwood) and Abhinav Goyal (@agoyal_DB2).