Using TSA/db2haicu to automate failover – Part 1: The Preparation

Edited 8/3/2012 to add a detail about db2cptsa and to put db2nodes.cfg in its own section.

I’ve actually been hating db2haicu lately. I had 3 different environments where I had to set it up over the course of about 2 weeks, and none of them went simply and smoothly. As of the writing of this post, I’ve managed to get one working by myself and one working with the help of one of our Linux gurus. The other is still hanging out there, waiting to be revisited. I’ve learned some things on the other two that I’m going to try, and if they don’t work, then it’ll be PMR-land for me.

Concepts Of db2haicu And Automated Failover

I originally heard that failover was automated in 9.5. Silly me for thinking it was fully integrated and could be set up via a few parameters. My bubble on that was burst a couple of years ago. Expect this to be like many DB2 features early in their lifecycle – like user exit was early on, this is not fully integrated or super easy.
The over all picture for automating failover is:

  1. Set up HADR using hostnames, not IP addresses
  2. Prepare for db2haicu
  3. Run db2haicu
  4. Test failovers

Behind the scenes, db2haicu is issuing Tivoli System Automation (TSA) commands. It doesn’t take much googling to find basically what it is doing, because there are details on the more mannual way there was of doing this back with 8.2. And you’ll still need root or sudo permissions for a few direct commands – both on setup and for ongoing support.

Prep Work, Part One: Having All Your Information Ready

The first thing to do is to make several decisions about your environment and how your setup will work. This generally includes:

    1. Will you have a Virtual IP for the database connection?
    2. What will you use for a network Quorum device? A network quorum is just a pingable IP address that is used to decide which node in the cluster will serve as the “active” node during a site failure, and which nodes will be offline. It should be a server that is very highly available such as a domain controller.
    3. Is there a private network between the two database servers that should be used by HADR and as a part of automated failover? If so, you’ll need details on that.

With those decisions in mind, here are the inputs you’ll need to have ready:


  1. Public IP address of both database servers
  2. Fully qualified host names of both database servers
  3. IP address of the Quorum device
  4. Names of network cards associated with the public network on the servers IF there is more than one network card on each server(for example, ‘eth0’)


  1. Virtual IP address if using one, along with the subnet mask
  2. Private IP addresses, if private network being used in addition to the public network
  3. Name of network cards for the private network, if one is being used (for example, ‘eth0’)

Prep Work, Part Two: Actual Actions to Take Before You Run db2haicu

Software Installed

Yes, by default, the TSA components are installed with db2. However, it is a fairly common failure that on installing db2 the TSA components fail to be installed. This is an installation message that can be ignored most of the time for non-production servers because you’re not likely to use TSA there. One interesting scenario that I’ve run into is that when installing on Red Hat 6, the version of TSA that is part of the 9.7 base code cannot be installed because it does not work with Red Hat 6. But with I think Fixpack 1 (though it could be 3 or 4), the version of TSA bundled with DB2 does work on Red Hat 6. But if you install from the base code and then apply the FixPack, you didn’t install the code so the FixPack won’t update it. In this case you have to manually install this component. It’s not that hard.

If you’re lucky, everything went OK on DB2 installation, and you won’t have to do this step.

To install the code(yes, the example below is from Red Hat 5.6 – it’s what I had available):
Log in as root
cd path_where_fixpack_code_is/universal/db2/linuxamd64/tsamp

# ./prereqSAM
prereqSAM: All prerequisites for the ITSAMP installation are met on operating system:
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
# ./installSAM
prereqSAM: All prerequisites for the ITSAMP installation are met on operating system:
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
SAM is currently not installed.
installSAM: The following package is not installed yet and needs to be installed:  ./Linux/i386/sam-

installSAM: A general License Agreement and License Information specifically for System Automation will be shown. Scroll down usi                             ng the Enter key (line by line) or Space bar (page by page). At the end you will be asked to accept the terms to be allowed to in                             stall the product. Select Enter to continue.

installSAM: To accept all terms of the preceding License Agreement and License Information type 'y', anything else to decline.

installSAM: You accepted the terms in License Agreement and License information. System Automation will now be installed.

installSAM: Installing System Automation on platform: x86_64
installSAM: Packages will be installed from directory:  ./Linux/i386

installSAM: Installing

installSAM: Installing

installSAM: Installing
package rsct.core.cimrm is not installed

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing

installSAM: Installing
installSAM: Installed System Automation package and prerequisites:

installSAM: The following license is installed:
Product: IBM Tivoli System Automation for Multiplatforms (Try & Buy License) 3.2
Evaluation Period: 90 days (90 days left)
Creation date: Wed 19 Aug 2009 12:00:01 AM CDT
Expiration date: Thu 31 Dec 2037 12:00:01 AM CST

Subsystem         Group            PID     Status
 ctrmc            rsct             1627    active
installSAM: Warning: Must set CT_MANAGEMENT_SCOPE=2

installSAM: All packages were installed successfully.


HADR Setup

It’s really just a straight forward HADR setup – you’ll want to make sure you have your HADR peer window set. See my post on HADR for more details on HADR.

Hosts File

TSA is really picky about the hosts file. You either need to have root (sudo is fine) access or have access to someone with root or sudo access so you can edit it if needed. Note that editing a hosts file in certain ways can break other things on the system, so you’ll likely want your Systems Admin or whatever other engineers you’re working with on the project aware that you’re making changes.

What TSA requires is that both hosts for the HADR pair have one line and only one line each in the hosts file. The entries must be of this format:

ipaddress  shortname   longname


ipaddress  longname    shortname

Where longname is the fully qualified server name and shortname is a shorter version. You need to have HADR set up with whatever comes first – shortname or longname.


Make sure that db2nodes.cfg (in $INSTHOME/sqllib) uses the same as hadr(longname or shortname), and that the hostname command returns the same of these two choices.

Longname vs. Shortname

It doesn’t really matter whether you pick the longname or the shortname, you just have to be 100% consistent. IP addresses don’t work. And the errors that db2haicu gives you are in no way helpful to determine you have a mismatch here. The errors are extremely generic.


Sometimes on Fixpack, the TSA scripts don’t get updated as they should. To make sure you’ve got them up to date, you always should do the following:

Log in as root
cd path_where_fixpack_code_is/universal/db2/linuxamd64/install
cd /opt/IBM/db2/V9.7/install

# ./db2cptsa
DBI1119I  The version of the DB2 High Availability (HA) scripts for the
      IBM Tivoli System Automation for Multiplatforms (SA MP) found in
      /usr/sbin/rsct/sapolicies/db2 is the same version as the version
      of the scripts on the current DB2 install media.


You need DB2 HA scripts to use SA MP with the DB2 HA feature.

SA MP and the DB2 HA feature being installed from the DB2 install media
require DB2 HA scripts with a version the same as or higher than the
version of the scripts also on the DB2 install media.

The version of the DB2 HA scripts currently installed the same as the
version of the scripts on the DB2 install media.

User response:

No action is required.

The above is what we expect to see. If you don’t see the above, you will most likely see it telling you that it has updated the scripts. If this returns an error, you need to investigate further. This must be successful before you can move on.


Ok, this is the last of the prepartion steps. preprpnode is a command that you run as root. I think that all it really does is trade public keys between the two servers. But it can also fail, and it must be run successfully before db2haicu will work. Here’s what it looks like. You don’t have to cd anywhere special to run it.

# preprpnode spp05db01r spp05db02r

Yep, that simple. It doesn’t return any output unless there is an error. The most common things to cause an error here are not having the TSA component installed (in which case you get a command not found), or to have a firewall either on the server or between the servers blocking traffic. The latter looks like this:

# preprpnode svp35db01b svp35db02b
/usr/sbin/rsct/bin/lsrsrc-api: 2612-022 A session could not be established with the RMC daemon on svp35db02b.
preprpnode: Unable to obtain the public key from svp35db02b.

If you see the above, db2haicu will also fail.

You need to check if iptables is running or look in your environment for another firewall. Assuming you are allowed to do so, you can check and stop iptables using this(as root):

# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --             state RELATED,ESTABLISHED
2    ACCEPT     icmp --  
3    ACCEPT     all  --  
4    ACCEPT     tcp  --             state NEW tcp dpt:22
5    REJECT     all  --             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

You have new mail in /var/spool/mail/root

# /etc/init.d/iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]

# /etc/init.d/iptables status
iptables: Firewall is not running.

Other Posts In This Series

This series consists of four posts:
Using TSA/db2haicu to automate failover – Part 1: The Preparation
Using TSA/db2haicu to automate failover – Part 2: How it looks if it goes smoothly
Using TSA/db2haicu to Automate Failover Part 3: Testing, Ways Setup can go Wrong and What to do.
“Using TSA/db2haicu to automate failover Part 4: Dealing with Problems After Setup

Search this blog on “TSA” for other posts on TSA issues and tips.


Great whitepaper on db2haicu setup:

Ember Crooks
Ember Crooks

Ember is always curious and thrives on change. She has built internationally recognized expertise in IBM Db2, spent a year working with high-volume MySQL, and is now learning Snowflake. Ember shares both posts about her core skill sets and her journey learning Snowflake.

Ember lives in Denver and work from home

Articles: 556


  1. How do we configure TSA for multiple instances on a server??

    can you provide sample xml file to configure multiple instances

    • Unfortunately, I don’t have any experience to share here. My best guess is that you would run db2haicu as each instance owner. (after doing all your prep work) I have done multiple databases on one instance. I haven’t been using the xml method, though I’ve been thinking lately that I should try it.
      We try very hard to stick to a one database one instance one server setup for our production environments, and while I’ve been also doing HADR/TSA in test environments now and then, usually that is a client that is willing to have separate servers even for test to better mimic prod.

  2. I had also run into the similar installation problem on Red Hat 6.3 while installing the version of TSA that is part of the 9.7 FP5, base code cannot be installed because it does not work with Red Hat 6. The TSA team suggested me to install “3.2.1-TIV-ITSAMP-FP0003, Tivoli System Automation for Multiplatforms”. That worked perfectly for me and I was able to install the TSA sucessfully. One more thing, I had to use the license file, sam32.lic , that comes with the DB2 9.7 install media, since the link provided was a fixpack ( no license included ).

    Hope this information would be useful to someone.

  3. Hello, I ended with a partial TSA installation , neither ./installSAM nor ./uninstallSAM finish succesfully. The version is bundled with v97fp7 .Can you propose a solution, this is the uninstall log :

    /usr/sbin/rsct/bin/lsrsrc-api: 2612-022 A session could not be established with the RMC daemon on “local_node”.
    uninstallSAM: Uninstalling System Automation on platform x86_64
    uninstallSAM: Uninstallin
    uninstallSAM: Error: Failed with return-code 255 : sam-
    uninstallSAM: Any packages failed uninstallation. See details below
    uninstallSAM: Error: Failed with return-code 255 : sam-
    error: %postun(sam- scriptlet failed, exit status 1

    Uninstalling IBM Tivoli System Automation for Multiplatforms (SA MP) …
    0513-084 There were no records that matched your request.
    0513-084 There were no records that matched your request.
    0513-084 There were no records that matched your request.
    0513-071 The ctrmc Subsystem has been added.
    Cannot add ctrmc to init script structure
    Warning: An error was returned by rmcctrl while attempting to restart the rmc subsystem, exit code=0

    uninstallSAM: No uninstallation was performed

    • My first thoughts are to go find a good Linux admin or to open a PMR. I’m sorry I don’t have any good answers on that for you – I have never seen it. If it was AIX, I’d be trying smit to remove it.

    • I know my response is too late 🙂 .. But if it can help someone thats great

      I had same issue. I removed /opt/rsct and ran uninstallSAM.. it worked fine

    • ACR and TSA serve different purposes. ACR does not automate failover in any way – with ACR, you have to have someone manually issue the takeover commands for HADR to failover. The only place you really choose between ACR and TSA is when you choose whether to use a virtual IP or not. Using the virtual IP is easier because not all applications do well with caching an alternate server.

  4. Hi Ember,
    Thanks for this detailed blog. I faced the below issue while installing TSAMP and the prereqSAM failed with this error:

    prereqSAM: Using default prerequisite checking on the following rpm package ‘libstdc++4.-32bit’ ‘x86_64’
    prereqSAM: Error: The rpm package is installed, but the following part was not found ‘libstdc++46-32bit-4.6.9-0.11.38.x86_64
    libstdc++43-32bit-4.6.9-0.11.38.x86_64’ ‘/usr/lib/’

    prereqSAM: Using default prerequisite checking on the following rpm package ‘pam-32bit’
    prereqSAM: rpm package and version installed ‘pam-32bit’

    1 missing package: libstdc++46-32bit

    I struggled for 2 days with it and found the following PMR with IBM:

    So the fix is, install TSAMP with this command and it works fine:
    ./installSAM –noprereqcheck

  5. Hi,
    I’ve tried to startup de Cluster on DB2 10.1 with a couple of VM on SUSE 10
    Everthing works fins untl I run db2haicu on the Principal server. The last step

    I receive this error:
    2014-07-23- E1380534G352 LEVEL: Error
    PID : 32289 TID : 2858981072 PROC : db2haicu
    INSTANCE: db2inst1 NODE : 000
    HOSTNAME: hadrprin
    FUNCTION: DB2 UDB, high avail services, sqlhaUIMain, probe:900
    MESSAGE : ECF=0x90000530=-1879046864=ECF_SQLHA_NOTINITED
    Library uninitialized error

    The only weird thing I found is that cluster finds 4 NICs when I only have 1 eth0 on each one and the lo.
    I also tried to do it with xml file and I get the same error

    Could be a kind of multipath on Linux?

    Best regards

  6. Hi,
    I have a question regarding the usage of Virtual IP. Concerning I have a HADR nodes in different subnets, I am unable to use the Virtual IP. Nevertheless, I need to connect to my HADR database from withing e.g. a script. Shall I for example catalog the DB2 HADR database on a machine running a script? If not, how shall I use the hadr database? thanks

    • ACR(automatic client reroute) still works, and can be used in those situations. It is easy to use if you’re using JDBC type 2 or traditional db2 clients. In those situations, all you have to do is to define an alternate server at the DB2 server, and the clients will pick it up on their next database connection. Functionality should be built in to handle failovers. In fact, we still define an alternate server (of the virtual IP) even when using a virtual IP to get that retry functionality.

      If your application is using type 4 JDBC drivers or some other connection methodology, your app may have a place to specify the alternate server manually.

  7. I have a problem with additng virutal IP:
    “Cannot create a virtual IP because there are no networks defined for this cluster. Configure the networks before adding virtual IPs.”
    Do you happen to see that error?

    • I have not seen that error. When you get it, have you defined at least one network card on each of the primary and the standby through db2haicu?

  8. Thanks Ember for another informative post.
    How do the resource groups get created/defined ?
    I am troubleshooting current TSA setup but when I run “lsrg” or “lssam” it says
    lsrg: There are no resource groups defined.
    lssam: No resource groups defined or cluster is offline!

    The lsrpdomain and lsrpnode comamnds return valid info.
    Or when I run $ lsrsrc -Ab IBM.ResourceGroup —all I get is..
    Resource Persistent and Dynamic Attributes for IBM.ResourceGroup

      • Maybe someone else may find this useful.
        After going down some rabbit holes, we finally got TSA working, the problem we were having stemmed from an invalid license

        /tmp $ samlicm -s
        Error: Product license is invalid and needs to be upgraded.

        Once we added a valid license and followed the setup steps again it worked.

        • Hi Johnf,
          I am in the same situation.How did you added the license ?let me know on the steps that you followed plese

          • Hi jayakrishna,

            samlicm -s …to check if you have a valid license
            samlicm -i …to add or install a new license file
            license files were stored in /var/opt/sam/lic

            Hope that helps.

    • I have not. I have gone for separate is for separate databases since that is what is stated as supported, though I do not like it, and would generally prefer the other configuration from a usability standpoint.

  9. Hello
    II have configured TSA with multiple instances. The right way is to create one XML file for each instance, and execute db2haicu -d filename.xml with the right instance user.

  10. Hi I am gettting command not found when run preprpnode server1 server2.
    how to resolve this issue.

    • You’re running that as root, right? Make sure that TSAMP installed properly when you installed DB2.

  11. Hello Ember,

    I have to setup TSA for single instance with 3 databases. Please let me me how many VIPs required and which IP whould be given as alternate server details ?

  12. samlicm -s
    Error: Product license is invalid and needs to be upgraded.
    I have nodelock file inside /var/opt/sam/lic:
    How do I update the license please

  13. Hi mrs Crooks

    Thanks for your helpful blog !

    I have issue with prepare TSA

    I am getting error when run preprpnode server1 server2.

    preprpnode: The node name XXXXXX cannot be resolved.


    • I’m late in responding here, but have you made sure your hosts files are all good and you can ping the other server using that hostname?

  14. Hi Ember,

    In our PROD environment , I have setup HADR/TSA in a single instance multiple databases.But now client requirement is to have different instance for each databases of different applications and setup HADR/TSA over there. I did create a new instance and tried running db2haicu and it gave some errors related to mount points.Hopefully what i think here is since we do not have different /db2home file systems in our server , all the instances are getting created in same /home filesystem and hence if we mount different db2home filesystems and then create instances on different home filesystems then we should be able to run db2haicu and setup TSA in each instance separately.

    Need you to shed some lights here if am wrong since am not able to find out any proper document to setup db2haicu on multiple instances on same server.

    Thanks & Regards,
    Athulya Sankar

    • For multiple instances, you generally only have one TSAMP domain. It shouldn’t matter where the instances’ home directories are. Each one has to have a separate home directory, but placing them in separate file systems is merely a best practice, and not technically required.

  15. Hi Ember Crooks,
    Recently We build HADR with Db2 on RHEL OS.Planning to use TSAMP for automatic failover on this environment.Successfully installed TSA while installing Db2. After that we applied Db2 Advanced Work Group server Edition licence on Db2 product.
    Checked samlicm -s command on both linux servers but is showing output like

    Product: IBM Tivoli System Automation for Multiplatforms (Try & Buy License)
    Evaluation Period: 90 days (60 days left)
    Creation date: Fri 16 Aug 2013 12:00:01 AM CEST
    Expiration date: Thu 31 Dec 2037 12:00:01 AM CET

    We already have licence for Db2 product.We will get licence for TSA with Purchasing Db2 licence or not.Whether we need to buy separate licence for TSAMP as per output of samlicm -s cmd. Please clarify me on my queries.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.