Edited 8/3/2012 to add a detail about db2cptsa and to put db2nodes.cfg in its own section.
I’ve actually been hating db2haicu lately. I had 3 different environments where I had to set it up over the course of about 2 weeks, and none of them went simply and smoothly. As of the writing of this post, I’ve managed to get one working by myself and one working with the help of one of our Linux gurus. The other is still hanging out there, waiting to be revisited. I’ve learned some things on the other two that I’m going to try, and if they don’t work, then it’ll be PMR-land for me.
Concepts Of db2haicu And Automated Failover
I originally heard that failover was automated in 9.5. Silly me for thinking it was fully integrated and could be set up via a few parameters. My bubble on that was burst a couple of years ago. Expect this to be like many DB2 features early in their lifecycle – like user exit was early on, this is not fully integrated or super easy.
The over all picture for automating failover is:
- Set up HADR using hostnames, not IP addresses
- Prepare for db2haicu
- Run db2haicu
- Test failovers
Behind the scenes, db2haicu is issuing Tivoli System Automation (TSA) commands. It doesn’t take much googling to find basically what it is doing, because there are details on the more mannual way there was of doing this back with 8.2. And you’ll still need root or sudo permissions for a few direct commands – both on setup and for ongoing support.
Prep Work, Part One: Having All Your Information Ready
The first thing to do is to make several decisions about your environment and how your setup will work. This generally includes:
- Will you have a Virtual IP for the database connection?
- What will you use for a network Quorum device? A network quorum is just a pingable IP address that is used to decide which node in the cluster will serve as the “active” node during a site failure, and which nodes will be offline. It should be a server that is very highly available such as a domain controller.
- Is there a private network between the two database servers that should be used by HADR and as a part of automated failover? If so, you’ll need details on that.
With those decisions in mind, here are the inputs you’ll need to have ready:
- Public IP address of both database servers
- Fully qualified host names of both database servers
- IP address of the Quorum device
- Names of network cards associated with the public network on the servers IF there is more than one network card on each server(for example, ‘eth0’)
- Virtual IP address if using one, along with the subnet mask
- Private IP addresses, if private network being used in addition to the public network
- Name of network cards for the private network, if one is being used (for example, ‘eth0’)
Prep Work, Part Two: Actual Actions to Take Before You Run db2haicu
Yes, by default, the TSA components are installed with db2. However, it is a fairly common failure that on installing db2 the TSA components fail to be installed. This is an installation message that can be ignored most of the time for non-production servers because you’re not likely to use TSA there. One interesting scenario that I’ve run into is that when installing on Red Hat 6, the version of TSA that is part of the 9.7 base code cannot be installed because it does not work with Red Hat 6. But with I think Fixpack 1 (though it could be 3 or 4), the version of TSA bundled with DB2 does work on Red Hat 6. But if you install from the base code and then apply the FixPack, you didn’t install the code so the FixPack won’t update it. In this case you have to manually install this component. It’s not that hard.
If you’re lucky, everything went OK on DB2 installation, and you won’t have to do this step.
To install the code(yes, the example below is from Red Hat 5.6 – it’s what I had available):
Log in as root
# ./prereqSAM prereqSAM: All prerequisites for the ITSAMP installation are met on operating system: Red Hat Enterprise Linux Server release 5.6 (Tikanga) # ./installSAM prereqSAM: All prerequisites for the ITSAMP installation are met on operating system: Red Hat Enterprise Linux Server release 5.6 (Tikanga) SAM is currently not installed. installSAM: The following package is not installed yet and needs to be installed: ./Linux/i386/sam-184.108.40.206-10347.i386.rpm installSAM: A general License Agreement and License Information specifically for System Automation will be shown. Scroll down usi ng the Enter key (line by line) or Space bar (page by page). At the end you will be asked to accept the terms to be allowed to in stall the product. Select Enter to continue. installSAM: To accept all terms of the preceding License Agreement and License Information type 'y', anything else to decline. y installSAM: You accepted the terms in License Agreement and License information. System Automation will now be installed. installSAM: Installing System Automation on platform: x86_64 installSAM: Packages will be installed from directory: ./Linux/i386 installSAM: Installing ./Linux/i386/src-220.127.116.11-10320.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils-18.104.22.168-10320.i386.rpm ./Linux/i386/rsct.core-22.214.171.124-10320.i386.rpm ./Linux/i386/rsct.basic-126.96.36.199-10320.i386.rpm ./Linux/i386/rsct.opt.storagerm-188.8.131.52-10320.i386.rpm installSAM: Installing ./Linux/i386/rsct.64bit-184.108.40.206-10313.x86_64.rpm package rsct.core.cimrm is not installed installSAM: Installing ./Linux/i386/src.msg.de_DE-220.127.116.11-0.i386.rpm ./Linux/i386/src.msg.de_DE.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/src.msg.de_DE@euro-22.214.171.124-0.i386.rpm ./Linux/i386/src.msg.de_DE.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.es_ES-188.8.131.52-0.i386.rpm ./Linux/i386/src.msg.es_ES.ISO-8859-1-184.108.40.206-0.i386.rpm ./Linux/i386/src.msg.es_ES@euro-220.127.116.11-0.i386.rpm ./Linux/i386/src.msg.es_ES.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.fr_FR-22.214.171.124-0.i386.rpm ./Linux/i386/src.msg.fr_FR.ISO-8859-1-126.96.36.199-0.i386.rpm ./Linux/i386/src.msg.fr_FR@euro-188.8.131.52-0.i386.rpm ./Linux/i386/src.msg.fr_FR.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.it_IT-220.127.116.11-0.i386.rpm ./Linux/i386/src.msg.it_IT.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/src.msg.it_IT@euro-22.214.171.124-0.i386.rpm ./Linux/i386/src.msg.it_IT.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.ja_JP.eucJP-188.8.131.52-0.i386.rpm ./Linux/i386/src.msg.ja_JP.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.ko_KR.eucKR-220.127.116.11-0.i386.rpm ./Linux/i386/src.msg.ko_KR.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.pt_BR-22.214.171.124-0.i386.rpm ./Linux/i386/src.msg.pt_BR.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.zh_CN.GB2312-188.8.131.52-0.i386.rpm ./Linux/i386/src.msg.zh_CN.GB18030-184.108.40.206-0.i386.rpm ./Linux/i386/src.msg.zh_CN.GBK-220.127.116.11-0.i386.rpm ./Linux/i386/src.msg.zh_CN.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/src.msg.zh_TW-22.214.171.124-0.i386.rpm ./Linux/i386/src.msg.zh_TW.Big5-126.96.36.199-0.i386.rpm ./Linux/i386/src.msg.zh_TW.eucTW-188.8.131.52-0.i386.rpm ./Linux/i386/src.msg.zh_TW.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.de_DE-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.de_DE.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.de_DE@euro-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.de_DE.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.es_ES-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.es_ES.ISO-8859-1-184.108.40.206-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.es_ES@euro-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.es_ES.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.fr_FR-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.fr_FR.ISO-8859-1-126.96.36.199-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.fr_FR@euro-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.fr_FR.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.it_IT-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.it_IT.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.it_IT@euro-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.it_IT.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.ja_JP.eucJP-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.ja_JP.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.ko_KR.eucKR-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.ko_KR.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.pt_BR-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.pt_BR.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.zh_CN.GB2312-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.zh_CN.GB18030-184.108.40.206-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.zh_CN.GBK-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.zh_CN.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.utils.msg.zh_TW-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.zh_TW.Big5-126.96.36.199-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.zh_TW.eucTW-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.utils.msg.zh_TW.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.de_DE-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.msg.de_DE.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/rsct.core.msg.de_DE@euro-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.msg.de_DE.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.es_ES-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.msg.es_ES.ISO-8859-1-184.108.40.206-0.i386.rpm ./Linux/i386/rsct.core.msg.es_ES@euro-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.msg.es_ES.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.fr_FR-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.msg.fr_FR.ISO-8859-1-126.96.36.199-0.i386.rpm ./Linux/i386/rsct.core.msg.fr_FR@euro-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.msg.fr_FR.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.it_IT-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.msg.it_IT.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/rsct.core.msg.it_IT@euro-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.msg.it_IT.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.ja_JP.eucJP-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.msg.ja_JP.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.ko_KR.eucKR-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.msg.ko_KR.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.pt_BR-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.msg.pt_BR.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.zh_CN.GB2312-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.msg.zh_CN.GB18030-184.108.40.206-0.i386.rpm ./Linux/i386/rsct.core.msg.zh_CN.GBK-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.core.msg.zh_CN.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.core.msg.zh_TW-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.core.msg.zh_TW.Big5-126.96.36.199-0.i386.rpm ./Linux/i386/rsct.core.msg.zh_TW.eucTW-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.core.msg.zh_TW.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.de_DE-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.basic.msg.de_DE.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/rsct.basic.msg.de_DE@euro-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.basic.msg.de_DE.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.es_ES-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.basic.msg.es_ES.ISO-8859-1-184.108.40.206-0.i386.rpm ./Linux/i386/rsct.basic.msg.es_ES@euro-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.basic.msg.es_ES.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.fr_FR-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.basic.msg.fr_FR.ISO-8859-1-126.96.36.199-0.i386.rpm ./Linux/i386/rsct.basic.msg.fr_FR@euro-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.basic.msg.fr_FR.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.it_IT-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.basic.msg.it_IT.ISO-8859-1-18.104.22.168-0.i386.rpm ./Linux/i386/rsct.basic.msg.it_IT@euro-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.basic.msg.it_IT.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.ja_JP.eucJP-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.basic.msg.ja_JP.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.ko_KR.eucKR-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.basic.msg.ko_KR.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.pt_BR-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.basic.msg.pt_BR.UTF-8-126.96.36.199-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.zh_CN.GB2312-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.basic.msg.zh_CN.GB18030-184.108.40.206-0.i386.rpm ./Linux/i386/rsct.basic.msg.zh_CN.GBK-220.127.116.11-0.i386.rpm ./Linux/i386/rsct.basic.msg.zh_CN.UTF-8-18.104.22.168-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.basic.msg.zh_TW-22.214.171.124-0.i386.rpm ./Linux/i386/rsct.basic.msg.zh_TW.Big5-126.96.36.199-0.i386.rpm ./Linux/i386/rsct.basic.msg.zh_TW.eucTW-188.8.131.52-0.i386.rpm ./Linux/i386/rsct.basic.msg.zh_TW.UTF-8-184.108.40.206-0.i386.rpm installSAM: Installing ./Linux/i386/sam-220.127.116.11-10347.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.de_DE-18.104.22.168-0.i386.rpm ./Linux/i386/sam.msg.de_DE.ISO-8859-1-22.214.171.124-0.i386.rpm ./Linux/i386/sam.msg.de_DE@euro-126.96.36.199-0.i386.rpm ./Linux/i386/sam.msg.de_DE.UTF-8-188.8.131.52-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.es_ES-184.108.40.206-0.i386.rpm ./Linux/i386/sam.msg.es_ES.ISO-8859-1-220.127.116.11-0.i386.rpm ./Linux/i386/sam.msg.es_ES@euro-18.104.22.168-0.i386.rpm ./Linux/i386/sam.msg.es_ES.UTF-8-22.214.171.124-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.fr_FR-126.96.36.199-0.i386.rpm ./Linux/i386/sam.msg.fr_FR.ISO-8859-1-188.8.131.52-0.i386.rpm ./Linux/i386/sam.msg.fr_FR@euro-184.108.40.206-0.i386.rpm ./Linux/i386/sam.msg.fr_FR.UTF-8-220.127.116.11-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.it_IT-18.104.22.168-0.i386.rpm ./Linux/i386/sam.msg.it_IT.ISO-8859-1-22.214.171.124-0.i386.rpm ./Linux/i386/sam.msg.it_IT@euro-126.96.36.199-0.i386.rpm ./Linux/i386/sam.msg.it_IT.UTF-8-188.8.131.52-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.ja_JP.eucJP-184.108.40.206-0.i386.rpm ./Linux/i386/sam.msg.ja_JP.UTF-8-220.127.116.11-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.ko_KR.eucKR-18.104.22.168-0.i386.rpm ./Linux/i386/sam.msg.ko_KR.UTF-8-22.214.171.124-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.pt_BR-126.96.36.199-0.i386.rpm ./Linux/i386/sam.msg.pt_BR.UTF-8-188.8.131.52-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.zh_CN.GB2312-184.108.40.206-0.i386.rpm ./Linux/i386/sam.msg.zh_CN.GB18030-220.127.116.11-0.i386.rpm ./Linux/i386/sam.msg.zh_CN.GBK-18.104.22.168-0.i386.rpm ./Linux/i386/sam.msg.zh_CN.UTF-8-22.214.171.124-0.i386.rpm installSAM: Installing ./Linux/i386/sam.msg.zh_TW-126.96.36.199-0.i386.rpm ./Linux/i386/sam.msg.zh_TW.Big5-188.8.131.52-0.i386.rpm ./Linux/i386/sam.msg.zh_TW.eucTW-184.108.40.206-0.i386.rpm ./Linux/i386/sam.msg.zh_TW.UTF-8-220.127.116.11-0.i386.rpm installSAM: Installing ./Linux/i386/rsct.opt.stackdump-18.104.22.168-10320.i386.rpm installSAM: Installing ./Linux/i386/sam.sappolicy-22.214.171.124-10347.i386.rpm installSAM: Installing ./Linux/i386/sam.adapter-126.96.36.199-10347.i386.rpm installSAM: Installed System Automation package and prerequisites: rsct.64bit-188.8.131.52-10313.x86_64 rsct.basic-184.108.40.206-10320.i386 rsct.basic.msg.de_DE-220.127.116.11-0.i386 rsct.basic.msg.de_DE@euro-18.104.22.168-0.i386 rsct.basic.msg.de_DE.ISO-8859-1-22.214.171.124-0.i386 rsct.basic.msg.de_DE.UTF-8-126.96.36.199-0.i386 rsct.basic.msg.es_ES-188.8.131.52-0.i386 rsct.basic.msg.es_ES@euro-184.108.40.206-0.i386 rsct.basic.msg.es_ES.ISO-8859-1-220.127.116.11-0.i386 rsct.basic.msg.es_ES.UTF-8-18.104.22.168-0.i386 rsct.basic.msg.fr_FR-22.214.171.124-0.i386 rsct.basic.msg.fr_FR@euro-126.96.36.199-0.i386 rsct.basic.msg.fr_FR.ISO-8859-1-188.8.131.52-0.i386 rsct.basic.msg.fr_FR.UTF-8-184.108.40.206-0.i386 rsct.basic.msg.it_IT-220.127.116.11-0.i386 rsct.basic.msg.it_IT@euro-18.104.22.168-0.i386 rsct.basic.msg.it_IT.ISO-8859-1-22.214.171.124-0.i386 rsct.basic.msg.it_IT.UTF-8-126.96.36.199-0.i386 rsct.basic.msg.ja_JP.eucJP-188.8.131.52-0.i386 rsct.basic.msg.ja_JP.UTF-8-184.108.40.206-0.i386 rsct.basic.msg.ko_KR.eucKR-220.127.116.11-0.i386 rsct.basic.msg.ko_KR.UTF-8-18.104.22.168-0.i386 rsct.basic.msg.pt_BR-22.214.171.124-0.i386 rsct.basic.msg.pt_BR.UTF-8-126.96.36.199-0.i386 rsct.basic.msg.zh_CN.GB18030-188.8.131.52-0.i386 rsct.basic.msg.zh_CN.GB2312-184.108.40.206-0.i386 rsct.basic.msg.zh_CN.GBK-220.127.116.11-0.i386 rsct.basic.msg.zh_CN.UTF-8-18.104.22.168-0.i386 rsct.basic.msg.zh_TW-22.214.171.124-0.i386 rsct.basic.msg.zh_TW.Big5-126.96.36.199-0.i386 rsct.basic.msg.zh_TW.eucTW-188.8.131.52-0.i386 rsct.basic.msg.zh_TW.UTF-8-184.108.40.206-0.i386 rsct.core-220.127.116.11-10320.i386 rsct.core.msg.de_DE-18.104.22.168-0.i386 rsct.core.msg.de_DE@euro-22.214.171.124-0.i386 rsct.core.msg.de_DE.ISO-8859-1-126.96.36.199-0.i386 rsct.core.msg.de_DE.UTF-8-188.8.131.52-0.i386 rsct.core.msg.es_ES-184.108.40.206-0.i386 rsct.core.msg.es_ES@euro-220.127.116.11-0.i386 rsct.core.msg.es_ES.ISO-8859-1-18.104.22.168-0.i386 rsct.core.msg.es_ES.UTF-8-22.214.171.124-0.i386 rsct.core.msg.fr_FR-126.96.36.199-0.i386 rsct.core.msg.fr_FR@euro-188.8.131.52-0.i386 rsct.core.msg.fr_FR.ISO-8859-1-184.108.40.206-0.i386 rsct.core.msg.fr_FR.UTF-8-220.127.116.11-0.i386 rsct.core.msg.it_IT-18.104.22.168-0.i386 rsct.core.msg.it_IT@euro-22.214.171.124-0.i386 rsct.core.msg.it_IT.ISO-8859-1-126.96.36.199-0.i386 rsct.core.msg.it_IT.UTF-8-188.8.131.52-0.i386 rsct.core.msg.ja_JP.eucJP-184.108.40.206-0.i386 rsct.core.msg.ja_JP.UTF-8-220.127.116.11-0.i386 rsct.core.msg.ko_KR.eucKR-18.104.22.168-0.i386 rsct.core.msg.ko_KR.UTF-8-22.214.171.124-0.i386 rsct.core.msg.pt_BR-126.96.36.199-0.i386 rsct.core.msg.pt_BR.UTF-8-188.8.131.52-0.i386 rsct.core.msg.zh_CN.GB18030-184.108.40.206-0.i386 rsct.core.msg.zh_CN.GB2312-220.127.116.11-0.i386 rsct.core.msg.zh_CN.GBK-18.104.22.168-0.i386 rsct.core.msg.zh_CN.UTF-8-22.214.171.124-0.i386 rsct.core.msg.zh_TW-126.96.36.199-0.i386 rsct.core.msg.zh_TW.Big5-188.8.131.52-0.i386 rsct.core.msg.zh_TW.eucTW-184.108.40.206-0.i386 rsct.core.msg.zh_TW.UTF-8-220.127.116.11-0.i386 rsct.core.utils-18.104.22.168-10320.i386 rsct.core.utils.msg.de_DE-22.214.171.124-0.i386 rsct.core.utils.msg.de_DE@euro-126.96.36.199-0.i386 rsct.core.utils.msg.de_DE.ISO-8859-1-188.8.131.52-0.i386 rsct.core.utils.msg.de_DE.UTF-8-184.108.40.206-0.i386 rsct.core.utils.msg.es_ES-220.127.116.11-0.i386 rsct.core.utils.msg.es_ES@euro-18.104.22.168-0.i386 rsct.core.utils.msg.es_ES.ISO-8859-1-22.214.171.124-0.i386 rsct.core.utils.msg.es_ES.UTF-8-126.96.36.199-0.i386 rsct.core.utils.msg.fr_FR-188.8.131.52-0.i386 rsct.core.utils.msg.fr_FR@euro-184.108.40.206-0.i386 rsct.core.utils.msg.fr_FR.ISO-8859-1-220.127.116.11-0.i386 rsct.core.utils.msg.fr_FR.UTF-8-18.104.22.168-0.i386 rsct.core.utils.msg.it_IT-22.214.171.124-0.i386 rsct.core.utils.msg.it_IT@euro-126.96.36.199-0.i386 rsct.core.utils.msg.it_IT.ISO-8859-1-188.8.131.52-0.i386 rsct.core.utils.msg.it_IT.UTF-8-184.108.40.206-0.i386 rsct.core.utils.msg.ja_JP.eucJP-220.127.116.11-0.i386 rsct.core.utils.msg.ja_JP.UTF-8-18.104.22.168-0.i386 rsct.core.utils.msg.ko_KR.eucKR-22.214.171.124-0.i386 rsct.core.utils.msg.ko_KR.UTF-8-126.96.36.199-0.i386 rsct.core.utils.msg.pt_BR-188.8.131.52-0.i386 rsct.core.utils.msg.pt_BR.UTF-8-184.108.40.206-0.i386 rsct.core.utils.msg.zh_CN.GB18030-220.127.116.11-0.i386 rsct.core.utils.msg.zh_CN.GB2312-18.104.22.168-0.i386 rsct.core.utils.msg.zh_CN.GBK-22.214.171.124-0.i386 rsct.core.utils.msg.zh_CN.UTF-8-126.96.36.199-0.i386 rsct.core.utils.msg.zh_TW-188.8.131.52-0.i386 rsct.core.utils.msg.zh_TW.Big5-184.108.40.206-0.i386 rsct.core.utils.msg.zh_TW.eucTW-220.127.116.11-0.i386 rsct.core.utils.msg.zh_TW.UTF-8-18.104.22.168-0.i386 rsct.opt.stackdump-22.214.171.124-10320.i386 rsct.opt.storagerm-126.96.36.199-10320.i386 sam-188.8.131.52-10347.i386 sam.adapter-184.108.40.206-10347.i386 sam.msg.de_DE-220.127.116.11-0.i386 sam.msg.de_DE@euro-18.104.22.168-0.i386 sam.msg.de_DE.ISO-8859-1-22.214.171.124-0.i386 sam.msg.de_DE.UTF-8-126.96.36.199-0.i386 sam.msg.es_ES-188.8.131.52-0.i386 sam.msg.es_ES@euro-184.108.40.206-0.i386 sam.msg.es_ES.ISO-8859-1-220.127.116.11-0.i386 sam.msg.es_ES.UTF-8-18.104.22.168-0.i386 sam.msg.fr_FR-22.214.171.124-0.i386 sam.msg.fr_FR@euro-126.96.36.199-0.i386 sam.msg.fr_FR.ISO-8859-1-188.8.131.52-0.i386 sam.msg.fr_FR.UTF-8-184.108.40.206-0.i386 sam.msg.it_IT-220.127.116.11-0.i386 sam.msg.it_IT@euro-18.104.22.168-0.i386 sam.msg.it_IT.ISO-8859-1-22.214.171.124-0.i386 sam.msg.it_IT.UTF-8-126.96.36.199-0.i386 sam.msg.ja_JP.eucJP-188.8.131.52-0.i386 sam.msg.ja_JP.UTF-8-184.108.40.206-0.i386 sam.msg.ko_KR.eucKR-220.127.116.11-0.i386 sam.msg.ko_KR.UTF-8-18.104.22.168-0.i386 sam.msg.pt_BR-22.214.171.124-0.i386 sam.msg.pt_BR.UTF-8-126.96.36.199-0.i386 sam.msg.zh_CN.GB18030-188.8.131.52-0.i386 sam.msg.zh_CN.GB2312-184.108.40.206-0.i386 sam.msg.zh_CN.GBK-220.127.116.11-0.i386 sam.msg.zh_CN.UTF-8-18.104.22.168-0.i386 sam.msg.zh_TW-22.214.171.124-0.i386 sam.msg.zh_TW.Big5-126.96.36.199-0.i386 sam.msg.zh_TW.eucTW-188.8.131.52-0.i386 sam.msg.zh_TW.UTF-8-184.108.40.206-0.i386 sam.sappolicy-220.127.116.11-10347.i386 src-18.104.22.168-10320.i386 src.msg.de_DE-22.214.171.124-0.i386 src.msg.de_DE@euro-126.96.36.199-0.i386 src.msg.de_DE.ISO-8859-1-188.8.131.52-0.i386 src.msg.de_DE.UTF-8-184.108.40.206-0.i386 src.msg.es_ES-220.127.116.11-0.i386 src.msg.es_ES@euro-18.104.22.168-0.i386 src.msg.es_ES.ISO-8859-1-22.214.171.124-0.i386 src.msg.es_ES.UTF-8-126.96.36.199-0.i386 src.msg.fr_FR-188.8.131.52-0.i386 src.msg.fr_FR@euro-184.108.40.206-0.i386 src.msg.fr_FR.ISO-8859-1-220.127.116.11-0.i386 src.msg.fr_FR.UTF-8-18.104.22.168-0.i386 src.msg.it_IT-22.214.171.124-0.i386 src.msg.it_IT@euro-126.96.36.199-0.i386 src.msg.it_IT.ISO-8859-1-188.8.131.52-0.i386 src.msg.it_IT.UTF-8-184.108.40.206-0.i386 src.msg.ja_JP.eucJP-220.127.116.11-0.i386 src.msg.ja_JP.UTF-8-18.104.22.168-0.i386 src.msg.ko_KR.eucKR-22.214.171.124-0.i386 src.msg.ko_KR.UTF-8-126.96.36.199-0.i386 src.msg.pt_BR-188.8.131.52-0.i386 src.msg.pt_BR.UTF-8-184.108.40.206-0.i386 src.msg.zh_CN.GB18030-220.127.116.11-0.i386 src.msg.zh_CN.GB2312-18.104.22.168-0.i386 src.msg.zh_CN.GBK-22.214.171.124-0.i386 src.msg.zh_CN.UTF-8-126.96.36.199-0.i386 src.msg.zh_TW-188.8.131.52-0.i386 src.msg.zh_TW.Big5-184.108.40.206-0.i386 src.msg.zh_TW.eucTW-220.127.116.11-0.i386 src.msg.zh_TW.UTF-8-18.104.22.168-0.i386 installSAM: The following license is installed: Product: IBM Tivoli System Automation for Multiplatforms (Try & Buy License) 3.2 Evaluation Period: 90 days (90 days left) Creation date: Wed 19 Aug 2009 12:00:01 AM CDT Expiration date: Thu 31 Dec 2037 12:00:01 AM CST Subsystem Group PID Status ctrmc rsct 1627 active installSAM: Warning: Must set CT_MANAGEMENT_SCOPE=2 installSAM: All packages were installed successfully.
It’s really just a straight forward HADR setup – you’ll want to make sure you have your HADR peer window set. See my post on HADR for more details on HADR.
TSA is really picky about the hosts file. You either need to have root (sudo is fine) access or have access to someone with root or sudo access so you can edit it if needed. Note that editing a hosts file in certain ways can break other things on the system, so you’ll likely want your Systems Admin or whatever other engineers you’re working with on the project aware that you’re making changes.
What TSA requires is that both hosts for the HADR pair have one line and only one line each in the hosts file. The entries must be of this format:
ipaddress shortname longname
ipaddress longname shortname
Where longname is the fully qualified server name and shortname is a shorter version. You need to have HADR set up with whatever comes first – shortname or longname.
Make sure that db2nodes.cfg (in $INSTHOME/sqllib) uses the same as hadr(longname or shortname), and that the hostname command returns the same of these two choices.
Longname vs. Shortname
It doesn’t really matter whether you pick the longname or the shortname, you just have to be 100% consistent. IP addresses don’t work. And the errors that db2haicu gives you are in no way helpful to determine you have a mismatch here. The errors are extremely generic.
Sometimes on Fixpack, the TSA scripts don’t get updated as they should. To make sure you’ve got them up to date, you always should do the following:
Log in as root
# ./db2cptsa DBI1119I The version of the DB2 High Availability (HA) scripts for the IBM Tivoli System Automation for Multiplatforms (SA MP) found in /usr/sbin/rsct/sapolicies/db2 is the same version as the version of the scripts on the current DB2 install media. Explanation: You need DB2 HA scripts to use SA MP with the DB2 HA feature. SA MP and the DB2 HA feature being installed from the DB2 install media require DB2 HA scripts with a version the same as or higher than the version of the scripts also on the DB2 install media. The version of the DB2 HA scripts currently installed the same as the version of the scripts on the DB2 install media. User response: No action is required.
The above is what we expect to see. If you don’t see the above, you will most likely see it telling you that it has updated the scripts. If this returns an error, you need to investigate further. This must be successful before you can move on.
Ok, this is the last of the prepartion steps. preprpnode is a command that you run as root. I think that all it really does is trade public keys between the two servers. But it can also fail, and it must be run successfully before db2haicu will work. Here’s what it looks like. You don’t have to cd anywhere special to run it.
# preprpnode spp05db01r spp05db02r
Yep, that simple. It doesn’t return any output unless there is an error. The most common things to cause an error here are not having the TSA component installed (in which case you get a command not found), or to have a firewall either on the server or between the servers blocking traffic. The latter looks like this:
# preprpnode svp35db01b svp35db02b /usr/sbin/rsct/bin/lsrsrc-api: 2612-022 A session could not be established with the RMC daemon on svp35db02b. preprpnode: Unable to obtain the public key from svp35db02b.
If you see the above, db2haicu will also fail.
You need to check if iptables is running or look in your environment for another firewall. Assuming you are allowed to do so, you can check and stop iptables using this(as root):
# /etc/init.d/iptables status Table: filter Chain INPUT (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 2 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 5 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) num target prot opt source destination 1 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) num target prot opt source destination You have new mail in /var/spool/mail/root # /etc/init.d/iptables stop iptables: Flushing firewall rules: [ OK ] iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Unloading modules: [ OK ] # /etc/init.d/iptables status iptables: Firewall is not running. #
Other Posts In This Series
This series consists of four posts:
Using TSA/db2haicu to automate failover – Part 1: The Preparation
Using TSA/db2haicu to automate failover – Part 2: How it looks if it goes smoothly
Using TSA/db2haicu to Automate Failover Part 3: Testing, Ways Setup can go Wrong and What to do.
“Using TSA/db2haicu to automate failover Part 4: Dealing with Problems After Setup
Search this blog on “TSA” for other posts on TSA issues and tips.
Great whitepaper on db2haicu setup: http://download.boulder.ibm.com/ibmdl/pub/software/dw/data/dm-0908hadrdb2haicu/HADR_db2haicu.pdf
How do we configure TSA for multiple instances on a server??
can you provide sample xml file to configure multiple instances
Unfortunately, I don’t have any experience to share here. My best guess is that you would run db2haicu as each instance owner. (after doing all your prep work) I have done multiple databases on one instance. I haven’t been using the xml method, though I’ve been thinking lately that I should try it.
We try very hard to stick to a one database one instance one server setup for our production environments, and while I’ve been also doing HADR/TSA in test environments now and then, usually that is a client that is willing to have separate servers even for test to better mimic prod.
I had also run into the similar installation problem on Red Hat 6.3 while installing the version of TSA that is part of the 9.7 FP5, base code cannot be installed because it does not work with Red Hat 6. The TSA team suggested me to install “3.2.1-TIV-ITSAMP-FP0003, Tivoli System Automation for Multiplatforms”. That worked perfectly for me and I was able to install the TSA sucessfully. One more thing, I had to use the license file, sam32.lic , that comes with the DB2 9.7 install media, since the link provided was a fixpack ( no license included ).
Hope this information would be useful to someone.
Hello, I ended with a partial TSA installation , neither ./installSAM nor ./uninstallSAM finish succesfully. The version is bundled with v97fp7 .Can you propose a solution, this is the uninstall log :
/usr/sbin/rsct/bin/lsrsrc-api: 2612-022 A session could not be established with the RMC daemon on “local_node”.
uninstallSAM: Uninstalling System Automation on platform x86_64
uninstallSAM: Error: Failed with return-code 255 : sam-22.214.171.124-10046
uninstallSAM: Any packages failed uninstallation. See details below
uninstallSAM: Error: Failed with return-code 255 : sam-126.96.36.199-10046
error: %postun(sam-188.8.131.52-10046.i386) scriptlet failed, exit status 1
Uninstalling IBM Tivoli System Automation for Multiplatforms (SA MP) …
0513-084 There were no records that matched your request.
0513-084 There were no records that matched your request.
0513-084 There were no records that matched your request.
0513-071 The ctrmc Subsystem has been added.
Cannot add ctrmc to init script structure
Warning: An error was returned by rmcctrl while attempting to restart the rmc subsystem, exit code=0
uninstallSAM: No uninstallation was performed
My first thoughts are to go find a good Linux admin or to open a PMR. I’m sorry I don’t have any good answers on that for you – I have never seen it. If it was AIX, I’d be trying smit to remove it.
I know my response is too late 🙂 .. But if it can help someone thats great
I had same issue. I removed /opt/rsct and ran uninstallSAM.. it worked fine
Benefits of using this over setting up ACR(automatic client reroute)?
ACR and TSA serve different purposes. ACR does not automate failover in any way – with ACR, you have to have someone manually issue the takeover commands for HADR to failover. The only place you really choose between ACR and TSA is when you choose whether to use a virtual IP or not. Using the virtual IP is easier because not all applications do well with caching an alternate server.
Thanks for this detailed blog. I faced the below issue while installing TSAMP and the prereqSAM failed with this error:
prereqSAM: Using default prerequisite checking on the following rpm package ‘libstdc++4.-32bit’ ‘x86_64’
prereqSAM: Error: The rpm package is installed, but the following part was not found ‘libstdc++46-32bit-4.6.9-0.11.38.x86_64
prereqSAM: Using default prerequisite checking on the following rpm package ‘pam-32bit’
prereqSAM: rpm package and version installed ‘pam-32bit’
1 missing package: libstdc++46-32bit
I struggled for 2 days with it and found the following PMR with IBM:
So the fix is, install TSAMP with this command and it works fine:
I’ve tried to startup de Cluster on DB2 10.1 with a couple of VM on SUSE 10
Everthing works fins untl I run db2haicu on the Principal server. The last step
I receive this error:
2014-07-23-00.18.24.544869-240 E1380534G352 LEVEL: Error
PID : 32289 TID : 2858981072 PROC : db2haicu
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, high avail services, sqlhaUIMain, probe:900
MESSAGE : ECF=0x90000530=-1879046864=ECF_SQLHA_NOTINITED
Library uninitialized error
The only weird thing I found is that cluster finds 4 NICs when I only have 1 eth0 on each one and the lo.
I also tried to do it with xml file and I get the same error
Could be a kind of multipath on Linux?
I have not seen that specific error.
I have a question regarding the usage of Virtual IP. Concerning I have a HADR nodes in different subnets, I am unable to use the Virtual IP. Nevertheless, I need to connect to my HADR database from withing e.g. a script. Shall I for example catalog the DB2 HADR database on a machine running a script? If not, how shall I use the hadr database? thanks
ACR(automatic client reroute) still works, and can be used in those situations. It is easy to use if you’re using JDBC type 2 or traditional db2 clients. In those situations, all you have to do is to define an alternate server at the DB2 server, and the clients will pick it up on their next database connection. Functionality should be built in to handle failovers. In fact, we still define an alternate server (of the virtual IP) even when using a virtual IP to get that retry functionality.
If your application is using type 4 JDBC drivers or some other connection methodology, your app may have a place to specify the alternate server manually.
I have a problem with additng virutal IP:
“Cannot create a virtual IP because there are no networks defined for this cluster. Configure the networks before adding virtual IPs.”
Do you happen to see that error?
I have not seen that error. When you get it, have you defined at least one network card on each of the primary and the standby through db2haicu?
Thanks Ember for another informative post.
How do the resource groups get created/defined ?
I am troubleshooting current TSA setup but when I run “lsrg” or “lssam” it says
lsrg: There are no resource groups defined.
lssam: No resource groups defined or cluster is offline!
The lsrpdomain and lsrpnode comamnds return valid info.
Or when I run $ lsrsrc -Ab IBM.ResourceGroup —all I get is..
Resource Persistent and Dynamic Attributes for IBM.ResourceGroup
db2haicu defines them when you run db2haicu.
Maybe someone else may find this useful.
After going down some rabbit holes, we finally got TSA working, the problem we were having stemmed from an invalid license
/tmp $ samlicm -s
Error: Product license is invalid and needs to be upgraded.
Once we added a valid license and followed the setup steps again it worked.
I am in the same situation.How did you added the license ?let me know on the steps that you followed plese
samlicm -s …to check if you have a valid license
samlicm -i …to add or install a new license file
license files were stored in /var/opt/sam/lic
Hope that helps.
Hello, In one of the replies above you have mentioned building TSA on multiple databases under a single instance.
Did you achieve it using a Single VIP on all DB’s?
I was wondering how this scenario works?
I have not. I have gone for separate is for separate databases since that is what is stated as supported, though I do not like it, and would generally prefer the other configuration from a usability standpoint.
II have configured TSA with multiple instances. The right way is to create one XML file for each instance, and execute db2haicu -d filename.xml with the right instance user.
Hi I am gettting command not found when run preprpnode server1 server2.
how to resolve this issue.
You’re running that as root, right? Make sure that TSAMP installed properly when you installed DB2.
I have to setup TSA for single instance with 3 databases. Please let me me how many VIPs required and which IP whould be given as alternate server details ?
Each database will need its own VIP.
Error: Product license is invalid and needs to be upgraded.
I have nodelock file inside /var/opt/sam/lic:
How do I update the license please
This link seems to cover it well: https://developer.ibm.com/answers/questions/240295/what-can-i-do-when-either-my-tsamp-license-is-expi.html
Hi mrs Crooks
Thanks for your helpful blog !
I have issue with prepare TSA
I am getting error when run preprpnode server1 server2.
preprpnode: The node name XXXXXX cannot be resolved.
I’m late in responding here, but have you made sure your hosts files are all good and you can ping the other server using that hostname?
In our PROD environment , I have setup HADR/TSA in a single instance multiple databases.But now client requirement is to have different instance for each databases of different applications and setup HADR/TSA over there. I did create a new instance and tried running db2haicu and it gave some errors related to mount points.Hopefully what i think here is since we do not have different /db2home file systems in our server , all the instances are getting created in same /home filesystem and hence if we mount different db2home filesystems and then create instances on different home filesystems then we should be able to run db2haicu and setup TSA in each instance separately.
Need you to shed some lights here if am wrong since am not able to find out any proper document to setup db2haicu on multiple instances on same server.
Thanks & Regards,
For multiple instances, you generally only have one TSAMP domain. It shouldn’t matter where the instances’ home directories are. Each one has to have a separate home directory, but placing them in separate file systems is merely a best practice, and not technically required.
Hi Ember Crooks,
Recently We build HADR with Db2 184.108.40.206 on RHEL OS.Planning to use TSAMP for automatic failover on this environment.Successfully installed TSA while installing Db2. After that we applied Db2 Advanced Work Group server Edition licence on Db2 product.
Checked samlicm -s command on both linux servers but is showing output like
Product: IBM Tivoli System Automation for Multiplatforms (Try & Buy License) 220.127.116.11
Evaluation Period: 90 days (60 days left)
Creation date: Fri 16 Aug 2013 12:00:01 AM CEST
Expiration date: Thu 31 Dec 2037 12:00:01 AM CET
We already have licence for Db2 product.We will get licence for TSA with Purchasing Db2 licence or not.Whether we need to buy separate licence for TSAMP as per output of samlicm -s cmd. Please clarify me on my queries.
The license for TSAM should be included in the Db2 code – this technote explains how to find it in the code and apply it on 10.5 – it should be similar on 11.4.5. https://www.ibm.com/support/pages/where-obtain-license-file-sam41lic-tivoli-system-automation-multiplatforms-41-tsamp-db2-105