How to Disable ASLR and Why you Should do it

Update: Oct 28, 2013 – the below applies only to DB2 before DB2 10.1 Fixpack 3. IBM says that ASLR can safely be used with DB2 on 10.1 Fixpack 3 and later. See the technote at http://www-01.ibm.com/support/docview.wss?uid=swg21365583. FixPack 3 was released on September 27th. Also, please see comment from a reader at the bottom about the security implications of disabling ASLR.

This post is specific to Linux. The commands shared have been tested on RedHat 6.2.

What is ASLR

Address Space Layout Randomization is a security feature that randomizes where in a process’s address space key data is stored. This enhances security by making it harder for a hacker to find these key memory areas. I believe it is enabled by default on recent RedHat distributions. You can find more information on the concept here: http://en.wikipedia.org/wiki/Address_space_layout_randomization

Why ASLR Should be Disabled on all DB2 Database Servers

Quite simply, enabling ASLR can cause odd behavoir for a number of DB2 processes other than the core db2sysc. IBM details it in this technote: https://www-304.ibm.com/support/docview.wss?uid=swg21365583. DB2 maps a shared memory object across processes, and apparently it can’t always handle the addressing if ASLR is enabled.

The error message from the db2 diagnostic log that I’ve most frequently seen is one of these two:

2012-01-12-15.38.24.853671-300 E304269938E671      LEVEL: Error (OS)
PID     : 12732                TID  : 46912562375120PROC : db2bp
INSTANCE: db2inst1             NODE : 000
APPID   : *LOCAL.db2inst1.120117015847
FUNCTION: DB2 UDB, oper system services, sqlowqueInternal, probe:40
MESSAGE : ZRC=0x870F003E=-2029060034=SQLO_QUE_BAD_HANDLE "Bad Queue Handle"
          DIA8555C An invalid message queue handle was encountered.
CALLED  : OS, -, msgsnd                           OSERR: EINVAL (22)
DATA #1 : system V message queue identifier., PD_TYPE_SYSV_QUEUE_ID, 4 bytes
0x02F7000C
DATA #2 : Pointer, 8 bytes
0x00000000007d7890
DATA #3 : unsigned integer, 8 bytes
3841

2011-08-17-12.14.51.565199-300 I657879649E972      LEVEL: Error
PID     : 15948                TID  : 47622807768096PROC : db2egcf
INSTANCE: db2inst1             NODE : 000
FUNCTION: <0>, <0>, <0>, probe:100
MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
          "No Storage Available for allocation"
          DIA8305C Memory allocation failure occurred.
DATA #1 : String, 22 bytes
sqlocshr.-2062614523..
CALLSTCK:
  [0] 0x00002B500232AA9D pdOSSeLoggingCallback + 0x91
  [1] 0x00002B50076EE5FB /opt/IBM/db2/v9.5/lib64/libdb2osse.so.1 + 0x1B25FB
  [2] 0x00002B50076EE3FC ossLog + 0xA6
  [3] 0x0000000000402BBE _Z21db2haAttachToDatabasePcP9SQLE_KRCBPP16sqeLocalDatabase + 0x22A
  [4] 0x0000000000402C58 _Z17db2haGetHADRStatePcP9HDR_STATES_ + 0x64
  [5] 0x0000000000402225 main + 0x1DD
  [6] 0x0000003ECC01D994 __libc_start_main + 0xF4
  [7] 0x0000000000401FAA pdOSSeLoggingCallback + 0x7A
  [8] 0x0000000000000000 ?unknown + 0x0
  [9] 0x0000000000000000 ?unknown + 0x0

In the second, you’ll note that the process – db2egcf – is a monitoring process for HADR. Since one of the possible failure scenarios with ASLR relates to unexpected HADR failovers, this fits right in with the tech note linked above.

How to Disable it

First, you must have root for this. Or use an SA as your Waldo if you are not able to get root yourself. There are two pieces to this – disabling it for your server currently and ensuring that change is persistent across reboots.

Runtime

This is quite simple – a single command:

# sysctl -w kernel.randomize_va_space=0

You do not have to restart db2 or the server for the change to take effect.

Saving it for all future reboots

To make your change persistent, you have to add a line to the /etc/sysctl.conf file (on RedHat). This can easily be accomplished using:

# echo "" >> /etc/sysctl.conf
# echo '# Disabling ASLR per IBM recommendation for DB2 servers:' >> /etc/sysctl.conf
# echo '#   ASLR may cause intermittent failures in shmat() for any db2 processes other than db2sysc.' >> /etc/sysctl.conf
# echo 'kernel.randomize_va_space = 0' >> /etc/sysctl.conf

Obviously, I’ve included comment lines here. They help to keep it clear why something is defined – my hope is that if some System Administrator sees that at some point in the future, they would research or contact a DBA before changing the setting