Tech Blog

These are blog entries written by the UNIX Health Check development team. Our team has extensive technical experience on both AIX and Red Hat systems, and we like to share our knowledge with our visitors.

Topics: AIX, Security

Heartbleed bug

OpenSSL on AIX can be impacted by the Heartbleed bug. Only OpenSSL 1.0.1e (IBM AIX VRMFs - 1.0.1.500 & 1.0.1.501) is vulnerable to the Heartbleed bug (CVE-2014-010). All OpenSSL v0.9.8.xxxx and v12.9.8.xxxx are NOT vulnerable to this CVE.

IBM released OpenSSL 1.0.1g by the end of April 2014, which is the official fix.

The following is information about an ifix that was made available by IBM. The ifix is just a workaround, and currently IBM recommends upgrading to OpenSSL 1.0.1.511 instead (see below).

  • This is a workaround compiled with the feature turned off.
  • This is not OS dependent. It only depends on the OpenSSL level.
Below are the download and install/uninstall instructions.

The OpenSSL ifix doesn't require a reboot. However... It's a shared library update, so any daemons that use it will need to be restarted such as sshd. If you aren't sure what applications running on your machine use OpenSSL, it's recommended to reboot.

To download it, go to: https://testcase.software.ibm.com/ and log in as "Anonymous" (no password needed). Click on the "fromibm" folder, and then click on the "aix" folder. Scroll down the list until you find the following file and click on it to download:
0160_ifix.140409.epkg.Z
Once the download is complete, transfer the file to your AIX system. Log on to your AIX system, go to the directory where you put the file, and run the following command as the root user.

To preview the installation of 0160_ifix.140409.epkg.Z, please do the following:
# emgr -p -e 0160_ifix.140409.epkg.Z
To install the ifix, run the following:
# emgr -X -e 0160_ifix.140409.epkg.Z
If you need to uninstall the iFix for some reason, run the following command as root:
# emgr -r -L 0160_ifix.140409.epkg.Z
The following is more information, updated on June 13, 2014:

IBM has released several new levels for OpenSSL that address both the Heartbleed bug, as well as several other security vulnerabilities that have been identified recently.

We currently recommend downloading OpenSSL 1.0.1.511. This level can be used on AIX 5.3, 6.1 and 7.1. You can find OpenSSL in the IBM Web Download Pack at:

http://www-03.ibm.com/systems/power/software/aix/expansionpack/

Click on Downloads (on the right), log in with your IBM user ID (or register for one, if you don't already have an IBM user ID). Select openssl on the next page, and click on Continue at the bottom. Click Submit to accept IBM's privacy statement on the next page, and you'll be forwarded to a list of possible downloads. Here, click on "Download using http", and select the OpenSSL images for openssl-1.0.1.511.tar.Z. You probably also want to review the Readme beneath it as well.

You will download the openssl-1.0.1.511.tar.Z file. Transfer that onto your AIX systems into a separate folder.

Uncompress the file:
# gzip -d openssl-1.0.1.511.tar.Z
Now you will have a tar file.

Un-tar it:
# tar xf openssl-1.0.1.511.tar
That will give you folder openssl-1.0.1.511 within your current folder. Go into that folder:
# cd openssl-1.0.1.511
Here you can find 3 filesets; run inutoc to generate the .toc file:
# ls
openssl.base       openssl.license    openssl.man.en_US
# inutoc .
Then install the filesets:
# update_all -d . -cY
Now, it should be installed. Before logging out, make sure you can access your system through ssh using a separate window.

For more information, see http://heartbleed.com. Please ensure your UNIX Health Check level is up to date. Version 14.04.10 and up includes a check for your AIX systems to see if any are impacted by the Heartbleed bug.

Topics: AIX, System Admin

Use machstat to identify power or cooling issues

Not very much known is the machstat command in AIX that can be used to display the status of the Power Status Register, and thus can be helpful to identify any issues with either Power or Cooling.

# machstat -f
0 0 0
If it returns all zeroes, everything is fine. Anything else is not good. The first digit (the so-called EPOW Event) indicates the type of problem:

EPOW EventDescription
0normal operation
1non-critical cooling problem
2non-critical power problem
3severe power problem - halt system
4severe problems - halt immediately
5unhandled issue
7unhandled issue

Another way to determine if the system may have a power or cooling issue, is by looking at a crontab entry in the root user's crontab:
# crontab -l root | grep -i powerfail
0 00,12 * * * wall%rc.powerfail:2::WARNING!!! The system is now operating with a power problem. This message will be walled every 12 hours. Remove this crontab entry after the problem is resolved.
If a powerfail message is present in the crontab of user root, this may indicate that there is an issue to be looked into. Contact your IBM representative to check the system out. Afterwards, make sure to remove the powerfail entry from the root user's crontab.

Topics: AIX, LVM, System Admin

LVM command history

Want to know which LVM commands were run on a system? Simply run the following command, and get a list of the LVM command history:

# alog -o -t lvmcfg
To filter out only the actual commands:
# alog -o -t lvmcfg | grep -v -E "workdir|exited|tellclvmd"
[S 06/11/13-16:52:02:236 lvmstat.c 468] lvmstat -v testvg
[S 06/11/13-16:52:02:637 lvmstat.c 468] lvmstat -v rootvg
[S 07/20/13-15:02:15:076 extendlv.sh 789] extendlv testlv 400
[S 07/20/13-15:02:33:199 chlv.sh 527] chlv -x 4096 testlv
[S 08/22/13-12:29:16:807 chlv.sh 527] chlv -e x testlv
[S 08/22/13-12:29:26:150 chlv.sh 527] chlv -e x fslv00
[S 08/22/13-12:29:46:009 chlv.sh 527] chlv -e x loglv00
[S 08/22/13-12:30:55:843 reorgvg.sh 590] reorgvg
The information for this LVM command history is stored in /var/adm/ras/lvmcfg.log. You can check the location for a circular log, by running:
# alog -t lvmcfg -L
#file:size:verbosity
/var/adm/ras/lvmcfg.log:51200:3
More detail can also be found in the lvmt log, by running:
# alog -t lvmt -o

Topics: AIX, System Admin

Suspending and resuming a process

You may be familiar with suspending a process that is running in the foreground by pressing CTRL-Z. It will suspend the process, until you type "fg", and the process will resume again.

# sleep 400
After pressing CTRL-Z, you'll see:
[1] + Stopped (SIGTSTP)        sleep 400
Then type "fg" to resume the process:
# fg
sleep 400
But what if you wish to suspend a process that is not attached to a terminal, and is running in the background? This is where the kill command is useful. Using signal 17, you can suspend a process, and using signal 19 you can resume a process.

This is how it works: First look up the process ID you wish to suspend:
# sleep 400 &
[1]     8913102
# ps -ef | grep sleep
    root  8913102 10092788   0 07:10:30  pts/1  0:00 sleep 400
    root 14680240 10092788   0 07:10:34  pts/1  0:00 grep sleep
Then suspend the process with signal 17:
# kill -17 8913102
[1] + Stopped (SIGSTOP)        sleep 400 &
To resume it again, send signal 19:
# kill -19 8913102

Topics: AIX, System Admin

RANDOM Korn Shell built-in

The use of $RANDOM in Korn Shell can be very useful. Korn shell built-in $RANDOM can generate random numbers in the range 0:32767. At every call a new random value is generated:

# echo $RANDOM
19962
# echo $RANDOM
19360
The $RANDOM Korn shell built-in can also be used to generate numbers within a certain range, for example, if you want to run the sleep command for a random number of seconds.

To sleep between 1 and 600 seconds (up to 10 minutes):
# sleep $(print $((RANDOM%600+1)))

Topics: AIX, System Admin

Number of active virtual processors

To know quickly how many virtual processors are active, run:

# echo vpm | kdb
For example:
# echo vpm | kdb
...
VSD Thread State.
 CPU VP_STATE   SLEEP_STATE  PROD_TIME: SECS   NSECS     CEDE_LAT

   0  ACTIVE    AWAKE        0000000000000000  00000000  00
   1  ACTIVE    AWAKE        0000000000000000  00000000  00
   2  ACTIVE    AWAKE        0000000000000000  00000000  00
   3  ACTIVE    AWAKE        0000000000000000  00000000  00
   4  DISABLED  AWAKE        00000000503536C7  261137E1  00
   5  DISABLED  SLEEPING     0000000051609EAF  036D61DC  02
   6  DISABLED  SLEEPING     0000000051609E64  036D6299  02
   7  DISABLED  SLEEPING     0000000051609E73  036D6224  02

Topics: AIX, System Admin

How to read the /var/adm/ras/diag log file

There are 2 ways for reading the Diagnostics log file, located in /var/adm/ras/diag:

The first option uses the diag tool. Run:

# diag
Then hit ENTER and select "Task Selection", followed by "Display Previous Diagnostic Results" and "Display Previous Results".

The second option is to use diagrpt. Run:
# /usr/lpp/diagnostics/bin/diagrpt -s 010101
To display only the last entry, run:
# /usr/lpp/diagnostics/bin/diagrpt -o

Topics: AIX, Backup & restore, System Admin, Virtual I/O Server, Virtualization

How to make a system backup of a VIOS

To create a system backup of a Virtual I/O Server (VIOS), run the following commands (as user root):

# /usr/ios/cli/ioscli viosbr -backup -file vios_config_bkup
-frequency daily -numfiles 10
# /usr/ios/cli/ioscli backupios -nomedialib -file /mksysb/$(hostname).mksysb -mksysb
The first command (viosbr) will create a backup of the configuration information to /home/padmin/cfgbackups. It will also schedule the command to run every day, and keep up to 10 files in /home/padmin/cfgbackups.

The second command is the mksysb equivalent for a Virtual I/O Server: backupios. This command will create the mksysb image in the /mksysb folder, and exclude any ISO repositiory in rootvg, and anything else excluded in /etc/exclude.rootvg.

Topics: AIX, Backup & restore, Storage, System Admin

Using mkvgdata and restvg in DR situations

It is useful to run the following commands before you create your (at least) weekly mksysb image:

# lsvg -o | xargs -i mkvgdata {}
# tar -cvf /sysadm/vgdata.tar /tmp/vgdata
Add these commands to your mksysb script, just before running the mksysb command. What this does is to run the mkvgdata command for each online volume group. This will generate output for a volume group in /tmp/vgdata. The resulting output is then tar'd and stored in the /sysadm folder or file system. This allows information regarding your volume groups, logical volumes, and file systems to be included in your mksysb image.

To recreate the volume groups, logical volumes and file systems:
  • Run:
    # tar -xvf /sysadm/vgdata.tar
  • Now edit /tmp/vgdata/{volume group name}/{volume group name}.data file and look for the line with "VG_SOURCE_DISK_LIST=". Change the line to have the hdisks, vpaths or hdiskpowers as needed.
  • Run:
    # restvg -r -d /tmp/vgdata/{volume group name}/{volume group name}.data
Make sure to remove file systems with the rmfs command before running restvg, or it will not run correctly. Or, you can just run it once, run the exportvg command for the same volume group, and run the restvg command again. There is also a "-s" flag for restvg that lets you shrink the file system to its minimum size needed, but depending on when the vgdata was created, you could run out of space, when restoring the contents of the file system. Just something to keep in mind.

Topics: AIX, System Admin

A quick way to remove all printer queues

Here's a quick way to remove all the printer queues from an AIX system:

/usr/lib/lpd/pio/etc/piolsvp -p | grep -v PRINTER | \
   while read queue device rest ; do
   echo $queue $device
   rmquedev -q$queue -d$device
   rmque -q$queue
done

Number of results found for topic AIX: 231.
Displaying results: 31 - 40.