If you run into the following error:

cl_mklv: Operation is not allowed because vg is a RAID concurrent volume group.

This may be caused by the volume group being varied on, on the other node. If it should not be varied on, on the other node, run:

# varyoffvg vg

And then retry the LVM command. If it continues to be a problem, then stop HACMP on both nodes, export the volume group and re-import the volume group on both nodes, and then restart the cluster.

If you get the following message when you open a vterm:

The session is reserved for physical serial port communication.

Then this may be caused by the fact that your system is still in MDC, or manufactoring default configuration mode. It can easily be resolved:

• Power down your frame.
• Power it back up to standby status.
• Then, when activating the default LPAR, choose "exit the MDC".

Some applications, for example Oracle when using raw logical volumes, may require specific access to logical volumes. Oracle will require that the raw logical volume is owned by the oracle account, and it may or may not require custom permissions.

The default values for a logical volume are: dev_uid=0 (owned by user root), dev_gid=0 (owned by group system) and dev_perm=432 (mode 660). You can check the current settings of a logical volume by using the readvgda command:

# readvgda vpath51 | egrep "lvname|dev_|Logical"
lvname:         testlv (i=2)
dev_uid:        0
dev_gid:        0
dev_perm:       432

If the logical volume was create with or has been modified to use customized owner/group/mode values, the dev_values will show the current uid/gid/perm values, for example:

# chlv -U user -G staff -P 777 testlv
# ls -als /dev/*testlv
   0 crwxrwxrwx 1 user staff 57, 3 Mar 10 14:39 /dev/rtestlv
   0 brwxrwxrwx 1 user staff 57, 3 Mar 10 14:39 /dev/testlv
# readvgda vpath51 | egrep "lvname|dev_|Logical"
lvname:         testlv (i=2)
dev_uid:        3878
dev_gid:        1
dev_perm:       511

When the volume group is exported, and re-imported, this information is lost:

# errpt
# exportvg testvg
# importvg -y testvg vpath51
testvg
# ls -als /dev/*testlv
   0 crw-rw---- 1 root system 57, 3 Mar 10 15:11 /dev/rtestlv
   0 brw-rw---- 1 root system 57, 3 Mar 10 15:11 /dev/testlv

To avoid this from happening, make sure to use the -R option, that will restore any specific settings:

# chlv -U user -G staff -P 777 testlv
# ls -als /dev/*testlv
   0 crwxrwxrwx 1 user staff 57, 3 Mar 10 15:11 /dev/rtestlv
   0 brwxrwxrwx 1 user staff 57, 3 Mar 10 15:11 /dev/testlv
# readvgda vpath51 | egrep "lvname|dev_|Logical"
lvname:         testlv (i=2)
dev_uid:        3878
dev_gid:        1
dev_perm:       511
# varyoffvg testvg
# exportvg testvg
importvg -Ry testvg vpath51
testvg
# ls -als /dev/*testlv
   0 crwxrwxrwx 1 user staff 57, 3 Mar 10 15:14 /dev/rtestlv
   0 brwxrwxrwx 1 user staff 57, 3 Mar 10 15:14 /dev/testlv

Never use the chown/chmod/chgrp commands to change the same settings on the logical volume. It will work, however, the updates will not be written to the VGDA, and as soon as the volume group is exported out and re-imported on the system, the updates will be gone:

# chlv -U root -G system -P 660 testlv
# ls -als /dev/*testlv
   0 crw-rw---- 1 root system 57, 3 Mar 10 15:14 /dev/rtestlv
   0 brw-rw---- 1 root system 57, 3 Mar 10 15:14 /dev/testlv
# chown user.staff /dev/testlv /dev/rtestlv
# chmod 777 /dev/testlv /dev/rtestlv
# ls -als /dev/*testlv
   0 crwxrwxrwx 1 user staff 57, 3 Mar 10 15:14 /dev/rtestlv
   0 brwxrwxrwx 1 user staff 57, 3 Mar 10 15:14 /dev/testlv
# readvgda vpath51 | egrep "lvname|dev_|Logical"
lvname:         testlv (i=2)
dev_uid:        0
dev_gid:        0
dev_perm:       360

Notice above how the chlv command changed the owner to root, the group to system, and the permissions to 660. Even after the chown and chmod commands are run, and the changes are visible on the device files in /dev, the changes are not seen in the VGDA. This is confirmed when the volume group is exported and imported, even with using the -R option:

# varyoffvg testvg
# exportvg testvg
# importvg -Ry testvg vpath51
testvg
# ls -als /dev/*testlv
   0 crw-rw---- 1 root system 57, 3 Mar 10 15:23 /dev/rtestlv
   0 brw-rw---- 1 root system 57, 3 Mar 10 15:23 /dev/testlv

So, when you have customized user/group/mode settings for logical volumes, and you need to export and import the volume group, always make sure to use the -R option when running importvg.

Also, make sure never to use the chmod/chown/chgrp commands on logical volume block and character devices in /dev, but use the chlv command instead, to make sure the VGDA is updated accordingly.

Note: A regular volume group does not store any customized owner/group/mode in the VGDA. It is only stored for Big or Scalable volume groups. In case you're using a regular volume group with customized owner/group/mode settings for logical volumes, you will have to use the chmod/chown/chgrp commands to update it, especially after exporting and re-importing the volume group.

Here are some color codes you can use in the Korn Shell:

## Reset to normal: \033[0m
NORM="\033[0m"

## Colors:
BLACK="\033[0;30m"
GRAY="\033[1;30m"
RED="\033[0;31m"
LRED="\033[1;31m"
GREEN="\033[0;32m"
LGREEN="\033[1;32m"
YELLOW="\033[0;33m"
LYELLOW="\033[1;33m"
BLUE="\033[0;34m"
LBLUE="\033[1;34m"
PURPLE="\033[0;35m"
PINK="\033[1;35m"
CYAN="\033[0;36m"
LCYAN="\033[1;36m"
LGRAY="\033[0;37m"
WHITE="\033[1;37m"

## Backgrounds
BLACKB="\033[0;40m"
REDB="\033[0;41m"
GREENB="\033[0;42m"
YELLOWB="\033[0;43m"
BLUEB="\033[0;44m"
PURPLEB="\033[0;45m"
CYANB="\033[0;46m"
GREYB="\033[0;47m"

## Attributes:
UNDERLINE="\033[4m"
BOLD="\033[1m"
INVERT="\033[7m"

## Cursor movements
CUR_UP="\033[1A"
CUR_DN="\033[1B"
CUR_LEFT="\033[1D"
CUR_RIGHT="\033[1C"

## Start of display (top left)
SOD="\033[1;1f"

Just copy everyting above and paste it into your shell or in a script. Then, you can use the defined variables:

## Example - Red underlined
echo "${RED}${UNDERLINE}This is a test!${NORM}"

## Example - different colors
echo "${RED}This ${YELLOW}is ${LBLUE}a ${INVERT}test!${NORM}"

## Example - cursor movement
# echo " ${CUR_LEFT}Test"

## Create a rotating thingy
while true ; do
printf "${CUR_LEFT}/"
perl -e "use Time::HiRes qw(usleep); usleep(100000)"
printf "${CUR_LEFT}-"
perl -e "use Time::HiRes qw(usleep); usleep(100000)"
printf "${CUR_LEFT}\\"
perl -e "use Time::HiRes qw(usleep); usleep(100000)"
printf "${CUR_LEFT}|"
perl -e "use Time::HiRes qw(usleep); usleep(100000)"
done

Note that the perl command used above will cause a sleep of 0.1 seconds. Perl is used here, because the sleep command can't be used to sleep less than 1 second.

If FIRMWARE_EVENT entries appear in the AIX error log without FRU or location code callout, these events are likely attributed to an AIX memory page deconfiguration event, which is the result of a single memory cell being marked as unusable by the system firmware. The actual error is and will continue to be handled by ECC; however, notification of the unusable bit is also passed up to AIX. AIX in turn migrates the data and deallocates the memory page associated with this event from its memory map. This process is an AIX RAS feature which became available in AIX 5.3 and provides extra memory resilience and is no cause for alarm. Since the failure represents a single bit, a hardware action is NOT warranted.

To suppress logging, the following command will have to be entered and the partition will have to be rebooted to make the change effective:

# chdev -l sys0 -a log_pg_dealloc=false

Check the current status:

# lsattr -El sys0 -a log_pg_dealloc

More information about this function can be found in the "Highly Available POWER Servers for Business-Critical Applications" document which is available at the following link:

ftp://ftp.software.ibm.com/common/ssi/rep_wh/n/POW03003USEN/POW03003USEN.PDF (see pages 17-22 specifically).

The compare_report command is a very useful utility to compare the software installed on two systems, for example for making sure the same software is installed on two nodes of a PowerHA cluster.

First, create the necessary reports:

# ssh node2 "lslpp -Lc" > /tmp/node2
# lslpp -Lc > /tmp/node1

Next, generate the report. There are four interesting options: -l, -h, -m and -n:

• -l   Generates a report of base system installed software that is at a lower level.
• -h   Generates a report of base system installed software that is at a higher level.
• -m   Generates a report of filesets not installed on the other system.
• -n   Generates a report of filesets not installed on the base system.

For example:

# compare_report -b /tmp/node1 -o /tmp/node2 -l
#(baselower.rpt)
#Base System Installed Software that is at a lower level
#Fileset_Name:Base_Level:Other_Level
bos.msg.en_US.net.ipsec:6.1.3.0:6.1.4.0
bos.msg.en_US.net.tcp.client:6.1.1.1:6.1.4.0
bos.msg.en_US.rte:6.1.3.0:6.1.4.0
bos.msg.en_US.txt.tfs:6.1.1.0:6.1.4.0
xlsmp.msg.en_US.rte:1.8.0.1:1.8.0.3

# compare_report -b /tmp/node1 -o /tmp/node2 -h
#(basehigher.rpt)
#Base System Installed Software that is at a higher level
#Fileset_Name:Base_Level:Other_Level
idsldap.clt64bit62.rte:6.2.0.5:6.2.0.4
idsldap.clt_max_crypto64bit62.rte:6.2.0.5:6.2.0.4
idsldap.cltbase62.adt:6.2.0.5:6.2.0.4
idsldap.cltbase62.rte:6.2.0.5:6.2.0.4
idsldap.cltjava62.rte:6.2.0.5:6.2.0.4
idsldap.msg62.en_US:6.2.0.5:6.2.0.4
idsldap.srv64bit62.rte:6.2.0.5:6.2.0.4
idsldap.srv_max_cryptobase64bit62.rte:6.2.0.5:6.2.0.4
idsldap.srvbase64bit62.rte:6.2.0.5:6.2.0.4
idsldap.srvproxy64bit62.rte:6.2.0.5:6.2.0.4
idsldap.webadmin62.rte:6.2.0.5:6.2.0.4
idsldap.webadmin_max_crypto62.rte:6.2.0.5:6.2.0.4
AIX-rpm:6.1.3.0-6:6.1.3.0-4

# compare_report -b /tmp/node1 -o /tmp/node2 -m
#(baseonly.rpt)
#Filesets not installed on the Other System
#Fileset_Name:Base_Level
Java6.sdk:6.0.0.75
Java6.source:6.0.0.75
Java6_64.samples.demo:6.0.0.75
Java6_64.samples.jnlp:6.0.0.75
Java6_64.source:6.0.0.75
WSBAA70:7.0.0.0
WSIHS70:7.0.0.0

# compare_report -b /tmp/node1 -o /tmp/node2 -n
#(otheronly.rpt)
#Filesets not installed on the Base System
#Fileset_Name:Other_Level
xlC.sup.aix50.rte:9.0.0.1

The iptrace command can be very useful to find out what network traffic flows to and from an AIX system.

You can use any combination of these options, but you do not need to use them all:

• -a   Do NOT print out ARP packets.
• -s [source IP]   Limit trace to source/client IP address, if known.
• -d [destination IP]   Limit trace to destination IP, if known.
• -b   Capture bidirectional network traffic (send and receive packets).
• -p [port]   Specify the port to be traced.
• -i [interface]   Only trace for network traffic on a specific interface.

Example:

Run iptrace on AIX interface en1 to capture port 80 traffic to file trace.out from a single client IP to a server IP:

# iptrace -a -i en1 -s clientip -b -d serverip -p 80 trace.out

This trace will capture both directions of the port 80 traffic on interface en1 between the clientip and serverip and sends this to the raw file of trace.out.

To stop the trace:

# ps -ef|grep iptrace
# kill 

The ipreport command can be used to transform the trace file generated by iptrace to human readable format:

# ipreport trace.out > trace.report

AIX-rpm is a "virtual" package which reflects what has been installed on the system by installp. It is created by the /usr/sbin/updtvpkg script when the rpm.rte is installed, and can be run anytime the administrator chooses (usually after installing something with installp that is required to satisfy some dependency by an RPM package).

Since AIX-rpm has to have some sort of version number, it simply reflects the level of bos.rte on the system where /usr/sbin/updtvpkg is being run. It's just informational - nothing should be checking the level of AIX-rpm.

AIX doesn't just automatically run /usr/sbin/updtvpkg every time that something gets installed or deinstalled because on some slower systems with lots of software installed, /usr/sbin/updtvpkg can take a LONG time.

If you want to run the command manually:

# /usr/sbin/updtvpkg

If you get an error similar to "cannot read header at 20760 for lookup" when running updtvpkg, run a rpm rebuilddb:

# rpm --rebuilddb

Once you run updtvpkg, you can run a rpm -qa to see your new AIX-rpm package.

If you get a message "PRNG is not SEEDED" when trying to run ssh, you probably have an issue with the /dev/random and/or /dev/urandom devices on your system. These devices are created during system installation, but may sometimes be missing after an AIX upgrade.

Check permissions on random numbers generators, the "others" must have "read" access to these devices:

# ls -l /dev/random /dev/urandom
crw-r--r-- 1 root system 39, 0 Jan 22 10:48 /dev/random
crw-r--r-- 1 root system 39, 1 Jan 22 10:48 /dev/urandom

If the permissions are not set correctly, change them as follows:

# chmod o+r /dev/random /dev/urandom

Now stop and start the SSH daemon again, and retry if ssh works.

# stopsrc -s sshd
# startsrc -s sshd

If this still doesn't allow users to use ssh and the same message is produced, or if devices /dev/random and/or /dev/urandom are missing:

# stopsrc -s sshd
# rm -rf /dev/random
# rm -rf /dev/urandom
# mknod /dev/random c 39 0
# mknod /dev/urandom c 39 1
# randomctl -l
# ls -ald /dev/random /dev/urandom
# startsrc -s sshd

One of the best tools to look at LVM usage is with lvmstat. It can report the bytes read and written to logical volumes. Using that information, you can determine which logical volumes are used the most.

Gathering LVM statistics is not enabled by default:

# lvmstat -v data2vg
0516-1309 lvmstat: Statistics collection is not enabled for
        this logical device. Use -e option to enable.

As you can see by the output here, it is not enabled, so you need to actually enable it for each volume group prior to running the tool using:

# lvmstat -v data2vg -e

The following command takes a snapshot of LVM information every second for 10 intervals:

# lvmstat -v data2vg 1 10

This view shows the most utilized logical volumes on your system since you started the data collection. This is very helpful when drilling down to the logical volume layer when tuning your systems.

# lvmstat -v data2vg

Logical Volume    iocnt   Kb_read  Kb_wrtn   Kbps
  appdatalv      306653  47493022   383822  103.2
  loglv00            34         0     3340    2.8
  data2lv           453    234543   234343   89.3         

What are you looking at here?

• iocnt: Reports back the number of read and write requests.
• Kb_read: Reports back the total data (kilobytes) from your measured interval that is read.
• Kb_wrtn: Reports back the amount of data (kilobytes) from your measured interval that is written.
• Kbps: Reports back the amount of data transferred in kilobytes per second.

You can use the -d option for lvmstat to disable the collection of LVM statistics.