The following is a description of how you can set up a private network between two VIO clients on one hardware frame.

Servers to set up connection: server1 and server2
Purpose: To be used for Oracle interconnect (for use by Oracle RAC/CRS)

IP Addresses assigned by network team:

192.168.254.141 (server1priv)
192.168.254.142 (server2priv)
Subnetmask: 255.255.255.0

• Log in to HMC GUI as hscroot.
• Change the default profile of server1, and add a new virtual Ethernet adapter. Set the port virtual Ethernet to 4 (PVID 4). Select "This adapter is required for virtual server activation". Configuration -> Manage Profiles -> Select "Default" -> Actions -> Edit -> Select "Virtual Adapters" tab -> Actions -> Create Virtual Adapter -> Ethernet adapter -> Set "Port Virtual Ethernet" to 4 -> Select "This adapter is required for virtual server activation." -> Click Ok -> Click Ok -> Click Close.
• Do the same for server2.
• Now do the same for both VIO clients, but this time do "Dynamic Logical Partitioning". This way, we don't have to restart the nodes (as we previously have only updated the default profiles of both servers), and still get the virtual adapter.
• Run cfgmgr on both nodes, and see that you now have an extra Ethernet adapter, in my case ent1.
• Run "lscfg -vl ent1", and note the adapter ID (in my case C5) on both nodes. This should match the adapter IDs as seen on the HMC.
• Now configure the IP address on this interface on both nodes.
• Add the entries for server1priv and server2priv in /etc/hosts on both nodes.
• Run a ping: ping server2priv (from server1) and vice versa.
• Done!

• On each node: deconfigure the en1 interface:

  # ifconfig en1 detach

• Rmdev the devices on each node:

  # rmdev -dl en1
  # rmdev -dl ent1
  

• Remove the virtual adapter with ID 5 from the default profile in the HMC GUI for server1 and server2.
• DLPAR the adapter with ID 5 out of server1 and server2.
• Run cfgmgr on both nodes to confirm the adapter does not re-appear. Check with:

  # lsdev -Cc adapter

• Done!

If clstat is not working, you may get the following error, when running clstat:

# clstat
Failed retrieving cluster information.

There are a number of possible causes:
clinfoES or snmpd subsystems are not active.
snmp is unresponsive.
snmp is not configured correctly.
Cluster services are not active on any nodes.

Refer to the HACMP Administration Guide for more information.
Additional information for verifying the SNMP configuration on AIX 6
can be found in /usr/es/sbin/cluster/README5.5.0.UPDATE

Commands clstat or cldump will not start if the internet MIB tree is not enabled in snmpdv3.conf file. This behavior is usually seen in AIX 6.1 onwards where this internet MIB entry was intentionally disabled as a security issue. This internet MIB entry is required to view/resolve risc6000clsmuxpd (1.3.6.1.4.1.2.3.1.2.1.5) MIB sub tree which is used by clstat or cldump functionality.

There are two ways to enable this MIB sub tree (risc6000clsmuxpd). They are:

1) Enable the main internet MIB entry by adding this line in /etc/snmpdv3.conf file:

VACM_VIEW defaultView internet - included -

But doing so is not recommended, as it unlocks the entire MIB tree.

2) Enable only the MIB sub tree for risc6000clsmuxpd without enabling the main MIB tree by adding this line in /etc/snmpdv3.conf file.

VACM_VIEW defaultView 1.3.6.1.4.1.2.3.1.2.1.5 - included -

Note: After enabling the MIB entry above snmp daemon must be restarted with the following commands as shown below:

# stopsrc -s snmpd
# startsrc -s snmpd

After snmp is restarted leave the daemon running for about two minutes before attempting to start clstat or cldump.

COMMUNITY public plubic noAuthNoPriv 0.0.0.0 0.0.0.0 -

smux 1.3.6.1.4.1.2.3.1.2.1.2 gated_password  # gated
smux 1.3.6.1.4.1.2.3.1.2.1.5 clsmuxpd_password # HACMP/ES for AIX ...

smux 1.3.6.1.4.1.2.3.1.2.1.2 gated_password
smux 1.3.6.1.4.1.2.3.1.2.1.5 clsmuxpd_password

# stopsrc -s hostmibd
# stopsrc -s snmpmibd
# stopsrc -s aixmibd
# stopsrc -s snmpd
# sleep 4
# chssys -s hostmibd -a "-c public"
# chssys -s aixmibd  -a "-c public"
# chssys -s snmpmibd  -a "-c public"
# sleep 4
# startsrc -s snmpd
# startsrc -s aixmibd
# startsrc -s snmpmibd
# startsrc -s hostmibd
# sleep 120
# stopsrc -s clinfoES
# startsrc -s clinfoES
# sleep 120

# snmpinfo -m dump -v -o /usr/es/sbin/cluster/hacmp.defs cluster

# smitty cm_ver_and_sync.select

To determine if the number of open files is growing over a period of time, issue lsof to report the open files against a PID on a periodic basis. For example:

# lsof -p (PID of process) -r (interval) > lsof.out

# df -kP filesystem_from_lsof | awk '{print $6}' | tail -1

# find filesystem_name -inum inode_from_lsof -print

# chuser nofiles=XXXXX user_id

# svmon -P java_pid -m | grep pers

If you try to estabilish a dsh session with a remote node sometimes you may get an error message like this:

# dsh -n server date
server.domain.com: Host key verification failed.
dsh:  2617-009 server.domain.com remote shell had exit code 255

# ssh server

# ssh server.domain.com date
The authenticity of host server.domain.com can't be established.
RSA key fingerprint is 1b:b1:89:c0:63:d5:f1:f1:41:fa:38:14:d8:60:ce.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added server.domain.com (RSA) 
to the list of known hosts.
Tue Sep  6 11:56:34 EDT 2011

# dsh -n server date
server.domain.com: Tue Sep  6 11:56:38 EDT 2011

Sometimes, you just need that one single file from a mksysb image backup. It's really not that difficult to accomplish this.

First of all, go to the directory that contains the mksysb image file:

# cd /sysadm/iosbackup

# restore -T -q -l -f vio1.mksysb | grep nimbck.ksh
New volume on vio1.mksysb:
Cluster size is 51200 bytes (100 blocks).
The volume number is 1.
The backup date is: Thu Jun  9 23:00:28 MST 2011
Files are backed up by name.
The user is padmin.
-rwxr-xr-x- 10   staff  May 23  08:37  1801 ./home/padmin/nimbck.ksh

# restore -x -q -f vio1.mksysb ./home/padmin/nimbck.ksh
x ./home/padmin/nimbck.ksh

# cd ./home/padmin
# ls -als nimbck.ksh
4 -rwxr-xr-x    1 10  staff  1801 May 23 08:37 nimbck.ksh

The savevg command can be used to backup user volume groups. All logical volume information is archived, as well as JFS and JFS2 mounted filesystems. However, this command cannot be used to backup raw logical volumes.

Save the contents of a raw logical volume onto a file using:

# dd if=/dev/lvname of=/file/system/lvname.dd

# dd if=/file/system/lvname.dd of=/dev/lvname

There is a LED which you can turn on to identify a device, which can be useful if you need to replace a device. It's the same binary as being used by diag.

To show the syntax:

# /usr/lpp/diagnostics/bin/usysident ? 
usage: usysident [-s {normal | identify}] 
                 [-l location code | -d device name]
       usysident [-t] 

# /usr/lpp/diagnostics/bin/usysident 
normal 

# /usr/lpp/diagnostics/bin/usysident -d hdisk1 
normal

# /usr/lpp/diagnostics/bin/usysident -s identify -d hdisk1 
# /usr/lpp/diagnostics/bin/usysident -d hdisk1 
identify

# /usr/lpp/diagnostics/bin/usysident -s normal -d hdisk1 
# /usr/lpp/diagnostics/bin/usysident -d hdisk1 
normal

Getting disk devices named the same way on, for example, 2 nodes of a PowerHA cluster, can be really difficult. For us humans though, it's very useful to have the disks named the same way on all nodes, so we can recognize the disks a lot faster, and don't have to worry about picking the wrong disk.

The way to get around this usually involved either creating dummy disk devices or running configuration manager on a specific adapter, like: cfgmgr -vl fcs0. This complicated procedure is not needed any more since AIX 7.1 and AIX 6.1 TL6, because a new command has been made available, called rendev, which is very easy to use for renaming devices:

# lspv
hdisk0  00c8b12ce3c7d496  rootvg  active
hdisk1  00c8b12cf28e737b  None

# rendev -l hdisk1 -n hdisk99

# lspv
hdisk0  00c8b12ce3c7d496  rootvg  active
hdisk99 00c8b12cf28e737b  None

There's a simple command to list information about a mksysb image, called lsmksysb:

# lsmksysb -lf mksysb.image
VOLUME GROUP:      rootvg
BACKUP DATE/TIME:  Mon Jun 6 04:00:06 MST 2011
UNAME INFO:        AIX testaix1 1 6 0008CB1A4C00
BACKUP OSLEVEL:    6.1.6.0
MAINTENANCE LEVEL: 6100-06
BACKUP SIZE (MB):  49920
SHRINK SIZE (MB):  17377
VG DATA ONLY:      no

rootvg:
LV NAME    TYPE     LPs  PPs  PVs  LV STATE      MOUNT POINT
hd5        boot     1    2    2    closed/syncd  N/A
hd6        paging   32   64   2    open/syncd    N/A
hd8        jfs2log  1    2    2    open/syncd    N/A
hd4        jfs2     8    16   2    open/syncd    /
hd2        jfs2     40   80   2    open/syncd    /usr
hd9var     jfs2     40   80   2    open/syncd    /var
hd3        jfs2     40   80   2    open/syncd    /tmp
hd1        jfs2     8    16   2    open/syncd    /home
hd10opt    jfs2     8    16   2    open/syncd    /opt
dumplv1    sysdump  16   16   1    open/syncd    N/A
dumplv2    sysdump  16   16   1    open/syncd    N/A
hd11admin  jfs2     1    2    2    open/syncd    /admin