These are blog entries written by the UNIX Health Check development team. Our team has extensive technical experience on both AIX and Red Hat systems, and we like to share our knowledge with our visitors.
It is very easy to clone your rootvg to another disk, for example for testing purposes. For example: If you wish to install a piece of software, without modifying the current rootvg, you can clone a rootvg disk to a new disk; start your system from that disk and do the installation there. If it succeeds, you can keep using this new rootvg disk; If it doesn't, you can revert back to the old rootvg disk, like nothing ever happened.
First, make sure every logical volume in the rootvg has a name that consists of 11 characters or less (if not, the alt_disk_copy command will fail).
To create a copy on hdisk1, type:
alt_disk_copy -d hdisk1
If you now restart your system from hdisk1, you will notice, that the original rootvg has been renamed to old_rootvg. To delete this volume group (in case you're satisfied with the new rootvg), type:
# alt_rootvg_op -X old_rootvg
A very good article about alternate disk installs can be found on developerWorks.
If you wish to copy a mirrored rootvg to two other disks, make sure to use quotes around the target disks, e.g. if you wish to create a copy on disks hdisk4 and hdisk5, run:
The AIX kernel has an "enter_dbg" variable in it that can be set at the beginning of the boot processing which will cause all boot process output to be sent to the system console. In some cases, this data can be useful in debugging boot issues. The procedure for setting the boot debugger is as follows:
First: Preparing the system.
Set up KDB to present an initial debugger screen
# bosboot -ad /dev/ipldevice -I
Reboot the server:
# shutdown -Fr
Setting up for Kernel boot trace:
When the debugger screen appears, set enter_dbg to the value we want to use:
An adapter that has previously been added to a LPAR and now needs to be removed, usually doesn't want to be removed from the LPAR, because it is in use by the LPAR. Here's how you find and remove the involved devices on the LPAR:
First, run:
# lsslot -c pci
This will find the adapter involved.
Then, find the parent device of a slot, by running:
# lsdev -Cl [adapter] -F parent
(Fill in the correct adapter, e.g. fcs0).
Now, remove the parent device and all its children:
# rmdev -Rl [parentdevice] -d
For example:
# rmdev -Rl pci8 -d
Now you should be able to remove the adapter via the HMC from the LPAR.
If you need to replace the adapter because it is broken and needs to be replaced, then you need to power down the PCI slot in which the adapter is placed:
After issuing the "rmdev" command, run diag and go into "Task Selection", "Hot Plug Task", "PCI Hot Plug Manager", "Replace/Remove a PCI Hot Plug Adapter". Select the adapter and choose "remove".
After the adapter has been replaced (usually by an IBM technician), run cfgmgr again to make the adapter known to the LPAR.
When you encounter an issue where ssh through a firewall works perfectly, but when doing scp of large files (for example mksysb images) the scp connection stalls, then there's a solution to this problem: Add "-l 8192" to the scp command.
The reason for scp to stall, is because scp greedily grabs as much bandwith of the network as possible when it transfers files, any delay caused by the network switch of the firewall can easily make the TCP connection stalled.
Adding the option "-l 8192" limits the scp session bandwith up to 8192 Kbit/second, which seems to work safe and fast enough (up to 1 MB/second):
If your AIX server level is below 5.3 TL06, the easiest way is just to upgrade your current OS to TL 06 at minimum (take note it will depend of configurations for Power6 processors) then clone your server and install it on the new p6.
But if you want to avoid an outage on your account, you can do the next using a NIM server (this is not official procedure for IBM, so they do not support this):
Create your mksysb resource and do not create a spot from mksysb.
Create an lppsource and spot with minimum TL required (I used TL08).
Once you do nim_bosinst, choose the mksysb, and the created spot. It will send a warning message about spot is not at same level as mksysb, just ignore it.
Do all necessary to boot from NIM.
Once restoring the mksysb, there's some point where it is not able to create the bootlist because it detects the OS level is not supported on p6. So It will ask to continue and fix it later via SMS or fix it right now.
Choose to fix it right now (it will open a shell). You will notice oslevel is as the same as mksysb.
Create a NFS from NIM server or another server where you have the necessary TL and mount it on the p6.
Proceed to do an upgrade, change the bootlist, exit the shell. Server will boot with new TL over the p6.
By default, AIX will avoid logging duplicate errpt entries. You can see the default settings using smitty errdemon, which will be set to checking duplicate entries within a 10000 milliseconds time interval (10 seconds). Also, the default duplicate error maximum is set to 1000, so after 1000 duplicates, an additional entry will be made, depending on which is reached earlier, the duplicate time interval of 10 seconds or the duplicate error maximum.
Date/Time: Sun May 17 22:11:46 PDT 2009
Sequence Number: 8539
Machine Id: 00GB214D4C00
Node Id: blahblah
Class: O
Type: INFO
Resource Name: RMCdaemon
Description
The default log file has been changed.
Probable Causes
The current default log file has been renamed and a new log file created.
Failure Causes
The current log file has become too large.
Recommended Actions
No action is required.
Detail Data
DETECTING MODULE
RSCT,rmcd_err.c,1.17,512
ERROR ID
6e0tBL/GsC28/gQH/ne1K//...................
REFERENCE CODE
File name
/var/ct/IW/log/mc/default
This error report entry refers to a file that was created, called /var/ct/IW/log/mc/default. Actually, when the file reaches 256 Kb, a new one is created, and the old one is renamed to default.last.
The following messages can be found in this file:
2610-217 Received 193 unrecognized messages in the last 10.183333 minutes.
Service is rmc.
This message more or less means:
"2610-217
Received count of unrecognized messages unrecognized messages in the last time minutes. Service is service_name.
Explanation:
The RMC daemon has received the specified number of unrecognized messages within the specified time interval. These messages were received on the UDP port, indicated by the specified service name, used for communication among RMC daemons. The most likely cause of this error is that this port number is being used by another application.
User Response:
Validate that the port number configured for use by the Resource Monitoring and Control daemon is only being used by the RMC daemon."
Check if something else is using the port of the RMC daemon:
# grep RMC /etc/services
rmc 657/tcp # RMC
rmc 657/udp # RMC
# lsof -i :657
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rmcd 1384574 root 3u IPv6 0xf35f20 0t0 UDP *:rmc
rmcd 1384574 root 14u IPv6 0xf2fd39 0t0 TCP *:rmc (LISTEN)
# netstat -Aan | grep 657
f1000600022fd398 tcp 0 0 *.657 *.* LISTEN
f10006000635f200 udp 0 0 *.657 *.*
The socket 0x22fd008 is being held by proccess 1384574 (rmcd).
No, it is actually the RMC daemon that is using this port, so this is fine.
Start an IP trace to find out who's transmitting to this port:
# startsrc -s iptrace -a "-b -p 657 /tmp/iptrace.bin"
To turn on PRM trace, on LPAR do:
# /usr/sbin/rsct/bin/rmctrace -s ctrmc -a PRM=100
Monitor /var/ct/3410054220/log/mc/default file on LPAR make sure
you see NEW errors for 2610-217 log after starting trace, may need to
wait for 10min (since every 10 minutes it logs one 2610-217 error entry).
To monitor default file, do:
# tail -f /var/ct/3410054220/log/mc/default
To stop iptrace, on LPAR do:
# stopsrc -s iptrace
To stop PRM trace, on LPAR do:
# /usr/sbin/rsct/bin/rmctrace -s ctrmc -a PRM=0
To format the iptraces, do:
# ipreport -rns /tmp/ipt > /tmp/ipreport.out
Collect ctsnap data, on LPAR do:
# ctsnap -x runrpttr
When analyzing the data you may find several nodeid's in the packets.
On HMC side, you can run: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc to find out if 22758085eb959fec was managed by HMC. You will need to have root access on the HMC to run this command. And you can get a temporary password from IBM to run with the pesh command as the hscpe user to get this root access.
This command will list the known managed systems to the HMC and their nodeid's.
Then, on the actual LPARs run /usr/sbin/rsct/bin/lsnodeid to determine the nodeid of that LPAR. If you find any discrepancies between the HMC listing of nodeid's and the nodeid's found on the LPAR's, then that is causing the errpt message to appear about the change of the log file.
To solve this, you have to recreate the RMC deamon databases on both the HMC and on the LPARs that have this issue:
On HMC side run:
# /usr/sbin/rsct/bin/rmcctrl -z
# /usr/sbin/rsct/bin/rmcctrl -A
# /usr/sbin/rsct/bin/rmcctrl -p
Then run /usr/sbin/rsct/install/bin/recfgct on the LPARs:
# /usr/sbin/rsct/install/bin/recfgct
0513-071 The ctcas Subsystem has been added.
0513-071 The ctrmc Subsystem has been added.
0513-059 The ctrmc Subsystem has been started.
Subsystem PID is 194568.
# /usr/sbin/rsct/bin/lsnodeid
6bcaadbe9dc8904f
Repeat this for every LPAR connected to the HMC.
After that, you can run on the HMC again:
A very usefull command to compary 2 files is sdiff. Let's say you want to compare the lslpp from 2 different hosts, then sdiff -s shows the differences between two files next to each other:
