Topics: AIX, Monitoring, Red Hat / Linux, Security, System Admin
Sudosh
Sudosh is designed specifically to be used in conjunction with sudo or by itself as a login shell. Sudosh allows the execution of a root or user shell with logging. Every command the user types within the root shell is logged as well as the output.
This is different from "sudo -s" or "sudo /bin/sh", because when you use one of these instead of sudosh to start a new shell, then this new shell does not log commands typed in the new shell to syslog; only the fact that a new shell started is logged.
If this newly started shell supports commandline history, then you can still find the commands called in the shell in a file such as .sh_history, but if you use a shell such as csh that does not support command-line logging you are out of luck.
Sudosh fills this gap. No matter what shell you use, all of the command lines are logged to syslog (including vi keystrokes). In fact, sudosh uses the script command to log all key strokes and output.
Setting up sudosh is fairly easy. For a Linux system, first download the RPM of sudosh, for example from rpm.pbone.net. Then install it on your Linux server:
Then, go to the /etc file system and open up /etc/sudosh.conf. Here you can adjust the default shell that is started, and the location of the log files. Default, the log directory is /var/log/sudosh. Make sure this directory exists on your server, or change it to another existing directory in the sudosh.conf file. This command will set the correct authorizations on the log directory:# rpm -ihv sudosh-1.8.2-1.2.el4.rf.i386.rpm Preparing... ########################################### [100%] 1:sudosh ########################################### [100%]
Then, if you want to assign a user sudosh access, edit the /etc/sudoers file by running visudo, and add the following line:# sudosh -i [info]: chmod 0733 directory /var/log/sudosh
username ALL=PASSWD:/usr/bin/sudoshNow, the user can login, and run the following command to gain root access:
Now, as a sys admin, you can view the log files created in /var/log/sudosh, but it is much cooler to use the sudosh-replay command to replay (like a VCR) the actual session, as run by the user with the sudosh access.$ sudo sudosh Password: # whoami root
First, run sudosh-replay without any paramaters, to get a list of sessions that took place using sudosh:
Now, you can actually replay the session, by (for example) running:# sudosh-replay Date Duration From To ID ==== ======== ==== == == 09/16/2010 6s root root root-root-1284653707-GCw26NSq Usage: sudosh-replay ID [MULTIPLIER] [MAXWAIT] See 'sudosh-replay -h' for more help. Example: sudosh-replay root-root-1284653707-GCw26NSq 1 2
# sudosh-replay root-root-1284653707-GCw26NSq 1 5The first paramtere is the session-ID, the second parameter is the multiplier. Use a higher value for multiplier to speed up the replay, while "1" is the actual speed. And the third parameter is the max-wait. Where there might have been wait times in the actual session, this parameter restricts to wait for a maximum max-wait seconds, in the example above, 5 seconds.
For AIX, you can find the necessary RPM here. It is slightly different, because it installs in /opt/freeware/bin, and also the sudosh.conf is located in this directory. Both Linux and AIX require of course sudo to be installed, before you can install and use sudosh.
Topics: AIX, Security, System Admin↑
SUID
Always watch out for files with the SUID bit set; especially if these are files that are not on the AIX system by default. Before any vendor or application team installs additional software on the AIX system, it may be worth file to run the following command, to discover any files with the SUID bit set:
# find / \( -perm -2000 -o -perm -4000 \) -type f -lsSave the output of this command for later reference. Once the vendor or application team is done installing their application and/or database software, run the same command again, to discover if any newly created files exist, especially those that are owned by user root and have the SUID bit set. This allows other users to run the command as if they we're root.
The SUID bit can only be set on binary executables on AIX (starting with release 3.2.5 of AIX). Other UNIX operating systems, such as Fedora, may allow scripts to run with SUID bits set. On AIX it is allowed to set the SUID bit on a script, but AIX simply ignores it, and runs the script as the user who started the script, not using the account that owns the script, because this would be a huge security hole.
However, it is still very easy to write a C program that does the trick. The following example is a program called "sushi". The source code of the program, sushi.c, looks like this:
#include <stdio.h>
#include <time.h>
char *getlogin();
#define LOG_FILE "/tmp/sushilog"
main(argc, argv)
int argc;
char **argv;
{
char buf[1024], *p=buf;
int i, t;
FILE *log;
char msg[BUFSIZ], *ct, *name;
*p='\0';
for (i=1; i<argc; ++i)
{
strcpy(p, argv[i]);
p += strlen(argv[i]);
if (i < argc-1)
{
*p = ' ';
++p;
*p = '\0';
}
}
time(&t);
ct = ctime(&t);
name = getlogin();
setuid(0);
log = fopen(LOG_FILE, "a");
if (!log)
printf("Couldn't open log file!\n");
else
{
sprintf(msg, "SUSHI: %s %s %s\n", name, buf, ct);
fputs(msg, log);
fclose(log);
system(buf);
}
}
The makefile looks like this (and makes sure the SUID bit is set when running "make"):
################################################
# Make rules #
################################################
all: sushi
sushi: sushi.o
$(CC) -o $@ sushi.o
clean:
rm -f *.o sushi
install:
cp -p sushi /bin
chown root /bin/sushi
chmod a+rx /bin/sushi
chmod u+s /bin/sushi
################################################
# Source/object rules #
################################################
sushi.o: sushi.c
gcc -c $*.c
Now, if this file is compiled as user root, a program called /bin/sushi will exists; it will be owned by user root, and the SUID will be set:
The sushi program basically takes everything entered as a parameter on the command line, and runs it. So if the file is owned by user root, it will run the parameter as user root. For example, if you would want to open a Korn shell as a regular user, and get root access:# ls -als /bin/sushi 8 -rwsr-xr-x 1 root root 6215 Sep 9 09:21 /bin/sushi
This is something that you want to avoid. Even vendors are known to build backdoors like these into their software. The find command shown at the beginning of this article will help you discover commands as these. Note that the good thing of the sushi program shown above is, that it will write an entry into log file /tmp/sushilog each time someone uses the command.$ /bin/sushi ksh # whoami root
To avoid users being able to run commands with the SUID set, you may want to add the "nosuid" option in /etc/filesystems for each file system:
/exports/install:
dev = "/exports/install"
vfs = nfs
nodename = fileserver.company.com
mount = true
options = ro,bg,hard,intr,nodev,nosuid,sec=sys
account = false
Especially for (permanently) NFS mounted file systems, it is a VERY good idea to have this nosuid option set, avoiding someone to create a sushi-like program on a NFS server, and being able to run the program as a regular user on the NFS client system, to gain root access on the NFS client; or if you want to mount a NFS share on a client temporarily, enable the nosuid by running:
Check if it is set by running:# mount -o nosuid server:/filesystem /mountpoint
# mount | grep nfs server /filesystem /mountpoint nfs3 Sep 09 09:30 nosuid
Topics: AIX, System Admin↑
Truss
To get more information on what a specific process is doing, you can get the truss command. That may be very useful, for example when a process appears to be hanging.
For example, if you want to know what the "recover" process is doing, first look up the PID of this process:
Then, run the truss command using that PID:# ps -ef | grep -i recover | grep -v grep root 348468 373010 0 17:30:25 pts/1 0:00 recover -f -a /etc
This way, you can see the process is actually sleeping.cscnimmaster# truss -p 348468 kreadv(0, 0x00000000, 0, 0x00000000) (sleeping...)
This is how to translate a hardware address to a physical location:
The command lscfg shows the hardware addresses of all hardware. For example, the following command will give you more detail on an individual device (e.g. ent1):
This ent1 device is an 'Internal Port'. If we check ent2 on the same box:# lscfg -pvl ent1 ent1 U788C.001.AAC1535-P1-T2 2-Port 10/100/1000 Base-TX PCI-X Adapter 2-Port 10/100/1000 Base-TX PCI-X Adapter: Network Address.............001125C5E831 ROM Level.(alterable).......DV0210 Hardware Location Code......U788C.001.AAC1535-P1-T2 PLATFORM SPECIFIC Name: ethernet Node: ethernet@1,1 Device Type: network Physical Location: U788C.001.AAC1535-P1-T2
This is a device on a PCI I/O card.# lscfg -pvl ent2 ent2 U788C.001.AAC1535-P1-C13-T1 2-Port 10/100/1000 Base-TX PCI-X 2-Port 10/100/1000 Base-TX PCI-X Adapter: Part Number.................03N5298 FRU Number..................03N5298 EC Level....................H138454 Brand.......................H0 Manufacture ID..............YL1021 Network Address.............001A64A8D516 ROM Level.(alterable).......DV0210 Hardware Location Code......U788C.001.AAC1535-P1-C13-T1 PLATFORM SPECIFIC Name: ethernet Node: ethernet@1 Device Type: network Physical Location: U788C.001.AAC1535-P1-C13-T1
For a physical address like U788C.001.AAC1535-P1-C13-T1:
- U788C.001.AAC1535 - This part identifies the 'system unit/drawer'. If your system is made up of several drawers, then look on the front and match the ID to this section of the address. Now go round the back of the server.
- P1 - This is the PCI bus number. You may only have one.
- C13 - Card Slot C13. They are numbered on the back of the server.
- T1 - This is port 1 of 2 that are on the card.
Topics: AIX, Installation, System Admin↑
install_all_updates
A usefull command to update software on your AIX server is install_all_updates. It is similar to running smitty update_all, but it works from the command line. The only thing you need to provide is the directory name, for example:
# install_all_updates -d .This installs all the software updates from the current directory. Of course, you will have to make sure the current directory contains any software. Don't worry about generating a Table Of Contents (.toc) file in this directory, because install_all_updates generates one for you.
By default, install_all_updates will apply the filesets. Use -c to commit any software. Also, by default, it will expand any file systems; use -x to prevent this behavior). It will install any requisites by default (use -n to prevent). You can use -p to run a preview, and you can use -s to skip the recommended maintenance or technology level verification at the end of the install_all_updates output. You may have to use the -Y option to agree to all licence agreements.
To install all available updates from the cdrom, and agree to all license agreements, and skip the recommended maintenance or technology level verification, run:
# install_all_updates -d /cdrom -Y -s
Use this procedure to quickly configure an HACMP cluster, consisting of 2 nodes and disk heartbeating.
Prerequisites:
Make sure you have the following in place:
- Have the IP addresses and host names of both nodes, and for a service IP label. Add these into the /etc/hosts files on both nodes of the new HACMP cluster.
- Make sure you have the HACMP software installed on both nodes. Just install all the filesets of the HACMP CD-ROM, and you should be good.
- Make sure you have this entry in /etc/inittab (as one of the last entries):
clinit:a:wait:/bin/touch /usr/es/sbin/cluster/.telinit
- In case you're using EMC SAN storage, make sure you configure you're disks correctly as hdiskpower devices. Or, if you're using a mksysb image, you may want to follow this procedure EMC ODM cleanup.
- Create the cluster and its nodes:
Enter a cluster name and select the nodes you're going to use. It is vital here to have the hostnames and IP address correctly entered in the /etc/hosts file of both nodes.# smitty hacmp Initialization and Standard Configuration Configure an HACMP Cluster and Nodes
- Create an IP service label:
Enter an IP Label/Address (press F4 to select one), and enter a Network name (again, press F4 to select one).# smitty hacmp Initialization and Standard Configuration Configure Resources to Make Highly Available Configure Service IP Labels/Addresses Add a Service IP Label/Address
- Set up a resource group:
Enter the name of the resource group. It's a good habit to make sure that a resource group name ends with "rg", so you can recognize it as a resource group. Also, select the participating nodes. For the "Fallback Policy", it is a good idea to change it to "Never Fallback". This way, when the primary node in the cluster comes up, and the resource group is up-and-running on the secondary node, you won't see a failover occur from the secondary to the primary node.# smitty hacmp Initialization and Standard Configuration Configure HACMP Resource Groups Add a Resource Group
Note: The order of the nodes is determined by the order you select the nodes here. If you put in "node01 node02" here, then "node01" is the primary node. If you want to have this any other way, now is a good time to correctly enter the order of node priority. - Add the Servie IP/Label to the resource group:
Select the resource group you've created earlier, and add the Service IP/Label.# smitty hacmp Initialization and Standard Configuration Configure HACMP Resource Groups Change/Show Resources for a Resource Group (standard)
- Run a verification/synchronization:
Just hit [ENTER] here. Resolve any issues that may come up from this synchronization attempt. Repeat this process until the verification/synchronization process returns "Ok". It's a good idea here to select to "Automatically correct errors".# smitty hacmp Extended Configuration Extended Verification and Synchronization
- Start the HACMP cluster:
Select both nodes to start. Make sure to also start the Cluster Information Daemon.# smitty hacmp System Management (C-SPOC) Manage HACMP Services Start Cluster Services
- Check the status of the cluster:
Wait until the cluster is stable and both nodes are up.# clstat -o # cldump
First, look at the available SAN disk devices on your nodes, and select a small disk, that won't be used to store any data on, but only for the purpose of doing the disk heartbeat. It is a good habit, to request your SAN storage admin to zone a small LUN as a disk heartbeating device to both nodes of the HACMP cluster. Make a note of the PVID of this disk device, for example, if you choose to use device hdiskpower4:
So, we're going to set up the disk heartbeat network on device hdiskpower4, with PVID 000a807f6b9cc8e5:# lspv | grep hdiskpower4 hdiskpower4 000a807f6b9cc8e5 None
- Create an concurrent volume group:
Select both nodes to create the concurrent volume group on by pressing F7 for each node. Then select the correct PVID. Give the new volume group a name, for example "hbvg".# smitty hacmp System Management (C-SPOC) HACMP Concurrent Logical Volume Management Concurrent Volume Groups Create a Concurrent Volume Group
- Set up the disk heartbeat network:
Select "diskhb" and accept the default Network Name.# smitty hacmp Extended Configuration Extended Topology Configuration Configure HACMP Networks Add a Network to the HACMP Cluster
- Run a discovery:
# smitty hacmp Extended Configuration Discover HACMP-related Information from Configured Nodes
- Add the disk device:
Select the disk device on both nodes by selecting the same disk on each node by pressing F7.# smitty hacmp Extended Configuration Extended Topology Configuration Configure HACMP Communication Interfaces/Devices Add Communication Interfaces/Devices Add Discovered Communication Interface and Devices Communication Devices
- Run a Verification & Synchronization again, as described earlier above. Then check with clstat and/or cldump again, to check if the disk heartbeat network comes online.
Topics: AIX, System Admin↑
MD5 for AIX
If you need to run an MD5 check-sum on a file on AIX, you will notice that there's not md5 or md5sum command available on AIX. Instead, use the following command to do this:
# csum -h MD5 [filename]Note: csum can't handle files larger than 2 GB.
When you want to mount an NFS file system on a node of an HACMP cluster, there are a couple of items you need check, before it will work:
- Make sure the hostname and IP address of the HACMP node are resolvable and provide the correct output, by running:
# nslookup [hostname] # nslookup [ip-address]
- The next thing you will want to check on the NFS server, if the node names of your HACMP cluster nodes are correctly added to the /etc/exports file. If they are, run:
# exportfs -va
- The last, and tricky item you will want to check is, if a service IP label is defined as an IP alias on the same adapter as your nodes hostname, e.g.:
The example above shows you that the default gateway is defined on the en1 interface. The next command shows you where your Service IP label lives:# netstat -nr Routing tables Destination Gateway Flags Refs Use If Exp Groups Route Tree for Protocol Family 2 (Internet): default 10.251.14.1 UG 4 180100 en1 - - 10.251.14.0 10.251.14.50 UHSb 0 0 en1 - - 10.251.14.50 127.0.0.1 UGHS 3 791253 lo0 - -
As you can see, the Service IP label (in the example above called "serviceip") is defined on en1. In that case, for NFS to work, you also want to add the "serviceip" to the /etc/exports file on the NFS server and re-run "exportfs -va". And you should also make sure that hostname "serviceip" resolves to an IP address correctly (and of course the IP address resolves to the correct hostname) on both the NFS server and the client.# netstat -i Name Mtu Network Address Ipkts Ierrs Opkts en1 1500 link#2 0.2.55.d3.75.77 2587851 0 940024 en1 1500 10.251.14 node01 2587851 0 940024 en1 1500 10.251.20 serviceip 2587851 0 940024 lo0 16896 link#1 1912870 0 1914185 lo0 16896 127 loopback 1912870 0 1914185 lo0 16896 ::1 1912870 0 1914185
Topics: AIX, Performance, System Admin↑
Nmon analyser - A free tool to produce AIX performance reports
Searching for an easy way to create high-quality graphs that you can print, publish to the Web, or cut and paste into performance reports? Look no further. The nmon_analyser tool takes files produced by the NMON performance tool, turns them into Microsoft Excel spreadsheets, and automatically produces these graphs.
You can download the tool here:
http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/
Restoring from tape:
First change the block size of the tape device to 512:
# chdev -l rmt0 -a block_size=512Check to make sure the block size of the tape drive has been changed:
# tctl -f /dev/rmt0 statusYou will receive output similar to this:
Change to the /tmp directory (or a directory where you would like to store the /image.data file from the mksysb image) and restore the /image.data file from the tape:rmt0 Available 09-08-00-0,0 LVD SCSI 4mm Tape Drive attribute value description user_settable block_size 512 BLOCK size (0=variable length) True compress yes Use data COMPRESSION True density_set_1 71 DENSITY setting #1 True density_set_2 38 DENSITY setting #2 True extfm yes Use EXTENDED file marks True mode yes Use DEVICE BUFFERS during writes True ret no RETENSION on tape change or reset True ret_error no RETURN error on tape change or reset True size_in_mb 36000 Size in Megabytes False
# cd /tmp # restore -s2 -xqvf /dev/rmt0.1 ./image.data
