Tech Blog

These are blog entries written by the UNIX Health Check development team. Our team has extensive technical experience on both AIX and Red Hat systems, and we like to share our knowledge with our visitors.

Topics: AIX, SAN, SDD, System Admin

Method error when running cfgmgr

If you see the following error when running cfgmgr:

Method error (/usr/lib/methods/fcmap >> /var/adm/essmap.out):
        0514-023 The specified device does not exist in the
                 customized device configuration database.
This is caused when you have ESS driver filesets installed, but no ESS (type 2105) disks in use on the system. Check the type of disks by running:
# lsdev -Cc disk | grep 2105
If no type 2105 disks are found, you can uninstall any ESS driver filesets:
# installp -u ibm2105.rte ibmpfe.essutil.fibre.data ibmpfe.essutil.rte

Topics: AIX, System Admin

Olson time zone support

"The public-domain time zone database contains code and data that represent the history of local time for many representative locations around the globe. It is updated periodically to reflect changes made by political bodies to time zone boundaries, UTC offsets, and daylight-saving rules. This database (often called tz or zoneinfo) is used by several implementations.

Each location in the database represents a national region where all clocks keeping local time have agreed since 1970. Locations are identified by continent or ocean and then by the name of the location, which is typically the largest city within the region. For example, America/New_York represents most of the US eastern time zone; America/Phoenix represents most of Arizona, which uses mountain time without daylight saving time (DST); America/Detroit represents most of Michigan, which uses eastern time but with different DST rules in 1975; and other entries represent smaller regions like Starke County, Indiana, which switched from central to eastern time in 1991 and switched back in 2006."

The public-domain time zone database is also widely known as the Olson time zone database and is the architecture on which the International Components for Unicode (ICU) and the Common Locale Data Repository (CLDR) time zone support relies.

In previous AIX releases, the method by which the operating system supports time zone conventions is based on the POSIX time zone specification. In addition to this industry standard approach, AIX V6.1 recognizes and processes the Olson time zone naming conventions to facilitate support for a comprehensive set of time zones.

This enhancement leverages the uniform time zone naming convention of the Olson database to offer an intuitive set of time zone values that can be assigned to the TZ time zone environment variable.

Note: Time zone definitions conforming to the POSIX specification are still supported and recognized by AIX. AIX checks the TZ environment variable to determine if the environment variable follows the POSIX specification rules. If the TZ environment variable does not match the POSIX convention, AIX calls the ICU library to get the Olson time zone translation.

The use of the Olson database for time zone support within AIX provides significant advantages over the traditional POSIX rules. One of the biggest advantages is that Olson database maintains a historical record of what the time zone rules were at given points in time, so that if the rules change in a particular location, dates and times can be interpreted correctly both in the present and past. A good example of this is the US state of Indiana, which just began using daylight saving time in the year 2006. Under the POSIX implementation, Indiana would have to set its time zone value to EST5EDT, which would format current dates correctly using daylight saving time, but would also format times from previous years as though they were on daylight saving time, which is incorrect. Use of the ICU API set for time zones also allows for localized display names for time zones. For example, Central Daylight Saving Time would have an abbreviation of CDT for all locales under a POSIX implementation, but under ICU/Olson, it displays properly as HAC (Heure Avancée du Centre) in a French locale.

As in previous AIX releases, system administrators can rely on the Systems Management Interface Tool (SMIT) to configure the time zone by using system defined values for the TZ environment variable. To accomplish this task, enter the main SMIT menu and select System Environments, Change / Show Date and Time to access the Change Time Zone Using System Defined Values menu. Alternatively, the SMIT fast path chtz_date will directly open the Change / Show Date and Time menu. Selecting the Change Time Zone Using System Defined Values option will prompt SMIT to open the Select COUNTRY or REGION menu.


SMIT uses the undocumented /usr/lib/nls/lstz -C command to produce the list of available countries and regions. Note that undocumented commands and features are not officially supported for customer use, are not covered by the AIX compatibility statement, and may be subject to change without notice.

After you have chosen the country or region in the Select COUNTRY or REGION menu, a new selection menu will list all available time zones for the country or region in question.


The selected value of the first column will be passed by SMIT to the chtz command, which in turn will change the TZ variable value in the /etc/environment system level configuration file. As with previous AIX releases, time zone configuration changes always require a system reboot to become effective.

SMIT uses the internal /usr/lib/nls/lstz -c command to produce the list of available time zones for a given country and region. The -c flag uses a country or region designation as the input parameter. The /usr/lib/nls/lstz -C command provides a list of available input parameters. The /usr/lib/nls/lstz command used without any flag provides a full list of all Olson time zones available on AIX. Note that undocumented commands and features are not officially supported for customer use, are not covered by the AIX compatibility statement, and may be subject to change without notice.

Topics: AIX, Security, System Admin

Changing a password without a prompt

If you have to change the password for a user, and you have a need to script this, for example if you have the password for multiple users, or on several different servers, then here's an easy way to change the password for a user, without having to type the password on the command line prompt:

# echo "user:password" | chpasswd

Topics: AIX, Backup & restore, System Admin, Veritas NetBackup

Client logging with Veritas NetBackup

If an AIX server is backed up by Veritas NetBackup, then this is how you can enable logging of the backups on your AIX client:

First, make sure the necessary folders exist in /usr/openv/netbackup/logs, and the access is set to 777, by running:

# mkdir bp bparchive bpbackup bpbkar bpcd bpdbsbora
# mkdir bpfilter bphdb bpjava-msvc bpjava-usvc bpkeyutil
# mkdir bplist bpmount bpnbat bporaexp bporaexp64 
# mkdir bporaimp bporaimp64 bprestore db_log dbclient
# mkdir symlogs tar user_ops
# chmod 777 *
Then, you have to change the default debug level in /usr/openv/netbackup/bp.conf, by adding:
VERBOSE = 2
By default, VERBOSE is set to one, which means there isn't any logging at all, so that is not helpful. You can go up to "VERBOSE = 5", but that may create very large log files, and this may fill up the file system. In any case, check how much disk space is available in /usr before enabling the logging of the Veritas NetBackup client.

Backups through Veritas NetBackup are initiated through inetd:
# egrep "bpcd" /etc/services
bpcd        13782/tcp # VERITAS NetBackup
bpcd        13782/udp # VERITAS NetBackup
# grep bpcd /etc/inetd.conf
bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd
Now all you have to do is wait for the NetBackup server (the one listed in /usr/openv/netbackup/bp.conf) to start the backup on the AIX client. After the backup has run, you should at least find a log file in the bpcd and bpbkar folders in /usr/openv/netbackup.

Topics: AIX, System Admin

Sendmail tips

To find out if sendmail is running:

# ps -ef | grep sendmail
To stop and restart sendmail:
# stopsrc -s sendmail
# startsrc -s sendmail -a "-bd -q30m"
Or:
# refresh -s sendmail
Use the -v flag on the mail command for "verbose" output. This is especially useful if you can't deliver mail, but also don't get any errors. E.g.:
# cat /etc/motd |mailx -v -s"test" email@address.com
To get sendmail to work on a system without DNS, create and/or edit /etc/netsvc.conf. It should contain 1 line only:
hosts=local
If you see the following error in the error report when starting sendmail:
DETECTING MODULE
'srchevn.c'@line:'355'
FAILING MODULE
sendmail
Then verify that your /etc/mail/sendmail.cf file is correct, and/or try starting the sendmail daemon as follows (instead of just running "startsrc -s sendmail"):
# startsrc -s sendmail -a "-bd -q30m"
More tips can be found here: http://www.angelfire.com/il2/sgillen/sendmail.html

Topics: AIX, PowerHA / HACMP, System Admin

NTP slewing in clusters

In order to keep the system time synchronized with other nodes in an HACMP cluster or across the enterprise, Network Time Protocol (NTP) should be implemented. In its default configuration, NTP will periodically update the system time to match a reference clock by resetting the system time on the node. If the time on the reference clock is behind the time of the system clock, the system clock will be set backwards causing the same time period to be passed twice. This can cause internal timers in HACMP and Oracle databases to wait longer periods of time under some circumstances. When these circumstances arise, HACMP may stop the node or the Oracle instance may shut itself down.

Oracle will log an ORA-29740 error when it shuts down the instance due to inconsistent timers. The hatsd daemon utilized by HACMP will log a TS_THREAD_STUCK_ER error in the system error log just before HACMP stops a node due to an expired timer.

To avoid this issue, system managers should configure the NTP daemon to increment time on the node slower until the system clock and the reference clock are in sync (this is called "slewing" the clock) instead of resetting the time in one large increment. The behavior is configured with the -x flag for the xntpd daemon.

To check the current running configuration of xntpd for the -x flag:

# ps -aef | grep xntpd | grep -v grep
    root  409632  188534   0 11:46:45      -  0:00 /usr/sbin/xntpd
To update the current running configuration of xntpd to include the -x flag:
# chssys -s xntpd -a "-x"
0513-077 Subsystem has been changed.
# stopsrc -s xntpd
0513-044 The /usr/sbin/xntpd Subsystem was requested to stop.
# startsrc -s xntpd
0513-059 The xntpd Subsystem has been started. Subsystem PID is 40932.
# ps -f | grep xntpd | grep -grep
    root  409632  188534   0 11:46:45      -  0:00 /usr/sbin/xntpd -x

Topics: AIX, HMC, System Admin

How to change the HMC password (of user hscroot)

You can ssh as user hscroot to the HMC, and change the password this way:

hscroot@hmc> chhmcusr -u hscroot -t passwd
Enter the new password:
Retype the new password:

Topics: AIX, Performance, Storage, System Admin

Creating a RAM disk on AIX

The AIX mkramdisk command allows system administrators to create memory-resident file systems. The performance benefits of using RAM disk can be astonishing. The unload of a large TSM database was reduced from 40 hours on SAN disks down to 10 minutes using RAM disk.

The configuration of a RAM disk file system is very simple and takes just a few minutes. Once the file system is mounted, it can be used like any other file system. There are three steps involved: creating the RAM disk, making the file system and then mounting the file system.

First, we create the RAM disk, specifying the size we want. Let's create a RAM disk of 4 GB:

# mkramdisk 4G
The system will assign the next available RAM disk. Since this is our first one, it will be assigned the name ramdisk0:
# ls -l /dev/ram*
brw-------    1 root system  46, 0 Sep 22 08:01 /dev/ramdisk0
If there isn't sufficient available memory to create the RAM disk you have requested, the mkramdisk command will alert you. Free up some memory or create a smaller size RAM disk. You can use Dynamic LPAR on the HMC or IVM to assign more memory to your partition.

We could use the RAM disk /dev/ramdisk0 as a raw logical volume, but here we

Topics: Performance, Red Hat / Linux, Storage, System Admin

Creating a RAM disk on Linux

On Linux, you can use the tmpfs to create a RAM disk:

# mkdir -p /mnt/tmp
# mount -t tmpfs -o size=20m tmpfs /mnt/tmp
This will create a 20 Megabyte sized RAM file system, mounted on /mtn/tmp. If you leave out the "-o size" option, by default half of the memory will be allocated. However, the memory will not be used, as long as no data is written to the RAM file system.

Topics: AIX, System Admin

Core file naming

Before AIX 5L Version 5.1, a core file was always stored in a file named core. If the same or another application generated another core file before you renamed the previous one, the original content was lost.

Beginning with AIX 5L Version 5.1, you can enable a unique naming of core files, but be aware that the default behavior is to name the files core. You apply the new enhancement by setting the environment variable CORE_NAMING to a non-NULL value, for example:

CORE_NAMING=yes
After setting CORE_NAMING, you can disable this feature by setting the variable to the NULL value. For example, if you are using the Korn shell, do the following:
export CORE_NAMING=
After setting CORE_NAMING, all new core will be stored in files of the format core.pid.ddhhmmss, where:
  • pid: Process ID
  • dd: Day of the month
  • hh: Hours
  • mm: Minutes
  • ss: Seconds
In the following example, two core files are generated by a process identified by PID 30480 at different times:
# ls -l core*
-rw-r--r--   1 user group   8179 Jan 28 2010  core.30480.28232347
-rw-r--r--   1 user group   8179 Jan 28 2010  core.30482.28232349
The time stamp used is in GMT and your time zone will not be used.

Also check out the lscore and the chcore commands, which can also be used to list and set core naming. These commands can also be set to define a core location, and to turn core compression on.

Number of results found for topic System Admin: 249.
Displaying results: 121 - 130.