Tech Blog

These are blog entries written by the UNIX Health Check development team. Our team has extensive technical experience on both AIX and Red Hat systems, and we like to share our knowledge with our visitors.

Topics: AIX, SSA, Storage

SSA batteries

To find the status of the batteries of an SSA adapter, enter as root:

# ssa_fw_status -a ssaX
X is the number of your adapter, for example:
# ssa_fw_status -a ssa0
Fast write cache size: 32
Expected battery life: 22000
Powered on hours: 20706
Battery state: Active
After installing a new battery, enter the following command:
# ssa_format - l ssaX - b
This will reset the lifetime counter.

Topics: AIX, Storage, System Admin

Using a DVD-RAM as a regular file system

To use a DVD-RAM for writing and reading like a file system, use the following commands. This will work only at AIX 5.2 and above:

  • Put a DVD-RAM into the drive
    # udfcreate -d /dev/cd0
  • Mount the DVD-RAM:
    # mount -V udfs /dev/cd0 /mnt
    If you get an error, ensure /etc/vfs contains this line (and retry the mount command after validating):
    udfs 34 /sbin/helpers/udfmnthelp
Then use this as a regular filesystem.

Topics: AIX, SSA, Storage, System Admin

Renaming pdisks

If, for some reason, the pdisk and hdisk numbering of SSA disks is not sequential anymore, then there's a way to bring order in to chaos. Usually, the pdisk and hdisk numbering order are screwed up when you replace multiple disks together. Especially on HACMP clusters, a correct numbering of pdisks and hdisks on all nodes of the cluster, comes in handy.

Unmount all file systems on the specific disks, then varyoff the volume group:

# /usr/lib/methods/cfgssar -l ssar
If this doesn't help (it sometimes will), then renumber the disks manually:

Write down the pdisk names, hdisk names, location of the disks in the SSA drawer and the connection ID's of the disks. You can use lsdev -Cc pdisk to show you all the pdisks and the drawer and location codes. Use lsdev -Clpdiskx -Fconnwhere to show the connection ID of a pdisk. Then, figure out, how you want all disks numbered.

Remove the pdisks and hdisks with the rmdev -dl command.
Create the pdisks again:
# mkdev -p ssar -t scsd -c pdisk -s ssar -w [connection-ID] -l pdisk1
Create the hdisks again:
# mkdev -p ssar -t hdisk -c disk -s ssar -w [connection-ID] -l hdisk3
Test with:
# ssaxlate -l pdisk1
if it shows hdisk3 (Usually the hdisk number is 2 higher than the pdisk number if you use 2 SCSI disks in the rootvg).

If you've done all disks this way, check with lsdev -Cc pdisk. If you're happy, then varyon the volume group again and mount all filesystems.

Topics: AIX, Performance, Storage, System Admin

Creating a large file

When you wish to create a very large file for test purposes, try this command:

# dd if=/dev/zero bs=1024 count=2097152 of=./test.large.file
This wil create a file consisting of 2097152 blocks of 1024 bytes, which is 2GB. You can change the count value to anything you like.

Be aware of the fact, that if you wish to create files larger than 2GB, that your file system needs to be created as a "large file enabled file system", otherwise the upper file size limit is 2GB (under JFS; under JFS2 the upper limit is 64GB). Also check the ulimit values of the user-id you use to create the large file: set the file limit to -1, which is unlimited. Usually, the file limit is default set to 2097151 in /etc/security/limits, which stands for 2097151 blocks of 512 bytes = 1GB.

Another way to create a large file is:
# /usr/sbin/lmktemp ./test.large.file 2147483648
This will create a file of 2147483648 bytes (which is 1024 * 2097152 = 2GB).

You can use this large file for adapter throughput testing purposes:

Write large sequential I/O test:
# cd /BIG
# time /usr/sbin/lmktemp 2GBtestfile 2147483648
Divide 2048/#seconds for MB/sec write speed.

Read large sequential I/O test:
# umount /BIG
(This will flush file from memory)
# mount /BIG
# time cp 2GBtestfile /dev/null
Divide 2048/#seconds for MB/sec read speed.

Tip: Run nmon (select a for adapter) in another window. You will see the throughput for each adapter.

More information on JFS and JFS2 can be found here.

Topics: AIX, Storage, System Admin

Finding and removing core dumps

Use the following command to interactively find and remove core dumps on your system:

# find / -name core -exec file {} \; -exec rm -i {} \;

Topics: EMC, SAN, Storage

EMC PowerPath key installation

This describes how to configure the EMC PowerPath registration keys.

First, check the current configuration of PowerPath:

# powermt config
Warning: all licenses for storage systems support are missing or expired.
The install the keys:
# emcpreg -install

=========== EMC PowerPath Registration ===========
Do you have a new registration key or keys to enter?[n] y
Enter the registration keys(s) for your product(s),
one per line, pressing Enter after each key.
After typing all keys, press Enter again.

Key (Enter if done): P6BV-4KDB-QET6-RF9A-QV9D-MN3V
1 key(s) successfully added.
Key successfully installed.

Key (Enter if done):
1 key(s) successfully registered.
(Note: the license key used in this example is not valid).

Topics: EMC, SAN, Storage

Reset reservation bit

If you run into not being able to access an hdiskpowerX disk, you may need to reset the reservation bit on it:

# /usr/lpp/EMC/Symmetrix/bin/emcpowerreset fscsiX hdiskpowerX

Topics: EMC, SAN, Storage

BCV issue with Solution Enabler

There is a known bug on AIX with Solution Enabler, the software responsible for BCV backups. Hdiskpower devices dissapear and you need to run the following command to make them come back. This will happen when a server is rebooted. BCV devices are only visible on the target servers.

# /usr/lpp/EMC/Symmetrix/bin/mkbcv -a ALL
hdisk2 Available
hdisk3 Available
hdisk4 Available
hdisk5 Available
hdisk6 Available
hdisk7 Available
hdisk8 Available
hdiskpower1 Available
hdiskpower2 Available
hdiskpower3 Available
hdiskpower4 Available

Topics: EMC, SAN, Storage

EMC Grab

EMC Grab is a utility that is run locally on each host and gathers storage-specific information (driver version, storage-technical details, etc). The EMC Grab report creates a zip file. This zip file can be used by EMC support.

You can download the "Grab Utility" from the following locations:

When you've downloaded EMCgrab, and stored in a temporary location on the server like /tmp/emc, untar it using:
tar -xvf *tar
Then run:
/tmp/emc/emcgrab/emcgrab.sh
The script is interactive and finishes after a couple of minutes.

Topics: EMC, Installation, SAN, Storage

EMC and MPIO

You can run into an issue with EMC storage on AIX systems using MPIO (No Powerpath) for your boot disks:

After installing the ODM_DEFINITONS of EMC Symmetrix on your client system, the system won't boot any more and will hang with LED 554 (unable to find boot disk).

The boot hang (LED 554) is not caused by the EMC ODM package itself, but by the boot process not detecting a path to the boot disk if the first MPIO path does not corresponding to the fscsiX driver instance where all hdisks are configured. Let me explain that more in detail:

Let's say we have an AIX system with four HBAs configured in the following order:

# lscfg -v | grep fcs
fcs2 (wwn 71ca) -> no devices configured behind this fscsi2 driver instance (path only configured in CuPath ODM table)
fcs3 (wwn 71cb) -> no devices configured behind this fscsi3 driver instance (path only configured in CuPath ODM table)
fcs0 (wwn 71e4) -> no devices configured behind this fscsi0 driver instance (path only configured in CuPath ODM table)
fcs1 (wwn 71e5) -> ALL devices configured behind this fscsi1 driver instance
Looking at the MPIO path configuration, here is what we have for the rootvg disk:
# lspath -l hdisk2 -H -F"name parent path_id connection status"
name   parent path_id connection                      status
hdisk2 fscsi0 0       5006048452a83987,33000000000000 Enabled
hdisk2 fscsi1 1       5006048c52a83998,33000000000000 Enabled
hdisk2 fscsi2 2       5006048452a83986,33000000000000 Enabled
hdisk2 fscsi3 3       5006048c52a83999,33000000000000 Enabled
The fscsi1 driver instance is the second path (pathid 1), then remove the 3 paths keeping only the path corresponding to fscsi1 :
# rmpath -l hdisk2 -p fscsi0 -d
# rmpath -l hdisk2 -p fscsi2 -d
# rmpath -l hdisk2 -p fscsi3 -d
# lspath -l hdisk2 -H -F"name parent path_id connection status"
Afterwards, do a savebase to update the boot lv hd5. Set up the bootlist to hdisk2 and reboot the host.

It will come up successfully, no more hang LED 554.

When checking the status of the rootvg disk, a new hdisk10 has been configured with the correct ODM definitions as shown below:
# lspv
hdisk10 0003027f7f7ca7e2 rootvg active
# lsdev -Cc disk
hdisk2 Defined   00-09-01 MPIO Other FC SCSI Disk Drive
hdisk10 Available 00-08-01 EMC Symmetrix FCP MPIO Raid6
To summarize, it is recommended to setup ONLY ONE path when installing an AIX to a SAN disk, then install the EMC ODM package then reboot the host and only after that is complete, add the other paths. Dy doing that we ensure that the fscsiX driver instance used for the boot process has the hdisk configured behind.

Number of results found for topic Storage: 51.
Displaying results: 31 - 40.