Tech Blog

These are blog entries written by the UNIX Health Check development team. Our team has extensive technical experience on both AIX and Red Hat systems, and we like to share our knowledge with our visitors.

Topics: Red Hat / Linux, Security

ILO access through SSH

This describes how to get SSH access to a Linux system on a HP blade system, which requires you to work through the ILO:

First of all, you need to know the ILO IP address. Simply open up an SSH session to this IP address:

# ssh -l ilo-admin 10.250.21.37
ilo-admin@10.250.21.37's password:
User:ilo-admin logged-in to 10.250.21.37
iLO 2 Advanced 1.60 at 16:05:58 Jul 11 2008
Server Name:
Server Power: On

</>hpiLO->
The next thing you need to do is type "VSP", hit ENTER and login to the server:
hpiLO-> VSP
Starting virtual serial port
Press 'ESC (' to return to the CLI Session
</>hpiLO-> Virtual Serial Port active: IO=0x02F8 INT=3
[ENTER]
</>hpiLO-> Virtual Serial Port active: IO=0x02F8 INT=3
Red Hat Enterprise Linux ES release 4 (Nahant Update 8)
Kernel 2.6.9-89.ELsmp on an i686
hostname login:
To make this magic happen, we need to spawn a getty on /dev/ttyS1. You might see somthing like this in /etc/inittab:
mo1::off:/sbin/mgetty -x 0 -D -s38400 -a /dev/ttyS1
The mgetty will not work. That expects a modem. Comment it out (it is off anyways). Add this line:
ilo:2345:respawn:/sbin/agetty ttyS1 115200 vt100
Then allows root to login on that tty:
# echo "ttyS1" >> /etc/securetty
Then reread the /etc/inittab and spawn any missing processes, like the new getty:
# kill -HUP 1
Now you should be able to ssh to the servers ILO IP address, login as ilo-admin, run VSP and get a login prompt.

Topics: Red Hat / Linux, System Admin

How to enable ntpd on Linux

This is a procedure to enable time synchronization (ntpd) on Linux (in this example, replace the IP address of the time server with the IP address of your time server):

  • Stop all applications on the server.
  • Check if you can access the time servers, e.g.:
    # ntpdate -q 10.250.9.11
  • Check if the current timezone setting is correct by simply running the date command.
  • Set the time and date correct:
    # ntpdate 10.250.9.11
  • Start the NTP server:
    # service ntpd start
  • Check the status:
    # service ntpd status
  • Check the time synchronization (it may take some time for the client to synchronize with its time server):
    # ntpq -p
  • Check that ntpd is started at system restart:
    # chkconfig ntpd on
    # chkconfig --list | grep ntpd
  • Check the process:
    # ps -ef | grep ntpd
  • Reboot the server:
    # reboot

Topics: Red Hat / Linux, System Admin

Enabling sendmail on Linux

  • Make sure the relay host, e.g. the Exchange server, allows incoming email from your Linux server.
  • Make sure no firewall is blocking SMTP traffic from the Linux host. You can use nmap for this purpose:
    # nmap -sS smtp.server.com
    (Replace "smtp.server.com" for the actual SMTP server hostname of your environment).
  • Check it the DNS configuration is correct in /etc/resolv.conf and make sure you can resolve the hostname and its IP address reversely:
    # nslookup hostname
    # nslookup ipaddress
    
    (use the IP address returned by the first DNS lookup on the hostname to reversely lookup the hostname by the IP address).
  • Make a copy of sendmail.mc and sendmail.cf in /etc/mail.
  • Edit sendmail.mc (add in the name of your SMTP server):
    define(`confTRUSTED_USER', `root')dnl
    define(`SMART_HOST', `esmtp:smtp.server.com')dnl
    MASQUERADE_AS(`hostname.com')dnl
    FEATURE(masquerade_envelope)dnl
    FEATURE(masquerade_entire_domain)dnl
    
  • Then run:
    # make -C /etc/mail
  • Edit sendmail.cf by modifying the "C{E}" line in sendmail.cf. Take any user listed on that line including root off that line, so mail sent from root gets masqueraded as well. Towards the bottom of sendmail.cf file, there is a section for Ruleset 94. Make sure that after "R$+" there is ONE tab (no space, or multiple spaces/tabs):
    SMasqEnv=94
    R$+ $@ $>MasqHdr $1
    
  • Clean out /var/spool/clientmqueue and /var/spool/mqueue (there may be lots of OLD emails there, we may not want to send these anymore).
  • Then restart sendmail:
    # service sendmail restart
    (or "service sendmail start" if it isn't running yet; check the status with: "service sendmail status").
  • Make sure that sendmail is started at system restart:
    # chkconfig sendmail on # chkconfig --list sendmail
  • Open a "tail -f /var/log/maillog" so you can watch any syslog activity for mail (of course there should be a "mail.*" entry in /etc/syslog.conf directing output to /var/log/maillog for this to work).
  • Send a test email message:
    # echo "test" | sendmail -v address@email.com
    (and check that the email message is actually accepted for delivery in the verbose output).
  • Wait for the mail to arrive in your mailbox.

Topics: Networking, Red Hat / Linux

Enabling bonding in Linux

To enable "etherchannel" or "bonding" in Linux nomenclature:

  • Add these two lines to /etc/modprobe.conf:
    alias bond0 bonding
    options bond0 miimon=100 mode=1 primary=eth0
    Entry "mode=1" simply means active/standby. Entry "miimon" is the number in milliseconds to wait before determining a link dead (Change eth0 to match your primary device, if it is different. Blades sometimes have eth4 as the primary device).
  • In /etc/sysconfig/network-scripts create ifcfg-bond0 with the following (of course, change the network info to match your own):
    DEVICE=bond0
    BROADCAST=10.250.19.255
    IPADDR=10.250.19.194
    NETMASK=255.255.255.0
    GATEWAY=10.250.19.1
    ONBOOT=yes
    BOOTPROTO=none
  • Change ifcfg-eth0 and ifcfg-eth1 (or whatever they are) to resemble this:
    DEVICE=eth0
    HWADDR=00:22:64:9B:54:9C
    USERCTL=no
    ONBOOT=yes
    MASTER=bond0
    SLAVE=yes
    BOOTPPROTO=none
    Leave the value of HWADDR to whatever it is in your file. This is important. It is this devices MAC Address.
  • Run /etc/init.d/network restart. You will want to do at least this part from the console, in case something goes wrong.
  • Once you get your "OK" and the prompt comes back, do an ifconfig -a. You should see bond0.
  • Make sure you can ping your default gateway. After that, all should be good.
Note: When making back up copies of the ifcfg-* files, you must either move the backup files out of this directory or change your backup copy strategy for these files. The primary network script that reads these files, basically runs: ls ifcg-*. It then creates an interface based on the part after the dash ("-"). So if you run, for example:
# cp ifcfg-eth0 ifcfg-eth0.bak
You will end up with an alias device of eth0 called eth0.bak. Instead do this:
# cp ifcfg-eth0 bak.$(date +%Y%m%d).ifcfg-eth0
That foils the configuration script and allows to keep backup/backout copies in the same directory with the working copies.

Topics: AIX, Backup & restore, Monitoring, Red Hat / Linux, Spectrum Protect

Report the end result of a TSM backup

A very easy way of getting a report from a backup is by using the POSTSchedulecmd entry in the dsm.sys file. Add the following entry to your dsm.sys file (which is usually located in /usr/tivoli/tsm/client/ba/bin or /opt/tivoli/tsm/client/ba/bin):

POSTSchedulecmd "/usr/local/bin/RunTsmReport"
This entry tells the TSM client to run script /usr/local/bin/RunTSMReport, as soon as it has completed its scheduled command. Now all you need is a script that creates a report from the dsmsched.log file, the file that is written to by the TSM scheduler:
#!/bin/bash
TSMLOG=/tmp/dsmsched.log
WRKDIR=/tmp
echo "TSM Report from `hostname`" >> ${WRKDIR}/tsmc
tail -100 ${TSMLOG} > ${WRKDIR}/tsma
grep -n "Elapsed processing time:" ${WRKDIR}/tsma > ${WRKDIR}/tsmb
CT2=`cat ${WRKDIR}/tsmb | awk -F":" '{print $1}'`
((CT3 = $CT2 - 14))
((CT5 = $CT2 + 1 ))
CT4=1
while read Line1 ; do
   if [ ${CT3} -gt ${CT4} ] ; then
      ((CT4 = ${CT4} + 1 ))
   else
      echo "${Line1}" >> ${WRKDIR}/tsmc
      ((CT4 = ${CT4} + 1 ))
      if [ ${CT4} -gt ${CT5} ] ; then
         break
      fi
   fi
done < ${WRKDIR}/tsma
mail -s "`hostname` Backup" email@address.com < ${WRKDIR}/tsmc
rm ${WRKDIR}/tsma ${WRKDIR}/tsmb ${WRKDIR}/tsmc

Topics: Networking, Red Hat / Linux

Linux bond interfaces

Linux allows binding multiple network interfaces into a single channel/NIC using special kernel module called bonding. According to official bonding documentation, The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical "bonded" interface. The behavior of the bonded interfaces depends upon the mode; generally speaking, modes provide either hot standby or load balancing services. Additionally, link integrity monitoring may be performed.

Setting up bounding is easy with RHEL v4.0. Red Hat Linux stores network configuration in /etc/sysconfig/network-scripts/ directory. First, you need to create bond0 config file:

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
Append following lines to it:
DEVICE=bond0
IPADDR=192.168.1.20
NETWORK=192.168.1.0
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
Replace above IP address with your actual IP address. Save file and exit to shell prompt. Now open the configuration files for eth0 and eth1 in the same directory using the vi text editor and make sure file read as follows for eth0 interface:
# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
Repeat the same for the ifcfg-eth1 file, of course, set the DEVICE to eth1. Then, make sure that the following two lines are added to either /etc/modprobe.conf or /etc/modules.conf (see this page or also this page for more information):
alias bond0 bonding
options bond0 mode=1 miimon=100
Then load the bonding module:
# modprobe bonding
Restart networking service in order to bring up bond0 interface:
# service network restart
Verify everything is working:
# less /proc/net/bonding/bond0
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:c6:be:59

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:c6:be:63

Topics: Red Hat / Linux, SAN, Storage

Emulex hbanyware

If you have Emulex HBA''s and the hbanyware software installed, for example on Linux, then you can use the following commands to retrieve information about the HBA''s:

To run a GUI version:

# /usr/sbin/hbanyware/hbanyware
To run the command-line verion:
# /usr/sbin/hbanyware/hbacmd listhbas
To get for attributes about a specific HBA:
# /usr/sbin/hbanyware/hbacmd listhbas 10:00:00:00:c9:6c:9f:d0

Topics: Red Hat / Linux

No SSH connectivity

If you've just installed a Red Hat (or Fedora) system and you find that even though the ssh daemon is running and you're very sure there's no firewall blocking your access, that you still are unable to ssh or telnet to the system, then you may have run into to firewall that gets installed with Red Hat (or Fedora).

A very easy way to flush all rules from that firewall can be done by running:

# iptables -F
Now check again if you access your system through ssh and/or telnet.

Topics: Red Hat / Linux

Authentication token lock busy

If you run into an error: "passwd: Authentication token lock busy" (E.g. on a RHEL server while trying to update the root password), then probably the root file system is set to read-only mode. You can make it read-write, by doing:

# mount -o remount,rw /

Topics: Red Hat / Linux

Determining which adapter has link

If you need to determine on a Linux system which network interface actually has link on it (meaning, that it is cabled correctly to the switch), then run the following command:

# mii-tool -v

Number of results found for topic Red Hat / Linux: 103.
Displaying results: 81 - 90.