Topics: PowerHA / HACMP
PowerHA UI
In PowerHA version 7.2.2, you can use a graphical user interface (GUI) to monitor your cluster environment.
The PowerHA GUI provides the following advantages over the PowerHA command line:
Monitor the status for all clusters, sites, nodes, and resource groups in your environment. Scan event summaries and read a detailed description for each event. If the event occurred because of an error or issue in your environment, you can read suggested solutions to fix the problem. Search and compare log files. Also, the format of the log file is easy to read and identify important information. View properties for a cluster such as the PowerHA SystemMirror version, name of sites and nodes, and repository disk information.
Check out a video that provides an overview for the PowerHA GUI at https://www.youtube.com/watch?v=d_QVvh2dcCM.
Information on how to install and start using it can be found on the IBM website.
The following procedure describes how to perform a command-line based upgrade of the Hardware Management Console (HMC) from version V8 R8.6.0 SP1 to V8 R8.7.0 SP1. This involves these two steps:
- First, upgrade to version V8 R8.7.0 (also known as MH01704).
- Next, update to service pack 1 of V8 R8.7.0
- The HMC, called hmc01, at IP address 172.16.52.100.
- A separate AIX system, called aix01, at IP address 172.16.52.101.
So, let's first perform the upgrade to version V8 R8.7.0. This is based on https://www-01.ibm.com/support/docview.wss?uid=nas8N1020108. Download files img2a, img3a, base.img, disk1.img and hmcnetworkfiles.sum into a folder on the separate AIX system. You may download these files directly from ftp://ftp.software.ibm.com/software/server/hmc/network/v8870/x86/ to the AIX system.
You can use FTP to download the files, by logging in anonymously to the IBM FTP server and using any password (it says to specify your complete email address, but in fact anything you type will be fine). For example:
Downloading these files may take a while as they are several gigabytes in size.# ftp ftp.software.ibm.com Connected to dispsd-40-www3.boulder.ibm.com. 220 ProFTPD 1.3.5b Server (proftpd) Name (ftp.software.ibm.com:root): anonymous 331 Anonymous login ok, send your complete email address as your password Password: 230 Anonymous access granted, restrictions apply ftp> bin 200 Type set to I ftp> cd software/server/hmc/network/v8870/x86/ 250 CWD command successful ftp> promp Interactive mode off. ftp> mget * 200 PORT command successful 150 Opening BINARY mode data connection for img3a (34015945 bytes) ...
Or, if you have wget installed on the AIX system, the following command can be used to get the individual files, for example:
# wget ftp://ftp.software.ibm.com/software/server/hmc/network/v8870/x86/*Now that you have downloaded all the required files, for example in folder /HMC on the AIX system, make sure that the files can be read by everyone:
Then, log in to the command line of the HMC:# chmod -R 755 /HMC # chown -R root.system /HMC
For this to work, you obviously need to know the password for the hscroot account on the HMC, and you need to have remote SSH access enabled on the HMC. If necessary to enable the remote SSH access, log in with a web browser to the GUI of the HMC at https://172.16.52.100 and change the remote access setting through the GUI (we'll not cover how to do this in this procedure though).# ssh -l hscroot 172.16.52.100 hscroot@172.16.52.100's password: Last login: Wed Jan 17 22:24:49 2018 hscroot@hmcw01:~>
On the HMC, run the following command to save the upgrade data to disk:
Then, tell it to go download the upgrade files through SFTP from the AIX server:# saveupgdata -r disk
Note here how the root password of the AIX system is set to P@ssw0rd, and that the files will be downloaded from the /HMC folder on the AIX system. Also note here, that with newer OpenSSH levels on AIX, root may not be allowed to start up a SFTP session to the AIX system remotely, and thus, in that case, it may be better to use a different user account (other than root) to download the files. Any account will do, as long as that account has access to the files in the /HMC folder (or any other folder name, where you have downloaded the HMC network installation files).# getupgfiles -r sftp -h 172.16.52.101 -u root --passwd 'P@ssw0rd' -d /HMC
Downloading these files to the HMC may take a while. If you want, you can start up an additional SSH session to the HMC (log in to the HMC in a separate window), and then run the following command to monitor the progress of the files download to file system /hmcdump:
After a while, the prompt will be returned.# monhmc -r disk 1
Then, set up the system for an altdisk boot:
Then, reboot the system to initiate the upgrade:# chhmc -c altdiskboot -s enable --mode upgrade
This upgrade may take a while, like 15 minutes or so, depending on the size of the upgrade and model of the HMC. You may set up a simple ping to the HMC, so you can monitor when it shows back up online after the upgrade:# hmcshutdown -r -t now
Once it start pinging again, you may start up a new SSH session to the HMC. Please note that even though you can log back in to the HMC, that the upgrade may not yet be entirely complete. Use the following command on the HMC to test if the upgrade is complete:# ping 172.16.52.100
If this command returns "A connection to the Command Server failed", then the upgrade is still not yet complete. Please wait a while before proceeding, and repeat the lshmc command after a few minutes again. Once the lshmc command properly outputs the version information, then you may proceed. For example:# lshmc -V
At this point, the upgrade to version V8 R8.7.0 is complete, and you can proceed with the next step: Updating the HMC to service Pack 1 (also known as MH01725).hscroot@hmc01:~> lshmc -V "version= Version: 8 Release: 8.7.0 Service Pack: 0 HMC Build level 1709071101 ","base_version=V8R8.7.0 "
This service pack can be downloaded from IBM Fix Central. On this site, search for your HMC model. For example, if you have a 7042 model HMC, type in the search window: "7042". Then select V8R8.7.0, and then download only MH01725. Do not download MH01704 (we already completed that step above). You'll have to download an update ISO image (for example: HMC_Update_V8R870_SP1_x86.iso), and 4 MH01725* files. Put these files in a separate folder on the AIX system, for example in /SP1.
On the HMC, run the following command to start the update:
# updhmc -t sftp -h 172.16.52.101 -u root --passwd 'P@ssw0rd' -f /SP1/HMC_Update_V8R870_SP1_x86.iso -rThis command will initiate the update, and the HMC will reboot by itself. This step may take another 15 minutes or so. You can check again, once the HMC is available after the reboot, that the update is complete, by running the "lshmc -V" command. The lshmc -V command should output that service pack 1 is installed.
For example:
At this point, both the upgrade and update are complete. You may want to log in to the GUI of the HMC using a web browser, and check for any alert messages, and close them out. Usually, the upgrade/update of an HMC may trigger a few alert messages, and there's no need for IBM to respond to them (if you're using the call-home feature of the HMC), as you already know that these messages occurred during the upgrade/update.hscroot@hmc01:~> lshmc -V "version= Version: 8 Release: 8.7.0 Service Pack: 1 HMC Build level 1712090351 MH01725 - HMC 870 Service Pack 1 Release [x86_64] ","base_version=V8R8.7.0 "
Please also note that with this release (V8 R8.7.0), there is no longer a classic interface, so the web-based GUI of the HMC may look somewhat different to you, if you're used to using the classic web-based GUI.
If using sftp isn't an option, for example because sftp isn't allowed or not available on any server, you can also first transfer the ISO image over to the HMC, and then run the update from the HMC itself.
This works as follows, assuming you want to update the HMC with fix MH01752:
First, download the ISO image from IBM fix central. You'll notice that for fix MH01752, the iso image has a filename called MH01752_x86.iso. Transfer this file over to the hardware management console - assuming here that your HMC is called "hmc01":
Now the iso image file is in the home directory of user hscroot on the HMC. If you log in through ssh to the HMC, and just do a "ls", you'll see the file right there.# scp MH01752_x86.sio hscroot@hmc01:~
Next, issue the upgrade from the HMC command line. Be sure to use the "-c" option as well, as that will tell the HMC to delete the iso image file once the update has been completed:
That's it - that will update the HMC using the local iso image file on the HMC itself.# updhmc -t disk -f /home/hscroot/MH01752_x86.sio -r -c
Configuring NTP on CentOS 6 (and similar versions) involves a number of steps - especially if you want to have it configured right and secure. Here's a quick guide how to do it:
First of all you have to determine the IP addresses of the NTP servers you are going to use. You may have to contact your network administrator to find out. Ensure that you get at least two time server IP addresses to use.
Then, install and verify the NTP packages:
Edit file /etc/ntp.conf and ensure that option "broadcastclient" is commented out (which it is by default with a new installation).# yum -y install ntp ntpdate # yum -q ntp ntpdate
Enable ntp and ntpdate at system boot time:
Ensure that file /etc/ntp/step-tickers is empty. This will make sure that if ntpdate is run, that it will use one of the time servers configured in /etc/ntp.conf.# chkconfig ntpd on # chkconfig ntpdate on
Add two time servers to /etc/ntp.conf, or use any of the pre-configured time servers in this file. Comment out the pre-configured servers, if you are using your own time servers.# cp /dev/null /etc/ntp/step-tickers
Do not copy the example above. Use the IP addresses for each time server that you've received from your network administrator instead.#server 0.centos.pool.ntp.org iburst #server 1.centos.pool.ntp.org iburst #server 2.centos.pool.ntp.org iburst #server 3.centos.pool.ntp.org iburst server 1.2.3.4 server 5.6.7.8
Enable NTP slewing (for slow time stepping if the time on the server is off, instead of suddenly making big time jump changes), by adding "-x" to OPTIONS in /etc/sysconfig/ntpd. Also add "SYNC_HWCLOCK=yes" in /etc/sysconfig/ntpdate to synchronize the hardware clock with any time changes.
Stop the NTP service, if it is running:
Start the ntpdate service (this will synchronize the system clock and the hardware clock):# service ntpd stop
Now, start the time service:# service ntpdate start
Wait a few minutes for the server to synchronize its time with the time servers. This may take anywhere between a few and 15 minutes. Then check the status of the time synchronization:# service ntpd start
The asterisk in front of the time server name in the "ntpq -p" output indicates that the client has reached time synchronization with that particular time server.# ntpq -p # ntpstat
Done!
Whenever you have to connect through SSH to a lot of different servers, and you create a command for it like this:
You may run into an error that stops your command, especially when a new server is added to $SERVER_LIST, like this:# for h in $SERVER_LIST; do ssh $h "uptime"; done
And you'll have to type "yes" every time this error is encountered.The authenticity of host 'myserver (1.2.3.4)' can't be established. RSA key fingerprint is ..... Are you sure you want to continue connecting (yes/no)?
So, how do you automate this, and not have to type "yes" with every new host?
The answer is to disable strict host key checking with the ssh command like this:
Please note that you should only do this with hosts that you're familiar with, and/or are in trusted networks, as it bypasses a security question.ssh -oStrictHostKeyChecking=no $h uptime
Red Hat Enterprise Linux 7 and similar Linux distrobutions have a new command to set the hostname of the system easily. The command is hostnamectl. For example, to set the hostname of a RHEL 7 system to "flores", run:
The hostnamectl command provides some other interesting features.# hostnamectl set-hostname flores
For example, it can be used to set the deployment type of the system, for example "development" or "production" or anything else you like to give it (as long as it's a single word. You can do so, for example by setting it to "production", by running:
Another option is to set the location of the system (and here you can use multiple words):# hostnamectl set-deployment production
To retrieve all this information, use hostnamectl as well to query the status:# hostnamectl set-location "third floor rack A12 U24"
root@(enemigo) selinux # hostnamectl status
Static hostname: flores
Icon name: computer-laptop
Chassis: laptop
Deployment: production
Location: third floor rack A12 U24
Machine ID: 4d8158f54d5166ff374bb372599351c4
Boot ID: ae8e7dccf14a492984fb5462c4da2aa2
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.2.2.el7.x86_64
Architecture: x86-64
Security Enhanced Linux, or short SELinux, is by default enabled on Red Hat Enterprise (and alike) Linux systems.
To determine the status of SELinux, simply run:
There will be times when it may be necessary to disable SELinux. Or for example, when a Linux system is not Internet facing, you may not need to have SELinux enabled.# sestatus
From the command line, you can edit the /etc/sysconfig/selinux file. This file is a symbolic link to file /etc/selinux/config.
By default, option SELINUX will be set to enforcing in this file:
By changing it to "permissive", you will disable SELinux:# This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing
SELINUX=permissive
A Red Hat Enterprise Linux system should have a single default gateway defined. However, sometimes, it does occur that a system has multiple default gateways. Here's information to detect multiple default gateways and how to get rid of them:
First, check the number of default gateways defined, by running the netstat command and looking for entries that start with 0.0.0.0:
In the example above, there are 2 default gateway entries, one to 192.168.0.1, and another one to 192.168.1.1.# netstat -nr | grep ^0.0.0.0 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 em1 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 em2
Quite often, more than 1 default gateways will be defined on a RHEL system, if there are multiple network interfaces present, and a GATEWAY entry is defined in each of the network interface files in /etc/sysconfig/network-script/ifcfg-*:
On a system with multiple network interfaces, it is best to define the default gateway in file /etc/sysconfig/network instead. This file is global network file. Put the following entries in this file, assuming your default gateway is 192.168.0.1 and the network interface to be used for the default gateway is em1:# grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-* ifcfg-em1:GATEWAY=192.168.0.1 ifcfg-em2:GATEWAY=192.168.1.1
Next, remove any GATEWAY entries in any of the ifcfg-* files in /etc/sysconfig/network-scripts.GATEWAY=192.168.0.1 GATEWAYDEV=em1
Finally, restart the network service:
This should resolve multiple default gateways, and the output of the netstat command should now only show one single entry with 0.0.0.0.# service network restart
Note: If the netstat command is not available on the system, you may also determine the number of default gateways, by running:
# ip route show | grep ^default
Topics: Networking, System Admin↑
Ping tricks
A few trick for the ping command to thoroughly test your network connectivity and check how much time a ping request takes:
Increase the interval of the ping requests from the default 1 second to, for example, 10 ping requests every second by using the -i option. As a test, to ping to 192.168.0.1, 10 times a second, run:
You can also go up to 1/100th of a second:# ping -i .1 192.168.0.1
To increase the default packet size of 64 bites, use -s option, for example to ping 1 KB with every ping request, run:# ping -i .01 192.168.0.1
Or combine the -i and -s options:# ping -s 1024 192.168.0.1
# ping -s 1024 -i .01 192.168.0.1
This is a quick NFS configuration using RHEL without too much concerts about security or any fine tuning and access control. In our scenario, there are two hosts:
- NFS Server, IP 10.1.1.100
- NFS Client, IP 10.1.1.101
On the NFS server, un the below commands to begin the NFS server installation:
Next, for this procedure, we export an arbitrary directory called /opt/nfs. Create /opt/nfs directory:[nfs-server] # yum install nfs-utils rpcbind
Edit the /etc/exports file (which is the NFS exports file) to add the below line to export folder /opt/nfs to client 10.1.1.101:[nfs-server] # mkdir -p /opt/nfs
Next, make sure to open port 2049 on your firewall to allow client requests:/opt/nfs 10.1.1.101(no_root_squash,rw)
Start the rpcbind and NFS server daemons in this order:[nfs-server] # firewall-cmd --zone=public --add-port=2049/tcp --permanent [nfs-server] # firewall-cmd --reload
Check the NFS server status:[nfs-server] # service rpcbind start; service nfs start
[nfs-server] # service nfs status
Redirecting to /bin/systemctl status nfs.service
nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled;
vendor preset: disabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
order-with-mounts.conf
Active: active (exited) since Tue 2017-11-14 09:06:21 CST; 1h 14min ago
Main PID: 2883 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/nfs-server.service
Next, export all the file systems configured in /etc/exports:
And check the currently exported file systems:[nfs-server] # exportfs -rav
Next, continue with the NFS client:[nfs-server] # exportfs -v
Install the required packages:
Create a mount point directory on the client, for example /mnt/nfs:[nfs-client] # yum install nfs-utils rpcbind [nfs-client]# service rpcbind start
Discover the NFS exported file systems:[nfs-client] # mkdir -p /mnt/nfs
Mount the previously NFS exported /opt/nfs directory:[nfs-client] # showmount -e 10.1.1.100 Export list for 10.1.1.100: /opt/nfs 10.1.1.101
Test the correctness of the setup between the NFS server and the NFS client by creating a file in the NFS mounted directory on the client side:[nfs-client] # mount 10.1.1.100:/opt/nfs /mnt/nfs
Move to the server side and check if the testfile file exists:[nfs-client] # cd /mnt/nfs/ [nfs-client] # touch testfile [nfs-client] # ls -l total 0 -rw-r--r--. 1 root root 0 Dec 11 08:13 testfile
At this point it is working, but it is not set up to remain there permanently (as in: it will be gone when either the NFS server or NFS client is rebooted. To ensure it remains working even after a reboot, perform the following steps:[nfs-server] # cd /opt/nfs/ [nfs-server] # ls -l total 0 -rw-r--r--. 1 root root 0 Dec 11 08:13 testfile
On the NFS server side, to have the NFS server service enabled at system boot time, run:
On the NFS server client side, add an entry to the /etc/fstab file, that will ensure the NFS file system is mounted at boot time:[nfs-server] # systemctl enable nfs-server
The options for the NFS file systems are as follows:10.1.1.100:/opt/nfs /mnt/nfs nfs4 soft,intr,nosuid 0 0
- soft = No hard mounting, avoids hanging file access commands on the NFS client, if the NFS servers is unavailable.
- intr = Allow NFS requests to be interrupted if the NFS server goes down or can't be reached.
- nosuid = This prevents remote users from gaining higher privileges by running a setuid program.
This will tell you the established connections for each of the clients, for example:[nfs-server] # netstat -an | grep 10.1.1.100:2049
In the example above you can see that IP address 10.1.1.101 on port 757 (NFS client) is connected to port 2049 on IP address 10.1.1.100 (NFS server).tcp 0 0 10.1.1.100:2049 10.1.1.101:757 ESTABLISHED
Topics: Red Hat / Linux, System Admin↑
Incrond
Incron is an interesting piece of software for Linux, that can monitor for file changes in a specific folder, and can act upon those file changes. For example, it's possible to wait for files to be written in a folder, and have a command run to process these files.
Incron is not installed by default and is part of the EPEL repository. For Red Hat and CentOS 7, it's also possible to just download the RPM package from https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/i/incron-0.5.12-11.el7.x86_64.rpm, for example using wget.
To install incron, run:
There are 4 files important for incron:# yum -y install /path/to/incron*rpm
- /etc/incron.conf - The main configuration file for incron, but this file can be left configured as default.
- /usr/sbin/incrond - This is the incron daemon that will have to run for incron to work. You can simply start it by executing this command, and it will automatically run in the background. When it's no longer needed, you can simply kill the process of /usr/sbin/incrond. However, its better to enable the service as system boot time and start the service:
# systemctl enable incrond.service # service incrond start
- /var/log/cron - This is the default location where the incron daemon will log its activities (through rsyslog). The file is also used by the cron daemon, so you may see other messages in this file. By using the tail command on this file, you can monitor what the incron daemon is doing. For example:
# tail -f /var/log/cron
- The incrontab file - You can edit this file by running:
This command will automatically load the incrontab file in an editor like VI, and you can add/modify/remove entries this way. Once you save the file, its contents will be automatically activated by the incron daemon. To list the entries in the incrontab file, run:# incrontab -e
# incrontab -l
[path] [mask] [command]
Where:
- [path] is the folder that the incron daemon will be monitoring for any new files (only in the folder itself, not in any sub-folders).
- [mask] is the activity that the incron daemon should respond to. There are several different available activities to choose from. For a list of options, see https://linux.die.net/man/5/incrontab. One option that can be used is "IN_CLOSE_WRITE", which means, act if a file is closed for writing, meaning, writing to a file in the folder has been completed.
- [command] is the command to be run by the incron daemon when a file activity takes place in the monitored path. For this command you can use available wildcards, such as:
- $@ : watched filesystem path
- $# : event-related file name
You can have multiple entries in the incrontab file, each on a separate line. In the example above, the incron daemon will start script /path/to/script.bash with two parameters (the path of the monitored folder, and the name of the file that was written to the folder), for each file that has been closed for writing in folder /path/to/my/folder./path/to/my/folder IN_CLOSE_WRITE /path/to/script.bash $@ $#
To monitor the status of the incron daemon, run:
To restart the incron daemon, run:# service incrond status
Or shorter:# service incrond stop # service incrond start
There is a downside to using incron, which is, that there is no way to limit the number of processes that can be started by the incron daemon. If a thousand files are written to the folder monitored by the incron daemon, then it will kick off the defined proces in the incrontab file for that folder a thousand times. This may place some serious CPU load on a system (or even hang up the system), especially if the command being run is CPU and/or memory intensive.# service incrond restart
Number of results found: 470.
Displaying results: 61 - 70.
Displaying results: 61 - 70.
