IBM General
-----------
1.Go to http://www.ibm.com. Click the Products tab, then click Systems > Power Systems (AIX, IBM i, Linux) in the context menu and explore this page. This page contains information about the current POWER processor-based models. A large number of links, useful documents, and detailed information are available. Depending on your interest, click the different links and explore the information that is available.
2.Go to http://www.ibm.com. Click the Support and downloads tab, follow the context menu, click Support by products, and then go to Servers & Systems. This will take you to the Support for IBM Systems page. From the Select product or service type for support list, select Power. Select a Power System model from the Hardware drop-down list, and click Go. Explore the page, follow the different links in the Configuration, Documentation, Diagnostic, and Problem resolution menus.
3.Use the following web address to access the IBM Systems Information Center: http://publib.boulder.ibm.com/eserver. This page is the entry point for hardware as well as software information. Click the IBM Systems Hardware Information Center link.
4.Go to the AIX Information Center. Use the following URL:
http://publib.boulder.ibm.com/eserver
Click AIX Information Center link and, if the next page provides choices, select the AIX 7.1 link.
When you are at the resulting IBM Systems Information Center page, select AIX PDFs. This topic contains links to PDF versions of the AIX documentation.
5.Go to the Power Systems Software website and see what is available: http://www.ibm.com/systems/power/software. Click the PowerVM - Virtualization without limits link. Navigate the page and the different tabs.
6.Find Linux information for POWER6 and POWER7 processor-based servers by visiting this site: http://www.ibm.com/systems/power/software/linux
----------------------------------------------------------------------------------------------------------------------------------------------------
Run level for the system is always read from /etc/inittab
----------------------------------------------------------
Syntax:
<ID>:<Run Level>:<Action>:<Command>
Run level is used for maintenance purpose.
In AIX, there are 10 run levels from 0 to 9.
0 and 1 is reserved run levels.
2 is default run level.
3 to 9 is user defined run level.
Eg:
Oracle:2/3/6/8:<once>/<wait>/<respawn>:startsrc –s oracle
Explanation:
Oracle is the ID,
2/3/6/8 is the run level for this particular process,
Once, wait, respawn is the action taken on this particular process where once means the process will run and stop, wait means will wait for the process to complete (if error occurs will stop, system needs to be rebooted then) & respawn means if the process does not exists, it will start it.
Startsrc –s oracled is the command given to start the subsystem oracle daemon.
To modify records of the /etc/inittab file, commands used are:
Mkitab : will add new record (entry) in /etc/inittab
To add an entry in /etc/inittab file,
# mkitab <rule>
For eg:
# mkitab “cdrom : 2 : respawn : startsrc –d cdromd”
Chitab : will change the existing records in /etc/inittab
To change an entry in /etc/inittab file,
# chitab <rule>
For eg:
# chitab “cdrom : 3 : respawn : startsrc –s cdromd”
Rmitab : will remove the records from /etc/inittab
To change an entry in /etc/inittab file,
# rmitab <rule>
For eg:
# rmitab “cdrom : 3 : respawn : startsrc –s cdromd”
Lsitab : will list the records available in /etc/inittab
To list entries in /etc/inittab file,
# lsitab <rule>
For eg:
# lsitab “cdrom : 3 : respawn : startsrc –s cdromd”
Here’s a question, check whether the records listed by lsitab command is same as # cat /etc/inittab?
To identify current run level, command used is:
# cat /etc/init.state
OR
# who –r
----------------------------------------------------------------------------------------------------------------------------------------------------
Boot and console messages can be used to identify and fix problems. These messages are automatically stored on disk by AIX. To view the stored messages, use the alog command.
-------------------------
For example,
To List the defined log types in AIX
alog -L # alog -L boot bosinst nim console cfg dumpsymp To View the boot log in AIX alog -o -t boot
To View the console log in AIX alog -o -t console
To find out the properties of boot log file in AIX
# alog -L -t boot file:size:verbosity /var/adm/ras/bootlog:131072:1
----------------------------------------------------------------------------------------------------------------------------------------------------
To View failed Login
---------------------
who -aH /etc/security/failedlogin | pg
To clear the password policy
----------------------------
#pwdadm -f NOCHECK vjayapal
----------------------------------------------------------------------------------------------------------------------------------------------------
To view large fie size through find command
------------------------------------------
find . -type f -size +1000000 -exec ls -al {} \;
Removing old files
------------------
find /prod/postoffice -type f -mtime +120 -name "*.retr" -depth -xdev -exec rm -ef {} \;
Run it as root
In the cron tab
---------------
03 17 * * 6 find /prod/postoffice -type f -mtime +120 -name "*.retr" -depth -xdev -exec rm -ef {} \\; >/stats/retr_cleanup.log 2>&1
To find the which process consuming more cpu.
----------------------------------------------
ps aux | head -1; ps aux | sort -rn +2 | head -20
To find the large file in the directory on aix
du -a /tmp | sort -n -r | head -n 10
find /tmp -size +2000 -exec ls -s {} \; | sort -nr | more
find /tmp -xdev -size +2048 -ls
find /tmp -xdev -mtime 0 -ls
----------------------------------------------------------------------------------------------------------------------------------------------------
Net Backup stop and start
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/netbackup/bin/goodies/netbackup start
To check backup process running ?
cd /usr/openv/netbackup/bin
./bpps -a -- > it is show you whether running or not.
Another method
>ps -ef | grep -i bpcd
hbo 23986252 4849818 0 12:35:25 - 0:00 /usr/openv/netbackup/bin/bpcd
root 55705830 39977140 0 12:36:07 pts/305 0:00 grep -i bpcd
/etc/rc.client.netbackup start
kill -9 `bpps -a | grep root | awk '{ print $2 }'`
To check any backup currently running
ps -ef | grep -i bpbkar
----------------------------------------------------------------------------------------------------------------------------------------------------
To Know the SP level in HACMP
------------------------------
> /usr/es/sbin/cluster/cspoc/cli_on_cluster -S halevel -s
rdcp01epica1: 7.1.1 SP6
rdcp01epica2: 7.1.1 SP5
root@rdcp01epica2 in /
Cluster freezing
------------------
Smitty cl_admin
select HACMP Resource Group and Application Management
select Suspend/Resume Application Monitoring
select Suspend Application Monitoring
select the Application server you want to suspend or freez.
It will disable the application monitoring and avoid the failover. Once you
are done with your oracle patching, you have to follow the same steps to
unfreez the application server.
Cluster filesystem increase.
----------------------------
root@chicondb1 in /usr/es/sbin/cluster/cspoc
> ./cli_chfs -a size=+100G /u01/app/oracle/data01
root@chicondb1 in /usr/es/sbin/cluster/cspoc
----------------------------------------------------------------------------------------------------------------------------------------------------
defunct process
---------------
Kill the defunct process in aix.
ps -ef|grep defunct|awk '{print $2}'|xargs -n 1 kill -9
ps -ef | grep "<defunct>" | grep -v grep | awk '{print $2}' | xargs kill -9
----------------------------------------------------------------------------------------------------------------------------------------------------
Fix the unixadm issue.(ssh password less authentication)
-----------------------------------------------------------
In destination server already unixadm account prsents ,then just reset the password for unixadm in the destination server and set the password is permanent.now try ssh.
If account is not there in destination server do the following
1.create the unixadm account in the destination server.
2.create a directory name as called as .ssh and inside of it create the file name called as authorized_keys
3.just copy the id_rsa.pub file(from /home/unixadm/.ssh/) and paste it in the destination server in the place of /home/unixadm/.ssh/
4.now give the command like this cat id_rsa.pub >> authorized_keys
5.Now remove the id_rsa.pub in the destination server.
Now checkit out from nim server it will works.
----------------------------------------------------------------------------------------------------------------------------------------------------
Nagios Administration
1.http://dmns04nag1.nitc.catholichealth.net/nagiosql/
Use the unixadm1 and unixadm123 the another user name password will be asked
user name : admin
Password :W1shB0ne
another account veena and the password is temp@123
2.Click supervision under that hosts will be there.By searching the hosts u can find out specific one.
Then select the specific host and through right side icons you can modify service and delete the hosts and copy the same to hosts like that.
copying is for safety purpose.
3.After all the changes done, go to tools under that click nagios control click the do it button one by one.
Write monitoing data --> after pressed do it.It should provide you completed.If you are facing any error you should not click next doit button.This is suitable for all the buttons.
Write additional data -- Doit
Check configuration files--Doit
Restart Nagios -- DOit
----------------------------------------------------------------------------------------------------------------------------------------------------
To check gpfs filesystems.
root@chiconapp1 in /
> mmlsnsd -m
Disk name NSD volume ID Device Node name Remarks
---------------------------------------------------------------------------------------
lawsond1 0AFABCC64AB38CC1 /dev/hdiskpower6 chicongpfs01
lawsond2 0AFABCC64AB38CC2 /dev/hdiskpower34 chicongpfs01
lawsond3 0AFABCC64AB38CC3 /dev/hdiskpower7 chicongpfs01
lawsonm1 0AFABCC64AB38CC4 /dev/hdiskpower35 chicongpfs01
lawsonm2 0AFABCC64AB3E028 /dev/hdiskpower58 chicongpfs01
----------------------------------------------------------------------------------------------------------------------------------------------------
Reduce filesystem in aix
-------------------------
> du -sm paranoid.log
84.11 paranoid.log
root@nitcife07 in /tmp
> cp -p paranoid.log paranoid.log.bkp
root@nitcife07 in /tmp
> cat paranoid.log | grep "/2009" | wc -l
410361
root@nitcife07 in /tmp
> tail +410361 paranoid.log.bkp > paranoid.log
root@nitcife07 in /tmp
> du -sm 410361
du: 410361: A file or directory in the path name does not exist.
root@nitcife07 in /tmp
> du -sm paranoid.log
48.32 paranoid.log
root@nitcife07 in /tmp
> cd /var/adm/syslog/proftpd
root@nitcife07 in /var/adm/syslog/proftpd
> ls -la
----------------------------------------------------------------------------------------------------------------------------------------------------
package installation through command line
1.inside of the folder just create the toc file using inutoc command the run the below command.it will install
install_all_updates -d . -Y
rdcp28kcisa1: root:/# instfix -ik IV37279
All filesets for IV37279 were found.
----------------------------------------------------------------------------------------------------------------------------------------------------
Nerworker backup tool start and stop service.
---------------------------------------------
To stop networker services: nsr_shutdown
If it is not stopping nsr_shutdown -f
To start networker services: /etc/rc.nsr
---------------------------------------------------------------------------------------------------------------------------------------------------
Device commands
----------------
lsattr -El ovpass0
lsattr -El fscsi3
lsdev -C -s fcp
lslpp -l | grep array
lslpp -l | grep -i fcp
lsdev -Cc tape
lsdev | grep -i VERITAS
lsdev -l ovpass0
lsdev -l ovpass0 -F parent
lsdev -l fscsi3 -F parent
fcstat -e fcs3
---------------------------------------------------------------------------------------------------------------------------------------------------
Changing the time zone on aix
root@10.250.134.124(/)$date
Tue Sep 16 12:44:33 PDT 2014
root@10.250.134.124(/)$grep -i tz /etc/environment
TZ=US/Pacific
root@10.250.134.124(/)$smit chtz_date
now select the americal/denver
root@10.250.134.124(/)$date
Tue Sep 16 12:44:33 PDT 2014
root@10.250.134.124(/)$grep -i tz /etc/environment
TZ=America/Denver
root@10.250.134.124(/)$export TZ=America/Denver
root@10.250.134.124(/)$date
Tue Sep 16 13:45:39 MDT 2014
root@10.250.134.124(/)$
----------------------------------------------------------------------------------------------------------------------------------------------------
To know the disk is not used by oracle
--------------------------------------
# extendvg -f oravg hdiskpower2
0516-1339 extendvg: Physical volume contains some 3rd party volume group.
0516-1397 extendvg: The physical volume hdisk20, will not be added to the volume group.
0516-792 extendvg: Unable to extend volume group.
Disks are showing used by the ASM.
----------------------------------
ORCLDISK means it is an ASM disk
DATA_000 is the ASM disk group name
> lquerypv -h /dev/hdiskpower2
00000000 00820101 00000000 80000000 FC40785C |.............@x\|
00000010 00000000 00000000 00000000 00000000 |................|
00000020 4F52434C 4449534B 00000000 00000000 |ORCLDISK........|
00000030 00000000 00000000 00000000 00000000 |................|
00000040 0B200000 00000103 44415441 5F303030 |. ......DATA_000|
00000050 30000000 00000000 00000000 00000000 |0...............|
00000060 00000000 00000000 44415441 00000000 |........DATA....|
00000070 00000000 00000000 00000000 00000000 |................|
00000080 00000000 00000000 44415441 5F303030 |........DATA_000|
00000090 30000000 00000000 00000000 00000000 |0...............|
000000A0 00000000 00000000 00000000 00000000 |................|
000000B0 00000000 00000000 00000000 00000000 |................|
000000C0 00000000 00000000 01F724AE 26BF6C00 |..........$.&.l.|
000000D0 01F724AE 31464800 02001000 00100000 |..$.1FH.........|
000000E0 0001BC80 00020000 00000003 00000001 |................|
000000F0 00000002 00000002 00000000 00000000 |................|
root@dmnt17hiea1 in /
Oracle RAC cluster
------------------
No need to set the pvid and vg, just shared disks
> /u01/app/11.2.0/grid/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE dmnp21i1a1
ora.FRA.dg ora....up.type ONLINE ONLINE dmnp21i1a1
ora....ER.lsnr ora....er.type ONLINE ONLINE dmnp21i1a1
ora....N1.lsnr ora....er.type ONLINE ONLINE dmnp22i1b1
ora....N2.lsnr ora....er.type ONLINE ONLINE dmnp22i1b1
ora....N3.lsnr ora....er.type ONLINE ONLINE dmnp22i1b1
ora.asm ora.asm.type ONLINE ONLINE dmnp21i1a1
ora.cvu ora.cvu.type ONLINE ONLINE dmnp21i1a1
ora....SM1.asm application ONLINE ONLINE dmnp21i1a1
ora....A1.lsnr application ONLINE ONLINE dmnp21i1a1
ora....1a1.gsd application OFFLINE OFFLINE
ora....1a1.ons application ONLINE OFFLINE
ora....1a1.vip ora....t1.type ONLINE ONLINE dmnp21i1a1
ora....SM2.asm application ONLINE ONLINE dmnp22i1b1
ora....B1.lsnr application ONLINE ONLINE dmnp22i1b1
ora....1b1.gsd application OFFLINE OFFLINE
ora....1b1.ons application ONLINE ONLINE dmnp22i1b1
ora....1b1.vip ora....t1.type ONLINE ONLINE dmnp22i1b1
ora.empiprd.db ora....se.type ONLINE ONLINE dmnp21i1a1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE dmnp21i1a1
ora.oc4j ora.oc4j.type ONLINE ONLINE dmnp21i1a1
ora.ons ora.ons.type ONLINE ONLINE dmnp22i1b1
ora....ry.acfs ora....fs.type ONLINE ONLINE dmnp21i1a1
ora.scan1.vip ora....ip.type ONLINE ONLINE dmnp22i1b1
ora.scan2.vip ora....ip.type ONLINE ONLINE dmnp22i1b1
ora.scan3.vip ora....ip.type ONLINE ONLINE dmnp22i1b1
root@dmnp22i1b1 in /usr/es/sbin/cluster
---------------------------------------------------------------------------------------------------------------------------------------------------
NTP
---
How to verify your ntp (Network Time Protocol) setup is working properly in Linux or UNIX Server?
# ntpq -c peers
remote refid st t when poll reach delay offset disp
==============================================================
*time1.domain.. time-a.nist.gov 1 u 38 64 377 1.16 -13.351 29.63
If you see a star (*) in the name of the time server, your time is being synchronised properly.
The third column, st is the stratum.
• If it is The lower the number, the closer you are to the time source.
• If the Stratum 16 means you are not synchronized.
Command Usage: ntpq -c peers
To synchronize the time manually,
# ntpdate -u <NTP_SERVER>
----------------------------------------------------------------------------------------------------------------------------------------------------
on aix
--------
lsslot -c pci
lscfg | grep disk
# lspath -Ht -l hdisk0
status name parent path_id
Enabled hdisk0 vscsi0 0
Enabled hdisk0 vscsi1 1
# chpath -l hdisk0 -p vscsi0 -a priority=2
path Changed
----------------------------------------------------------------------------------------------------------------------------------------------------
Ulimit change on aix
---------------------
ulimit -a < user id >
ulimit -aS <user id >
ulimit -aH <user id >
chuser nofiles=32768 nofiles_hard=65536 admin_user_ID
chuser fsize=-1 fsize_hard=-1 admin_user_ID
chuser data=-1 data_hard=-1 admin_user_ID
chuser stack=-1 stack_hard=-1 admin_user_ID
chuser rss=-1 rss_hard=-1 admin_user_ID
chuser core=-1 core_hard=-1 admin_user_ID
----------------------------------------------------------------------------------------------------------------------------------------------------
System dump
------------
A system dump is a snapshot of operating system state at the time of crash or a manually initiated dump.When a manually initiated or unexpected
system halt occurs the system dump facility automatically copies selected areas of kernel data to the primary(or secondary) dump device .
These areas include kernel memory as well as other areas registered in a stucture called the master dump table by kernel modules or kernel extensions.
It provides a mechanism to capture suffient information abt the AIX kernel to expert analysis.Once the preserved image is written to disk,
the system will be booted and returned to production.The dump is then typically submitted to IBM for analysis.
Types of dump
-------------
1.Traditional
2.Firmware assisted
3.Live dump facility
Traditional dump
----------------
AIX generates dump prior to halt .
Firmware assisted(From power6 or AIX 6.1 onwards)
-----------------
POWER firmware generates dump in parallel with AIX halt process.(full memory dump)
Live dump facility
------------------
Called by livedumpstart & dumpctrl
Selective dump of registered components without need for a system restart.
livedumpstart
-------------
If the individual compoents is having problem(such as hung),a livedumpstart command may be run to dump the needed diagnosic info.
dumpctrl
--------
The management of live dump(such as enabling a component or controlling the dump directory)is handled with the dumpctrl command.
-The raw dump can be formatted into readable output through the kdb command.
-sysdumpdev command
systemdump in an LPAR environment
---------------------------------
In an LPAR env, a dump can be initiated from the HMC.
888 - crass progress code of dump(HMC will not flash this code)
Primary dump device.
--------------------
If an AIX kernel crash occurs , kernel data is written to the primary dump deice, which is by default /dev/hd6, the primary paging device.
After the kernel crash, AIX may need to be rebooted.If the auto start system attribute is set to TRUE, the system will automatically reboot after a crash.
During the next reboot, the dump is copied into the dump directory.The default location is /var/adm/ras.The dump file name is vmcore.x where x indcates the number of the dump.
#sysdumpdev -l
#sysdumpdev -p /dev/sysdumpnull --> Deactivate primary dump.
#sysdumpdev -P -s /dev/rmt0 --> Change secondary dump device(permanent)
#sysdumpdev -L --> Displays the information abt the last dump(including the date and dump status).To need extra detail enter the below command
#sysdumpstart -p --> to get extra info from the above command.
Dump status
-----------
0 = 0c0 -- dump completed
-1 = 0c8 -- no primary dump device
-2 = 0c4 -- partial dump
-3 = 0c5 -- dump failed to start
Two types of dump
-----------------
Primary and secondary dump
Primary dump
------------
Usually used when you wish to save the dump data.
Secondary dump
--------------
An alternate dump device , often used to discard dump data ( using /dev/sysdumpnull)
dumpinfo
--------
Do not mirror the dump lv.Mirrorvg command will not mirror a dumplv in the rootvg unless it is the paging space.
System dumps are usually recorded in the error log with the DUMP_STATUS label.
The servers more than 4GB of real memory will have dedicatted dump device created at the installation time.
lg_dumplv is the default name of the dump device logical volume.
Creating dump device
--------------------
#mklv -y /dev/ded_dumplv -t sysdump rootvg 64
#sysdumpdev -P -p /dev/ded_dumplv
#sysdumpdev -l
#sysdumpdev -e --> Estimated dump size.
#sysdumpdev -C --> Turn on dump compression.
#sysdumpdev -c --> Turn off compression.
#sysdumpdev -K --> To make always allow dump as TRUE
#sysdumpdev -k --> To make always allow dump as FALSE
In AIX 6.1 and the later, dumps are always compressed.
/var filesystem will have dump info
Dumpcheck utility
-----------------
/usr/lib/ras/dumpcheck
#/usr/lib/ras/dumpcheck -p
#crontab -l ; grep dumpcheck
0 15 * * * /usr/lib/ras/dumpcheck > /dev/null 2>&1
#/usr/lib/ras/dumpcheck -t "0 14 * * *" --> to add the entry in crontab if already does not exists.
sysdumpstart command
--------------------
A system dump may be automatically created by the system.If we want we can do it by sysdumpstart command
smitty sysdumpstart
Specifying automatic reboot
---------------------------
#chdev -l sys0 -a autorestart=true
or
#smit chgsys
Dump restart
-------------
Before
dmnp32test1> sysdumpdev -L
0453-019 No previous dumps recorded.
Scanning device /dev/lg_dumplv for existing dump.
root@dmnp32test1 in /
While selecting restart for the lpar select dump then reboot it.
After dump reboot system will show like below.
root@dmnp32test1(/)#sysdumpdev -L
Device name: /dev/lg_dumplv
Major device number: 10
Minor device number: 11
Size: 189527552 bytes
Uncompressed Size: 969789042 bytes
Date/Time: Thu Dec 11 11:21:56 2014
Dump status: 0
Type of dump: traditional
dump completed successfully
root@dmnp32test1(/)#
After this we have to take snap and provide to IBM for the analysis.
----------------------------------------------------------------------------------------------------------------------------------------------------
To verify the mksysb file
-------------------------
> lsmksysb -l -f csclaw9mig_mksysb
VOLUME GROUP: rootvg
BACKUP DATE/TIME: Mon Nov 24 04:17:06 EST 2014
UNAME INFO: AIX csclaw9mig 1 6 00F6E8BD4C00
BACKUP OSLEVEL: 6.1.6.15
MAINTENANCE LEVEL: 6100-06
BACKUP SIZE (MB): 52736
SHRINK SIZE (MB): 29918
VG DATA ONLY: no
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 1 1 closed/syncd N/A
hd6 paging 64 64 1 open/syncd N/A
hd8 jfs2log 1 1 1 open/syncd N/A
hd4 jfs2 8 8 1 open/syncd /
hd2 jfs2 16 16 1 open/syncd /usr
hd9var jfs2 8 8 1 open/syncd /var
hd3 jfs2 8 8 1 open/syncd /tmp
hd1 jfs2 16 16 1 open/syncd /home
hd10opt jfs2 40 40 1 open/syncd /opt
hd11admin jfs2 4 4 1 open/syncd /admin
fwdump jfs2 8 8 1 open/syncd /var/adm/ras/platform
dr jfs2 92 92 1 closed/syncd /dr
livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump
fslv00 jfs2 4 4 1 open/syncd /roothome
dumplv1 sysdump 16 16 1 open/syncd N/A
dumplv2 sysdump 16 16 1 open/syncd N/A
root@dmns03nim1 in /nim/mksysb/dmnp32test1/dmnp32test1/csclaw9mig
>
Restoring specific frile from mksysb
-------------------------------------
specific file from the mksysb backup can be restored using the restorevgfiles command. In the following example the file will be restored to the current directory (/tmp/restore). Using the -d flag a alternative restore location can be specified.
Path to the mksysb image file
|
(/tmp/restore) # restorevgfiles -f /export2/P2_1202_TL7.mk ./root/j1
|
The file to be extracted from the mksysb image.
New volume on /export2/P2_1202_TL7.mk:
Cluster size is 51200 bytes (100 blocks).
The volume number is 1.
The backup date is: Tue Feb 21 18:09:12 GMT+01:00 2012
Files are backed up by name.
The user is root.
x 6 ./root/j1
The total size is 6 bytes.
The number of restored files is 1.
==================================================================
(/tmp/restore) # ls -la */*
-rw-r--r-- 1 root system 6 Feb 17 11:16 root/j1
----------------------------------------------------------------------------------------------------------------------------------------------------
Multibos note
--------------
Note : - Remember doing the Mutibos os updates always applied to the standby lvs.After took the mutibos just reboot the server to the bos_lvs and
install the updates to the original lvs(hd5,hd4,hd2,hd9var.hd10opt).After the update installation we can wake up the standby using the multibos -S
and then type the oslevel -s.Once if it is success reboot the server to teh original lv.
Notr :- multibos only for the os upgrade not for the migration.
Multibos boot device check
--------------------------
root@dmnp32test1(/)#bootinfo -v
bos_hd5
From here you have apply the os update files.So updates will reflect to the orininal lv.
----------------------------------------------------------------------------------------------------------------------------------------------------
To Check the specific package is available in the lpp_source or spot.Here we are checking for the alt_clone fileset.
#nim -o showres 'AIX61_TL6_Full_spot' | grep "bos.alt_disk_install.rte"
bos.alt_disk_install.rte 6.1.6.0 C F Alternate Disk Installation
If you do not see the Alternate Disk Installation fileset you will need to install it manually into your spot from the lpp_source. To do this run the following:
nim -o cust -a lpp_source=AIX61_TL6_Full_lpp -a filesets=bos.alt_disk_install AIX61_TL6_Full_spot
----------------------------------------------------------------------------------------------------------------------------------------------------
Network
-------
Resolve umserv01 to ip address (from /etc/hosts file):
#host umserv01
To change the host name to umserv01:
#hostname umserv01
To the status of ethernet device en0:
#entstat en0
To list the detailed status of device en0:
#entstat -d en0
To list all net configurable attributes and their values:
#no -a
To change umserv01wall parameter to its default value:
#no -d umserv01wall
To make the machine as router in tcpip networks:
#no -o ipforwarding=1
To trace the route to umserv01:
#traceroute umserv01
To tcp ping to the machine umserv01:
#ping umserv01
To show the status of all network interfaces:
#ifconfig -a
To show the status of en0:
#ifconfig en0
Turns on network card en0:
#ifconfig en0 up
Turns off network card en0:
#ifconfig en0 down
Removes en0 card from the network interface list:
#ifconfig en0 detach
Configure en0 starts immediately:
Temporarily:# ifconfig en0 inet 192.168.100.9 netmask 255.255.255.0 up
Permanently:# chdev -l en0 -a netaddr=192.168.100.10 -a netmask=0xffffff00
Create alias ip address for en0:
Temporarily:# ifconfig en0 alias 192.168.100.10 netmask 255.255.255.0
Permanently:# chdev -l en0 -a alias4=192.168.100.10,255.255.255.0
Remove a permanently added alias:
# chdev -l en0 -a delalias4=192.168.100.10,255.255.255.0
Via SMIT:
# smitty tcpip -> further Configuration -> Network Interfaces -> Network Interface Selection -> Configure Aliases
To make 192.168.100.1 as default gateway for entire network:
Temporarily:#route add 0 192.168.100.1
Permanently:#chdev -l inet0 -a route=0,192.168.100.1
To make 200.7 as gateway for 300.0 network:
#route add 192.100.300.0 192.100.200.7
To clear the gateway table:
#route -f
To change the host name to umserv01 permanently:
#chdev -l inet0 -a hostname=umserv01
To set the MTU to 1500 on en69:
#chdev -l en69 -a mtu=1500
To show the state of all sockets:
#netstat -a
To show the network buffers cache:
#netstat -c
To show the net drops of packets:
#netstat -D
To display interface statistics:
#netstat -i
To show the routing table:
#netstat -r
To show routing table (ip will be given instead of host names)
#netstat -rn
To show the statistics of the protocols
#netstat -s
To show the statistics of respective protocols
#netstat -s -p < tcp/udp/ipv6>
----------------------------------------------------------------------------------------------------------------------------------------------------
How to check the etherchannel works good ?
# entstat -d en7 | grep -i sync
Synchronization: IN_SYNC
Synchronization: IN_SYNC
Synchronization: IN_SYNC
Synchronization: IN_SYNC
# entstat -d en7 | grep -i agg
Device Type: IEEE 802.3ad Link Aggregation
Statistics for every adapter in the IEEE 802.3ad Link Aggregation:
IEEE 802.3ad Link Aggregation Statistics:
Aggregation status: Aggregated
Receive TCP Segment Aggregation: Enabled
TCP Packets Aggregated into Large Packets: 2968288
TCP Payload Bytes Aggregated into Large Packets: 4158020008
TCP Segment Aggregation Large Packets Created: 1367426
TCP Segment Aggregation Average Packets Aggregated: 2
TCP Segment Aggregation Maximum Packets Aggregated: 15
Aggregation: Aggregatable
Aggregation: Aggregatable
Receive TCP Segment Aggregation: Enabled
TCP Packets Aggregated into Large Packets: 1465534
TCP Payload Bytes Aggregated into Large Packets: 2129560180
TCP Segment Aggregation Large Packets Created: 116145
TCP Segment Aggregation Average Packets Aggregated: 12
TCP Segment Aggregation Maximum Packets Aggregated: 17
Aggregation: Aggregatable
Aggregation: Aggregatable
#
----------------------------------------------------------------------------------------------------------------------------------------------------
Howto configure AIX syslogd (/etc/syslog.conf)
----------------------------
The following steps will guide you trough configuring syslog on an AIX system.
This info is valid for AIX 4.3.3, AIX 5.1, AIX 5.2 and AIX 5.3.
By default, an AIX system will not do syslog'ging. For a strange reason, the default install on an AIX install will not place entry's in /etc/syslog.conf, leaving a totl userless syslogd.
Even more, comparing the syslogd on AIX to another Unix, (ex: Linux), you will notice that <facility>.* doesn't work. So make sure that your priority level is never *.
Comming from a Linux world, you most probably want the following config:
mail.debug /var/log/mail
user.debug /var/log/user
kern.debug /var/log/kern
syslog.debug /var/log/syslog
daemon.debug /var/log/daemon
auth.debug /var/log/secure
local2.debug /var/log/sudo
As AIX standard has no logrotate daemon running, you want to adapt syslogd todo the rotating on his own, an example would be:
mail.debug /var/log/mail rotate size 100k files 4 # 4 files, 100kB each
user.debug /var/log/user rotate files 12 time 1m # 12 files, monthly rotate
kern.debug /var/log/kern rotate files 12 time 1m compress # 12 files, monthly rotate, compress
...
When adaptions are made to /etc/syslog.conf, make sure to restart syslogd.
[root@sys /] refresh -s syslogd
0513-095 The request for subsystem refresh was completed successfully.
[root@sys /]
In the dmns03nim1 server inside of syslog.conf
------------------------
*.info @10.250.160.57
mail.info /var/adm/syslog/mail.log rotate size 5m compress archive /stats/syslog/archive
daemon.debug /var/adm/syslog/daemon.log rotate size 5m compress archive /stats/syslog/archive
auth.info /var/adm/syslog/security.log rotate size 5m compress archive /stats/syslog/archive
local2.debug /var/adm/syslog/sudo.log rotate size 5m compress archive /stats/syslog/archive
local6.debug /var/adm/syslog/ldap/ldap.log rotate size 100m compress
*.info;mail.none,daemon.none,auth.none,local2.none,local4.none /var/adm/syslog/syslog.log rotate size 5m compress archive /stats/syslog/archive
# ASO log configuration
aso.notice /var/log/aso/aso.log rotate size 1m files 8 compress
aso.info /var/log/aso/aso_process.log rotate size 1m files 8 compress
aso.debug /var/log/aso/aso_debug.log rotate size 32m files 8 compress
auth.info @10.250.160.57
----------------------------------------------------------------------------------------------------------------------------------------------------
AIX temperature sensors
By executing the command /usr/lpp/diagnostics/bin/uesensor, you can, on supported platform, get the fan speeds and temperature of your system:
[root@sys /] /usr/lpp/diagnostics/bin/uesensor -a
3 0 11 31 P1
9001 0 11 2100 F1
9001 1 11 2760 F2
9001 2 11 1890 F3
9001 3 11 1890 F4
9002 0 11 5129 P1
9002 1 11 3129 P1
9002 2 11 5129 P1
9002 3 11 12077 P1
9004 0 11 3 P3-V1
9004 1 11 3 P3-V2
9004 2 11 3 P3-V3
[root@sys /]
----------------------------------------------------------------------------------------------------------------------------------------------------
How to change hostname on AIX, “Cleanly”
Changing the hostname at a Windows Server will require a reboot. However, changing a hostname on a Unix/Linux server do not require a reboot! But, it is not so straightforward to change the hostname of a server online. This is because you will need to change a couple of things, like the entry in ODM, nodename and current hostname.
oldhostname:/:>chdev -l inet0 -a hostname=newhostname
inet0 changed
oldhostname:/:>uname -S newhostname => Change the network name of this machine.
oldhostname:/:>hostname newhostname => Sets the name of the current host system.
newhostname
oldhostname:/:>su -
newhostname:/:>exit
----------------------------------------------------------------------------------------------------------------------------------------------------
Difference between sudo su and sudo su -
sudo su --> if you do it will call the present user profile and run the root commands.so it will not have all the rights.
sudo su - --> if you do it will call the root user profile so ow you have full rights as a real root account.
----------------------------------------------------------------------------------------------------------------------------------------------------
Find the child devices for the particular device.
lsdev -Cc adapter | grep fcs0
fcs0 Available 08-00 4Gb FC PCI Express Adapter (df1000fe)
lsdev -C | grep -i 08-00
----------------------------------------------------------------------------------------------------------------------------------------------------
-----------
1.Go to http://www.ibm.com. Click the Products tab, then click Systems > Power Systems (AIX, IBM i, Linux) in the context menu and explore this page. This page contains information about the current POWER processor-based models. A large number of links, useful documents, and detailed information are available. Depending on your interest, click the different links and explore the information that is available.
2.Go to http://www.ibm.com. Click the Support and downloads tab, follow the context menu, click Support by products, and then go to Servers & Systems. This will take you to the Support for IBM Systems page. From the Select product or service type for support list, select Power. Select a Power System model from the Hardware drop-down list, and click Go. Explore the page, follow the different links in the Configuration, Documentation, Diagnostic, and Problem resolution menus.
3.Use the following web address to access the IBM Systems Information Center: http://publib.boulder.ibm.com/eserver. This page is the entry point for hardware as well as software information. Click the IBM Systems Hardware Information Center link.
4.Go to the AIX Information Center. Use the following URL:
http://publib.boulder.ibm.com/eserver
Click AIX Information Center link and, if the next page provides choices, select the AIX 7.1 link.
When you are at the resulting IBM Systems Information Center page, select AIX PDFs. This topic contains links to PDF versions of the AIX documentation.
5.Go to the Power Systems Software website and see what is available: http://www.ibm.com/systems/power/software. Click the PowerVM - Virtualization without limits link. Navigate the page and the different tabs.
6.Find Linux information for POWER6 and POWER7 processor-based servers by visiting this site: http://www.ibm.com/systems/power/software/linux
----------------------------------------------------------------------------------------------------------------------------------------------------
Run level for the system is always read from /etc/inittab
----------------------------------------------------------
Syntax:
<ID>:<Run Level>:<Action>:<Command>
Run level is used for maintenance purpose.
In AIX, there are 10 run levels from 0 to 9.
0 and 1 is reserved run levels.
2 is default run level.
3 to 9 is user defined run level.
Eg:
Oracle:2/3/6/8:<once>/<wait>/<respawn>:startsrc –s oracle
Explanation:
Oracle is the ID,
2/3/6/8 is the run level for this particular process,
Once, wait, respawn is the action taken on this particular process where once means the process will run and stop, wait means will wait for the process to complete (if error occurs will stop, system needs to be rebooted then) & respawn means if the process does not exists, it will start it.
Startsrc –s oracled is the command given to start the subsystem oracle daemon.
To modify records of the /etc/inittab file, commands used are:
Mkitab : will add new record (entry) in /etc/inittab
To add an entry in /etc/inittab file,
# mkitab <rule>
For eg:
# mkitab “cdrom : 2 : respawn : startsrc –d cdromd”
Chitab : will change the existing records in /etc/inittab
To change an entry in /etc/inittab file,
# chitab <rule>
For eg:
# chitab “cdrom : 3 : respawn : startsrc –s cdromd”
Rmitab : will remove the records from /etc/inittab
To change an entry in /etc/inittab file,
# rmitab <rule>
For eg:
# rmitab “cdrom : 3 : respawn : startsrc –s cdromd”
Lsitab : will list the records available in /etc/inittab
To list entries in /etc/inittab file,
# lsitab <rule>
For eg:
# lsitab “cdrom : 3 : respawn : startsrc –s cdromd”
Here’s a question, check whether the records listed by lsitab command is same as # cat /etc/inittab?
To identify current run level, command used is:
# cat /etc/init.state
OR
# who –r
----------------------------------------------------------------------------------------------------------------------------------------------------
Boot and console messages can be used to identify and fix problems. These messages are automatically stored on disk by AIX. To view the stored messages, use the alog command.
-------------------------
For example,
To List the defined log types in AIX
alog -L # alog -L boot bosinst nim console cfg dumpsymp To View the boot log in AIX alog -o -t boot
To View the console log in AIX alog -o -t console
To find out the properties of boot log file in AIX
# alog -L -t boot file:size:verbosity /var/adm/ras/bootlog:131072:1
----------------------------------------------------------------------------------------------------------------------------------------------------
To View failed Login
---------------------
who -aH /etc/security/failedlogin | pg
To clear the password policy
----------------------------
#pwdadm -f NOCHECK vjayapal
----------------------------------------------------------------------------------------------------------------------------------------------------
To view large fie size through find command
------------------------------------------
find . -type f -size +1000000 -exec ls -al {} \;
Removing old files
------------------
find /prod/postoffice -type f -mtime +120 -name "*.retr" -depth -xdev -exec rm -ef {} \;
Run it as root
In the cron tab
---------------
03 17 * * 6 find /prod/postoffice -type f -mtime +120 -name "*.retr" -depth -xdev -exec rm -ef {} \\; >/stats/retr_cleanup.log 2>&1
To find the which process consuming more cpu.
----------------------------------------------
ps aux | head -1; ps aux | sort -rn +2 | head -20
To find the large file in the directory on aix
du -a /tmp | sort -n -r | head -n 10
find /tmp -size +2000 -exec ls -s {} \; | sort -nr | more
find /tmp -xdev -size +2048 -ls
find /tmp -xdev -mtime 0 -ls
----------------------------------------------------------------------------------------------------------------------------------------------------
Net Backup stop and start
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/netbackup/bin/goodies/netbackup start
To check backup process running ?
cd /usr/openv/netbackup/bin
./bpps -a -- > it is show you whether running or not.
Another method
>ps -ef | grep -i bpcd
hbo 23986252 4849818 0 12:35:25 - 0:00 /usr/openv/netbackup/bin/bpcd
root 55705830 39977140 0 12:36:07 pts/305 0:00 grep -i bpcd
/etc/rc.client.netbackup start
kill -9 `bpps -a | grep root | awk '{ print $2 }'`
To check any backup currently running
ps -ef | grep -i bpbkar
----------------------------------------------------------------------------------------------------------------------------------------------------
To Know the SP level in HACMP
------------------------------
> /usr/es/sbin/cluster/cspoc/cli_on_cluster -S halevel -s
rdcp01epica1: 7.1.1 SP6
rdcp01epica2: 7.1.1 SP5
root@rdcp01epica2 in /
Cluster freezing
------------------
Smitty cl_admin
select HACMP Resource Group and Application Management
select Suspend/Resume Application Monitoring
select Suspend Application Monitoring
select the Application server you want to suspend or freez.
It will disable the application monitoring and avoid the failover. Once you
are done with your oracle patching, you have to follow the same steps to
unfreez the application server.
Cluster filesystem increase.
----------------------------
root@chicondb1 in /usr/es/sbin/cluster/cspoc
> ./cli_chfs -a size=+100G /u01/app/oracle/data01
root@chicondb1 in /usr/es/sbin/cluster/cspoc
----------------------------------------------------------------------------------------------------------------------------------------------------
defunct process
---------------
Kill the defunct process in aix.
ps -ef|grep defunct|awk '{print $2}'|xargs -n 1 kill -9
ps -ef | grep "<defunct>" | grep -v grep | awk '{print $2}' | xargs kill -9
----------------------------------------------------------------------------------------------------------------------------------------------------
Fix the unixadm issue.(ssh password less authentication)
-----------------------------------------------------------
In destination server already unixadm account prsents ,then just reset the password for unixadm in the destination server and set the password is permanent.now try ssh.
If account is not there in destination server do the following
1.create the unixadm account in the destination server.
2.create a directory name as called as .ssh and inside of it create the file name called as authorized_keys
3.just copy the id_rsa.pub file(from /home/unixadm/.ssh/) and paste it in the destination server in the place of /home/unixadm/.ssh/
4.now give the command like this cat id_rsa.pub >> authorized_keys
5.Now remove the id_rsa.pub in the destination server.
Now checkit out from nim server it will works.
----------------------------------------------------------------------------------------------------------------------------------------------------
Nagios Administration
1.http://dmns04nag1.nitc.catholichealth.net/nagiosql/
Use the unixadm1 and unixadm123 the another user name password will be asked
user name : admin
Password :W1shB0ne
another account veena and the password is temp@123
2.Click supervision under that hosts will be there.By searching the hosts u can find out specific one.
Then select the specific host and through right side icons you can modify service and delete the hosts and copy the same to hosts like that.
copying is for safety purpose.
3.After all the changes done, go to tools under that click nagios control click the do it button one by one.
Write monitoing data --> after pressed do it.It should provide you completed.If you are facing any error you should not click next doit button.This is suitable for all the buttons.
Write additional data -- Doit
Check configuration files--Doit
Restart Nagios -- DOit
----------------------------------------------------------------------------------------------------------------------------------------------------
To check gpfs filesystems.
root@chiconapp1 in /
> mmlsnsd -m
Disk name NSD volume ID Device Node name Remarks
---------------------------------------------------------------------------------------
lawsond1 0AFABCC64AB38CC1 /dev/hdiskpower6 chicongpfs01
lawsond2 0AFABCC64AB38CC2 /dev/hdiskpower34 chicongpfs01
lawsond3 0AFABCC64AB38CC3 /dev/hdiskpower7 chicongpfs01
lawsonm1 0AFABCC64AB38CC4 /dev/hdiskpower35 chicongpfs01
lawsonm2 0AFABCC64AB3E028 /dev/hdiskpower58 chicongpfs01
----------------------------------------------------------------------------------------------------------------------------------------------------
Reduce filesystem in aix
-------------------------
> du -sm paranoid.log
84.11 paranoid.log
root@nitcife07 in /tmp
> cp -p paranoid.log paranoid.log.bkp
root@nitcife07 in /tmp
> cat paranoid.log | grep "/2009" | wc -l
410361
root@nitcife07 in /tmp
> tail +410361 paranoid.log.bkp > paranoid.log
root@nitcife07 in /tmp
> du -sm 410361
du: 410361: A file or directory in the path name does not exist.
root@nitcife07 in /tmp
> du -sm paranoid.log
48.32 paranoid.log
root@nitcife07 in /tmp
> cd /var/adm/syslog/proftpd
root@nitcife07 in /var/adm/syslog/proftpd
> ls -la
----------------------------------------------------------------------------------------------------------------------------------------------------
package installation through command line
1.inside of the folder just create the toc file using inutoc command the run the below command.it will install
install_all_updates -d . -Y
rdcp28kcisa1: root:/# instfix -ik IV37279
All filesets for IV37279 were found.
----------------------------------------------------------------------------------------------------------------------------------------------------
Nerworker backup tool start and stop service.
---------------------------------------------
To stop networker services: nsr_shutdown
If it is not stopping nsr_shutdown -f
To start networker services: /etc/rc.nsr
---------------------------------------------------------------------------------------------------------------------------------------------------
Device commands
----------------
lsattr -El ovpass0
lsattr -El fscsi3
lsdev -C -s fcp
lslpp -l | grep array
lslpp -l | grep -i fcp
lsdev -Cc tape
lsdev | grep -i VERITAS
lsdev -l ovpass0
lsdev -l ovpass0 -F parent
lsdev -l fscsi3 -F parent
fcstat -e fcs3
---------------------------------------------------------------------------------------------------------------------------------------------------
Changing the time zone on aix
root@10.250.134.124(/)$date
Tue Sep 16 12:44:33 PDT 2014
root@10.250.134.124(/)$grep -i tz /etc/environment
TZ=US/Pacific
root@10.250.134.124(/)$smit chtz_date
now select the americal/denver
root@10.250.134.124(/)$date
Tue Sep 16 12:44:33 PDT 2014
root@10.250.134.124(/)$grep -i tz /etc/environment
TZ=America/Denver
root@10.250.134.124(/)$export TZ=America/Denver
root@10.250.134.124(/)$date
Tue Sep 16 13:45:39 MDT 2014
root@10.250.134.124(/)$
----------------------------------------------------------------------------------------------------------------------------------------------------
To know the disk is not used by oracle
--------------------------------------
# extendvg -f oravg hdiskpower2
0516-1339 extendvg: Physical volume contains some 3rd party volume group.
0516-1397 extendvg: The physical volume hdisk20, will not be added to the volume group.
0516-792 extendvg: Unable to extend volume group.
Disks are showing used by the ASM.
----------------------------------
ORCLDISK means it is an ASM disk
DATA_000 is the ASM disk group name
> lquerypv -h /dev/hdiskpower2
00000000 00820101 00000000 80000000 FC40785C |.............@x\|
00000010 00000000 00000000 00000000 00000000 |................|
00000020 4F52434C 4449534B 00000000 00000000 |ORCLDISK........|
00000030 00000000 00000000 00000000 00000000 |................|
00000040 0B200000 00000103 44415441 5F303030 |. ......DATA_000|
00000050 30000000 00000000 00000000 00000000 |0...............|
00000060 00000000 00000000 44415441 00000000 |........DATA....|
00000070 00000000 00000000 00000000 00000000 |................|
00000080 00000000 00000000 44415441 5F303030 |........DATA_000|
00000090 30000000 00000000 00000000 00000000 |0...............|
000000A0 00000000 00000000 00000000 00000000 |................|
000000B0 00000000 00000000 00000000 00000000 |................|
000000C0 00000000 00000000 01F724AE 26BF6C00 |..........$.&.l.|
000000D0 01F724AE 31464800 02001000 00100000 |..$.1FH.........|
000000E0 0001BC80 00020000 00000003 00000001 |................|
000000F0 00000002 00000002 00000000 00000000 |................|
root@dmnt17hiea1 in /
Oracle RAC cluster
------------------
No need to set the pvid and vg, just shared disks
> /u01/app/11.2.0/grid/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE dmnp21i1a1
ora.FRA.dg ora....up.type ONLINE ONLINE dmnp21i1a1
ora....ER.lsnr ora....er.type ONLINE ONLINE dmnp21i1a1
ora....N1.lsnr ora....er.type ONLINE ONLINE dmnp22i1b1
ora....N2.lsnr ora....er.type ONLINE ONLINE dmnp22i1b1
ora....N3.lsnr ora....er.type ONLINE ONLINE dmnp22i1b1
ora.asm ora.asm.type ONLINE ONLINE dmnp21i1a1
ora.cvu ora.cvu.type ONLINE ONLINE dmnp21i1a1
ora....SM1.asm application ONLINE ONLINE dmnp21i1a1
ora....A1.lsnr application ONLINE ONLINE dmnp21i1a1
ora....1a1.gsd application OFFLINE OFFLINE
ora....1a1.ons application ONLINE OFFLINE
ora....1a1.vip ora....t1.type ONLINE ONLINE dmnp21i1a1
ora....SM2.asm application ONLINE ONLINE dmnp22i1b1
ora....B1.lsnr application ONLINE ONLINE dmnp22i1b1
ora....1b1.gsd application OFFLINE OFFLINE
ora....1b1.ons application ONLINE ONLINE dmnp22i1b1
ora....1b1.vip ora....t1.type ONLINE ONLINE dmnp22i1b1
ora.empiprd.db ora....se.type ONLINE ONLINE dmnp21i1a1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE dmnp21i1a1
ora.oc4j ora.oc4j.type ONLINE ONLINE dmnp21i1a1
ora.ons ora.ons.type ONLINE ONLINE dmnp22i1b1
ora....ry.acfs ora....fs.type ONLINE ONLINE dmnp21i1a1
ora.scan1.vip ora....ip.type ONLINE ONLINE dmnp22i1b1
ora.scan2.vip ora....ip.type ONLINE ONLINE dmnp22i1b1
ora.scan3.vip ora....ip.type ONLINE ONLINE dmnp22i1b1
root@dmnp22i1b1 in /usr/es/sbin/cluster
---------------------------------------------------------------------------------------------------------------------------------------------------
NTP
---
How to verify your ntp (Network Time Protocol) setup is working properly in Linux or UNIX Server?
# ntpq -c peers
remote refid st t when poll reach delay offset disp
==============================================================
*time1.domain.. time-a.nist.gov 1 u 38 64 377 1.16 -13.351 29.63
If you see a star (*) in the name of the time server, your time is being synchronised properly.
The third column, st is the stratum.
• If it is The lower the number, the closer you are to the time source.
• If the Stratum 16 means you are not synchronized.
Command Usage: ntpq -c peers
To synchronize the time manually,
# ntpdate -u <NTP_SERVER>
----------------------------------------------------------------------------------------------------------------------------------------------------
on aix
--------
lsslot -c pci
lscfg | grep disk
# lspath -Ht -l hdisk0
status name parent path_id
Enabled hdisk0 vscsi0 0
Enabled hdisk0 vscsi1 1
# chpath -l hdisk0 -p vscsi0 -a priority=2
path Changed
----------------------------------------------------------------------------------------------------------------------------------------------------
Ulimit change on aix
---------------------
ulimit -a < user id >
ulimit -aS <user id >
ulimit -aH <user id >
chuser nofiles=32768 nofiles_hard=65536 admin_user_ID
chuser fsize=-1 fsize_hard=-1 admin_user_ID
chuser data=-1 data_hard=-1 admin_user_ID
chuser stack=-1 stack_hard=-1 admin_user_ID
chuser rss=-1 rss_hard=-1 admin_user_ID
chuser core=-1 core_hard=-1 admin_user_ID
----------------------------------------------------------------------------------------------------------------------------------------------------
System dump
------------
A system dump is a snapshot of operating system state at the time of crash or a manually initiated dump.When a manually initiated or unexpected
system halt occurs the system dump facility automatically copies selected areas of kernel data to the primary(or secondary) dump device .
These areas include kernel memory as well as other areas registered in a stucture called the master dump table by kernel modules or kernel extensions.
It provides a mechanism to capture suffient information abt the AIX kernel to expert analysis.Once the preserved image is written to disk,
the system will be booted and returned to production.The dump is then typically submitted to IBM for analysis.
Types of dump
-------------
1.Traditional
2.Firmware assisted
3.Live dump facility
Traditional dump
----------------
AIX generates dump prior to halt .
Firmware assisted(From power6 or AIX 6.1 onwards)
-----------------
POWER firmware generates dump in parallel with AIX halt process.(full memory dump)
Live dump facility
------------------
Called by livedumpstart & dumpctrl
Selective dump of registered components without need for a system restart.
livedumpstart
-------------
If the individual compoents is having problem(such as hung),a livedumpstart command may be run to dump the needed diagnosic info.
dumpctrl
--------
The management of live dump(such as enabling a component or controlling the dump directory)is handled with the dumpctrl command.
-The raw dump can be formatted into readable output through the kdb command.
-sysdumpdev command
systemdump in an LPAR environment
---------------------------------
In an LPAR env, a dump can be initiated from the HMC.
888 - crass progress code of dump(HMC will not flash this code)
Primary dump device.
--------------------
If an AIX kernel crash occurs , kernel data is written to the primary dump deice, which is by default /dev/hd6, the primary paging device.
After the kernel crash, AIX may need to be rebooted.If the auto start system attribute is set to TRUE, the system will automatically reboot after a crash.
During the next reboot, the dump is copied into the dump directory.The default location is /var/adm/ras.The dump file name is vmcore.x where x indcates the number of the dump.
#sysdumpdev -l
#sysdumpdev -p /dev/sysdumpnull --> Deactivate primary dump.
#sysdumpdev -P -s /dev/rmt0 --> Change secondary dump device(permanent)
#sysdumpdev -L --> Displays the information abt the last dump(including the date and dump status).To need extra detail enter the below command
#sysdumpstart -p --> to get extra info from the above command.
Dump status
-----------
0 = 0c0 -- dump completed
-1 = 0c8 -- no primary dump device
-2 = 0c4 -- partial dump
-3 = 0c5 -- dump failed to start
Two types of dump
-----------------
Primary and secondary dump
Primary dump
------------
Usually used when you wish to save the dump data.
Secondary dump
--------------
An alternate dump device , often used to discard dump data ( using /dev/sysdumpnull)
dumpinfo
--------
Do not mirror the dump lv.Mirrorvg command will not mirror a dumplv in the rootvg unless it is the paging space.
System dumps are usually recorded in the error log with the DUMP_STATUS label.
The servers more than 4GB of real memory will have dedicatted dump device created at the installation time.
lg_dumplv is the default name of the dump device logical volume.
Creating dump device
--------------------
#mklv -y /dev/ded_dumplv -t sysdump rootvg 64
#sysdumpdev -P -p /dev/ded_dumplv
#sysdumpdev -l
#sysdumpdev -e --> Estimated dump size.
#sysdumpdev -C --> Turn on dump compression.
#sysdumpdev -c --> Turn off compression.
#sysdumpdev -K --> To make always allow dump as TRUE
#sysdumpdev -k --> To make always allow dump as FALSE
In AIX 6.1 and the later, dumps are always compressed.
/var filesystem will have dump info
Dumpcheck utility
-----------------
/usr/lib/ras/dumpcheck
#/usr/lib/ras/dumpcheck -p
#crontab -l ; grep dumpcheck
0 15 * * * /usr/lib/ras/dumpcheck > /dev/null 2>&1
#/usr/lib/ras/dumpcheck -t "0 14 * * *" --> to add the entry in crontab if already does not exists.
sysdumpstart command
--------------------
A system dump may be automatically created by the system.If we want we can do it by sysdumpstart command
smitty sysdumpstart
Specifying automatic reboot
---------------------------
#chdev -l sys0 -a autorestart=true
or
#smit chgsys
Dump restart
-------------
Before
dmnp32test1> sysdumpdev -L
0453-019 No previous dumps recorded.
Scanning device /dev/lg_dumplv for existing dump.
root@dmnp32test1 in /
While selecting restart for the lpar select dump then reboot it.
After dump reboot system will show like below.
root@dmnp32test1(/)#sysdumpdev -L
Device name: /dev/lg_dumplv
Major device number: 10
Minor device number: 11
Size: 189527552 bytes
Uncompressed Size: 969789042 bytes
Date/Time: Thu Dec 11 11:21:56 2014
Dump status: 0
Type of dump: traditional
dump completed successfully
root@dmnp32test1(/)#
After this we have to take snap and provide to IBM for the analysis.
----------------------------------------------------------------------------------------------------------------------------------------------------
To verify the mksysb file
-------------------------
> lsmksysb -l -f csclaw9mig_mksysb
VOLUME GROUP: rootvg
BACKUP DATE/TIME: Mon Nov 24 04:17:06 EST 2014
UNAME INFO: AIX csclaw9mig 1 6 00F6E8BD4C00
BACKUP OSLEVEL: 6.1.6.15
MAINTENANCE LEVEL: 6100-06
BACKUP SIZE (MB): 52736
SHRINK SIZE (MB): 29918
VG DATA ONLY: no
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 1 1 closed/syncd N/A
hd6 paging 64 64 1 open/syncd N/A
hd8 jfs2log 1 1 1 open/syncd N/A
hd4 jfs2 8 8 1 open/syncd /
hd2 jfs2 16 16 1 open/syncd /usr
hd9var jfs2 8 8 1 open/syncd /var
hd3 jfs2 8 8 1 open/syncd /tmp
hd1 jfs2 16 16 1 open/syncd /home
hd10opt jfs2 40 40 1 open/syncd /opt
hd11admin jfs2 4 4 1 open/syncd /admin
fwdump jfs2 8 8 1 open/syncd /var/adm/ras/platform
dr jfs2 92 92 1 closed/syncd /dr
livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump
fslv00 jfs2 4 4 1 open/syncd /roothome
dumplv1 sysdump 16 16 1 open/syncd N/A
dumplv2 sysdump 16 16 1 open/syncd N/A
root@dmns03nim1 in /nim/mksysb/dmnp32test1/dmnp32test1/csclaw9mig
>
Restoring specific frile from mksysb
-------------------------------------
specific file from the mksysb backup can be restored using the restorevgfiles command. In the following example the file will be restored to the current directory (/tmp/restore). Using the -d flag a alternative restore location can be specified.
Path to the mksysb image file
|
(/tmp/restore) # restorevgfiles -f /export2/P2_1202_TL7.mk ./root/j1
|
The file to be extracted from the mksysb image.
New volume on /export2/P2_1202_TL7.mk:
Cluster size is 51200 bytes (100 blocks).
The volume number is 1.
The backup date is: Tue Feb 21 18:09:12 GMT+01:00 2012
Files are backed up by name.
The user is root.
x 6 ./root/j1
The total size is 6 bytes.
The number of restored files is 1.
==================================================================
(/tmp/restore) # ls -la */*
-rw-r--r-- 1 root system 6 Feb 17 11:16 root/j1
----------------------------------------------------------------------------------------------------------------------------------------------------
Multibos note
--------------
Note : - Remember doing the Mutibos os updates always applied to the standby lvs.After took the mutibos just reboot the server to the bos_lvs and
install the updates to the original lvs(hd5,hd4,hd2,hd9var.hd10opt).After the update installation we can wake up the standby using the multibos -S
and then type the oslevel -s.Once if it is success reboot the server to teh original lv.
Notr :- multibos only for the os upgrade not for the migration.
Multibos boot device check
--------------------------
root@dmnp32test1(/)#bootinfo -v
bos_hd5
From here you have apply the os update files.So updates will reflect to the orininal lv.
----------------------------------------------------------------------------------------------------------------------------------------------------
To Check the specific package is available in the lpp_source or spot.Here we are checking for the alt_clone fileset.
#nim -o showres 'AIX61_TL6_Full_spot' | grep "bos.alt_disk_install.rte"
bos.alt_disk_install.rte 6.1.6.0 C F Alternate Disk Installation
If you do not see the Alternate Disk Installation fileset you will need to install it manually into your spot from the lpp_source. To do this run the following:
nim -o cust -a lpp_source=AIX61_TL6_Full_lpp -a filesets=bos.alt_disk_install AIX61_TL6_Full_spot
----------------------------------------------------------------------------------------------------------------------------------------------------
Network
-------
Resolve umserv01 to ip address (from /etc/hosts file):
#host umserv01
To change the host name to umserv01:
#hostname umserv01
To the status of ethernet device en0:
#entstat en0
To list the detailed status of device en0:
#entstat -d en0
To list all net configurable attributes and their values:
#no -a
To change umserv01wall parameter to its default value:
#no -d umserv01wall
To make the machine as router in tcpip networks:
#no -o ipforwarding=1
To trace the route to umserv01:
#traceroute umserv01
To tcp ping to the machine umserv01:
#ping umserv01
To show the status of all network interfaces:
#ifconfig -a
To show the status of en0:
#ifconfig en0
Turns on network card en0:
#ifconfig en0 up
Turns off network card en0:
#ifconfig en0 down
Removes en0 card from the network interface list:
#ifconfig en0 detach
Configure en0 starts immediately:
Temporarily:# ifconfig en0 inet 192.168.100.9 netmask 255.255.255.0 up
Permanently:# chdev -l en0 -a netaddr=192.168.100.10 -a netmask=0xffffff00
Create alias ip address for en0:
Temporarily:# ifconfig en0 alias 192.168.100.10 netmask 255.255.255.0
Permanently:# chdev -l en0 -a alias4=192.168.100.10,255.255.255.0
Remove a permanently added alias:
# chdev -l en0 -a delalias4=192.168.100.10,255.255.255.0
Via SMIT:
# smitty tcpip -> further Configuration -> Network Interfaces -> Network Interface Selection -> Configure Aliases
To make 192.168.100.1 as default gateway for entire network:
Temporarily:#route add 0 192.168.100.1
Permanently:#chdev -l inet0 -a route=0,192.168.100.1
To make 200.7 as gateway for 300.0 network:
#route add 192.100.300.0 192.100.200.7
To clear the gateway table:
#route -f
To change the host name to umserv01 permanently:
#chdev -l inet0 -a hostname=umserv01
To set the MTU to 1500 on en69:
#chdev -l en69 -a mtu=1500
To show the state of all sockets:
#netstat -a
To show the network buffers cache:
#netstat -c
To show the net drops of packets:
#netstat -D
To display interface statistics:
#netstat -i
To show the routing table:
#netstat -r
To show routing table (ip will be given instead of host names)
#netstat -rn
To show the statistics of the protocols
#netstat -s
To show the statistics of respective protocols
#netstat -s -p < tcp/udp/ipv6>
----------------------------------------------------------------------------------------------------------------------------------------------------
How to check the etherchannel works good ?
# entstat -d en7 | grep -i sync
Synchronization: IN_SYNC
Synchronization: IN_SYNC
Synchronization: IN_SYNC
Synchronization: IN_SYNC
# entstat -d en7 | grep -i agg
Device Type: IEEE 802.3ad Link Aggregation
Statistics for every adapter in the IEEE 802.3ad Link Aggregation:
IEEE 802.3ad Link Aggregation Statistics:
Aggregation status: Aggregated
Receive TCP Segment Aggregation: Enabled
TCP Packets Aggregated into Large Packets: 2968288
TCP Payload Bytes Aggregated into Large Packets: 4158020008
TCP Segment Aggregation Large Packets Created: 1367426
TCP Segment Aggregation Average Packets Aggregated: 2
TCP Segment Aggregation Maximum Packets Aggregated: 15
Aggregation: Aggregatable
Aggregation: Aggregatable
Receive TCP Segment Aggregation: Enabled
TCP Packets Aggregated into Large Packets: 1465534
TCP Payload Bytes Aggregated into Large Packets: 2129560180
TCP Segment Aggregation Large Packets Created: 116145
TCP Segment Aggregation Average Packets Aggregated: 12
TCP Segment Aggregation Maximum Packets Aggregated: 17
Aggregation: Aggregatable
Aggregation: Aggregatable
#
----------------------------------------------------------------------------------------------------------------------------------------------------
Howto configure AIX syslogd (/etc/syslog.conf)
----------------------------
The following steps will guide you trough configuring syslog on an AIX system.
This info is valid for AIX 4.3.3, AIX 5.1, AIX 5.2 and AIX 5.3.
By default, an AIX system will not do syslog'ging. For a strange reason, the default install on an AIX install will not place entry's in /etc/syslog.conf, leaving a totl userless syslogd.
Even more, comparing the syslogd on AIX to another Unix, (ex: Linux), you will notice that <facility>.* doesn't work. So make sure that your priority level is never *.
Comming from a Linux world, you most probably want the following config:
mail.debug /var/log/mail
user.debug /var/log/user
kern.debug /var/log/kern
syslog.debug /var/log/syslog
daemon.debug /var/log/daemon
auth.debug /var/log/secure
local2.debug /var/log/sudo
As AIX standard has no logrotate daemon running, you want to adapt syslogd todo the rotating on his own, an example would be:
mail.debug /var/log/mail rotate size 100k files 4 # 4 files, 100kB each
user.debug /var/log/user rotate files 12 time 1m # 12 files, monthly rotate
kern.debug /var/log/kern rotate files 12 time 1m compress # 12 files, monthly rotate, compress
...
When adaptions are made to /etc/syslog.conf, make sure to restart syslogd.
[root@sys /] refresh -s syslogd
0513-095 The request for subsystem refresh was completed successfully.
[root@sys /]
In the dmns03nim1 server inside of syslog.conf
------------------------
*.info @10.250.160.57
mail.info /var/adm/syslog/mail.log rotate size 5m compress archive /stats/syslog/archive
daemon.debug /var/adm/syslog/daemon.log rotate size 5m compress archive /stats/syslog/archive
auth.info /var/adm/syslog/security.log rotate size 5m compress archive /stats/syslog/archive
local2.debug /var/adm/syslog/sudo.log rotate size 5m compress archive /stats/syslog/archive
local6.debug /var/adm/syslog/ldap/ldap.log rotate size 100m compress
*.info;mail.none,daemon.none,auth.none,local2.none,local4.none /var/adm/syslog/syslog.log rotate size 5m compress archive /stats/syslog/archive
# ASO log configuration
aso.notice /var/log/aso/aso.log rotate size 1m files 8 compress
aso.info /var/log/aso/aso_process.log rotate size 1m files 8 compress
aso.debug /var/log/aso/aso_debug.log rotate size 32m files 8 compress
auth.info @10.250.160.57
----------------------------------------------------------------------------------------------------------------------------------------------------
AIX temperature sensors
By executing the command /usr/lpp/diagnostics/bin/uesensor, you can, on supported platform, get the fan speeds and temperature of your system:
[root@sys /] /usr/lpp/diagnostics/bin/uesensor -a
3 0 11 31 P1
9001 0 11 2100 F1
9001 1 11 2760 F2
9001 2 11 1890 F3
9001 3 11 1890 F4
9002 0 11 5129 P1
9002 1 11 3129 P1
9002 2 11 5129 P1
9002 3 11 12077 P1
9004 0 11 3 P3-V1
9004 1 11 3 P3-V2
9004 2 11 3 P3-V3
[root@sys /]
----------------------------------------------------------------------------------------------------------------------------------------------------
How to change hostname on AIX, “Cleanly”
Changing the hostname at a Windows Server will require a reboot. However, changing a hostname on a Unix/Linux server do not require a reboot! But, it is not so straightforward to change the hostname of a server online. This is because you will need to change a couple of things, like the entry in ODM, nodename and current hostname.
oldhostname:/:>chdev -l inet0 -a hostname=newhostname
inet0 changed
oldhostname:/:>uname -S newhostname => Change the network name of this machine.
oldhostname:/:>hostname newhostname => Sets the name of the current host system.
newhostname
oldhostname:/:>su -
newhostname:/:>exit
----------------------------------------------------------------------------------------------------------------------------------------------------
Difference between sudo su and sudo su -
sudo su --> if you do it will call the present user profile and run the root commands.so it will not have all the rights.
sudo su - --> if you do it will call the root user profile so ow you have full rights as a real root account.
----------------------------------------------------------------------------------------------------------------------------------------------------
Find the child devices for the particular device.
lsdev -Cc adapter | grep fcs0
fcs0 Available 08-00 4Gb FC PCI Express Adapter (df1000fe)
lsdev -C | grep -i 08-00
----------------------------------------------------------------------------------------------------------------------------------------------------
