The
topas command is part of AIX and has been for years. Each new release
adds new features and if you are starting now it can be hard to
understand. There are also related commands that post-process topas
output files like topasrec and topasout.
Since late 2008
- AIX 5.3 TL09
- AIX 6.1 TL02
topas also includes the features of nmon - a popular downloaded AIX performance monitoring and data saving program. So now you have the best of both worlds. AIX support and the nmon features inside topas.
Topas has many modes that you need to understand and then mode options in each mode
- Default summary or workstation mode
- Detailed mode for specific sorts of data (one at a time but in detail)
- Cross-Partition mode - covering the Logical Partitions (LPARs) of a single machine
- Cluster mode like the above but you name the LPARs and it can be across multiple machines
- nmon mode
- Save data to a topas format
- Save data to nmon format
1) Default Mode
If you just type in: topas
Then you get the familiar default view - also known as workstation view as it has limited summary stats, for a server just listing one or two disks or networks is not enough.
Then you get the familiar default view - also known as workstation view as it has limited summary stats, for a server just listing one or two disks or networks is not enough.
If you now type: cnsfp
You get the left hand clear and on the right what is known as the "Fixed Set" - these are kernel details that can't be removed.
Also note if you are running an AIX6 Workload Partition (WPAR) that some of these will be in inverse video - this is to highlight that those statistics are global AIX level numbers and not WPAR specific numbers.
You get the left hand clear and on the right what is known as the "Fixed Set" - these are kernel details that can't be removed.
Also note if you are running an AIX6 Workload Partition (WPAR) that some of these will be in inverse video - this is to highlight that those statistics are global AIX level numbers and not WPAR specific numbers.
The Fixed Set stats, below:
Default View - CPU
Now type: c
and you get start cycling through a number of CPU details at different levels. Below are the individual Logical CPUs (assuming this machine has SMT switched on, otherwise this is the physical CPUS):
and you get start cycling through a number of CPU details at different levels. Below are the individual Logical CPUs (assuming this machine has SMT switched on, otherwise this is the physical CPUS):
A
second "c" gives you the simple graphs of CPU Utilisation broken out to
User (programs), System (AIX Kernel system calls and interrupts), wait
for I/O and Idle time. Also the Physc (Physical CPU time) showing in
this case under one CPU is in use and the %Entc (percentage of Entitled
CPU time consumed):
and then one more "c", gives you the CPU totals in numbers only and using minimal screen space:
A further "c" switches off the CPU statistics.
Similar, toggling of statistics happens with
Similar, toggling of statistics happens with
- n for networks
- d for disks
- f for file systems
- p for processes
Also some that are not shown here
- t for tape - this is for supported ATAPE drives
- w for Workload Manager (WLM) data
- @ for Workload Partition(WPAR) data
Below are images showing the various statistics available on my machine ...
Default View - Network
 On the first "n" you get all the networks listed
|  A second "n" gives you the Totals
|
Default View - Disks
 On the first "d" you get all the disks listed
|  A second "d" gives you the Totals
|
Default View - File Systems
 On the first "f" you get all the files systems listed
|  A second "f" gives you the Totals
|
Default View - Processes
On the first "p" you get the list of processes which will fill up any remaining screen space on the low left.
 |
A second "p" will switch off the processor stats.
Default View - Mixed stats
If
you request more than one it will try to display the busy parts of each
and share screen space among the requests as in the below example
showing only some CPUs:
Finally, at any time you can hit "a" and get back to the start up Default screen. We think "a" means All.
Active Memory Expansion (AME)
This just appears if your LPAR is
- running AIX 6.1 TL04 service pack 2 or higher and
- you have switched on AME at the HMC and
- cold restarted your LPAR with the new option.
Notes:
- TMEM is true memory actually in the LPAR in MB
- CMEM is compress memory used for the compressed pages in MB - this is the only why to finf this out.
- EFT is Expansion Factor Target - i.e. what you asked for
- EFA is Expansion Factor Actual - i.e. what it is at the moment. If different to the above it could be compressing pages at the moment or failing to get a high enough compression ratio as the Expansion Factor is unrealistic/optimistic.
- CI is the Compressed Page In in the number of pages (like paging in from the paging space)
- CO is the Compressed Page Out in the number of pages
2) Detailed Mode
Detailed
mode shows more details but only one resource at a time. All the
Detailed views are requested but UPPER CASE characters - again using a
suitable character to switch the resource.
Detailed Process view
Below is the busy process statistics that you get by typing: P
This is like a ps command where the busy processes are at the top:
This is like a ps command where the busy processes are at the top:
You
can reorder the process listing by moving the cursor to the column. You
do this using the Left-Arrow and Right-Arrow keys on your keyboard.
Below we went one left to the TIME column (the CPU seconds used by the
process since it started).
One
more column to the left is the PAGE-SPACE column. This is an
interesting column as it details the amount of the paging space used by
the process - I don't know any other commands that can show this
statistic.
Detailed Ethernet view
To show Ethernet statistics type: E
and you get a whole page of network details on the whole page as below.
and you get a whole page of network details on the whole page as below.
Detailed Ethernet On a VIOS view
To get this you just use "E" on the VIOS - nothing else is needed.
Note: if you use this Ethernet view on a Virtual I/O Server (VIOS) then you also get
Note: if you use this Ethernet view on a Virtual I/O Server (VIOS) then you also get
- Physical network
- Virtual network
- Shared Ethernet adapter (SEA bridge)
- Control channel information used to co-ordinate Virtual I/O Servers running Ethernet fail-over.
Here are two examples:
| border=1 |
 |
Detailed File Systems view
To show file system statistics type: F
Below is the file system detailed view - this explains which filesystems are performing the most disk I/O.
Detailed Logical Partition (LPAR) view
To show LPAR statistics type: L
Below
is the LPAR view - this is similar to the AIX mpstat command output and
includes many interesting details like Entitlement, SMT mode,
capped/uncapped mode, virtual CPUs and memory size.
Detailed Volume Group view
To show volume group (VG) statistics type: V
Below is the volume group detailed view - this explains which groups of disks are performing the most disk I/O.
This machine does not have many volume group but if you move with the Up-Arrow key and Down-Arrow key to a volume group and then hit "f" you will see the logical volumes (LV) of that volume group.
This machine does not have many volume group but if you move with the Up-Arrow key and Down-Arrow key to a volume group and then hit "f" you will see the logical volumes (LV) of that volume group.
Below is an example from a VIOS with three volume groups:
Having moved the cursor down to the "rootvg" and then type "f". We can see below the root volume group logical volumes.
Detailed Tape view
To show tape statistics type: T
Sorry but I don't have the right type of tape drive i.e. a ATAPE.
Sorry but I don't have the right type of tape drive i.e. a ATAPE.
Detailed Disk view
To show disk statistics type: D
Below is the disk detailed view:
Detailed Virtual SCSI Disk on a VIOS view
To get this you just use "d" while seeing the Detailed Disk view
Note: you can go down to particular "vhost" with the Down-Arrow key and type "f" to bring up the matching virtual SCSI device.
Note: you can go down to particular "vhost" with the Down-Arrow key and type "f" to bring up the matching virtual SCSI device.
Detailed Memory Affinity stats
For
the super large POWER7 machines, AIX will use affinity scheduling to
localise as much as possible the process running on a core to the memory
attached to the core. Hence below we see memory is marked as Local,
Near and Far. My machine has a single POWER7 so its not that interesting
as all memory is local. On larger machines we can monitor the success
of localising processes to memory.
Detailed Workload Partition (WPAR) view
- Sorry no screen capture yet
3) Cross-Partition Mode
To enter the Cross-Partition view - also known as CEC mode or whole machine view
- Start topas with: topas -C
- Once started type: C
- If you are using a Virtual I/O Server (VIOS) and logged on as the padmin user, type: topas -cecdisp
This
is actually a very different data source as the topas has to connect to
the other LPARs on the same machine but first has to query the HMC to
work the hostnames involved.
Below is the example from a VIOS.
When in Cross-Partition mode, type: g
Then you gets lots more Global data about the whole machine - which gives you a good view of the machine set up.
You can also switch off/on the Shared CPU and Dedicated CPU sections with "s" and "d".
Then you gets lots more Global data about the whole machine - which gives you a good view of the machine set up.
You can also switch off/on the Shared CPU and Dedicated CPU sections with "s" and "d".
When in Cross-Partition mode, type:v
then you get the virtual I/O server (VIOS) and Client relationships view
then you get the virtual I/O server (VIOS) and Client relationships view
If
you when in the VIOS server/client view then move Down-Arrows and
Up-Arrow keys to get the cursor to select a particular VIOS and then
type "f" then you see the VIOS clients attached to it.
When in Cross-Partition mode, type: m
then you get the memory pool information for Active Memory Sharing (AMS) statistics - not sure what you get if you have not activated AMS!
then you get the memory pool information for Active Memory Sharing (AMS) statistics - not sure what you get if you have not activated AMS!
When in Cross-Partition mode, type: p
then you get the CPU pool information for POWER6 and above based machines, where you can have more than one CPU pool typically for largely licence reasons or perhaps performance control.
then you get the CPU pool information for POWER6 and above based machines, where you can have more than one CPU pool typically for largely licence reasons or perhaps performance control.
4) Cross-Cluster Mode
More required here
5) nmon Mode
Arh - now you you have a whole new ball game 
How to enter nmon mode:
- If in Default, Detailed or Cross-Partition mode then type: ~
- At the command line type: nmon
- This starts /usr/bin/nmon which is a shell script that starts topas_nmon
- Which is a hard link to /usr/bin/topas - when it starts with the name topas-nmon it knows to go straight to nmon mode
There are two places to get more information:
- On the command line type: nmon -h
- While running nmon online, get the help information by typing: h
- Use "h" again to switch it off.
Here is a brief list of the options
Interactive Mode Commands
- Use "h" again to switch it off.
- h = Online help information
- r = Resources type, machine name, cache & AIX version
- p = Partitions stats
- c = CPU by processor stats with bar graphs then #=toggle PURR based stats (shared CPUs only)
- C = CPU by processor stats for high numbers of CPU
- l = long term CPU (over 75 snapshots) with bar graphs
- m = Memory and Paging stats
- M = Multiple Page Size stats in pages - 2nd time in MB's
- k = Kernel Internal stats
- n = Network stats
- = = For Network & Disk Toggle KB/s to MB/s
- O = Shared Ethernet Adapter VIOS only
- N = NFS Network File System stats (2nd N gets you NFSv4)
- d = Disk I/O Graphs (see -k to limit to specific disks)
- D = Disk I/O Stats - multiple times gets you more stats
- o = Disk I/O Map (one char per disk showing how busy it is)
- g = Disk Group I/O Stats (have to use -g cmdline option)
- a = Adapter I/O Stats
- ^ = Fibre Channel Adapter via fcstat command
- e = ESS vpath Logical Disk I/O Stats
- V = Disk Volume Group stats
- P = Paging Space stats
- j = JFS Stats
- t = Top Process Stats 1=Basic-Details 2=Accumulated-CPU then Performance sorted by 3=CPU 4=Size 5=I/O
- u = Top but with command arguments shown
- U = as u plus Workload Classes/WPAR
- W = Workload Management (WLM) Stats
- S = WLM with SubClasses
- @ = Workload Partition (WPAR) Stats
- w = use with top to show AIX wait processes
- A = Summarise Async I/O (aioserver) processes
- v = Verbose this highlights problems on the machine
- . = minimum mode i.e. only busy disks and processes
Key — Other Controls —
- + or - = double or halve the screen refresh time
- q = quit (also x, e or control-C)
- 0 = reset peak counts to zero
6) topas recording to file
The topasout command is used to convert topas binary files to text reports and more.
But the manual page is a little hard to understand:
But the manual page is a little hard to understand:
Machine or LPAR - Data reports
- topasout -c|-s|-a /etc/perf/daily/hostname_YYMMDD.topas
- Example: topasout -a /etc/perf/daily/silver8_090304.topas
- The output overwrites the file /etc/perf/daily/hostname_YYMMDD.topas_01
- c = comma separated but one timestamp and value per line
- s = spreadsheet space separated 100's of stats per line
- a = analyser .csv file for nmon analyser - or - use topas_nmon directly
Machine or LPAR - Text reports
- topasout -R OPTION [-i MM -b HHMM -e HHMM] /etc/perf/daily/hostname_YYMMDD.topas
- OPTION = detailed|summary|poolinfo|disk |lan|mempool|adapter|vadapter|vios|vios_adapter
- Example: topasout -R summary /etc/perf/daily/silver8_090304.topas >/tmp/LPARsummary.txt
- Text reports via standard out, so you need to redirect the output to a file
Cross-Partition -Text reports
- topasout -R detailed | -R summary [-i MM -b HHMM -e HHMM] /etc/perf/topas_cec.YYMMDD
- Example: topasout -R detailed topas_cec.090305 >/tmp/CECdetailed.txt
- Text reports via standard out, so you need to redirect the output to a file
For system use only, do NOT run these manually.
- topasout -R daily | -R weekly
- Notes: YYMMDD = year, month, day MM is minutes HHMM is hours & minutes
7) nmon recording to file
Briefly these are the options:
Mandatory For Data-Collect-Mode = comma separated values format
- -f output file is <hostname>_YYYYMMDD_HHMM.nmon Note: -f ot -F should be the FIRST option on the line as they set many defaults that later options can override
- -s <seconds> between snap shots
- -c <number> of refreshes
Optional
- -F <filename> same as -f but user supplied filename
- -r <runname> goes into spreadsheet file
- -t include top processes in the output
- -T as -t plus saves command line arguments in UARG
- -Y like -t but sum cmds with same name
- -w <number> Timestamp size (Tnnnn),
- -l <dpl> disks per line - avoid spreadsheet issues.
- -g <filename> User decided Disk Groups
- -d Include Disk Service time sections
- -k <disklist> Only report these disks also works online
- -G Use UTC/GMT standard time (not local time)
- -K Include RAW Kernel & LPAR sections
- -D Skip disk configuration sections
- -E Skip ESS configuration sections
- -J Skip JFS sections
- -V Include disk Volume Group section
- -P Include Paging Space section
- -M Include MEMPAGES section
- -N Include NFS section, use -NN for NFSv4 stats.
- -W Include WLM sections
- -S Include WLM sections with SubClasses
- -@ Include WPAR sections
- -^ Include Fibre Channel (FC) sections
- -O Include Shared Ethernet Adapter (SEA) VIOS
- -L Include LARGE page section
- -I <percent> Ignore process percent threshold
- -A Include Async I/O Section (implies -t as we collect process stats to work this out)
- -m <dir> nmon changes to this directory
- -Z <priority> set nice priority -20=important to 20=unimportant
Notes
- WPAR %Resc
- It is relative to the WPAR cpu limits. It is meant to be similar to %entc which tells you how much of your entitlement you have consumed. In case of cpu limits, since those are not really an entitlement, it is named differently.
- Back to %RESC, it is the same as vmstat's RC column (which replaces the normal EC column when a WPAR has cpu limits).
- Turning Continuous Disk Busy stats on/off
- iostat and sar do the same (turn on disk history while running) by the way.
- Default topas file directory
- /etc/perf
