a clever trick found here
find . -exec grep -n hello /dev/null {} \;
question:
I want both standard output and standard error of my command cmd to go to the same file log.txt
answer:
cmd >log.txt 2>&1
mpstat 5 5 command will print the CPU statistics 5 times at 5 second intervals.
CPU | minf | mjf | xcal | intr | ithr | csw | icsw | migr | smtx | srw | syscl | usr | sys | wt | idl |
0 | 1 | 0 | 0 | 345 | 224 | 589 | 220 | 0 | 0 | 0 | 799 | 29 | 1 | 0 | 70 |
0 | 1 | 0 | 0 | 302 | 200 | 752 | 371 | 0 | 0 | 0 | 1191 | 99 | 1 | 0 | 0 |
0 | 0 | 0 | 0 | 341 | 221 | 767 | 375 | 0 | 0 | 0 | 1301 | 100 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 411 | 256 | 776 | 378 | 0 | 0 | 0 | 1313 | 99 | 1 | 0 | 0 |
0 | 0 | 0 | 0 | 382 | 241 | 738 | 363 | 0 | 0 | 0 | 1163 | 97 | 3 | 0 | 0 |
In the output sample above, 4 of the 5 samples have CPU 0 with a combined user time and sytem time at 100 and idle time at 0 (column headings usr, sys, idl). This indicates that the CPU is completely consumed on this system.
you can use prstat to identify which processes are consuming the CPU resources. prstat -s cpu -n 5 command is used to list the five processes that are consuming the most CPU resources. The -s cpu flag tells prstat to sort the output by CPU usage. The -n 5 flag tells prstat to restrict the output to the top five processes.
PID | USERNAME | SIZE | RSS | STATE | PRI | NICE | TIME | CPU | PROCESS/NLWP |
13974 | kincaid | 888K | 432K | run | 40 | 0 | 36:14.51 | 67% | cpuhog/1 |
27354 | kincaid | 2216K | 1928K | run | 31 | 0 | 314:48.51 | 27% | server/5 |
14690 | root | 136M | 46M | sleep | 59 | 0 | 0:00.59 | 2.3% | Xsun/1 |
14797 | kincaid | 9192K | 7496K | sleep | 59 | 0 | 0:00.10 | 0.9% | dtwm/8 |
14851 | kincaid | 24M | 14M | sleep | 48 | 0 | 0:00.03 | 0.3% | netscape/1 |
Total: 97 processes, 190 lwps, load averages: 2.18, 2.15, 2.11 |
In the example output, there is a process named cpuhog that is consuming the majority (67 %) of the CPU cycles. If this process is killed using the kill -9 13974 command, and the mpstat command is subsequently repeated, the output shows that the CPU is idle the majority of the time.
Note: The -s cpu option is the default setting for prstat. Therefore, if the intent is to sort output by CPU usage, specifying the -s cpu option is not necessary. For the purpose of this article, the -s cpu option is set to distinguish it from other ways of sorting the output produced by prstat.
command prstat -s cpu -a -n 8 asks for the top 8 processes consuming the CPU and a list of resource consumption statistics for each user.
PID | USERNAME | SIZE | RSS | STATE | PRI | NICE | TIME | CPU | PROCESS/NLWP |
17005 | larry | 888K | 432K | run | 21 | 0 | 0:03.15 | 38% | cpuhog/1 |
17015 | larry | 888K | 432K | run | 21 | 0 | 0:03.06 | 36% | cpuhog/1 |
17175 | larry | 944K | 872K | run | 24 | 0 | 0:00.37 | 5.7% | find/1 |
16911 | moe | 944K | 872K | sleep | 58 | 0 | 0:00.48 | 3.3% | find/1 |
16915 | moe | 944K | 872K | sleep | 59 | 0 | 0:00.43 | 3.3% | find/1 |
17849 | curly | 944K | 872K | run | 31 | 0 | 0:00.00 | 3.0% | find/1 |
16472 | root | 132M | 42M | sleep | 59 | 0 | 0:01.00 | 0.9% | Xsun/1 |
16827 | kincaid | 6864K | 4704K | sleep | 48 | 0 | 0:00.05 | 0.4% | dtterm/1 |
NPROC | USERNAME | SIZE | RSS | MEMORY | TIME | CPU |
7 | larry | 7504K | 5656K | 0.6% | 0:06.58 | 80% |
8 | moe | 8248K | 6800K | 0.7% | 0:01.31 | 6.6% |
3 | curly | 3336K | 2832K | 0.3% | 0:00.00 | 3.0% |
34 | root | 213M | 95M | 9.5% | 0:03.05 | 1.0% |
78 | kincaid | 433M | 294M | 30% | 0:00.38 | 0.7% |
Total: 132 processes, 218 lwps, load averages: 3.90, 4.29, 2.45 |
A good tool for determining if a system is paging is sar. The sar -g command will yield the paging statistics for a given system.
$ sar -g 5 5
Getting the Resource Statistics for Each Thread Within a Process
SunOS tartan 5.8 Generic_108528-01 sun4u 02/12/01
13:20:37 | pgout/s | ppgout/s | pgfree/s | pgscan/s | %ufs_ipf |
13:20:42 | 39.92 | 538.72 | 670.26 | 1147.31 | 0.00 |
13:20:47 | 36.60 | 483.80 | 515.40 | 353.80 | 0.00 |
13:20:52 | 40.20 | 508.20 | 632.00 | 1125.20 | 0.00 |
13:20:57 | 35.80 | 462.60 | 580.40 | 1141.60 | 0.00 |
13:21:02 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
Average | 30.51 | 398.72 | 479.69 | 753.74 | 0.00 |
A common cause for system paging is a process or group of processes that are using a majority of the system’s memory. prstat is an excellent tool for identifying which processes are consuming the majority of a system’s memory. Use prstat -s size, which is similar to the previous command, but which sorts prstat output by size instead of by CPU usage.
top five processes on a system in terms of virtual memory consumption
$ prstat -s size -n 5
PID | USERNAME | SIZE | RSS | STATE | PRI | NICE | TIME | CPU | PROCESS/NLWP |
21307 | kincaid | 1001M | 616M | run | 2 | 0 | 0:01.16 | 32% | memhog/1 |
16472 | root | 138M | 43M | sleep | 59 | 0 | 0:17.28 | 1.2% | Xsun/1 |
18133 | kincaid | 92M | 31M | sleep | 49 | 0 | 0:01.42 | 0.0% | soffice.bin/9 |
16574 | kincaid | 44M | 24M | sleep | 49 | 0 | 0:10.37 | 0.2% | .netscape.bin/1 |
16674 | kincaid | 36M | 25M | sleep | 49 | 0 | 0:00.08 | 0.0% | sdtperfmeter/1 |
Total: 130 processes, 220 lwps, load averages: 0.51, 0.36, 0.23 |
$ sar -g 5 5
SunOS tartan 5.8 Generic_108528-01 sun4u 02/12/01
13:20:02 | pgout/s | ppgout/s | pgfree/s | pgscan/s | %ufs_ipf |
13:20:07 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
13:20:12 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
13:20:17 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
13:20:22 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
13:20:27 | 0.00 | 0.00 | 3.80 | 590.60 | 0.00 |
Average | 0.00 | 0.00 | 0.76 | 118.07 | 0.00 |
by using the -L switch, prstat will report statistics for each thread of a process.
$ prstat -L -p 3295
PID | USERNAME | SIZE | RSS | STATE | PRI | NICE | TIME | CPU | PROCESS/LWPID |
3295 | kincaid | 28M | 10M | sleep | 38 | 0 | 0:00.01 | 2.1% | java/16 |
3295 | kincaid | 28M | 10M | sleep | 55 | 0 | 0:00.01 | 1.9% | java/17 |
3295 | kincaid | 28M | 10M | sleep | 48 | 0 | 0:00.01 | 1.8% | java/15 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.01 | 1.8% | java/23 |
3295 | kincaid | 28M | 10M | sleep | 52 | 0 | 0:00.01 | 1.7% | java/12 |
3295 | kincaid | 28M | 10M | sleep | 48 | 0 | 0:00.01 | 1.6% | java/22 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.01 | 1.5% | java/13 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.01 | 1.5% | java/14 |
3295 | kincaid | 28M | 10M | sleep | 48 | 0 | 0:00.01 | 1.4% | java/19 |
3295 | kincaid | 28M | 10M | sleep | 48 | 0 | 0:00.01 | 1.4% | java/18 |
3295 | kincaid | 28M | 10M | sleep | 38 | 0 | 0:00.01 | 1.4% | java/21 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.01 | 1.3% | java/24 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.01 | 1.2% | java/20 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.00 | 0.0% | java/1 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.00 | 0.0% | java/11 |
3295 | kincaid | 28M | 10M | sleep | 0 | 0 | 0:00.00 | 0.0% | java/10 |
3295 | kincaid | 28M | 10M | sleep | 59 | 0 | 0:00.00 | 0.0% | java/9 |
3295 | kincaid | 28M | 10M | sleep | 0 | 0 | 0:00.00 | 0.0% | java/8 |
3295 | kincaid | 28M | 10M | sleep | 0 | 0 | 0:00.00 | 0.0% | java/7 |
3295 | kincaid | 28M | 10M | sleep | 59 | 0 | 0:00.00 | 0.0% | java/6 |
3295 | kincaid | 28M | 10M | sleep | 58 | 0 | 0:00.00 | 0.0% | java/5 |
Total: 1 processes, 24 lwps, load averages: 1.30, 1.22, 1.21 |
use the -m option to have prstat print out the micro statistics of the process.
Column Heading | Meaning |
USR | The percentage of time the process has spent in user mode |
SYS | The percentage of time the process has spent in system mode |
TRP | The percentage of time the process has spent in processing system traps |
DFL | The percentage of time the process has spent processing data page faults |
LCK | The percentage of time the process has spent waiting for user locks |
SLP | The percentage of time the process has spent sleeping |
TFL | The percentage of time the process has spent processing text page faults |
$ prstat -m -p 3295
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
3295 kincaid 0.8 0.0 0.0 0.0 0.0 0.0 99 0.0 22 4 25 0 java/24
In addition, the -m option can be used with the -L option so the micro statistics of each thread of a process can be monitored.
Some of the other Solaris tools that are worth reading about are: sar, iostat, netstat, and mpstat. All of these are great tools for identifying system performance issues. prstat is a nice complement to these tools because it helps further identify the processes and threads that may be responsible for system performance problems.
iostat
vmstat
netstat
Continue reading