Tag Archives: unix

find matching files, grep content with filename

Posted on by

a clever trick found here

	find . -exec grep -n hello /dev/null {} \;
Advertisements

prstat

Posted on by

Finding Processes that are Using Up the CPU


mpstat 5 5 command will print the CPU statistics 5 times at 5 second intervals.

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 1 0 0 345 224 589 220 0 0 0 799 29 1 0 70
0 1 0 0 302 200 752 371 0 0 0 1191 99 1 0 0
0 0 0 0 341 221 767 375 0 0 0 1301 100 0 0 0
0 0 0 0 411 256 776 378 0 0 0 1313 99 1 0 0
0 0 0 0 382 241 738 363 0 0 0 1163 97 3 0 0

In the output sample above, 4 of the 5 samples have CPU 0 with a combined user time and sytem time at 100 and idle time at 0 (column headings usr, sys, idl). This indicates that the CPU is completely consumed on this system.

you can use prstat to identify which processes are consuming the CPU resources. prstat -s cpu -n 5 command is used to list the five processes that are consuming the most CPU resources. The -s cpu flag tells prstat to sort the output by CPU usage. The -n 5 flag tells prstat to restrict the output to the top five processes.

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13974 kincaid 888K 432K run 40 0 36:14.51 67% cpuhog/1
27354 kincaid 2216K 1928K run 31 0 314:48.51 27% server/5
14690 root 136M 46M sleep 59 0 0:00.59 2.3% Xsun/1
14797 kincaid 9192K 7496K sleep 59 0 0:00.10 0.9% dtwm/8
14851 kincaid 24M 14M sleep 48 0 0:00.03 0.3% netscape/1
Total: 97 processes, 190 lwps, load averages: 2.18, 2.15, 2.11


In the example output, there is a process named cpuhog that is consuming the majority (67 %) of the CPU cycles. If this process is killed using the kill -9 13974 command, and the mpstat command is subsequently repeated, the output shows that the CPU is idle the majority of the time.

Note: The -s cpu option is the default setting for prstat. Therefore, if the intent is to sort output by CPU usage, specifying the -s cpu option is not necessary. For the purpose of this article, the -s cpu option is set to distinguish it from other ways of sorting the output produced by prstat.

command prstat -s cpu -a -n 8 asks for the top 8 processes consuming the CPU and a list of resource consumption statistics for each user.

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
17005 larry 888K 432K run 21 0 0:03.15 38% cpuhog/1
17015 larry 888K 432K run 21 0 0:03.06 36% cpuhog/1
17175 larry 944K 872K run 24 0 0:00.37 5.7% find/1
16911 moe 944K 872K sleep 58 0 0:00.48 3.3% find/1
16915 moe 944K 872K sleep 59 0 0:00.43 3.3% find/1
17849 curly 944K 872K run 31 0 0:00.00 3.0% find/1
16472 root 132M 42M sleep 59 0 0:01.00 0.9% Xsun/1
16827 kincaid 6864K 4704K sleep 48 0 0:00.05 0.4% dtterm/1
NPROC USERNAME SIZE RSS MEMORY TIME CPU
7 larry 7504K 5656K 0.6% 0:06.58 80%
8 moe 8248K 6800K 0.7% 0:01.31 6.6%
3 curly 3336K 2832K 0.3% 0:00.00 3.0%
34 root 213M 95M 9.5% 0:03.05 1.0%
78 kincaid 433M 294M 30% 0:00.38 0.7%
Total: 132 processes, 218 lwps, load averages: 3.90, 4.29, 2.45

Identifying Virtual Memory Usage

A good tool for determining if a system is paging is sar. The sar -g command will yield the paging statistics for a given system.

$ sar -g 5 5

Getting the Resource Statistics for Each Thread Within a Process

SunOS tartan 5.8 Generic_108528-01 sun4u 02/12/01

13:20:37 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
13:20:42 39.92 538.72 670.26 1147.31 0.00
13:20:47 36.60 483.80 515.40 353.80 0.00
13:20:52 40.20 508.20 632.00 1125.20 0.00
13:20:57 35.80 462.60 580.40 1141.60 0.00
13:21:02 0.00 0.00 0.00 0.00 0.00
Average 30.51 398.72 479.69 753.74 0.00

A common cause for system paging is a process or group of processes that are using a majority of the system’s memory. prstat is an excellent tool for identifying which processes are consuming the majority of a system’s memory. Use prstat -s size, which is similar to the previous command, but which sorts prstat output by size instead of by CPU usage.

top five processes on a system in terms of virtual memory consumption

$ prstat -s size -n 5

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
21307 kincaid 1001M 616M run 2 0 0:01.16 32% memhog/1
16472 root 138M 43M sleep 59 0 0:17.28 1.2% Xsun/1
18133 kincaid 92M 31M sleep 49 0 0:01.42 0.0% soffice.bin/9
16574 kincaid 44M 24M sleep 49 0 0:10.37 0.2% .netscape.bin/1
16674 kincaid 36M 25M sleep 49 0 0:00.08 0.0% sdtperfmeter/1
Total: 130 processes, 220 lwps, load averages: 0.51, 0.36, 0.23


$ sar -g 5 5

SunOS tartan 5.8 Generic_108528-01 sun4u 02/12/01

13:20:02 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
13:20:07 0.00 0.00 0.00 0.00 0.00
13:20:12 0.00 0.00 0.00 0.00 0.00
13:20:17 0.00 0.00 0.00 0.00 0.00
13:20:22 0.00 0.00 0.00 0.00 0.00
13:20:27 0.00 0.00 3.80 590.60 0.00
Average 0.00 0.00 0.76 118.07 0.00

Getting the Resource Statistics for Each Thread Within a Process

by using the -L switch, prstat will report statistics for each thread of a process. 

$ prstat -L -p 3295

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
3295 kincaid 28M 10M sleep 38 0 0:00.01 2.1% java/16
3295 kincaid 28M 10M sleep 55 0 0:00.01 1.9% java/17
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.8% java/15
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.8% java/23
3295 kincaid 28M 10M sleep 52 0 0:00.01 1.7% java/12
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.6% java/22
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.5% java/13
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.5% java/14
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.4% java/19
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.4% java/18
3295 kincaid 28M 10M sleep 38 0 0:00.01 1.4% java/21
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.3% java/24
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.2% java/20
3295 kincaid 28M 10M sleep 58 0 0:00.00 0.0% java/1
3295 kincaid 28M 10M sleep 58 0 0:00.00 0.0% java/11
3295 kincaid 28M 10M sleep 0 0 0:00.00 0.0% java/10
3295 kincaid 28M 10M sleep 59 0 0:00.00 0.0% java/9
3295 kincaid 28M 10M sleep 0 0 0:00.00 0.0% java/8
3295 kincaid 28M 10M sleep 0 0 0:00.00 0.0% java/7
3295 kincaid 28M 10M sleep 59 0 0:00.00 0.0% java/6
3295 kincaid 28M 10M sleep 58 0 0:00.00 0.0% java/5
Total: 1 processes, 24 lwps, load averages: 1.30, 1.22, 1.21

Getting the micro statistics on a process

use the -m option to have prstat print out the micro statistics of the process.

Column Heading Meaning
USR The percentage of time the process has spent in user mode
SYS The percentage of time the process has spent in system mode
TRP The percentage of time the process has spent in processing system traps
DFL The percentage of time the process has spent processing data page faults
LCK The percentage of time the process has spent waiting for user locks
SLP The percentage of time the process has spent sleeping
TFL The percentage of time the process has spent processing text page faults 

$ prstat -m -p 3295

     PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  3295 kincaid  0.8 0.0 0.0 0.0 0.0 0.0  99 0.0  22   4  25   0 java/24


In addition, the -m option can be used with the -L option so the micro statistics of each thread of a process can be monitored.

Some of the other Solaris tools that are worth reading about are: sar, iostat, netstat, and mpstat. All of these are great tools for identifying system performance issues. prstat is a nice complement to these tools because it helps further identify the processes and threads that may be responsible for system performance problems.

http://developers.sun.com/solaris/articles/prstat.html