prstat

Finding Processes that are Using Up the CPU


mpstat 5 5 command will print the CPU statistics 5 times at 5 second intervals.

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 1 0 0 345 224 589 220 0 0 0 799 29 1 0 70
0 1 0 0 302 200 752 371 0 0 0 1191 99 1 0 0
0 0 0 0 341 221 767 375 0 0 0 1301 100 0 0 0
0 0 0 0 411 256 776 378 0 0 0 1313 99 1 0 0
0 0 0 0 382 241 738 363 0 0 0 1163 97 3 0 0

In the output sample above, 4 of the 5 samples have CPU 0 with a combined user time and sytem time at 100 and idle time at 0 (column headings usr, sys, idl). This indicates that the CPU is completely consumed on this system.

you can use prstat to identify which processes are consuming the CPU resources. prstat -s cpu -n 5 command is used to list the five processes that are consuming the most CPU resources. The -s cpu flag tells prstat to sort the output by CPU usage. The -n 5 flag tells prstat to restrict the output to the top five processes.

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13974 kincaid 888K 432K run 40 0 36:14.51 67% cpuhog/1
27354 kincaid 2216K 1928K run 31 0 314:48.51 27% server/5
14690 root 136M 46M sleep 59 0 0:00.59 2.3% Xsun/1
14797 kincaid 9192K 7496K sleep 59 0 0:00.10 0.9% dtwm/8
14851 kincaid 24M 14M sleep 48 0 0:00.03 0.3% netscape/1
Total: 97 processes, 190 lwps, load averages: 2.18, 2.15, 2.11


In the example output, there is a process named cpuhog that is consuming the majority (67 %) of the CPU cycles. If this process is killed using the kill -9 13974 command, and the mpstat command is subsequently repeated, the output shows that the CPU is idle the majority of the time.

Note: The -s cpu option is the default setting for prstat. Therefore, if the intent is to sort output by CPU usage, specifying the -s cpu option is not necessary. For the purpose of this article, the -s cpu option is set to distinguish it from other ways of sorting the output produced by prstat.

command prstat -s cpu -a -n 8 asks for the top 8 processes consuming the CPU and a list of resource consumption statistics for each user.

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
17005 larry 888K 432K run 21 0 0:03.15 38% cpuhog/1
17015 larry 888K 432K run 21 0 0:03.06 36% cpuhog/1
17175 larry 944K 872K run 24 0 0:00.37 5.7% find/1
16911 moe 944K 872K sleep 58 0 0:00.48 3.3% find/1
16915 moe 944K 872K sleep 59 0 0:00.43 3.3% find/1
17849 curly 944K 872K run 31 0 0:00.00 3.0% find/1
16472 root 132M 42M sleep 59 0 0:01.00 0.9% Xsun/1
16827 kincaid 6864K 4704K sleep 48 0 0:00.05 0.4% dtterm/1
NPROC USERNAME SIZE RSS MEMORY TIME CPU
7 larry 7504K 5656K 0.6% 0:06.58 80%
8 moe 8248K 6800K 0.7% 0:01.31 6.6%
3 curly 3336K 2832K 0.3% 0:00.00 3.0%
34 root 213M 95M 9.5% 0:03.05 1.0%
78 kincaid 433M 294M 30% 0:00.38 0.7%
Total: 132 processes, 218 lwps, load averages: 3.90, 4.29, 2.45

Identifying Virtual Memory Usage

A good tool for determining if a system is paging is sar. The sar -g command will yield the paging statistics for a given system.

$ sar -g 5 5

Getting the Resource Statistics for Each Thread Within a Process

SunOS tartan 5.8 Generic_108528-01 sun4u 02/12/01

13:20:37 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
13:20:42 39.92 538.72 670.26 1147.31 0.00
13:20:47 36.60 483.80 515.40 353.80 0.00
13:20:52 40.20 508.20 632.00 1125.20 0.00
13:20:57 35.80 462.60 580.40 1141.60 0.00
13:21:02 0.00 0.00 0.00 0.00 0.00
Average 30.51 398.72 479.69 753.74 0.00

A common cause for system paging is a process or group of processes that are using a majority of the system’s memory. prstat is an excellent tool for identifying which processes are consuming the majority of a system’s memory. Use prstat -s size, which is similar to the previous command, but which sorts prstat output by size instead of by CPU usage.

top five processes on a system in terms of virtual memory consumption

$ prstat -s size -n 5

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
21307 kincaid 1001M 616M run 2 0 0:01.16 32% memhog/1
16472 root 138M 43M sleep 59 0 0:17.28 1.2% Xsun/1
18133 kincaid 92M 31M sleep 49 0 0:01.42 0.0% soffice.bin/9
16574 kincaid 44M 24M sleep 49 0 0:10.37 0.2% .netscape.bin/1
16674 kincaid 36M 25M sleep 49 0 0:00.08 0.0% sdtperfmeter/1
Total: 130 processes, 220 lwps, load averages: 0.51, 0.36, 0.23


$ sar -g 5 5

SunOS tartan 5.8 Generic_108528-01 sun4u 02/12/01

13:20:02 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
13:20:07 0.00 0.00 0.00 0.00 0.00
13:20:12 0.00 0.00 0.00 0.00 0.00
13:20:17 0.00 0.00 0.00 0.00 0.00
13:20:22 0.00 0.00 0.00 0.00 0.00
13:20:27 0.00 0.00 3.80 590.60 0.00
Average 0.00 0.00 0.76 118.07 0.00

Getting the Resource Statistics for Each Thread Within a Process

by using the -L switch, prstat will report statistics for each thread of a process.

$ prstat -L -p 3295

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
3295 kincaid 28M 10M sleep 38 0 0:00.01 2.1% java/16
3295 kincaid 28M 10M sleep 55 0 0:00.01 1.9% java/17
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.8% java/15
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.8% java/23
3295 kincaid 28M 10M sleep 52 0 0:00.01 1.7% java/12
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.6% java/22
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.5% java/13
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.5% java/14
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.4% java/19
3295 kincaid 28M 10M sleep 48 0 0:00.01 1.4% java/18
3295 kincaid 28M 10M sleep 38 0 0:00.01 1.4% java/21
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.3% java/24
3295 kincaid 28M 10M sleep 58 0 0:00.01 1.2% java/20
3295 kincaid 28M 10M sleep 58 0 0:00.00 0.0% java/1
3295 kincaid 28M 10M sleep 58 0 0:00.00 0.0% java/11
3295 kincaid 28M 10M sleep 0 0 0:00.00 0.0% java/10
3295 kincaid 28M 10M sleep 59 0 0:00.00 0.0% java/9
3295 kincaid 28M 10M sleep 0 0 0:00.00 0.0% java/8
3295 kincaid 28M 10M sleep 0 0 0:00.00 0.0% java/7
3295 kincaid 28M 10M sleep 59 0 0:00.00 0.0% java/6
3295 kincaid 28M 10M sleep 58 0 0:00.00 0.0% java/5
Total: 1 processes, 24 lwps, load averages: 1.30, 1.22, 1.21

Getting the micro statistics on a process

use the -m option to have prstat print out the micro statistics of the process.

Column Heading Meaning
USR The percentage of time the process has spent in user mode
SYS The percentage of time the process has spent in system mode
TRP The percentage of time the process has spent in processing system traps
DFL The percentage of time the process has spent processing data page faults
LCK The percentage of time the process has spent waiting for user locks
SLP The percentage of time the process has spent sleeping
TFL The percentage of time the process has spent processing text page faults

$ prstat -m -p 3295

PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
3295 kincaid 0.8 0.0 0.0 0.0 0.0 0.0 99 0.0 22 4 25 0 java/24


In addition, the -m option can be used with the -L option so the micro statistics of each thread of a process can be monitored.

Some of the other Solaris tools that are worth reading about are: sar, iostat, netstat, and mpstat. All of these are great tools for identifying system performance issues. prstat is a nice complement to these tools because it helps further identify the processes and threads that may be responsible for system performance problems.

http://developers.sun.com/solaris/articles/prstat.html

Advertisements

getting started google app engine using python

download and install python 2.5
http://www.python.org/ftp/python/2.5.4/python-2.5.4.msi

download and install app engine sdk for pyton
http://googleappengine.googlecode.com/files/GoogleAppEngine_1.2.3.msi

2 scripts of interest in C:/tools/google/appengine
* dev_appserver.py, the development web server
* appcfg.py, for uploading your app to App Engine

C:/tools/google/appengine>mkdir helloworld
C:/tools/google/appengine>cd helloworld
C:/tools/google/appengine/helloworld>
C:/tools/google/appengine/helloworld>vi helloworld.py

copy this and paste in helloworld.py

print 'Content-Type: text/plain'
print ''
print 'Hello, world!'

C:/tools/google/appengine/helloworld>vi app.yaml

copy this and paste in app.yaml

application: helloworld
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: helloworld.py

start the webserver with following cmd

    google_appengine/dev_appserver.py helloworld/

QUESTION: how does dev_appserver.py knows where is helloworld/
on file system ?
ANSWER: dev_appserver.py calls dev_appserver_main.py

run dev_appserver.py with --debug argument

C:/tools/google/appengine>python dev_appserver.py --debug helloworld/
INFO     2009-06-21 13:05:43,641 appengine_rpc.py:157] Server: appengine.google.com
Allow dev_appserver to check for updates on startup? (Y/n): n
dev_appserver will not check for updates on startup.  To change this setting, edit C:/Documents and Settings/Administrator/.appcfg_nag
WARNING  2009-06-21 13:05:53,095 datastore_file_stub.py:404] Could not read datastore data from c:/temp/user/dev_appserver.datastore
WARNING  2009-06-21 13:05:53,095 datastore_file_stub.py:404] Could not read datastore data from c:/temp/user/dev_appserver.datastore.history
WARNING  2009-06-21 13:05:53,296 dev_appserver.py:3296] Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO     2009-06-21 13:05:53,426 dev_appserver_main.py:465] Running application helloworld on port 8080: http://localhost:8080
=====================================================
I've added following line
    print ("=========path=%s" % path);
on line# 393 (in main method)
seems like path is relative to the dir from where we are running script

if I run from c:/

C:/>python C:/tools/google/appengine/dev_appserver.py helloworld/
=========path=helloworld/app.yaml
=========path=helloworld/app.yml
ERROR:root:Application configuration file not found in helloworld/

It cant find it.

It works with absolute path

C:/>python C:/tools/google/appengine/dev_appserver.py C:/tools/google/appengine/helloworld/
=======path=C:/tools/google/appengine/helloworld/app.yaml
INFO     2009-06-21 13:13:06,308 appengine_rpc.py:157] Server: appengine.google.com
WARNING  2009-06-21 13:13:06,328 datastore_file_stub.py:404] Could not read datastore data from c:/temp/user/dev_appserver.datastore
WARNING  2009-06-21 13:13:06,328 datastore_file_stub.py:404] Could not read datastore data from c:/temp/user/dev_appserver.datastore.history
WARNING  2009-06-21 13:13:06,348 dev_appserver.py:3296] Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO     2009-06-21 13:13:06,378 dev_appserver_main.py:466] Running application helloworld on port 8080: http://localhost:8080

C:/tools/google/appengine>python dev_appserver.py --debug helloworld/
=========path=helloworld/app.yaml
============================
C:/tools/google/appengine>python dev_appserver.py --help
Runs a development application server for an application.

dev_appserver.py [options] 

Application root must be the path to the application to run in this server.
Must contain a valid app.yaml or app.yml file.

Options:
  --help, -h                 View this helpful message.
  --debug, -d                Use debug logging. (Default false)
  --clear_datastore, -c      Clear the Datastore on startup. (Default false)
  --address=ADDRESS, -a ADDRESS
                             Address to which this server should bind. (Default
                             localhost).
  --port=PORT, -p PORT       Port for the server to run on. (Default 8080)
  --datastore_path=PATH      Path to use for storing Datastore file stub data.
                             (Default c:/temp/user/dev_appserver.datastore)
  --history_path=PATH        Path to use for storing Datastore history.
                             (Default c:/temp/user/dev_appserver.datastore.history)
  --require_indexes          Disallows queries that require composite indexes
                             not defined in index.yaml.
  --smtp_host=HOSTNAME       SMTP host to send test mail to.  Leaving this
                             unset will disable SMTP mail sending.
                             (Default '')
  --smtp_port=PORT           SMTP port to send test mail to.
                             (Default 25)
  --smtp_user=USER           SMTP user to connect as.  Stub will only attempt
                             to login if this field is non-empty.
                             (Default '').
  --smtp_password=PASSWORD   Password for SMTP server.
                             (Default '')
  --enable_sendmail          Enable sendmail when SMTP not configured.
                             (Default false)
  --show_mail_body           Log the body of emails in mail stub.
                             (Default false)
  --auth_domain              Authorization domain that this app runs in.
                             (Default gmail.com)
  --debug_imports            Enables debug logging for module imports, showing
                             search paths used for finding modules and any
                             errors encountered during the import process.
  --allow_skipped_files      Allow access to files matched by app.yaml's
                             skipped_files (default False)
  --disable_static_caching   Never allow the browser to cache static files.
                             (Default enable if expiration set in app.yaml)


=====================================================
The web server is now running, listening for requests on port 8080.
Test the application by visiting the following URL in your web browser:

    * http://localhost:8080/

You can leave the web server running while you develop your application.
The web server knows to watch for changes in your source files and reload
them if necessary.

To shut down the web server, make sure the terminal window is active,
then press Control-C (or the appropriate "break" key for your console).

NOTE: if you get error
application configuration file not found

say you have an C:\MyDemo\HelloWorld dir which include 2 files:
helloworld.py and app.yaml

then you should go to C:\MyDemo rather than C:\MyDemo\HelloWorld to
perform the 
cmd("dev_appserver.py helloworld/").