1. top
2. free -m
3. less /proc/meminfo
4. vmstat -s
5. sar
6. ps
7. smem
8. process level info - pmap
9. scripts to gather memory info on Linux
==================
1. top
==================
top command displays dynamic real-time view of system information and running tasks.
top - 14:58:12 up 8 days, 22:01, 6 users, load average: 10.34, 10.31, 10.35
Tasks: 406 total, 10 running, 396 sleeping, 0 stopped, 0 zombie
Cpu(s): 58.4%us, 4.3%sy, 0.0%ni, 36.9%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 14364508k total, 14315820k used, 48688k free, 153900k buffers
Swap: 2097144k total, 2097136k used, 8k free, 1090736k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11041 iu 15 0 240m 4416 2844 S 100.6 0.0 492:23.53 notification.ex
1978 root 25 0 729m 681m 664 R 99.2 4.9 4791:17 aide
15690 root 25 0 931m 883m 664 R 99.2 6.3 7658:11 aide
15875 root 25 0 1102m 1.0g 664 R 99.2 7.5 10527:19 aide
29496 root 25 0 1178m 1.1g 668 R 99.2 8.1 11962:01 aide
31509 root 25 0 1019m 972m 668 R 99.2 6.9 9091:25 aide
12776 root 25 0 607m 559m 664 R 98.9 4.0 3358:35 aide
12936 root 25 0 834m 786m 664 R 98.9 5.6 6212:07 aide
29103 root 25 0 456m 408m 664 R 98.9 2.9 1927:31 aide
16361 root 25 0 232m 184m 720 R 98.2 1.3 425:50.84 aide
11779 iu 22 0 5247m 2.8g 6860 S 5.3 20.5 35:42.43 java
32115 oracle 15 0 2256m 73m 70m S 2.0 0.5 0:16.31 oracle
4377 root 34 19 0 0 0 S 1.0 0.0 166:01.26 kipmi0
11072 iu 15 0 241m 6104 2796 S 0.7 0.0 0:48.57 notification.ex
11386 oracle 15 0 90120 1788 1044 S 0.3 0.0 0:00.14 sshd
PID – ID of the process
USER – The user that is the owner of the process
PR – priority of the process
NI – The “NICE” value of the process
VIRT – virtual memory used by the process
RES – physical memory used from the process
SHR – shared memory of the process
S – indicates the status of the process: S=sleep R=running Z=zombie
%CPU – This is the percentage of CPU used by this process
%MEM – This is the percentage of RAM used by the process
TIME+ –This is the total time of activity of this process
COMMAND – And this is the name of the process
to run top in batch:
top -b -n1 > top.log
top does one iteration, and exists.
To sort by memory:
SHIFT+M
To sort by any field:
SHIFT+F => Select field => Enter
==================
2. free -m
==================
free
free command displays amount of total, free and used physical memory (RAM) in the system, as well as showing information on shared memory, buffers, cached memory and swap space used by the Linux kernel.
By default the output is in Kb
free -m The output is in Mb
free -g The output is in Bb
free -m The output is in Mb
free -mt The output is in Mb with totals
free -mto The output is in Mb with totals
free -ms 5 Run free -m every 5 seconds
watch free - Run free every 2 seconds (default)
watch -n 1 -d free - Run free every 1 second, and highlight difference between runs
Understanding free output
~>free -m
total used free shared buffers cached
Mem: 14027 13984 43 0 166 1077
-/+ buffers/cache: 12739 1288
Swap: 2047 2047 0
total: Server total (physical) RAM (excluding a small bit that the kernel.
used: memory in use by the OS.
free: memory not in use.
shared, buffers, cached: This shows memory usage for specific purposes.
These values already included in "used".
shared - ?
buffers - Memory used for file system metadata.
cached - Pages with actual contents of files or block devices.
Pre allocation of bufferes and cached memory RAM helps the system to run faster because disk information is already in memory which saves I/O operations.
-/+ buffers/cache: Adjustment of values in first line. These values are more meaningful than the first line.
12739 = 13984 - (166+1077)
1288 = 43 + (166+1077)Swap: gives information about swap space usage
Linux (like most OS) will always try to use free RAM for caching.
So Mem: free will almost always be very low.
Therefore the line -/+ buffers/cache:free shows how much memory is free when ignoring caches.
caches will be freed automatically if memory gets scarce, so they do not really matter.
A Linux system is really low on memory if the free value in -/+ buffers/cache: line gets low.
==================
3. less /proc/meminfo
==================
The /proc file system does not contain real files.
They are rather virtual files that contain dynamic information about the kernel and the system.
less /proc/meminfo
MemTotal: 14364508 kB
MemFree: 3994220 kB
Buffers: 551096 kB
Cached: 2424648 kB
SwapCached: 379240 kB
Active: 6037092 kB
Inactive: 3469056 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 14364508 kB
LowFree: 3994220 kB
SwapTotal: 2097144 kB
SwapFree: 619160 kB
Dirty: 2340 kB
Writeback: 64 kB
AnonPages: 6157432 kB
Mapped: 750808 kB
Slab: 675316 kB
PageTables: 137172 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 9279396 kB
Committed_AS: 14622668 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 264892 kB
VmallocChunk: 34359473231 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
==================
4. vmstat
==================
vmstat reports report virtual memory statistics.
The information is about processes, swap, free, buffer and cache memory, paging space.
vmstat
The command will display report based on averages since last reboot.
vmstat 5
vmstat reports report virtual memory statistics.
The information is about processes, swap, free, buffer and cache memory, paging space.
vmstat
The command will display report based on averages since last reboot.
vmstat 5
The command will pool average system resources usage level for a sampling period of 5 seconds at interval of 5 seconds, except the first result that is averages since the last reboot.
vmstat -s
The output of vmstat -s is similar to /proc/meminfo
The output of vmstat -s is similar to /proc/meminfo
Default units are Kb
vmstat
procs -----------memory------------ ---swap-- -----io---- --system----- ---cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 2835188 38428 16336 6560160 0 0 51 70 0 0 7 6 85 2 0
vmstat -s
procs -----------memory------------ ---swap-- -----io---- --system----- ---cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 2835188 38428 16336 6560160 0 0 51 70 0 0 7 6 85 2 0
14364508 total memory
10392404 used memory
6060160 active memory
3469716 inactive memory
3972104 free memory
553064 buffer memory
2439024 swap cache
2097144 total swap
1465904 used swap
631240 free swap
377010091 non-nice user cpu ticks
2975 nice user cpu ticks
59577453 system cpu ticks
788782124 idle cpu ticks
18921537 IO-wait cpu ticks
29664 IRQ cpu ticks
176165 softirq cpu ticks
0 stolen cpu ticks
312184009 pages paged in
371292265 pages paged out
2711222 pages swapped in
3370874 pages swapped out
917256990 interrupts
1512285390 CPU context switches
1417539417 boot time
14021020 forks
==================
5. sar
==================
SAR stands for System Activity Reporter
It is part of sysstat installation.
It is running at intervals by crontab, samples system information, and logs that information into files.
One file per day.
One file per day.
sa01 for first day of the month, sa22 - for 22nd day of month, and so on.
By default - it is running every 10 minutes, and stores information for 7 days.
configuration file: /etc/sysconfig/sysstat
crontab file: /etc/cron.d/sysstat
files are written to: /var/log/sa/ folder
cd /var/log/sa
>% ls -1
sa18
sa19
sa20
sa21
sa22
When invoking sar - it is reading gathered information from pre generated binary files.
In the above output most of the columns are self explanatory(most of the outputs are in KB).
Values
|
Meaning
|
kbmemfree
|
Amount of free memory available in kilobytes
|
kbmemused
|
Amount of used memory in kilobytes
|
%memused
|
Percentage of used memory
|
kbbuffers
|
Amount of memory used as buffers by the kernel in kilobytes
|
kbcached
|
Amount of memory used to cache data by the kernel in kilobytes
|
kbswpfree
|
Amount of free swap space in kilobytes
|
kbswpused
|
Amount of used swap space in kilobytes
|
%swpused
|
Percentage of used swap space
|
kbswpcad
|
Amount of cached swap memory in kilobytes.
|
Additional fields:
kbcommit: Memory in kilobytes needed for current workload.%commit: Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap).
Some examples:
sar with no parameter - fetch memory info for current day.
>% sar
Linux 2.6.18-274.el5 (my_server) 03/26/2015
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 0.33 0.00 0.31 0.64 0.00 98.72
12:20:01 AM all 0.31 0.00 0.30 0.33 0.00 99.06
12:30:01 AM all 0.29 0.00 0.30 0.28 0.00 99.13
sar -f sa<day> fetch memory info for that day.
>% cd /var/log/sa>% sar -f sa22
Linux 2.6.18-274.el5 (my_server) 03/22/2015
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 0.36 0.00 0.31 0.67 0.00 98.65
12:20:01 AM all 0.35 0.00 0.30 0.36 0.00 98.99
12:30:01 AM all 0.31 0.00 0.29 0.27 0.00 99.12
12:40:01 AM all 0.31 0.00 0.29 0.30 0.00 99.10
sar -r sa<day> fetch extended memory info for that day.
sar -r -f sa22
Linux 2.6.18-274.el5 (my_server) 03/22/2015
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
12:10:01 AM 459976 9765036 95.50 195912 8156376 8889112 1596640 15.23 285264
12:20:01 AM 418156 9806856 95.91 199908 8195956 8889132 1596620 15.23 285312
12:30:01 AM 399384 9825628 96.09 203832 8205748 8889172 1596580 15.23 285272
12:40:01 AM 356488 9868524 96.51 207860 8248572 8889172 1596580 15.23 285272
sar -r
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
05:40:01 AM 515704 65157200 99.21 8 52898780 22709820 27.90 18171748 40886116 932
05:50:01 AM 546036 65126868 99.17 8 52869684 22709452 27.90 18144272 40884768 984
06:00:01 AM 515536 65157368 99.21 8 52838316 22783136 27.99 18149564 40889020 1720
06:10:01 AM 37281288 28391616 43.23 8 16407268 22695028 27.88 18133764 4423772 1064
Average: 25449404 40223500 61.25 8 28122914 22756175 27.96 18436755 15869233 1245
kbmemfree - Amount of free memory available in kilobytes.
kbmemused - Amount of used memory in kilobytes.
This does not take into account memory used by the kernel itself.
%memused - Percentage of used memory.
kbbuffers - Amount of memory used as buffers by the kernel in kilobytes.
kbmemused - Amount of used memory in kilobytes.
This does not take into account memory used by the kernel itself.
%memused - Percentage of used memory.
kbbuffers - Amount of memory used as buffers by the kernel in kilobytes.
kbcached - Amount of memory used to cache data by the kernel in kilobytes.
kbcommit - Amount of memory in kilobytes needed for current workload.
kbcommit - Amount of memory in kilobytes needed for current workload.
This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory.
%commit - Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap). This number may be greater than 100% because the kernel usually overcommits memory.
kbactive - Amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).
kbactive - Amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).
kbinact - Amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes).
For a specific day, say 22:
sar - r -f /var/log/sa/sa21
See SWAP memory usage:
sar -r for current day
sar -r -f sa14 for specific day:
sar -r for current day
sar -r -f sa14 for specific day:
Or
root@my_server:/var/log/sa>% sar -S -f sa14
Linux 2.6.32-504.8.1.el6.x86_64 (ind-ida-6-dbu-1) 05/14/2016 _x86_64_ (12 CPU)
12:00:01 AM kbswpfree kbswpused %swpused kbswpcad %swpcad
12:10:01 AM 10485756 0 0.00 0 0.00
12:20:01 AM 10485756 0 0.00 0 0.00
12:30:01 AM 10485756 0 0.00 0 0.00
12:40:01 AM 10485756 0 0.00 0 0.00
12:50:01 AM 10485756 0 0.00 0 0.00
01:00:01 AM 10485756 0 0.00 0 0.00
01:10:01 AM 10485756 0 0.00 0 0.00
01:20:01 AM 10485756 0 0.00 0 0.00
01:30:01 AM 10485756 0 0.00 0 0.00
01:40:01 AM 10485756 0 0.00 0 0.00
01:50:01 AM 10485756 0 0.00 0 0.00
02:00:01 AM 10485756 0 0.00 0 0.00
02:10:01 AM 10485756 0 0.00 0 0.00
02:20:01 AM 10485756 0 0.00 0 0.00
02:30:01 AM 10485756 0 0.00 0 0.00
02:40:01 AM 10485756 0 0.00 0 0.00
02:50:01 AM 10485756 0 0.00 0 0.00
03:00:01 AM 10485756 0 0.00 0 0.00
03:10:01 AM 10485756 0 0.00 0 0.00
03:20:01 AM 10485756 0 0.00 0 0.00
03:30:01 AM 10485756 0 0.00 0 0.00
03:40:01 AM 10485756 0 0.00 0 0.00
03:50:01 AM 10485756 0 0.00 0 0.00
04:00:01 AM 10485756 0 0.00 0 0.00
For more options, there is this nice reference: sar reference
==================
6. ps
==================
Useful ps References:
www.computerhope.com
ps options may have three formats:
1 UNIX options, which may be grouped and must be preceeded by a dash.
2 BSD options, which may be grouped and must not be used with a dash.
3 GNU long options, which are preceeded by two dashes.
Some useful options
ps command reports a snapshot on information of the current active processes.
ps
ps -a
List all running processes of current user. (Only processes that are attached to a terminal.)
ps -au
List all running processes of current user. (Whether or not they're attached to a terminal.)
ps -ax
List all running processes of all users.
ps -ax<user> or ps -ax <user>
List all running processes of user <user>
ps -aux
List all running processes, with additional information about their resource usage.
ps -eo pid,ppid,rss,size,vsize,pcpu,pmem,cmd -ww --sort=vsize
List all running processes, with additional information about their resource usage, and sort by virtual memory.
ps -eo pid,ppid,user,args,rsz,vsz,cmd --sort user
ps -eo pid,ppid,user,args,rsz,vsz,cmd --sort rsz
ps -eo pid,ppid,user,args,rsz,vsz,cmd --sort rsz
List all running processes, with additional information about their resource usage, and sort by user.
To see only the memory resources occupied by each category of processes, such as Apache httpd, MySQL mysqld or Java, use the following command:
ps -aux | awk '{print $4"\t"$11}' | sort | uniq -c | awk '{print $2" "$1" "$3}' | sort -nr
pid = Process ID
ppid = Parent Process ID
rss = Resident Set Size. It is the portion of memory occupied by a process that is held in main memory (RAM).
vsize = Virtual Memory size of entire process, in Kb.
size = The virtual size of the data section of the process.
pmem = ratio of the process’s rss to the physical memory on the machine.
pcpu = It is the CPU time used divided by the time the process has been running.
args = Command command with all its arguments as a string.
args = Command command with all its arguments as a string.
==================
7. smem
==================
smem is a utility to view physical memory usage.
smem reports physical memory usage.
It does not include memory that has been swapped out to disk.
It does not include memory that has been swapped out to disk.
It can report USS or PSS
USS - Unique Set Size - Unshared memory
PSS - Proportional Set Size - Unshared memory + process's proportion of shared memory
USS is the process unshared memory.
PSS is process USS + process's proportion of shared memory
usage:
smem -t -s uss
====================================
8. process level info - pmap
====================================
When ps is reporting on a memory,rss,size,vsize it is not possible to determine the actual process memory, from the loaded shared libraries.
To see the actual process memory usage, there is a command pmap
This is a sample of pmap -d <pid> output
root@hostname:~>% pmap -d 411
411: /starhome/igate/hostname ig1 vip_monitor.real.exe --PROCESSNAME=AUT-MOBOV-99_MON_IG2_1
Address Kbytes Mode Offset Device Mapping
0000000000400000 736 r-x-- 0000000000000000 0fd:0000c vip_monitor.real.exe
00000000006b8000 12 rw--- 00000000000b8000 0fd:0000c vip_monitor.real.exe
00000000006bb000 4 rw--- 00000000006bb000 000:00000 [ anon ]
000000001d600000 1672 rw--- 000000001d600000 000:00000 [ anon ]
00000032b3a00000 112 r-x-- 0000000000000000 0fd:00001 ld-2.5.so
00000032b3c1b000 4 r---- 000000000001b000 0fd:00001 ld-2.5.so
00000032b3c1c000 4 rw--- 000000000001c000 0fd:00001 ld-2.5.so
00000032b3e00000 80 r-x-- 0000000000000000 0fd:00002 libz.so.1.2.3
00000032b3e14000 2044 ----- 0000000000014000 0fd:00002 libz.so.1.2.3
00000032b4013000 4 rw--- 0000000000013000 0fd:00002 libz.so.1.2.3
00000032b4200000 1336 r-x-- 0000000000000000 0fd:00001 libc-2.5.so
00000032b434e000 2044 ----- 000000000014e000 0fd:00001 libc-2.5.so
00000032b454d000 16 r---- 000000000014d000 0fd:00001 libc-2.5.so
00000032b4551000 4 rw--- 0000000000151000 0fd:00001 libc-2.5.so
00002b82af0b8000 40 r-x-- 0000000000000000 0fd:00001 libnss_files-2.5.so
00002b82af0c2000 2044 ----- 000000000000a000 0fd:00001 libnss_files-2.5.so
00002b82af2c1000 4 r---- 0000000000009000 0fd:00001 libnss_files-2.5.so
00002b82af2c2000 4 rw--- 000000000000a000 0fd:00001 libnss_files-2.5.so
00002b82af2c3000 400 r---- 0000000000000000 0fd:00008 timezlrg.dat
00002b82af327000 32 r-x-- 0000000000000000 0fd:00008 libnque11.so
00002b82af32f000 1020 ----- 0000000000008000 0fd:00008 libnque11.so
00002b82af42e000 4 rw--- 0000000000007000 0fd:00008 libnque11.so
00002b82af42f000 332 rw--- 00002b82af42f000 000:00000 [ anon ]
00002b82af482000 55124 r---- 0000000000000000 0fd:00002 locale-archive
00007fff30766000 76 rwx-- 00007ffffffea000 000:00000 [ stack ]
00007fff30779000 8 rw--- 00007fffffffd000 000:00000 [ anon ]
ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ]
...
...
mapped: 215452K writeable/private: 22944K shared: 0K
Each shared library is listed twice; once for its code segment and once for its data segment. The code segments have a mode of "r-x--", while the data is set to "rw---".
The writeable/private part in the last line "mapped: 215452K writeable/private: 22944K" is a summary of the process memory segments, without shared liraries segments.. This is the actual process memory consumption.
Compare this to the output of ps aux
root@hostname:~>% ps aux | grep 411 | grep my_user
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
my_user 411 0.0 0.1 207260 33076 ? S Jan19 0:00 /starhome/igate/aut-mob-ovm ig1 vip_monitor.real.exe --PROCESSNAME=AUT-MOBOV-99_MON_IG2_1
RSS VSZ
pmap 22944Kb 215452Kb
ps -aux 33076Kb 207260Kb
9. scripts to gather memory info on Linux
====================================
Collect data from top, free, ps commands into a log
#!/bin/bash
touch top_trace.log
touch free.log
touch sorted_processes.log
while [ 1 -eq 1 ] ; do
date >> top_trace.log
top -b -n 1 | head -30 >> top_trace.log
date >> free.log
free >> free.log
date >> sorted_processes.log
ps -eo pid,ppid,rss,size,vsize,pcpu,pmem,cmd -ww --sort=vsize > sorted_processes.log
sleep 10
#------------------------------------------
#prevent the log files from growing forever
#------------------------------------------
mv -f `find . -name "sorted_processes.log" -size +1000000k` sorted_processes.log_bak
mv -f `find . -name "top_trace.log" -size +1000000k` top_trace.log_bak
mv -f `find . -name "free.log" -size +1000000k` free.log_bak
==================
References
==================
No comments:
Post a Comment