Understand output of vmstat memory utilization

Question

I have a solaris box and im trying to know whether its running out of memory or if its stable.

below is the output of vmstat.

 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr vc vc vc vc   in   sy   cs us sy id
 1 0 0 11426696 4603520 613 1477 449 6 6 0 0 78 22 28 29 8970 37714 22961 43 6 51
 4 0 0 4975280    0 1747 3487 805 0 0 0  0 233 41 33 44 9558 53713 15845 74 8 18
 4 0 0 4936944    0 933 1837 0  0  0  0  0 56 28 12 39 9317 46898 14648 82 7 11
 5 0 0 4943080    0 1056 2806 805 0 0 0  0 103 21 18 18 9286 46900 14866 78 8 14
 5 0 0 4942264    0 1088 2173 804 6 6 0  0 109 8 40 31 9927 56484 16495 84 8 8
 3 0 0 4942520    0 308 1018 1756 3 3 0  0 166 87 29 44 10638 64146 21413 83 9 8
 0 0 0 4942512    0 156 326 1740 0 0  0  0 370 12 33 52 11554 40375 21897 75 9 16
 2 0 0 4947384    0 294 560 845 0  0  0  0 121 18 23 20 9445 52382 17016 77 6 17

I can see the free column shows 0 however the sr column also shows 0

And output from top command doesn't show how much free memory available. Swap shows 0.0%

load averages:  11.4,  9.12,  9.24;                   
9021 processes: 9018 sleeping, 1 running, 2 on cpu
CPU states:  0.0% idle, 71.4% user, 28.6% kernel,  0.0% iowait,  0.0% swap
Memory: 24G phys mem, 16G total swap, 13G free swap

Am i running out of RAM? Please suggest how to interpret this data. Do i need to increase my physical memory?

Appreciate some insights.

Answer 1

This can help you https://www.howtogeek.com/424334/how-to-use-the-vmstat-command-on-linux/. There is explanation of those shorts.

Memory

swpd: the amount of virtual memory used. In other words, how much memory has been swapped out.,
free: the amount of idle (currently unused) memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.

Swap

si: Amount of virtual memory swapped in from swap space.
so: Amount of virtual memory swapped out to swap space.

IO

bi: Blocks received from a block device. The number of data blocks used to swap virtual memory back into RAM.
bo: Blocks sent to a block device. The number of data blocks used to swap virtual memory out of RAM and into swap space.

System

in: The number of interrupts per second, including the clock.
cs: The number of context switches per second. A context switch is when the kernel swaps from system mode processing into user mode processing.

Answer 2

From the Solaris 11.4 vmstat man page, there's one important thing to note:

Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.

That also applies to the first line of output from Solaris vmstat: it's a summary of all activity since the system was booted.

A good description of the output fields is found in the EXAMPLES section of the Solaris man vmstat page:

Examples

Example 1 Using vmstat

The following command displays a summary of what the system is doing every five seconds.

example% vmstat 5

kthr   memory          page             disk      faults        cpu
r b w swap  free re mf pi p fr de sr s0 s1 s2 s3  in  sy  cs us sy id
0 0 0 11456 4120 1  41 19 1  3  0  2  0  4  0  0  48 112 130  4 14 82
0 0 1 10132 4280 0   4 44 0  0  0  0  0 23  0  0 211 230 144  3 35 62
0 0 1 10132 4616 0   0 20 0  0  0  0  0 19  0  0 150 172 146  3 33 64
0 0 1 10132 5292 0   0  9 0  0  0  0  0 21  0  0 165 105 130  1 21 78
1 1 1 10132 5496 0   0  5 0  0  0  0  0 23  0  0 183  92 134  1 20 79
1 0 1 10132 5564 0   0 25 0  0  0  0  0 18  0  0 131 231 116  4 34 62
1 0 1 10124 5412 0   0 37 0  0  0  0  0 22  0  0 166 179 118  1 33 67
1 0 1 10124 5236 0   0 24 0  0  0  0  0 14  0  0 109 243 113  4 56 39

example%

The fields of vmstat's display are

kthr

Report the number of kernel threads in each of the three following states:

r

    the number of kernel threads in run queue
b

    the number of blocked kernel threads that are waiting for
    resources I/O, paging, and so forth
w

    the number of swapped out lightweight processes (LWPs) that
    are waiting for processing resources to finish.

memory

Report on usage of virtual and real memory.

swap

    available swap space (Kbytes)
free

    size of the free list (Kbytes)

page

Report information about page faults and paging activity. The information on each of the following activities is given in units per second.

re

    page reclaims — but see the –S option for how this field is modified.
mf

    minor faults — but see the –S option for how this field is modified.
pi

    kilobytes paged in
po

    kilobytes paged out
fr

    kilobytes freed
de

    anticipated short-term memory shortfall (Kbytes)
sr

    pages scanned by clock algorithm

When executed in a zone and if the pools facility is active, all of the above (except for ‘de’) only report activity on the processors in the processor set of the zone's pool.

disk

Report the number of disk operations per second. There are slots for up to four disks, labeled with a single letter and number. The letter indicates the type of disk (s = SCSI, i = IPI, and so forth); the number is the logical unit number.

faults

Report the trap/interrupt rates (per second).

in

    interrupts
sy

    system calls
cs

    CPU context switches

When executed in a zone and if the pools facility is active, all of the above only report activity on the processors in the processor set of the zone's pool.

cpu

Give a breakdown of percentage usage of CPU time. On MP systems, this is an average across all processors.

us

    user time
sy

    system time
id

    idle time

When executed in a zone and if the pools facility is active, all of the above only report activity on the processors in the processor set of the zone's pool.

Answer 3

"0" is not a valid free memory value.

By design, Solaris always makes sure a minimal amount of free memory is available. The fact the sr column is also equals to zero suggests there is no memory shortage. In any case, you wouldn't have been able to run vmstat or top in the first place with such an extreme RAM shortage.

You should investigate further to understand why the free memory is reported a zero. mdb's ::memstat command would be a good start:

# echo "::memstat" | mdb -k