Where's all my RAM?


So what is really going on?

Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory. The disk cache also known a dcache is an important part of your system that is generally split into two separate components, Buffers and Cache. Each of these portions of the memory store information about the topology of your disk. When an application or user makes a request for a piece of information on disk the Linux Kernel will first check these memory locations for information on that file or data storage facility. This allows the system to more accurately schedule and queue input and output to the physical drive while simultaneously improving the performance a user might experience. We can get a much better idea of how memory in a system is allocated by looking at the information provided by /proc on a system though there maybe infinitely more information here than some of you are ready to process, so in this article I will stick primarily to the information you might find when running free -m.

[[email protected] ~]# cat /proc/meminfo
MemTotal:        6120540 kB
MemFree:          169944 kB
Buffers:          162928 kB
Cached:          5008668 kB
SwapCached:          500 kB
Active:          3119404 kB
Inactive:        2572516 kB
Active(anon):     394940 kB
Inactive(anon):   129456 kB
Active(file):    2724464 kB
Inactive(file):  2443060 kB
Unevictable:        1880 kB
Mlocked:            1880 kB
SwapTotal:       7924728 kB
SwapFree:        7922836 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:        521724 kB
Mapped:            47920 kB
Shmem:              2492 kB
Slab:             193240 kB
SReclaimable:     175696 kB
SUnreclaim:        17544 kB
KernelStack:        3056 kB
PageTables:         7444 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10984996 kB
Committed_AS:    1598188 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      304620 kB
VmallocChunk:   34359428804 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6656 kB
DirectMap2M:     6275072 kB

Buffer Pages

The buffer cache ("Buffers:" in cat /proc/meminfo or free -m) is a close relative to the dentry/inode caches. The dentries and inodes in memory represent structures on disk, but are laid out very differently. When we need to fetch an inode or dentry to populate the caches, we must first pull in a page from the disk. This page has information on which those structures are represented. This information can not be a part of the page cache itself because it is not actually the contents of a file, but rather it is the raw contents of the disk. A page in the buffer cache may have dozens of on-disk inodes inside of it, although we may have only created an in-memory inode for one. The buffer has information that the kernel believe it will need not only for this inode or request but for others in the same group as well. This ultimate saves time and resources since the kernel will not have to make additional request to the disk to get information on these requested structures. There can be a variety of reasons that the system may choose to buffer an individual block or group of blocks but here are two common cases in which the kernel may create buffer pages.

  • When reading or writing pages of a file that are not stored in contiguous disk blocks.
    This happens either because the filesystem has allocated non-contiguous blocks to the file, or because the file contains "holes"
  • When accessing a single disk block.
    This can happen when reading a superblock or an inode block.

Raw disk operations such as dd use buffers.

[[email protected] ~]# free -m
              total       used       free     shared    buffers     cached
Mem:          5977       5811        165          0        159       4891
-/+ buffers/cache:        760       5216
Swap:         7738          1       7737

For example reading 10M of raw disk blocks.

[[email protected] ~]# dd if=/dev/sda6 of=/dev/zero bs=1024k count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.209051 seconds, 50.2 MB/s

Buffers size increased by 10MB (159M-169M).

[[email protected] ~]# free -m
              total       used       free     shared    buffers     cached
Mem:          5977       5811        165          0        169       4891
-/+ buffers/cache:        760       5216
Swap:         7738          1       7737

Cached

When ever files or blocks of data are accessed frequently on a system the Linux Kernel will cache information about that block or file so that future request for it can be processed more efficiently. This can include the Dentry Cache, Inode Cache, and Page Cache which each store some component information about those blocks or files.

 

Page Cache

The Linux Page Cache ("Cached:" in cat /proc/meminfo or free -m) is the largest single consumer of RAM on most systems. It is not uncommon to see this memory segment exceed 50% of the total available RAM on a healthy system.

Any time you or an application request a file be read from the disk that data is first read into memory and then goes into the page cache. After this initial read is completed, the kernel has the option to simply throw the page away since it is not being used or store that information in memory for later retrieval. If the kernel elects to keep this information in your cache and a request for data from the same area in that file is again requested by any application the data can be read directly out of memory. This shortcut allows the system to reduce disk I/O and significantly speed-up operations as the kernel will not need to schedule an additional read from the disk to retrieve that information again. This is the reason Linux uses its page cache so extensively. The kernel is effectively betting that after you or an application accesses a page on disk a single time, you will soon access it again.

The same is true for mmap()'d files ("Mapped:" cat /proc/meminfo or free -m). The first time the mmap()'d area is accessed, the page will be brought in from the disk and mapped into memory. The kernel could choose to immediately discard that page after the instruction that touched the page has completed. However, the kernel makes the same bets that it did for simple reads of a file. It keeps the page mapped into memory and bets that you or an application will soon access it again. This behavior at times can manifest itself in confusing ways.

It might be assumed that mmap()'d memory is not "Cached" because it is in active use and that "Cached" means "Completely unused right now". However, Linux itself does not define it that way. The Linux definition of "Cached" is closer to "This is a copy of data from the disk that we have here to save you time." It implies nothing about how the page is actually being used. This is why we have both "Cached:" and "Mapped:" in cat /proc/meminfo. All "Mapped:" memory is "Cached:", but not all "Cached:" memory is "Mapped:".

 

Dentry/Inode Cache

Each time you do an ls (or any other operation: open, stat, etc...) on a filesystem, the kernel needs to obtain data which  may be stored on the disk. The kernel parses that data and puts it in some filesystem-independent structures so that it can be handled in the same way across all filesystems. In the same fashion as the page cache in the above examples, the kernel has the option of throwing away these structures once the ls is completed. However, it makes the same bets as before: if you or that application has read it once, you're bound to read it again. The kernel stores this information in several "Cached:" segments called the dentry and inode caches. dentries are common across all filesystems, but each filesystem has its own cache for inodes. These individual components store information about the location of a file on disk such as the block and sector ID. These structures also store additional information such as the human readable path to a file, the owner of that file, and it's permission set.

You can view the different caches and their sizes by executing this command.
(This RAM is a component of "Slab:" in cat /proc/meminfo)

[[email protected] ~]head -2 /proc/slabinfo; cat /proc/slabinfo | egrep dentry\|inode
slabinfo - version: 2.1
# name            <active_objs> <num_objs>    : tunables    : slabdata <active_slabs> <num_slabs> 
ext4_inode_cache   12171  15411    984   33    8 : tunables    0    0    0 : slabdata    467    467      0
mqueue_inode_cache     36     36    896   36    8 : tunables    0    0    0 : slabdata      1      1      0
hugetlbfs_inode_cache     26     26    616   26    4 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache     434    460    704   46    8 : tunables    0    0    0 : slabdata     10     10      0
shmem_inode_cache    997   1000    808   40    8 : tunables    0    0    0 : slabdata     25     25      0
proc_inode_cache     873    925    648   25    4 : tunables    0    0    0 : slabdata     37     37      0
inode_cache         1116   1323    600   27    4 : tunables    0    0    0 : slabdata     49     49      0
dentry             18542  18564    192   42    2 : tunables    0    0    0 : slabdata    442    442      0

Older kernels (around 2.6.9) left some structures in the slab cache for longer periods of time than newer kernels do. That means that even though they may have been unused, they're left there until there is sufficient memory pressure. This happens more frequently with proc_inodes. /proc inodes also pin task_structs, which means that each one can effectively occupy over 2 KBytes of RAM. This RAM won't show up as "Cached:" and may appear to be a kernel memory leak. On a system with only 100 tasks (with little memory pressure) there can be hundreds of thousands of these left lying around though this is a normal condition.

 

What if I want to run more applications?

Run as many applications as you want to, the disk cache in no way affects the over availability of memory on a system but it can certainly affect the overall performance especially if your system is experiencing a memory related performance problem. If your applications want more memory, they simply take back a chunk that the disk cache has effectively borrowed. Disk cache can always be given back to applications immediately!

 

How do I stop Linux from doing this?

You can't really disable disk caching as it is part of the core virtual memory systems of Linux. You can however control caching to some degree with kernel level parameters if you feel it is really required. In most cases the only reason anyone ever wants to disable disk caching is because they don't really understand what those segments of memory are being used for or how the system can choose to allocate or de-allocate those segments. Disk caches by design are meant to make applications load faster and run smoother.

 

Why does top and free say all the ram is used?

This is just a misunderstanding of how Linux classifies memory usage. In most cases you will find that both you and the Linux kernel actually agree that memory taken by an application is "Used", while memory that isn't used for anything is "Free". So what do you call memory that is both used for something and available to applications, such as the disk cache? Most people would likely call it "Free Memory" but Linux classifies it as "Used Memory" which often leads to confusion.

Memory that is You would call it Linux calls it
taken by applications Used Used
available for applications, and used for something Free Used
not used for anything Free Free

 

How do I see how much free ram I really have?

To see how much ram is free to use for your applications, run free -m and look at the row that says "-/+ buffers/cache" in the column that says "free". That is your answer in megabytes.

[[email protected] ~]# free -m
              total       used       free     shared    buffers     cached
Mem:          5977       5811        165          0        159       4891
-/+ buffers/cache:        760       5216
Swap:         7738          1       7737

Some information sourced from http://www.linuxatemyram.com/