Where's all my RAM?
Luke Hebert
- 10 minutes read - 1970 wordsSo what is really going on?
Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory. The disk cache also known a dcache is an important part of your system that is generally split into two separate components, Buffers and Cache. Each of these portions of the memory store information about the topology of your disk. When an application or user makes a request for a piece of information on disk the Linux Kernel will first check these memory locations for information on that file or data storage facility. This allows the system to more accurately schedule and queue input and output to the physical drive while simultaneously improving the performance a user might experience. We can get a much better idea of how memory in a system is allocated by looking at the information provided by /proc on a system though there maybe infinitely more information here than some of you are ready to process, so in this article I will stick primarily to the information you might find when running free -m.
[root@server ~]# cat /proc/meminfo MemTotal: 6120540 kB MemFree: 169944 kB Buffers: 162928 kB Cached: 5008668 kB SwapCached: 500 kB Active: 3119404 kB Inactive: 2572516 kB Active(anon): 394940 kB Inactive(anon): 129456 kB Active(file): 2724464 kB Inactive(file): 2443060 kB Unevictable: 1880 kB Mlocked: 1880 kB SwapTotal: 7924728 kB SwapFree: 7922836 kB Dirty: 16 kB Writeback: 0 kB AnonPages: 521724 kB Mapped: 47920 kB Shmem: 2492 kB Slab: 193240 kB SReclaimable: 175696 kB SUnreclaim: 17544 kB KernelStack: 3056 kB PageTables: 7444 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10984996 kB Committed_AS: 1598188 kB VmallocTotal: 34359738367 kB VmallocUsed: 304620 kB VmallocChunk: 34359428804 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 6656 kB DirectMap2M: 6275072 kB
Buffer Pages
The buffer cache (“Buffers:” in cat /proc/meminfo or free -m) is a close relative to the dentry/inode caches. The dentries and inodes in memory represent structures on disk, but are laid out very differently. When we need to fetch an inode or dentry to populate the caches, we must first pull in a page from the disk. This page has information on which those structures are represented. This information can not be a part of the page cache itself because it is not actually the contents of a file, but rather it is the raw contents of the disk. A page in the buffer cache may have dozens of on-disk inodes inside of it, although we may have only created an in-memory inode for one. The buffer has information that the kernel believe it will need not only for this inode or request but for others in the same group as well. This ultimate saves time and resources since the kernel will not have to make additional request to the disk to get information on these requested structures. There can be a variety of reasons that the system may choose to buffer an individual block or group of blocks but here are two common cases in which the kernel may create buffer pages.
- When reading or writing pages of a file that are not stored in contiguous disk blocks. This happens either because the filesystem has allocated non-contiguous blocks to the file, or because the file contains “holes”
- When accessing a single disk block. This can happen when reading a superblock or an inode block.
Raw disk operations such as dd use buffers.
[root@server ~]# free -m total used free shared buffers cached Mem: 5977 5811 165 0 159 4891 -/+ buffers/cache: 760 5216 Swap: 7738 1 7737
For example reading 10M of raw disk blocks.
[root@server ~]# dd if=/dev/sda6 of=/dev/zero bs=1024k count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.209051 seconds, 50.2 MB/s
Buffers size increased by 10MB (159M-169M).
[root@server ~]# free -m total used free shared buffers cached Mem: 5977 5811 165 0 169 4891 -/+ buffers/cache: 760 5216 Swap: 7738 1 7737
Cached
When ever files or blocks of data are accessed frequently on a system the Linux Kernel will cache information about that block or file so that future request for it can be processed more efficiently. This can include the Dentry Cache, Inode Cache, and Page Cache which each store some component information about those blocks or files.
Page Cache
The Linux Page Cache (“Cached:” in cat /proc/meminfo or free -m) is the largest single consumer of RAM on most systems. It is not uncommon to see this memory segment exceed 50% of the total available RAM on a healthy system.
Any time you or an application request a file be read from the disk that data is first read into memory and then goes into the page cache. After this initial read is completed, the kernel has the option to simply throw the page away since it is not being used or store that information in memory for later retrieval. If the kernel elects to keep this information in your cache and a request for data from the same area in that file is again requested by any application the data can be read directly out of memory. This shortcut allows the system to reduce disk I/O and significantly speed-up operations as the kernel will not need to schedule an additional read from the disk to retrieve that information again. This is the reason Linux uses its page cache so extensively. The kernel is effectively betting that after you or an application accesses a page on disk a single time, you will soon access it again.
The same is true for mmap()'d files (“Mapped:” cat /proc/meminfo or free -m). The first time the mmap()'d area is accessed, the page will be brought in from the disk and mapped into memory. The kernel could choose to immediately discard that page after the instruction that touched the page has completed. However, the kernel makes the same bets that it did for simple reads of a file. It keeps the page mapped into memory and bets that you or an application will soon access it again. This behavior at times can manifest itself in confusing ways.
It might be assumed that mmap()'d memory is not “Cached” because it is in active use and that “Cached” means “Completely unused right now”. However, Linux itself does not define it that way. The Linux definition of “Cached” is closer to “This is a copy of data from the disk that we have here to save you time.” It implies nothing about how the page is actually being used. This is why we have both “Cached:” and “Mapped:” in cat /proc/meminfo. All “Mapped:” memory is “Cached:”, but not all “Cached:” memory is “Mapped:”.
Dentry/Inode Cache
Each time you do an ls (or any other operation: open, stat, etc…) on a filesystem, the kernel needs to obtain data which may be stored on the disk. The kernel parses that data and puts it in some filesystem-independent structures so that it can be handled in the same way across all filesystems. In the same fashion as the page cache in the above examples, the kernel has the option of throwing away these structures once the ls is completed. However, it makes the same bets as before: if you or that application has read it once, you’re bound to read it again. The kernel stores this information in several “Cached:” segments called the dentry and inode caches. dentries are common across all filesystems, but each filesystem has its own cache for inodes. These individual components store information about the location of a file on disk such as the block and sector ID. These structures also store additional information such as the human readable path to a file, the owner of that file, and it’s permission set.
You can view the different caches and their sizes by executing this command. (This RAM is a component of “Slab:” in cat /proc/meminfo)
[root@server ~]head -2 /proc/slabinfo; cat /proc/slabinfo | egrep dentry\|inode slabinfo - version: 2.1 # name <active_objs> <num_objs> : tunables : slabdata <active_slabs> <num_slabs> ext4_inode_cache 12171 15411 984 33 8 : tunables 0 0 0 : slabdata 467 467 0 mqueue_inode_cache 36 36 896 36 8 : tunables 0 0 0 : slabdata 1 1 0 hugetlbfs_inode_cache 26 26 616 26 4 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 434 460 704 46 8 : tunables 0 0 0 : slabdata 10 10 0 shmem_inode_cache 997 1000 808 40 8 : tunables 0 0 0 : slabdata 25 25 0 proc_inode_cache 873 925 648 25 4 : tunables 0 0 0 : slabdata 37 37 0 inode_cache 1116 1323 600 27 4 : tunables 0 0 0 : slabdata 49 49 0 dentry 18542 18564 192 42 2 : tunables 0 0 0 : slabdata 442 442 0
Older kernels (around 2.6.9) left some structures in the slab cache for longer periods of time than newer kernels do. That means that even though they may have been unused, they’re left there until there is sufficient memory pressure. This happens more frequently with proc_inodes. /proc inodes also pin task_structs, which means that each one can effectively occupy over 2 KBytes of RAM. This RAM won’t show up as “Cached:” and may appear to be a kernel memory leak. On a system with only 100 tasks (with little memory pressure) there can be hundreds of thousands of these left lying around though this is a normal condition.
What if I want to run more applications?
Run as many applications as you want to, the disk cache in no way affects the over availability of memory on a system but it can certainly affect the overall performance especially if your system is experiencing a memory related performance problem. If your applications want more memory, they simply take back a chunk that the disk cache has effectively borrowed. Disk cache can always be given back to applications immediately!
How do I stop Linux from doing this?
You can’t really disable disk caching as it is part of the core virtual memory systems of Linux. You can however control caching to some degree with kernel level parameters if you feel it is really required. In most cases the only reason anyone ever wants to disable disk caching is because they don’t really understand what those segments of memory are being used for or how the system can choose to allocate or de-allocate those segments. Disk caches by design are meant to make applications load faster and run smoother.
Why does top and free say all the ram is used?
This is just a misunderstanding of how Linux classifies memory usage. In most cases you will find that both you and the Linux kernel actually agree that memory taken by an application is “Used”, while memory that isn’t used for anything is “Free”. So what do you call memory that is both used for something and available to applications, such as the disk cache? Most people would likely call it “Free Memory” but Linux classifies it as “Used Memory” which often leads to confusion.
| Memory that is | You would call it | Linux calls it |
|---|---|---|
| taken by applications | Used | Used |
| available for applications, and used for something | Free | Used |
| not used for anything | Free | Free |
How do I see how much free ram I really have?
To see how much ram is free to use for your applications, run free -m and look at the row that says “-/+ buffers/cache” in the column that says “free”. That is your answer in megabytes.
[root@server ~]# free -m total used free shared buffers cached Mem: 5977 5811 165 0 159 4891 -/+ buffers/cache: 760 5216 Swap: 7738 1 7737
Some information sourced from http://www.linuxatemyram.com/