Have you fallen victim to the out-of-memory (OOM) killer? Are you looking for memory leaks? Use ProTop's "dirty memory" script on Linux to discover which processes are holding more private memory than others.
NOTE: If you are facing memory issues, be sure to establish monitoring and alerting for available memory (memAvailable). If you are running the dirty memory report on a scheduled basis as we suggest below, when an alert reaches you regarding available memory you can quickly scan the dirty memory reports to find the highest memory using processes. These processes can then be evaluated for appropriate memory consumption.
This logic reports on non-shared resident memory per process - this is the unique memory burden of that process. This memory is returned to the system when the process ends.
We want to keep the sum of this memory plus the sum of all shared memory low enough to avoid any excuse for the OOM killer to act. As a rule of thumb, aim to leave 25%-50% of memory free for other purposes.
This report can also be helpful when looking for memory leaks over time. You can see if you have processes with slow but persistent memory growth by comparing hour-by-hour or day-by-day samples.
Auto-create the reports
To create these samples on a regular basis, execute the commands below from root's crontab.
CAUTION: If you configure cron to run these reports automatically, be sure to add logic to clean them up on a regular basis.
To run this run report (as root):
sudo bin/dirtymem.sh
-or-
sudo ${DLC}/bin/_progres -b -p util/private_dirty.p
Then look in $PROTOP/rpt for a file named private_dirty.rpt*
You will only get data for your processes if you do not run the report as root.
WARNING: On large systems, it can take several minutes to parse /proc/<PID>/smaps.
What it does
(from this Stackexchange
and
https://gist.github.com/sameo/d49c50772d616ae00e96c9967e676976 )
For a given process in /proc/<pid>/smaps, for a given mapping entry, what are:- Shared_Clean
- Shared_Dirty
- Private_Clean
- Private_Dirty
Clean pages are pages that have not been modified since they were mapped. Typically, text sections from shared libraries are only read from disk when necessary and never modified, so they'll be in shared, clean pages.
Dirty pages are pages that are not clean (i.e. have been modified).
Private pages are available only to that process; shared pages are mapped by other processes*.
RSS is the total number of pages, shared or not, currently mapped into the process. So Shared_Clean + Shared_Dirty would be the shared part of the RSS (i.e. the part of RSS that is also mapped into other processes), and Private_Clean + Private_Dirty are the private part of RSS (i.e. only mapped in this process).
*Note that a "share-able" page is counted as a private mapping until it is actually shared. i.e. if there is only one process currently using libfoo, that library's text section will appear in the process's private mappings. It will be accounted for in the shared mappings (and removed from the private ones) only if/when another process starts using that library.
Another option
A very similar, and quite helpful, tool is "smem" which is a Python script. The download is just 60k. It has a lot of very useful options. To download smem:
sudo apt-get install smem
- or -
sudo su -
cd /tmp
wget http://www.selenic.com/smem/download/smem-1.4.tar.gz
tar xvzf smem-1.4.tar.gz
cp /tmp/smem-1.4/smem /usr/local/bin/
chmod +x /usr/local/bin/smem