Re: [Hampshire] Troubleshooting Memory Leak

Hi Stephen,

> have a CentOS 5 server running nginx, httpd and a dozen or so
> mongrel instances. Since this morning I've been experiencing a
> serious memory leak - about 2G in 2 hours, until the machine needs to
> be rebooted. I've tried to answer the question "what changed", and I
> can't think of anything on my side...
>
> In top, and related tools, I see no processes growing in size - all I
> see is the amount of memory available constantly reducing.
>
> The sort of thing I can think of that fits the evidence would be
> something like a process is being created and then dying without
> returning the memory to the system.
>
> However, I am unable to see the malloc's, and I can't currently think
> of a way to test this theory.
>
> I suspect systemtap may be the tool to use - anyone have experience of
> using it in this way, or have a recipe they could point me to?
>
> Any other ideas for troubleshooting and gathering info?
>
> Thanks in advance,
>
> S.

If you are not seeing the memory being allocated to user-space processes
then it must be in the kernel: probably a miss-configured or buggy
device driver. For a quick check, you can check using "vmstat -m" or
just "cat /proc/slabinfo" and see which of the caches is being added to.

Chris Simmonds