Detecting Memory Leaks in Kernel

 

This doc is from

http://psankar.blogspot.com/2010/11/detecting-memory-leaks-in-kernel.html

 

 

A memory leak is a behavior of a program when it consumes memory but never releases it. In user-space, these days, new applications are written mostly in sophisticated, evolved, modern programming languages like C#, Java etc. This releases the burden of memory management from the programmer. Programmers need to manage memory themselves, only when they code in pre-historic programming languages like C ;-) There are nice tools (like Valgrind) that can detect memory leaks, if they happen in user-space. Valgrind won't work in Kernel space. 

The Linux kernel is predominantly written in C, so that programmers can stay close to the hardware. Also there were no large-scale managed languages at the time the project was started. 

There are some interesting projects that help in writing FUSE filesystems via Mono/C# etc. However your grumpy blogger didn't find them to be active and for all practical purposes, C is the default Kernel programming language. 

Few days back, I was tasked with detecting memory leaks in a legacy kernel module that is not in Linus' tree. Code that goes into Linus' tree is usually of high quality and won't have memory leaks because of the rigorous reviews that are performed in LKML. So, any kernel code that is not merged upstream has a high chance of having leaks, among other bad things (so upstream your code, NOW). The simple tutorial below will explain how to detect memory leaks in kernel modules.

kmemleak

There is a nice tool named 'kmemleak' available in the Linux kernel since 2.6.31 to detect memory leaks. This tool is claimed to report a few false positives but that should not stop someone from using it. I was trying to find a kernel-space-leak-detector but did not find any links via Google. Some kernel hackers told me about this tool over IRC and this post is more as a pointer to the "kmemleak" docs, when somone googles for "kernel memory leak detection".

Pre-requisites: 
+ You need to know how to build and install your own kernel. 
+ You need to have 2.6.31 or newer version of the linux kernel
+ It is good if you know how to compile a kernel module. But don't worry if you don't know. You can refer to my previous tutorial for a simple hello-world kernel module. 

So, without further ado, the steps are: 

Step 1: Compile kernel with "CONFIG_DEBUG_KMEMLEAK" option enabled. You can get to this option via: make menuconfig, "Kernel Hacking", "Kernel Memory Leak Detector" , while compiling your kernel. 

Step 2: Increase the config option "Maximum kmemleak early log entires" value to a sufficiently large number like 1200. The default value of 400 may not work correctly in all configurations.

Step 3: Install this kernel and Reboot to this newly configured kernel. Do not be alarmed if your machine is slow.

Step 4: Upon reboot, Check if your debugfs is mounted. Otherwise mount it. If all is well, you should see a file kmemleak under your debugfs mounted location.

mount -t debugfs nodev /sys/kernel/debug/
cat /sys/kernel/debug/kmemleak

The above /sys/kernel/debug/kmemleak file will contain information about any memory leak that has been detected so far since the machine booted. Ideally there should be none, until this point in time. 

Step 5: Now we will see how we can detect a memory leak in a dummy kernel module as follows. Write a dummy kernel module with the following source (hello.c):


#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/vmalloc.h>

/* Never write a function like this ;) */
void myfunc(void)
{
        char *ptr;
        ptr = vmalloc(512);
        ptr = vmalloc(512);
        ptr = vmalloc(512);
}

int hello_init(void)
{
        printk(KERN_ALERT "Hello World");
        myfunc();
        return 0;
}

static void hello_exit(void)
{
        printk(KERN_ALERT "Goodbye World");
}

module_init(hello_init);
module_exit(hello_exit);

MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("Your Name");


Now the most important line in the above code snippet is:
ptr = vmalloc(512);
We allocate memory, as above, in the kernel module but never free this memory. 

Step 6: vi Makefile
EXTRA_CFLAGS=-g
obj-m := hello-kernel.o
hello-kernel-objs := hello.o

Step 7: Generate the kernel-object file hello-kernel.ko in your current directory:
make -C /lib/modules/`uname -r`/build M=`pwd`

All commands from now on require root permission.

Step 7.5: [Optional] At any stage, if you want to clear the memory profiler output so far created, so that we can focus just on the leaks reported from then on, you can do:
echo clear > /sys/kernel/debug/kmemleak

Step 8: Now insert the kernel object
insmod hello-kernel.ko

Step 9: The memory leak detection thread runs periodically. If you want to perform a test at any instant you want, Do:
echo scan > /sys/kernel/debug/kmemleak

Step 10: Now we will check if the leak is detected. Do:
cat /sys/kernel/debug/kmemleak
You should see:

unreferenced object 0xf9061000 (size 512):
comm "insmod", pid 12750, jiffies 14401507 (age 110.217s)
hex dump (first 32 bytes):
1c 0f 00 00 01 12 00 00 2a 0f 00 00 01 12 00 00 ........*.......
38 0f 00 00 01 12 00 00 bc 0f 00 00 01 12 00 00 8...............
backtrace:
[< c10b0001>] create_object+0x114/0x1db
[< c148b4d0>] kmemleak_alloc+0x21/0x3f
[< c10a43e9>] __vmalloc_node+0x83/0x90
[< c10a44b9>] vmalloc+0x1c/0x1e
[< f9055021>] myfunc+0x21/0x23 [hello_kernel]
[< f9058012>] 0xf9058012
[< c1001226>] do_one_initcall+0x71/0x113
[< c1056c48>] sys_init_module+0x1241/0x1430
[< c100284c>] sysenter_do_call+0x12/0x22
[< ffffffff>] 0xffffffff


As you can see in the bold text above, the leak is detected in myfunc function.

Caution: The memory leak detector code may take some time to identify the leaks. So repeat steps 9 and 10, after few minutes, if you don't get the leaks reported first time. You can try to kiss your hand elbow to pass time meanwhile ;-)

Further Reading
+ LWN Article about kmemleak - http://lwn.net/Articles/187979/ 
+ Under Kernel sources directory: Documentation/kmemleak.txt

Thanks a lot to Catalin Marinas for kmemleak and the people at kernelnewbies for helping, not just for this problem but for nuuuumerous people.