Config Huge Pages

Get a performance boost by backing your KVM guest with hugetlbfs

Tuesday, September 9, 2008 - 08:08 Haydn Solomon

Support for hugetlbfs was added with the release of kvm-62 and it can give your kvm guest machine a performance boost anywhere up to 10%. KVM provides you with an option of using hugetlbfs by using the –mem-path option when starting your kvm guest machine. Just as a sidenote, the mem-path option also supports tmpfs filesystems.

 

 

What is hugetlbfs and why would you want to use it?

It is common nowadays for architectures to support more than one page size. As an example, x86 allows you to choose 4kb or 2MB pages. In the past, Linux traditionally used large pages for supporting the actual kernel image only but in kernel 2.6 linux started allowing processes to use huge pages. The whole purpose for huge page support is performance and you can get up to a 10% performance boost based on the application you’re running. Huge pages tend to give a bigger performance boost to applications that are memory and compute intensive. In the following sections, I’ll walk through the procedure for backing your kvm guest with huge pages.

 

1. Check that your kernel has support for Huge Pages

First thing you need to do is check that your kernel has support for huge pages. If you’re running a 2.6 kernel then you most likely have huge page support already. You can confirm by running the following command.

# grep HUGETLB /boot/config-`uname -r`
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y

I ran this command on a Fedora Linux host but it should be similar for your distribution. Your output should look similar if your kernel has hugetlbfs support enabled. 
 

2. Calculate the number of huge pages you require

On x86, large pages are 2MB in size so you need to find out how many 2MB pages you need to back your kvm guest. Let’s say you want to use 512 MB for your guest machine then divide 512 by 2 to get 256 pages. Add a few extra pages for additional memory requirements, lets say 40 pages to give a total of 296 pages. Record this value as you’ll need it later on.

 

3. Setup your hugetlbfs filesystem

Now you need to setup your hugetlbfs filesystem on your host. As root, create a directory called hugepages like so.

mkdir /hugepages

Next add an entry to your /etc/fstab so that when you reboot your computer, your hugetlbfs will mount automatically. Your entry should look like the following.

hugetlbfs       /hugepages  hugetlbfs       defaults        0 0

Because it is difficult to reserve large pages on a busy system, it’s always best to reserve your huge pages shortly after restart.

 

4. Reboot and reserve your huge pages

As mentioned in the previous section, reserving huge pages is best done right after a reboot as it is difficult to reserve large pages on a busy system. Right after your computer has rebooted and using the value you calculated above for the number of huge pages you want to reserve, execute the following command.

echo 296 >  /proc/sys/vm/nr_hugepages

If you want this to be a permanent setup , you can also add it to your /etc/rc.local script so that it is always reserved on startup. To verify that you have large pages reserved inspect the contents of the /proc/meminfo file as follows.

# tail -n 5 /proc/meminfo 
HugePages_Total:   296
HugePages_Free:    296
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

Now you are ready to start backing your virtual machine with huge pages.

 

5. Start your kvm guest with huge page support

You have verified everything to this point and ready to back your virtual machine with large pages. Start your virtual machine with using the guest.xml option as follows:

  <memoryBacking>
    <hugepages/>
  </memoryBacking>

You will probably notice a performance increase in your virtual machine. You can check that you’re actually using huge pages by inspecting the /proc/meminfo file again. For example, this is what mine looked like while running a kvm guest backed with the values in this post.

# tail -n 5 /proc/meminfo 
HugePages_Total:   296
HugePages_Free:     39
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

 That concludes the procedure for backing your guest with huge pages. It doesn't take that long to setup so give it a try and see if you get a performance boost.

2、KVM Huge Page Backed Memory

intel 的X86 CPU通常使用4Kb内存页，当是经过配置，也能够使用巨页(huge page):

(4MB on x86_32, 2MB on x86_64 and x86_32 PAE)

使用巨页，KVM的虚拟机的页表将使用更少的内存，并且将提高CPU的效率。最高情况下，可以提高20%的效率！

使用方法，需要三部：

mount -t hugetlbfs hugetlbfs /dev/hugepages
#保留一些内存给巨页
sysctl vm.nr_hugepages=516
#给 kvm 传递参数 hugepages
qemu-kvm - qemu-kvm -mem-path /dev/hugepages

其中第三步，也可以在配置文件里加入：

<memoryBacking>
<hugepages/>
</memoryBacking>

验证方式，当虚拟机正常启动以后，在虚拟机里查看：

cat /proc/meminfo |grep -i HugePages_Free