Hugepages is a good way to improve performance on virtual machines, as it will pre-allocate lots of RAM dedicated for use by libvirt/qemu, or something else that can use hugepages.

Hardware requirements

To be able to use hugepages your CPU must support the PSE flag and/or the PDPE1GB flag.

$ cat /proc/cpuinfo | grep flags

Will get all the flags supported by your CPU.

Install software

Next step is to install some softwares

$ sudo apt-get install -y libhugetlbfs-bin

This package will install the huegadm command, which is usefull to check the state of the hugepages.

for example this command

# hugeadm --explain
Total System Memory: 16029 MB

Mount Point          Options
/dev/hugepages       rw,relatime,pagesize=2M

Huge page pools:
      Size  Minimum  Current  Maximum  Default
   2097152     4096     4096     4096        *
1073741824        0        0        0         

Huge page sizes with configured pools:
2097152

The /proc/sys/vm/min_free_kbytes of 67584 is too small. To maximiuse efficiency
of fragmentation avoidance, there should be at least one huge page free per zone
in the system which minimally requires a min_free_kbytes value of 112640

The recommended shmmax for your currently allocated huge pages is 8589934592 bytes.
To make shmmax settings persistent, add the following line to /etc/sysctl.conf:
  kernel.shmmax = 8589934592

To make your hugetlb_shm_group settings persistent, add the following line to /etc/sysctl.conf:
  vm.hugetlb_shm_group = 106

Note: Permanent swap space should be preferred when dynamic huge page pools are used.

it will tell you some few tweaks needs to be done.

The vm.hugetlb_shm_group should be set to the kvm group, in my case it is 106 just like it says above because I already have the vm.hugetlb_shm_group in my /etc/sysctl.conf

A shorter version of the above command would be

# huegadm --pool-list
      Size  Minimum  Current  Maximum  Default
   2097152     4096     4096     4096        *
1073741824        0        0        0         

As you can see this will only show the current state. I have 4096 blocks of 2097152 bytes each, that is 2Mb * 4096 = 8Gb.

Allocate hugepages on Debian

Need to change the kernel boot command line to allocate hugepages. So edit /etc/default/grub and append hugepages=X, where X is the number of pages, to the CMDLINE.

By default each page is 2Mb to allocate 8Gb for a VM that needs 8Gb ram you will need 4096 pages.

GRUB_CMDLINE_LINUX_DEFAULT="... hugepages=4096"

You can allocate 8Gb using 1Gb sized pages like this

GRUB_CMDLINE_LINUX_DEFAULT="... hugepagesz=1G hugepages=8"

then run update-grub and reboot.

You don’t need both. I am using 2Mb pages just beacause it did not work with 1Gb pages for me. Qemu/kvm failed to allocate the memory when I was using 1Gb huge pages. I think the default is 2Mb blocks, thats why there is a * in the Default column when checking the hugeadm --pool-list command. I don’t know how to change the default pool, and I have not yet researched that.

after reboot you should be able to see the allocation by using the hugeadm command

# huegadm --pool-list
      Size  Minimum  Current  Maximum  Default
   2097152     4096     4096     4096        *
1073741824        0        0        0   

The min/cur/max values of 4096 will be the value you set in the Grub command line.

Configure a virtual machine to use huge pages

This is the easy part, just open the XML file for the machine you want to change. If you don’t have any XML file you can run the command virsh dumpxml vm-name > vm-name.xml where vm-name is the name of the virtual machine you want to dump the XML for.

add this part

<domain ...>
  ...
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  ...
</domain>

to your XML file as a child node to the main <domain ...> tag. To let libvirt know about the changes redefine the virtual machine by runnin this command

virsh define vm-name.xml