0.NUMA基础
numa会把内存分为n组,每一组和特定的一组cpu有对应关系,即cpu访问与之相应的内存组会更快
所以,最开始,我们需要指代cpu和内存组的对应关系,可以使用一下命令
$ lscpu
(省略)
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19
(省略)
$ dmesg|grep -i numa
[ 0.000000] NUMA: Initialized distance table, cnt=2
[ 0.000000] mempolicy: Enabling automatic NUMA balancing. Configure with num _balancing= or the kernel.numa_balancing sysctl
[ 0.812716] pci_bus 0000:00: on NUMA node 0
[ 0.822274] pci_bus 0000:80: on NUMA node 1
$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18
node 0 size: 128708 MB
node 0 free: 120496 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19
node 1 size: 129010 MB
node 1 free: 117856 MB
node distances:
node 0 1
0: 10 21
1: 21 10
1.numastat
$ numastat
node0 node1
numa_hit 10199402 11499638
numa_miss 0 0
numa_foreign 0 0
interleave_hit 65774 66145
local_node 9775609 11037623
other_node 423793 462015
numa_hit
是打算在该节点上分配内存,最后从这个节点分配的次数;
num_miss
是打算在该节点分配内存,最后却从其他节点分配的次数;
num_foregin
是打算在其他节点分配内存,最后却从这个节点分配的次数;
interleave_hit
是采用interleave策略最后从该节点分配的次数;
local_node
该节点上的进程在该节点上分配的次数
other_node
是其他节点进程在该节点上分配的次数
2.numactl
感觉这个命令,主要是控制程序运行时cpu和内存的使用。
比如,依照上面的数据,我们的程序可以有如下的设置
bind0 = numactl -m 0 --physcpubind=0
bind1 = numactl -m 1 --physcpubind=1
bind2 = numactl -m 0 --physcpubind=2
bind3 = numactl -m 1 --physcpubind=3
bind4 = numactl -m 0 --physcpubind=4
bind5 = numactl -m 1 --physcpubind=5
bind6 = numactl -m 0 --physcpubind=6
bind7 = numactl -m 1 --physcpubind=7
bind8 = numactl -m 0 --physcpubind=8
bind9 = numactl -m 1 --physcpubind=9
bind10 = numactl -m 0 --physcpubind=10
bind11 = numactl -m 1 --physcpubind=11
bind12 = numactl -m 0 --physcpubind=12
bind13 = numactl -m 1 --physcpubind=13
bind14 = numactl -m 0 --physcpubind=14
bind15 = numactl -m 1 --physcpubind=15
bind16 = numactl -m 0 --physcpubind=16
bind17 = numactl -m 1 --physcpubind=17
bind18 = numactl -m 0 --physcpubind=18
bind19 = numactl -m 1 --physcpubind=19
速度测试命令,上面的命令运行更快
numactl --cpubind=0 --membind=0 dd if=/dev/zero of=/dev/shm/A bs=1M count=1024
numactl --cpubind=0 --membind=1 dd if=/dev/zero of=/dev/shm/A bs=1M count=1024
另外就是参数--interleave
是程序的内存在不同numa组中均匀分布
$ numactl --interleave=all mongod
参考:
1.概念和问题
http://cenalulu.github.io/linux/numa/
2.命令使用
https://blog.youkuaiyun.com/jollyjumper/article/details/17168175