Erlang 打开和关闭SMP选项之差异观察

最新推荐文章于 2023-11-13 13:40:46 发布

原创最新推荐文章于 2023-11-13 13:40:46 发布 · 6.9k 阅读

1 ·

CC 4.0 BY-SA版权

本文探讨了在Erlang otp_19.1版本中开启和关闭SMP（Symmetric MultiProcessing）选项对系统性能的差异。通过实验观察，分析SMP在并发处理、内存使用和响应时间等方面的表现变化。

##以下观察是基于版本otp_19.1

#观察SMP enabled

Step 1. 启动erl with SMP enabled (erl -smp enable)

sexxx484: ~/erlang/test > erl
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V8.1 (abort with ^G)
1>

#[smp:8:8] 表示在8核的机器上启动了8个scheduler, 每个core对应一个scheduler, 每一个scheduler对应于独立的一个操作系统线程. 启动erlang的时候可以通过选项 +S来设置启动多少个scheduler, 也可以在运行时通过erlang:system_flag 来调整.

+S Schedulers:SchedulerOnline

Sets the amount of scheduler threads to create and scheduler threads to set online when SMP support has been enabled. Valid range for both values are 1-1024. If the Erlang runtime system is able to determine the amount of logical processors configured and logical processors available, Schedulers will default to logical processors configured, and SchedulersOnline will default to logical processors available; otherwise, the default values will be 1. Schedulers may be omitted if :SchedulerOnline is not and vice versa. The amount of schedulers online can be changed at run time via erlang:system_flag(schedulers_online, SchedulersOnline).

Note: the results are similar whether symmetric multiprocessing is enabled or not. To prove it, you can just test it out by starting the Erlang VM with $ erl -smp disable.

To see if your Erlang VM runs with or without SMP support in the first place, start a new VM without any options and look for the first line output. If you can spot the text [smp:2:2] [rq:2], it means you're running with SMP enabled, and that you have 2 run queues (rq, or schedulers) running on two cores. If you only see [rq:1], it means you're running with SMP disabled.

If you wanted to know, [smp:2:2] means there are two cores available, with two schedulers. [rq:2] means there are two run queues active. In earlier versions of Erlang, you could have multiple schedulers, but with only one shared run queue. Since R13B, there is one run queue per scheduler by default; this allows for better parallelism.

#[async-threads:10] 表示启动了异步线程池, 含有10个异步IO线程. 启动erlang的时候可以通过选项+A来设置启动多少个异步IO线程.

既然说到async thread pool, 这里就解释一下，这个异步线程池的作用. 当你有大量的文件IO操作的时候，这个异步线程池可以改善系统的响应效率，因为这些相对比较耗时的IO操作都放到异步线程池里面来完成了，不会影响到正常的scheduler调度.

下面一段话引用自stackoverflow.

The BEAM runs Erlang code in special threads it calls schedulers. By default it will start a scheduler for every core in your processor. This can be controlled and start up time, for instance if you don't want to run Erlang on all cores but "reserve" some for other things. Normally when you do a file I/O operation then it is run in a scheduler and as file I/O operations are relatively slow they will block that scheduler while they are running. Which can affect the real-time properties. Normally you don't do that much file I/O so it is not a problem.

The asynchronous thread pool are OS threads which are used for I/O operations. Normally the pool is empty but if you use the +A at startup time then the BEAM will create extra threads for this pool. These threads will then only be used for file I/O operations which means that the scheduler threads will no longer block waiting for file I/O and the real-time properties are improved. Of course this costs as OS threads aren't free. The threads don't mix so scheduler threads are just scheduler threads and async threads are just async threads.

If you are writing linked-in drivers for ports these can also use the async thread pool. But you have to detect when they have been started yourself.

How many you need is very much up to your application. By default none are started. Like @demeshchuk I have also heard that Riak likes to have a large async thread pool as they open many files. My only advice is to try it and measure. As with all optimisation?

Step 2. 找到进程beam.smp的进程ID

sexxx484: ~/erlang/test >ps -ef | grep beam

Step 3. 查看改进程包括的所有线程

sexxx484: ~/erlang/test > top -H -p 699

top - 10:41:40 up 205 days, 22:31, 72 users, load average: 1.22, 1.65, 1.97
Tasks: 22 total, 0 running, 22 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.0%us, 4.7%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st
Mem: 32241M total, 28556M used, 3684M free, 842M buffers
Swap: 0M total, 0M used, 0M free, 14000M cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
699 20 0 1757m 20m 2980 S 0 0.1 0:00.05 beam.smp
703 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
704 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
705 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
706 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
707 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
708 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
709 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
710 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
711 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
712 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
713 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
714 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
716 20 0 1757m 20m 2980 S 0 0.1 0:00.03 beam.smp
717 20 0 1757m 20m 2980 S 0 0.1 0:00.22 beam.smp
718 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
719 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
720 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
721 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp
722 20 0 1757m 20m 2980 S 0 0.1 0:00.02 beam.smp
723 20 0 1757m 20m 2980 S 0 0.1 0:00.02 beam.smp
724 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp

#观察SMP disabled

Step 1. 启动erl with SMP enabled (erl -smp disable)

sexxx484: ~/erlang/test > erl -smp disable
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V8.1 (abort with ^G)
1>

这里没有了smp的标志,所以不会在每个core上都启动一scheduler,因为每个scheduler对应于一个独立的操作系统线程, 所以beam进程的内部线程数量相比于打开smp时就会少一些.

Step 2. 找到进程beam.smp的进程ID

sexxx484: ~/erlang/test >ps -ef | grep beam

Step 3. 查看改进程包括的所有线程

484: ~/erlang/test > top -H -p 31599

top - 10:39:35 up 205 days, 22:29, 72 users, load average: 1.21, 1.88, 2.08
Tasks: 11 total, 0 running, 11 sleeping, 0 stopped, 0 zombie
Cpu(s): 50.0%us, 0.0%sy, 0.0%ni, 50.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32241M total, 28549M used, 3691M free, 842M buffers
Swap: 0M total, 0M used, 0M free, 14000M cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31599 20 0 1579m 15m 2836 S 0 0.0 0:00.32 beam
31607 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31608 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31609 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31610 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31611 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31612 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31613 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31614 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31615 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
31616 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam

#观察SMP enabled

# spawn 4个Erlang processes, 分别打开和关闭SMP选项，观察processes在CPU上的执行情况差异

1. 启动erl with SMP enabled (erl -smp enable)

2. 编译运行以下程序（test_smp.erl）, test_smp:start(99999999).

3. 找到进程beam.smp的进程ID

4. 查看该进程包括的所有线程

5. 查看top with type 1 输出

# step 4 输出 , 有四个线程全力运行中（100%）, 在本例中，我启动了四个erlang processes，所以说明这四个processes被分配到了不同的core上同时运行.

top - 10:31:49 up 205 days, 22:21, 72 users, load average: 3.49, 2.40, 2.16
Tasks: 22 total, 4 running, 18 sleeping, 0 stopped, 0 zombie
Cpu(s): 54.1%us, 2.3%sy, 0.0%ni, 43.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 32241M total, 28554M used, 3686M free, 842M buffers
Swap: 0M total, 0M used, 0M free, 14000M cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3448 20 0 1695m 21m 2988 R 100 0.1 0:41.07 beam.smp
3449 20 0 1695m 21m 2988 R 100 0.1 0:40.60 beam.smp
3450 20 0 1695m 21m 2988 R 100 0.1 0:40.75 beam.smp
3451 20 0 1695m 21m 2988 R 100 0.1 0:40.54 beam.smp
3428 20 0 1695m 21m 2988 S 0 0.1 0:00.05 beam.smp
3432 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3433 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3434 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3435 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3436 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3437 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3438 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3439 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3440 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3442 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3443 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3444 20 0 1695m 21m 2988 S 0 0.1 0:00.00 beam.smp
3452 20 0 1695m 21m 2988 S 0 0.1 0:00.20 beam.smp
3453 20 0 1695m 21m 2988 S 0 0.1 0:00.21 beam.smp
3454 20 0 1695m 21m 2988 S 0 0.1 0:00.24 beam.smp
3455 20 0 1695m 21m 2988 S 0 0.1 0:00.04 beam.smp
3456 20 0 1695m 21m 2988 S 0 0.1 0:00.01 beam.smp

# step 5 输出, beam.smp 占用cpu百分比是398，基本上是4个cpu core在全速运行

top - 10:27:42 up 205 days, 22:17, 72 users, load average: 3.25, 2.35, 2.11
Tasks: 562 total, 1 running, 538 sleeping, 21 stopped, 2 zombie
Cpu0 : 42.2%us, 6.3%sy, 0.0%ni, 51.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 78.1%us, 0.0%sy, 0.0%ni, 21.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 13.9%us, 7.3%sy, 0.0%ni, 78.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 99.7%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu4 : 79.1%us, 1.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu5 : 59.6%us, 6.4%sy, 0.0%ni, 34.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 14.7%us, 7.4%sy, 0.0%ni, 77.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 72.8%us, 0.3%sy, 0.0%ni, 26.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32241M total, 28552M used, 3689M free, 842M buffers
Swap: 0M total, 0M used, 0M free, 14000M cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23244 20 0 1759m 20m 2992 S 398 0.1 1:56.44 beam.smp
15903 20 0 88656 42m 3760 S 41 0.1 9427:46 .vasd
27877 20 0 24932 5728 1876 S 2 0.0 1:00.69 dtworker
26756 20 0 24932 5624 1864 S 2 0.0 0:58.43 dtworker
15886 20 0 24932 5708 1868 S 1 0.0 0:55.55 dtworker
16540 20 0 24836 5616 1868 S 1 0.0 0:56.49 dtworker

#观察SMP disabled

1. 启动erl with SMP disabled (erl -smp disable)

2. 编译运行以下程序（test_smp.erl）, test_smp:start(99999999).

3. 找到进程beam.smp的进程ID

4. 查看该进程包括的所有线程

5. 查看top with type 1 输出

# step 4 输出 , 只有一个线程全力运行中（100%）, 在本例中，我启动了四个erlang processes，所以说明这四个processes只有一个处于运行中，另外三个处于等待中.

top - 10:33:20 up 205 days, 22:23, 72 users, load average: 3.00, 2.72, 2.31
Tasks: 11 total, 1 running, 10 sleeping, 0 stopped, 0 zombie
Cpu(s): 20.0%us, 3.8%sy, 0.0%ni, 76.1%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 32241M total, 28549M used, 3692M free, 842M buffers
Swap: 0M total, 0M used, 0M free, 14000M cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11044 20 0 1579m 15m 2836 R 100 0.0 0:23.10 beam
11048 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11050 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11051 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11052 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11053 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11056 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11057 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11058 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11059 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam
11060 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam

# step 5 输出, beam.smp 占用cpu百分比是 98, 只有一个cpu core在全速运行，剩下的CPU虽然空闲，但是不会被调度，因为SMP选项是 disabled.

top - 10:26:35 up 205 days, 22:16, 72 users, load average: 2.12, 2.07, 2.01
Tasks: 563 total, 2 running, 538 sleeping, 21 stopped, 2 zombie
Cpu0 : 14.6%us, 0.7%sy, 0.0%ni, 84.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 21.1%us, 0.7%sy, 0.0%ni, 77.9%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu2 : 14.3%us, 1.7%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 14.7%us, 1.7%sy, 0.0%ni, 83.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 99.0%us, 1.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 15.0%us, 0.3%sy, 0.0%ni, 84.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 15.3%us, 1.3%sy, 0.0%ni, 83.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 14.8%us, 0.7%sy, 0.0%ni, 84.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32241M total, 28548M used, 3692M free, 842M buffers
Swap: 0M total, 0M used, 0M free, 14000M cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11851 20 0 1579m 15m 2844 R 98 0.0 2:54.56 beam
25650 20 0 2120m 685m 33m S 2 2.1 592:22.53 java
27877 20 0 24932 5728 1876 S 1 0.0 1:00.18 dtworker
4804 20 0 828m 50m 7596 S 1 0.2 19:23.46 java
15781 20 0 191m 81m 2756 S 1 0.3 1:20.85 beam.smp
20531 20 0 22424 1928 1164 R 1 0.0 0:00.19 top
1570 20 0 87352 2204 1188 S 0 0.0 264:23.39 vmtoolsd
3207 20 0 32032 2452 540 S 0 0.0 101:07.79 avahi-daemon

# test_smp.erl

-module(test_smp).
-compile(export_all).

start(N)->
spawn(?MODULE, tail_fac, [N]),
spawn(?MODULE, tail_fac, [N]),
spawn(?MODULE, tail_fac, [N]),
spawn(?MODULE, tail_fac, [N]).

tail_fac(N) -> tail_fac(N,1).

tail_fac(0,Acc) -> Acc;
tail_fac(N,Acc) when N > 0 -> tail_fac(N-1,N+Acc).

##博客仅作个人记录##