本部分主要对ES的一套测试流程有一个熟悉,不断分析和学习各个指标和统计方法,找到性能瓶颈
一、装备
Linux环境
ElasticSearch单结点,无副本单分片
Kibana + x-pack 监控集群
jmeter压测
nmon监测Linux性能
二、服务器环境:
# cat /proc/version
Linux version 3.10.0-514.2.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Dec 6 23:06:41 UTC 2016
----------------------------------------------------------------------------------------------------------------
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
座: 8
NUMA 节点: 1
厂商 ID: GenuineIntel
CPU 系列: 6
型号: 42
型号名称: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
步进: 1
CPU MHz: 2394.454
BogoMIPS: 4788.90
超管理器厂商: KVM
虚拟化类型: 完全
L1d 缓存: 32K
L1i 缓存: 32K
L2 缓存: 4096K
----------------------------------------------------------------------------------------------------------------
# free -m
total used free shared buff/cache available
Mem: 32013 16583 1027 198 14402 14461
Swap: 0 0 0
----------------------------------------------------------------------------------------------------------------
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 20G 0 disk
└─vda1 253:1 0 20G 0 part /
vdb 253:16 0 200G 0 disk
└─vdb1 253:17 0 200G 0 part
# fdisk -l
磁盘 /dev/vda:21.5 GB, 21474836480 字节,41943040 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
磁盘标签类型:dos
磁盘标识符:0x000a0566
设备 Boot Start End Blocks Id System
/dev/vda1 * 2048 41943039 20970496 83 Linux
磁盘 /dev/vdb:214.7 GB, 214748364800 字节,419430400 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
磁盘标签类型:dos
磁盘标识符:0x739398fb
设备 Boot Start End Blocks Id System
/dev/vdb1 2048 419430399 209714176 83 Linux
----------------------------------------------------------------------------------------------------------------
三、准备过程
1、elasticsearch5.4.2,kibana,x-pack
elasticsearch即插即用,下下来改改配置就能用;
kibana类似,在配置文件里修改为本地的ES地址即可;
原本是打算用marvel监控,但貌似自5.0之后的ES的HEAD啊,marvel啊这些插件全没了,全集成到x-pack里面了
elasticsearch中运行 elasticsearch-plugin install x-pack
kibana中运行kibana-plugin install x-pack(时间稍有些慢)
即把ES相关的性能图添加到kibana中
2、jmeter
jmeter是java开发的一块集成测试的软件,通过它可以构建测试用例,保存jmx测试用例并交到服务器上执行。
jmeter支持远程启动,在服务器上启动jmeter-server(可能需要参数指定ip),在本机jmeter.properites中添加remote_server:远程ip:1099(1099为默认端口),即可在jmeter运行脚本时使用远程启动。
由于服务器不在局域网内,因此上传jmx文件给服务器使用
操作:
右键测试计划,添加线程组,修改线程数,循环次数,运行时间等关键参数
在线程组中添加Sampler,选择HTTP请求
在HTTP请求中添加聚合报告和结果树
在文件=>保存为测试计划 中保存jmx文件
3、nmon
nmon是一款linux下的系统性能指标工具,提供CPU,内存,磁盘IO,网络等多项指标的界面化展示。
安装:
wget http://sourceforge.net/projects/nmon/files/nmon16d_x86.tar.gz
tar xvzf nmon16d_x86.tar.gz
得到一系列nmon对应的各种版本的文件
找到相应的版本的nmon文件
chmod +x 赋予可执行权限
运行:
其中每个参数可控制展示的开关,按下c,m,d,n
即可监控系统性能
四、开始测试
ES结点数 | 分片数 | 副本数 | ES配置 | 线程数 | ES数据 | 索引setting | JVM参数 | 逐条insert写入速率范围(条/s) | 5分钟写入数据条数/数据量 | ||
Sample1 | 1 | 1 | 0 | A | 1 | A | 无 | A | |||
Sample2 | 1 | 1 | 0 | A | 200 | A | 无 | A | |||
Sample3 | 1 | 1 | 0 | A | 1000 | B | A | A | |||
Sample4 | 1 | 1 | 0 | A | 200 | B | A | A | |||
Sample5 | 1 | 1 | 0 | A | 200 | C | A | A | |||
Sample6 | |||||||||||
Sample7 |
样本项目描述
——ES配置:
A:
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: false
indices.memory.index_buffer_size: "50%"
indices.memory.min_index_buffer_size: "1024mb"
http.max_initial_line_length: "64kb"
http.max_header_size: "64kb"
action.search.shard_count.limit: 50000
script.inline: true
script.stored: true
thread_pool.bulk.queue_size: 5000
——JVM参数:
A:
-Xms2g -Xmx2g -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError
——ES数据:
A:
{
"_MCSJnlNo": "45139477"
}
B:
{
"_MCSJnlNo": "45139477",
"_raw": "2017-05-01 04:51:10.723 [[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO c.c.p.s.c.t.TransformTransport - recv:<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<Message>\r\n<Head>\r\n<_MCSJnlNo>45139477</_MCSJnlNo>\r\n<_ResponseSystemId>MCA</_ResponseSystemId>\r\n<_RejCode>000000</_RejCode>\r\n<_RejMsg></_RejMsg>\r\n<_MCHJnlNo>3973735</_MCHJnlNo>\r\n<_TransactionId>per.MCLoanInterestRateQuery</_TransactionId>\r\n<_MCHTimestamp>2017-05-01 04:51:10.61</_MCHTimestamp>\r\n</Head>\r\n<Body>\r\n<InterestRate>10.00</InterestRate>\r\n</Body>\r\n</Message>\r",
"_context_id": 472,
"_host": "192.168.0.34",
"type": "recv",
"_MCHTimestamp": "2017-05-01 04:51:10.61",
"_MCHJnlNo": "3973735",
"_RejCode": "000000",
"_ResponseSystemId": "MCA",
"InterestRate": "10.00",
"_TransactionId": "per.MCLoanInterestRateQuery",
"_ts": 1493585470000
}
C:
{
"_raw": "[2017-07-23 22:09:49,435] [elasticsearch] [DEBUG] [< {\"took\":12,\"timed_out\":false,\"_shards\":{\"total\":10,\"successful\":10,\"failed\":0},\"hits\":{\"total\":196699,\"max_score\":null,\"hits\":[{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3QT\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:35 test_longtime1/longtimetxt.0 three line log for one 296963 \",\"_context_id\":1088331,\"_host\":\"192.168.0.33\",\"_ts\":1500818975000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:35 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296963 \"]},\"sort\":[1500818975000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3QQ\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:34 test_longtime1/longtimetxt.0 three line log for one 296962 \",\"_context_id\":1088328,\"_host\":\"192.168.0.33\",\"_ts\":1500818974000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:34 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296962 \"]},\"sort\":[1500818974000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3QN\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:33 test_longtime1/longtimetxt.0 three line log for one 296961 \",\"_context_id\":1088325,\"_host\":\"192.168.0.33\",\"_ts\":1500818973000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:33 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296961 \"]},\"sort\":[1500818973000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3OF\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:32 test_longtime1/longtimetxt.0 three line log for one 296960 \",\"_context_id\":1088322,\"_host\":\"192.168.0.33\",\"_ts\":1500818972000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:32 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296960 \"]},\"sort\":[1500818972000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3OC\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:31 test_longtime1/longtimetxt.0 three line log for one 296959 \",\"_context_id\":1088319,\"_host\":\"192.168.0.33\",\"_ts\":1500818971000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:31 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296959 \"]},\"sort\":[1500818971000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3N_\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:30 test_longtime1/longtimetxt.0 three line log for one 296958 \",\"_context_id\":1088316,\"_host\":\"192.168.0.33\",\"_ts\":1500818970000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:30 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296958 \"]},\"sort\":[1500818970000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Lx\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:29 test_longtime1/longtimetxt.0 three line log for one 296957 \",\"_context_id\":1088313,\"_host\":\"192.168.0.33\",\"_ts\":1500818969000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:29 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296957 \"]},\"sort\":[1500818969000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Lu\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:28 test_longtime1/longtimetxt.0 three line log for one 296956 \",\"_context_id\":1088310,\"_host\":\"192.168.0.33\",\"_ts\":1500818968000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:28 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296956 \"]},\"sort\":[1500818968000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Lr\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:27 test_longtime1/longtimetxt.0 three line log for one 296955 \",\"_context_id\":1088307,\"_host\":\"192.168.0.33\",\"_ts\":1500818967000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:27 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296955 \"]},\"sort\":[1500818967000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Jr\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:26 test_longtime1/longtimetxt.0 three line log for one 296954 \",\"_context_id\":1088304,\"_host\":\"192.168.0.33\",\"_ts\":1500818966000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:26 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296954 \"]},\"sort\":[1500818966000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Jo\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:25 test_longtime1/longtimetxt.0 three line log for one 296953 \",\"_context_id\":1088301,\"_host\":\"192.168.0.33\",\"_ts\":1500818965000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:25 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296953 \"]},\"sort\":[1500818965000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Jl\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:24 test_longtime1/longtimetxt.0 three line log for one 296952 \",\"_context_id\":1088298,\"_host\":\"192.168.0.33\",\"_ts\":1500818964000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:24 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296952 \"]},\"sort\":[1500818964000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Hl\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:23 test_longtime1/longtimetxt.0 three line log for one 296951 \",\"_context_id\":1088295,\"_host\":\"192.168.0.33\",\"_ts\":1500818963000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:23 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296951 \"]},\"sort\":[1500818963000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Hi\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:22 test_longtime1/longtimetxt.0 three line log for one 296950 \",\"_context_id\":1088292,\"_host\":\"192.168.0.33\",\"_ts\":1500818962000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:22 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296950 \"]},\"sort\":[1500818962000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Hf\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:21 test_longtime1/longtimetxt.0 three line log for one 296949 \",\"_context_id\":1088289,\"_host\":\"192.168.0.33\",\"_ts\":1500818961000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:21 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296949 \"]},\"sort\":[1500818961000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3FU\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:20 test_longtime1/longtimetxt.0 three line log for one 296948 \",\"_context_id\":1088286,\"_host\":\"192.168.0.33\",\"_ts\":1500818960000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:20 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296948 \"]},\"sort\":[1500818960000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3FR\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:19 test_longtime1/longtimetxt.0 three line log for one 296947 \",\"_context_id\":1088283,\"_host\":\"192.168.0.33\",\"_ts\":1500818959000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:19 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296947 \"]},\"sort\":[1500818959000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3FO\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:18 test_longtime1/longtimetxt.0 three line log for one 296946 \",\"_context_id\":1088280,\"_host\":\"192.168.0.33\",\"_ts\":1500818958000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:18 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296946 \"]},\"sort\":[1500818958000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Cr\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:17 test_longtime1/longtimetxt.0 three line log for one 296945 \",\"_context_id\":1088277,\"_host\":\"192.168.0.33\",\"_ts\":1500818957000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:17 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296945 \"]},\"sort\":[1500818957000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Co\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:16 test_longtime1/longtimetxt.0 three line log for one 296944 \",\"_context_id\":1088274,\"_host\":\"192.168.0.33\",\"_ts\":1500818956000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:16 test_longtime1/longtimetxt.0 three line log for <em>one</em> 296944 \"]},\"sort\":[1500818956000]}]}}]",
"_context_id": 1247368,
"_host": "192.168.0.34",
"_ts": 1500818989000,
"data1": 重复raw,
"data2": 重复raw,
"data3": 重复raw,
"data4": 重复raw,
"data5": 重复raw,
"data6": 重复raw,
"data7": 重复raw,
"data8": 重复raw,
"data9": 重复raw,
"data10": 重复raw,
}
——索引settings:
A:
PUT index1
{
"template": "*",
"order": 1,
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.query.default_field": "_raw"
},
"mappings": {
"_default_": {
"date_detection": false,
"numeric_detection": false,
"_all": {
"enabled": false
},
"properties": {
"_host": {
"type": "keyword",
"norms": false
},
"_raw": {
"type": "text"
},
"_ts": {
"type": "long"
},
"_context_id": {
"type": "long"
}
},
"dynamic_templates": [
{
"strings_as_keywords": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}
}
----------------------------------------------------------------------------------------------------------------
Sample 1:
16:53:07 —— 16:58:07
性能抽样:
写入日志:
数据量:
x-pack图:
----------------------------------------------------------------------------------------------------------------
Sample 2:
17:07:15 —— 17:12:15
性能抽样:
写入日志:
数据量:
x-pack图:
----------------------------------------------------------------------------------------------------------
Sample 3:
17:24:03——17:29:03
性能抽样:
写入日志:
数据量:
x-pack图:
----------------------------------------------------------------------------------------------------------
Sample 4:
23:07:04——23:12:04
性能抽样:
写入日志
数据量:
x-pack图:
----------------------------------------------------------------------------------------------------------
Sample 5:
23:35:06——23:40:06
性能抽样:
写入日志:
数据量
x-pack图: