今天分享一下在性能测试统计中,各种参数和性能指数对性能测试误差的影响,以及各种减少误差方法效果,以便知道以后的性能测试改如何改进。
演示Demo
我写了一个模拟正常单接口性能测试的时候的Demo
,我使用了ThreadLimitTimesCount<String>
模型,使用sleep()
方法模拟请求和响应,具体代码如下:
/**
* 性能测试统计误差测试类
* by:FunTester
*/
class DeviationTest extends SourceCode {
static void main(String[] args) {
def ts = []
10.times {
ts << new FunTester(StringUtil.getString(10), 60)
}
new Concurrent(ts, "FunTester误差分析").start()
}
private static class FunTester extends ThreadLimitTimesCount<String> {
FunTester(String s, int times) {
super(s, times, null)
}
@Override
protected void doing() throws Exception {
sleep(0.1)
}
@Override
ThreadBase clone() {
new FunTester(StringUtil.getString(10), times)
}
}
}
下面开始针对之前提到的误差来源进行对比分析了。
由于模拟的时间比较短,这里就不进行标准的演示了。在我自己测试空转的过程中也很难在ms
级别统计代码运行,所以我也放弃了对代码运行时间的对比。
线程数
首先来研究一下,线程数对性能测试误差的影响。首先我们先来一组配置20
线程、50
请求次数在固定请求时间的情况下的模拟,看一下结果。
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
> {
> ① . "rt":103,
> ① . "failRate":0.0,
> ① . "threads":20,
> ① . "deviation":"3.29%",
> ① . "errorRate":0.0,
> ① . "executeTotal":1000,
> ① . "qps2":187.79342723004694,
> ① . "total":1000,
> ① . "qps":194.1747572815534,
> ① . "startTime":"2021-04-05 15:40:58",
> ① . "endTime":"2021-04-05 15:41:04",
> ① . "mark":"FunTester误差分析051540",
> ① . "table":"eJztjzEKwkAQRfvA3mEOkEC0sMgBPIDkAgEHXHBXySYQSxElpVVKj2BK7xOIeAxX7SzcFUQZmeHDm2r+PBGAx4xLnaIpML+0bX869vXufNjDMIZilmM29TkBIhCvuyZolgttEFKpMIEqMpjLbA66VCGsIoVTmWlXh/sPJTU8biWDOAZlQpVVdh3Z9QMWjumajY0DWxs/1Db2p65Z28AzPKreLr43fhs3w58UsyEbsiEb/hHYkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD5EcAVmaHb1"
> }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
实测平均耗时103ms
,总的误差是3.29%
,不算很高。下面看一下配置40
线程、50
请求次数在固定请求时间的情况下的模拟,看一下结果。
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
> {
> ① . "rt":103,
> ① . "failRate":0.0,
> ① . "threads":40,
> ① . "deviation":"2.7%",
> ① . "errorRate":0.0,
> ① . "executeTotal":2000,
> ① . "qps2":377.8575477045154,
> ① . "total":2000,
> ① . "qps":388.3495145631068,
> ① . "startTime":"2021-04-05 15:43:19",
> ① . "endTime":"2021-04-05 15:43:24",
> ① . "mark":"FunTester误差分析051543",
> ① . "table":"eJztz08KgkAUBvC94B3eARQUooUH6ADhBQQHEpopHANbRv9ctnLZEXLZfQSjYzTVrmiUCsbFNzz43Pje97Mt6vBGCxEymbH0WlXN+dQUu8vxQAOPsknKorjLCrItW39rzOR8JiSjMOEsoNyVLE2iKYkFd2jpchYnkWi70d6DJ4KeuwLf84hLh0d54PtD9fkHxa+vLgs1nSS9vFKXKzWfQnf1wy9rNW+xVdMt9mq+iUdTXdhWXW7U0Gvo2rRu7VPchT2oASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEWuENvcRZ6Q=="
> }
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
实测平均耗时103ms
,总的误差是2.7%
,可以说很相近,再试一组100
线程的,其他条件不变。
~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~ JSON ~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~~☢~
> {
> ① . "rt":102,
> ① . "failRate":0.0,
> ① . "threads":100,
> ① . "deviation":"3.7%",
> ① . "errorRate":0.0,
> ① . "executeTotal":5000,
> ① . "qps2":944.1087613293051,
> ① . "total":5000,
> ① . "qps":980.3921568627451,
> ① . "startTime":"2021-04-05 15:44:33",
> ① . "endTime":"2021-04-05 15:44:38",
> ① . "mark":"FunTester误差分析051544",
> ① . "table":"eJztzz0KwjAUwPE9kDu8A7RQF4cewANILxAwYMBEaVqoo4jS0amjR7Cj9ylUPIYpblXaih/w5IUHv0x5+XMGA84kNZG0iYyvZVmfT3W+vxwPMAoCSOaxFLMhb3DGu3dNpV0tjZUQKS