[性能对比] :Go语言内置正则库与grok、re2、hyperscan

已于 2024-10-14 11:22:49 修改
阅读量1.2k 收藏 8
点赞数 19
分类专栏： Go 文章标签： golang
于 2024-10-12 19:30:12 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/lcadna/article/details/142880703
版权

本项目的目的是测试和比较不同的正则表达式库在匹配手机号、银行卡号、身份证号等常见数据类型时的性能表现。我们选用了四个常见的正则表达式库：

  1. Go 原生正则库：Go 语言自带的 regexp 包。

  2. RE2 库：高效且安全的正则表达式库，Go 原生正则库的底层实现。

  3. Hyperscan 库：高性能、多模式匹配引擎，适用于需要处理大量数据的场景。

  4. Grok 库：专为日志解析设计的正则表达式库，支持丰富的模式匹配。

我们通过不同的测试样例（手机号、银行卡号、身份证号），对每个库的性能进行对比分析，包括：

  • 时间：执行匹配操作所耗费的时间。

  • 内存使用：匹配操作过程中占用的内存资源。

测试结果将帮助我们评估各库在不同场景下的性能差异，并为实际应用中的库选择提供参考

正则表达式

在该项目中，我们重点比较了四个不同的正则表达式库在匹配常见数据类型（如手机号、银行卡号、身份证号）时的性能表现。这些数据类型使用了以下三个正则表达式：

手机号正则表达式（phoneRegex）:

^1[3-9]\d{9}$

  • 匹配以1开头的中国大陆手机号，其中第二位数字为3-9，后面跟随9位数字。

银行卡号正则表达式（bankCardRegex）:

^\d{16,19}$

  1. 匹配长度为16到19位的纯数字，这通常是银行卡号的格式。

身份证号正则表达式（idCardRegex）:

^\d{17}[\dX]$

  • 匹配大陆身份证号，由17位数字和最后一位校验码（可能为数字或X）组成。

代码

package main

import (
    "fmt"
    "regexp"
    "runtime"
    "time"

    "github.com/flier/gohs/hyperscan"
    "github.com/trivago/grok"
    "github.com/wasilibs/go-re2"
)

// 测试样例：手机号、银行卡号、身份证号列表
var phoneNumbers = []string{
    "13812345678", "15898765432", "18912344321", "13987654321",
}

var bankCardNumbers = []string{
    "6217003810042196610", "6222023810001009123", "6217003810042196611",
}

var idCardNumbers = []string{
    "11010519880605371X", "32038219880509581X", "110105198806053719",
}

// 正则表达式
var phoneRegex = `^1[3-9]\d{9}$`
var bankCardRegex = `^\d{16,19}$`
var idCardRegex = `^\d{17}[\dX]$`

// 匹配函数类型
type matchFunc func(string, string) bool

// Go 自带的正则库
func matchWithGoRegexp(s string, regex string) bool {
    re := regexp.MustCompile(regex)
    return re.MatchString(s)
}

// 使用 re2 库
func matchWithre2(s string, regex string) bool {
    r, _ := re2.Compile(regex)
    return r.MatchString(s)
}

// 使用 hyperscan 库
func matchWithhyperscan(s string, regex string) bool {
    matched, _ := hyperscan.Match(regex, []byte(s))
    return matched
}

// 使用 grok 库
func matchWithgrok(s string, regex string) bool {
    g, err := grok.New(grok.Config{Patterns: grok.DefaultPatterns})
    if err != nil {
        panic(err)
    }
    compiledGrok, err := g.Compile(regex)
    if err != nil {
        panic(err)
    }
    return compiledGrok.MatchString(s)
}

// 性能测试
func testRegexPerformance(name string, fn matchFunc, testCases []string, regex string, iterations int) {
    fmt.Printf("Testing %s...\n", name)

    // 测试开始时间和内存
    start := time.Now()
    var memStart runtime.MemStats
    runtime.ReadMemStats(&memStart)

    // 生成足够的测试用例
    totalCases := make([]string, 0, iterations)
    for i := 0; i < iterations; i++ {
        totalCases = append(totalCases, testCases[i%len(testCases)]) // 循环使用测试案例
    }

    for _, testCase := range totalCases {
        fn(testCase, regex)
    }

    // 测试结束时间和内存
    elapsed := time.Since(start)
    var memEnd runtime.MemStats
    runtime.ReadMemStats(&memEnd)

    memUsed := int64(memEnd.TotalAlloc) - int64(memStart.TotalAlloc) // 转换为 int64

    // 使用更大单位显示内存使用
    if memUsed < 1024 {
        fmt.Printf("%s took %s, Memory used: %.2f bytes\n", name, elapsed, float64(memUsed))
    } else if memUsed < 1024*1024 {
        fmt.Printf("%s took %s, Memory used: %.2f KB\n", name, elapsed, float64(memUsed)/1024)
    } else if memUsed < 1024*1024*1024 {
        fmt.Printf("%s took %s, Memory used: %.2f MB\n", name, elapsed, float64(memUsed)/(1024*1024))
    } else {
        fmt.Printf("%s took %s, Memory used: %.2f GB\n", name, elapsed, float64(memUsed)/(1024*1024*1024))
    }

    // // 垃圾回收并测量内存
    // runtime.GC()
    // runtime.ReadMemStats(&memEnd)
    // // 计算 GC 后的内存使用
    // memAfterGC := int64(memEnd.TotalAlloc) - int64(memStart.TotalAlloc) // 转换为 int64

    // // 使用更大单位显示内存使用
    // if memAfterGC < 1024 {
    //  fmt.Printf("%s Memory used after GC:%.2f bytes\n\n", name, float64(memAfterGC))
    // } else if memAfterGC < 1024*1024 {
    //  fmt.Printf("%s Memory used after GC: %.2f KB\n\n", name, float64(memAfterGC)/1024)
    // } else if memAfterGC < 1024*1024*1024 {
    //  fmt.Printf("%s Memory used after GC: %.2f MB\n\n", name, float64(memAfterGC)/(1024*1024))
    // } else {
    //  fmt.Printf("%s Memory used after GC: %.2f GB\n\n", name, float64(memAfterGC)/(1024*1024*1024))
    // }
}

func main() {
    var iterationsList []int = []int{1, 10, 100, 1000, 10000, 100000}

    for _, iterations := range iterationsList {
        fmt.Printf("\nTesting with %d iterations...\n", iterations)

        // 测试手机号正则匹配
        testRegexPerformance("Go Regex (Phone)", matchWithGoRegexp, phoneNumbers, phoneRegex, iterations)
        testRegexPerformance("grok (Phone)", matchWithgrok, phoneNumbers, phoneRegex, iterations)
        testRegexPerformance("RE2 (Phone)", matchWithre2, phoneNumbers, phoneRegex, iterations)
        testRegexPerformance("Hyperscan (Phone)", matchWithhyperscan, phoneNumbers, phoneRegex, iterations)

        // 测试银行卡号正则匹配
        testRegexPerformance("Go Regex (Bank Card)", matchWithGoRegexp, bankCardNumbers, bankCardRegex, iterations)
        testRegexPerformance("grok (Bank Card)", matchWithgrok, bankCardNumbers, bankCardRegex, iterations)
        testRegexPerformance("RE2 (Bank Card)", matchWithre2, bankCardNumbers, bankCardRegex, iterations)
        testRegexPerformance("Hyperscan (Bank Card)", matchWithhyperscan, bankCardNumbers, bankCardRegex, iterations)

        // 测试身份证号正则匹配
        testRegexPerformance("Go Regex (ID Card)", matchWithGoRegexp, idCardNumbers, idCardRegex, iterations)
        testRegexPerformance("grok (ID Card)", matchWithgrok, idCardNumbers, idCardRegex, iterations)
        testRegexPerformance("RE2 (ID Card)", matchWithre2, idCardNumbers, idCardRegex, iterations)
        testRegexPerformance("Hyperscan (ID Card)", matchWithhyperscan, idCardNumbers, idCardRegex, iterations)
    }
}

输出


Testing with 1 iterations...
Testing Go Regex (Phone)...
Go Regex (Phone) took 147µs, Memory used: 5.67 KB
Testing grok (Phone)...
grok (Phone) took 434µs, Memory used: 234.77 KB
Testing RE2 (Phone)...
RE2 (Phone) took 338.4µs, Memory used: 169.16 KB
Testing Hyperscan (Phone)...
Hyperscan (Phone) took 961.1µs, Memory used: 800.00 bytes
Testing Go Regex (Bank Card)...
Go Regex (Bank Card) took 47.6µs, Memory used: 8.27 KB
Testing grok (Bank Card)...
grok (Bank Card) took 388.6µs, Memory used: 194.93 KB
Testing RE2 (Bank Card)...
RE2 (Bank Card) took 323µs, Memory used: 1.07 KB
Testing Hyperscan (Bank Card)...
Hyperscan (Bank Card) took 757.7µs, Memory used: 760.00 bytes
Testing Go Regex (ID Card)...
Go Regex (ID Card) took 174.6µs, Memory used: 7.36 KB
Testing grok (ID Card)...
grok (ID Card) took 554.9µs, Memory used: 193.61 KB
Testing RE2 (ID Card)...
RE2 (ID Card) took 215.6µs, Memory used: 1.07 KB
Testing Hyperscan (ID Card)...
Hyperscan (ID Card) took 598.5µs, Memory used: 760.00 bytes

Testing with 10 iterations...
Testing Go Regex (Phone)...
Go Regex (Phone) took 169.1µs, Memory used: 50.55 KB
Testing grok (Phone)...
grok (Phone) took 3.647201ms, Memory used: 1.87 MB
Testing RE2 (Phone)...
RE2 (Phone) took 616µs, Memory used: 10.70 KB
Testing Hyperscan (Phone)...
Hyperscan (Phone) took 4.3829ms, Memory used: 7.31 KB
Testing Go Regex (Bank Card)...
Go Regex (Bank Card) took 235.3µs, Memory used: 82.73 KB
Testing grok (Bank Card)...
grok (Bank Card) took 4.0071ms, Memory used: 1.90 MB
Testing RE2 (Bank Card)...
RE2 (Bank Card) took 837.1µs, Memory used: 10.70 KB
Testing Hyperscan (Bank Card)...
Hyperscan (Bank Card) took 5.6729ms, Memory used: 7.34 KB
Testing Go Regex (ID Card)...
Go Regex (ID Card) took 199.4µs, Memory used: 73.59 KB
Testing grok (ID Card)...
grok (ID Card) took 3.9797ms, Memory used: 1.89 MB
Testing RE2 (ID Card)...
RE2 (ID Card) took 782.1µs, Memory used: 10.70 KB
Testing Hyperscan (ID Card)...
Hyperscan (ID Card) took 5.2922ms, Memory used: 7.34 KB

Testing with 100 iterations...
Testing Go Regex (Phone)...
Go Regex (Phone) took 1.3595ms, Memory used: 505.66 KB
Testing grok (Phone)...
grok (Phone) took 35.803201ms, Memory used: 18.85 MB
Testing RE2 (Phone)...
RE2 (Phone) took 2.9211ms, Memory used: 107.22 KB
Testing Hyperscan (Phone)...
Hyperscan (Phone) took 37.658501ms, Memory used: 72.84 KB
Testing Go Regex (Bank Card)...
Go Regex (Bank Card) took 735.6µs, Memory used: 827.62 KB
Testing grok (Bank Card)...
grok (Bank Card) took 33.308501ms, Memory used: 19.32 MB
Testing RE2 (Bank Card)...
RE2 (Bank Card) took 5.6718ms, Memory used: 107.22 KB
Testing Hyperscan (Bank Card)...
Hyperscan (Bank Card) took 56.628102ms, Memory used: 73.62 KB
Testing Go Regex (ID Card)...
Go Regex (ID Card) took 706.4µs, Memory used: 736.12 KB
Testing grok (ID Card)...
grok (ID Card) took 33.489901ms, Memory used: 19.07 MB
Testing RE2 (ID Card)...
RE2 (ID Card) took 4.8854ms, Memory used: 107.22 KB
Testing Hyperscan (ID Card)...
Hyperscan (ID Card) took 48.555101ms, Memory used: 74.23 KB

Testing with 1000 iterations...
Testing Go Regex (Phone)...
Go Regex (Phone) took 6.3989ms, Memory used: 4.94 MB
Testing grok (Phone)...
grok (Phone) took 296.184308ms, Memory used: 188.85 MB
Testing RE2 (Phone)...
RE2 (Phone) took 35.4893ms, Memory used: 1.26 MB
Testing Hyperscan (Phone)...
Hyperscan (Phone) took 353.388309ms, Memory used: 734.77 KB
Testing Go Regex (Bank Card)...
Go Regex (Bank Card) took 8.666801ms, Memory used: 8.09 MB
Testing grok (Bank Card)...
grok (Bank Card) took 325.838308ms, Memory used: 192.00 MB
Testing RE2 (Bank Card)...
RE2 (Bank Card) took 56.139501ms, Memory used: 1.05 MB
Testing Hyperscan (Bank Card)...
Hyperscan (Bank Card) took 537.913414ms, Memory used: 742.56 KB
Testing Go Regex (ID Card)...
Go Regex (ID Card) took 9.1288ms, Memory used: 7.24 MB
Testing grok (ID Card)...
grok (ID Card) took 369.78511ms, Memory used: 191.27 MB
Testing RE2 (ID Card)...
RE2 (ID Card) took 51.905201ms, Memory used: 1.05 MB
Testing Hyperscan (ID Card)...
Hyperscan (ID Card) took 609.0739ms, Memory used: 742.58 KB

Testing with 10000 iterations...
Testing Go Regex (Phone)...
Go Regex (Phone) took 97.1449ms, Memory used: 49.43 MB
Testing grok (Phone)...
grok (Phone) took 3.047004091s, Memory used: 1.84 GB
Testing RE2 (Phone)...
RE2 (Phone) took 286.305799ms, Memory used: 10.83 MB
Testing Hyperscan (Phone)...
Hyperscan (Phone) took 3.42459599s, Memory used: 7.34 MB
Testing Go Regex (Bank Card)...
Go Regex (Bank Card) took 75.740699ms, Memory used: 80.81 MB
Testing grok (Bank Card)...
grok (Bank Card) took 2.978225733s, Memory used: 1.87 GB
Testing RE2 (Bank Card)...
RE2 (Bank Card) took 596.521708ms, Memory used: 10.96 MB
Testing Hyperscan (Bank Card)...
Hyperscan (Bank Card) took 5.02266107s, Memory used: 7.29 MB
Testing Go Regex (ID Card)...
Go Regex (ID Card) took 71.491502ms, Memory used: 71.88 MB
Testing grok (ID Card)...
grok (ID Card) took 2.973578556s, Memory used: 1.86 GB
Testing RE2 (ID Card)...
RE2 (ID Card) took 542.933711ms, Memory used: 10.93 MB
Testing Hyperscan (ID Card)...
Hyperscan (ID Card) took 4.130735778s, Memory used: 7.31 MB

Testing with 100000 iterations...
Testing Go Regex (Phone)...
Go Regex (Phone) took 560.546612ms, Memory used: 493.68 MB
Testing grok (Phone)...
grok (Phone) took 31.42383057s, Memory used: 18.38 GB
Testing RE2 (Phone)...
RE2 (Phone) took 3.28167285s, Memory used: 109.65 MB
Testing Hyperscan (Phone)...
Hyperscan (Phone) took 36.851407785s, Memory used: 71.94 MB
Testing Go Regex (Bank Card)...
Go Regex (Bank Card) took 752.315092ms, Memory used: 808.03 MB
Testing grok (Bank Card)...
grok (Bank Card) took 31.07751491s, Memory used: 18.69 GB
Testing RE2 (Bank Card)...
RE2 (Bank Card) took 5.978553846s, Memory used: 109.83 MB
Testing Hyperscan (Bank Card)...
Hyperscan (Bank Card) took 53.647871594s, Memory used: 72.52 MB
Testing Go Regex (ID Card)...
Go Regex (ID Card) took 737.583502ms, Memory used: 718.77 MB
Testing grok (ID Card)...
grok (ID Card) took 32.139839778s, Memory used: 18.61 GB
Testing RE2 (ID Card)...
RE2 (ID Card) took 5.738499301s, Memory used: 109.50 MB
Testing Hyperscan (ID Card)...
Hyperscan (ID Card) took 44.914807112s, Memory used: 72.86 MB

时间表 -电话

迭代次数Go Regex (Phone)grok (Phone)RE2 (Phone)Hyperscan (Phone)耗时最短引擎耗时最长引擎
1147µs434µs338.4µs961.1µsGo Regex (Phone)Hyperscan (Phone)
10169.1µs3.647201ms616µs4.3829msGo Regex (Phone)grok (Phone)
1001.3595ms35.803201ms2.9211ms37.658501msGo Regex (Phone)grok (Phone)
10006.3989ms296.184308ms35.4893ms353.388309msGo Regex (Phone)grok (Phone)
1000097.1449ms3.047004091s286.305799ms3.42459599sGo Regex (Phone)grok (Phone)
100000560.546612ms31.42383057s3.28167285s36.851407785sGo Regex (Phone)grok (Phone)

内存表 -电话

迭代次数Go Regex (Phone)grok (Phone)RE2 (Phone)Hyperscan (Phone)内存占用最少引擎内存占用最多引擎
15.67 KB234.77 KB169.16 KB800.00 bytesHyperscan (Phone)grok (Phone)
1050.55 KB1.87 MB10.70 KB7.31 KBRE2 (Phone)grok (Phone)
100505.66 KB18.85 MB107.22 KB72.84 KBHyperscan (Phone)grok (Phone)
10004.94 MB188.85 MB1.26 MB734.77 KBRE2 (Phone)grok (Phone)
1000049.43 MB1.84 GB10.83 MB7.34 MBRE2 (Phone)grok (Phone)
100000493.68 MB18.38 GB109.65 MB71.94 MBRE2 (Phone)grok (Phone)

时间表 - 银行卡

迭代次数Go Regex (Bank Card)grok (Bank Card)RE2 (Bank Card)Hyperscan (Bank Card)耗时最短引擎耗时最长引擎
147.6µs388.6µs323µs757.7µsGo Regex (Bank Card)Hyperscan (Bank Card)
10235.3µs4.0071ms837.1µs5.6729msGo Regex (Bank Card)Hyperscan (Bank Card)
100735.6µs33.308501ms5.6718ms56.628102msGo Regex (Bank Card)Hyperscan (Bank Card)
10008.666801ms325.838308ms56.139501ms537.913414msGo Regex (Bank Card)Hyperscan (Bank Card)
1000075.740699ms2.978225733s596.521708ms5.02266107sGo Regex (Bank Card)grok (Bank Card)
100000752.315092ms31.07751491s5.978553846s53.647871594sGo Regex (Bank Card)grok (Bank Card)

内存表 - 银行卡

迭代次数Go Regex (Bank Card)grok (Bank Card)RE2 (Bank Card)Hyperscan (Bank Card)内存占用最少引擎内存占用最多引擎
18.27 KB194.93 KB1.07 KB760.00 bytesRE2 (Bank Card)grok (Bank Card)
1082.73 KB1.90 MB10.70 KB7.34 KBHyperscan (Bank Card)grok (Bank Card)
100827.62 KB19.32 MB107.22 KB73.62 KBHyperscan (Bank Card)grok (Bank Card)
10008.09 MB192.00 MB1.05 MB742.56 KBRE2 (Bank Card)grok (Bank Card)
1000080.81 MB1.87 GB10.96 MB7.29 MBRE2 (Bank Card)grok (Bank Card)
100000808.03 MB18.69 GB109.83 MB72.52 MBRE2 (Bank Card)grok (Bank Card)

时间表 - 身份证

迭代次数Go Regex (ID Card)grok (ID Card)RE2 (ID Card)Hyperscan (ID Card)耗时最短引擎耗时最长引擎
1174.6µs554.9µs215.6µs598.5µsRE2 (ID Card)grok (ID Card)
10199.4µs3.9797ms782.1µs5.2922msGo Regex (ID Card)Hyperscan (ID Card)
100706.4µs33.489901ms4.8854ms48.555101msRE2 (ID Card)Hyperscan (ID Card)
10009.1288ms369.78511ms51.905201ms609.0739msRE2 (ID Card)grok (ID Card)
1000071.491502ms2.973578556s542.933711ms4.130735778sRE2 (ID Card)grok (ID Card)
100000737.583502ms32.139839778s5.738499301s44.914807112sRE2 (ID Card)grok (ID Card)

内存表 - 身份证

迭代次数Go Regex (ID Card)grok (ID Card)RE2 (ID Card)Hyperscan (ID Card)内存占用最多引擎内存占用最多引擎
17.36 KB193.61 KB1.07 KB760.00 bytesHyperscan (ID Card)grok (ID Card)
1073.59 KB1.89 MB10.70 KB7.34 KBHyperscan (ID Card)grok (ID Card)
100736.12 KB19.07 MB107.22 KB74.23 KBHyperscan (ID Card)grok (ID Card)
10007.24 MB191.27 MB1.05 MB742.58 KBRE2 (ID Card)grok (ID Card)
1000071.88 MB1.86 GB10.93 MB7.31 MBRE2 (ID Card)grok (ID Card)
100000718.77 MB18.61 GB109.50 MB72.86 MBRE2 (ID Card)grok (ID Card)

时间表

迭代次数Go Regex (Phone)grok (Phone)RE2 (Phone)Hyperscan (Phone)Go Regex (Bank Card)grok (Bank Card)RE2 (Bank Card)Hyperscan (Bank Card)Go Regex (ID Card)grok (ID Card)RE2 (ID Card)Hyperscan (ID Card)
1147µs434µs338.4µs961.1µs47.6µs388.6µs323µs757.7µs174.6µs554.9µs215.6µs598.5µs
10169.1µs3.647201ms616µs4.3829ms235.3µs4.0071ms837.1µs5.6729ms199.4µs3.9797ms782.1µs5.2922ms
1001.3595ms35.803201ms2.9211ms37.658501ms735.6µs33.308501ms5.6718ms56.628102ms706.4µs33.489901ms4.8854ms48.555101ms
10006.3989ms296.184308ms35.4893ms353.388309ms8.666801ms325.838308ms56.139501ms537.913414ms9.1288ms369.78511ms51.905201ms609.0739ms
1000097.1449ms3.047004091s286.305799ms3.42459599s75.740699ms2.978225733s596.521708ms5.02266107s71.491502ms2.973578556s542.933711ms4.130735778s
100000560.546612ms31.42383057s3.28167285s36.851407785s752.315092ms31.07751491s5.978553846s53.647871594s737.583502ms32.139839778s5.738499301s44.914807112s

内存表

迭代次数Go Regex (Phone)grok (Phone)RE2 (Phone)Hyperscan (Phone)Go Regex (Bank Card)grok (Bank Card)RE2 (Bank Card)Hyperscan (Bank Card)Go Regex (ID Card)grok (ID Card)RE2 (ID Card)Hyperscan (ID Card)
15.67 KB234.77 KB169.16 KB800.00 bytes8.27 KB194.93 KB1.07 KB760.00 bytes7.36 KB193.61 KB1.07 KB760.00 bytes
1050.55 KB1.87 MB10.70 KB7.31 KB82.73 KB1.90 MB10.70 KB7.34 KB73.59 KB1.89 MB10.70 KB7.34 KB
100505.66 KB18.85 MB107.22 KB72.84 KB827.62 KB19.32 MB107.22 KB73.62 KB736.12 KB19.07 MB107.22 KB74.23 KB
10004.94 MB188.85 MB1.26 MB734.77 KB8.09 MB192.00 MB1.05 MB742.56 KB7.24 MB191.27 MB1.05 MB742.58 KB
1000049.43 MB1.84 GB10.83 MB7.34 MB80.81 MB1.87 GB10.96 MB7.29 MB71.88 MB1.86 GB10.93 MB7.31 MB
100000493.68 MB18.38 GB109.65 MB71.94 MB808.03 MB18.69 GB109.83 MB72.52 MB718.77 MB18.61 GB109.50 MB72.86 MB

性能测试结果分析

  • Go 自带正则：在较小的测试集上表现良好，内存占用较低，时间性能也不错，但随着迭代数增加，内存使用显著上升。

  • grok：grok 的性能不如其他库，尤其在多次迭代时，内存消耗巨大，时间开销也较高。因为 grok 通常用于更复杂的日志解析，它的开销在这种简单匹配场景下会更大。

  • RE2：re2 的性能和内存使用都很稳定，尤其在大量迭代下，表现优于 Go 原生正则，并且有较低的内存占用。

  • Hyperscan：对于手机号、银行卡号和身份证号的匹配，hyperscan 表现出色，尤其是内存使用最低，但随着迭代增加，时间开销有所上升。

从结果可以看出：

  1. Go 自带正则库：适合中等规模的正则匹配任务，如果对时间和内存要求不特别苛刻，可以放心使用。

  2. RE2：更适合需要在较大规模数据集上进行正则匹配的情况，尤其是它的内存使用更高效。

  3. Hyperscan：在内存使用上最为出色，适合超大规模数据匹配任务，虽然时间开销有所增加，但仍然是优秀的选择。

  4. grok：如果仅仅是简单的正则匹配，不建议使用 grok，因为它是为复杂的日志解析设计的工具。

配置注意

sudo apt-get update
sudo apt-get install pkg-config
sudo apt-get install libhyperscan-dev

参考

hyperscan package - github.com/flier/gohs/hyperscan - Go Packages

grok package - github.com/trivago/grok - Go Packages  Comparison of regular expression typesgrok package - github.com/trivago/grok - Go Packages  Performance comparison of regular expression enginesgrok package - github.com/trivago/grok - Go Packages 

Logstash数据处理服务的过滤插件Grok正则匹配概念以及正则语法
江晓龙的博客
07-13 1026
收集来的日志往往都是非结构的日志，当然也可以通过配置将日志输出成结构化日志，Nginx就支持将日志输出成结构化的功能，但是很多一些开源软件是不支持将日志结构化的，针对这种需求，我们就可以使用正则将日志数据输出成功格式化数据。Grok正则插件是作用通过正则表达式将非结构的日志输出成结构化的日志，便于我们在kibana上对日志数据进行筛选。使用Grok正则匹配配置之前，需要对正则表达式有一定的基础。Grok插件自带了一些内置的正则匹配模式，将常用的一些匹配表达式（获取IP、文本字符串、请求URL、数字）写入到了
RE2 - 高效安全的正则表达式库
最新发布
gitblog_00853的博客
11-05 689
RE2 - 高效安全的正则表达式库 re2 Ruby bindings to RE2, a "fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, ...
gohs:HyperScan https的GoLang绑定
03-21
哥斯 Golang与Intel HyperScan regex匹配库的绑定： 建造 注意：默认情况下， gohs将使用Hyperscan v5 API，您也可以使用hyperscan_v4标签为Hyperscan v4进行构建。 go get -u -tags hyperscan_v4 github.com/flier/gohs/hyperscan 执照 您可以选择根据Apache许可（ ）或MIT许可（ ）该项目。 贡献 除非您明确声明，否则您有意提交以Apache-2.0许可中定义的方式包含在期货中的任何供款，应如上所述获得双重许可，而无需任何其他条款或条件。
正则表达式库
weixin_34128839的博客
11-17 100
2019独角兽企业重金招聘Python工程师标准>>> ...
C/C++中的正则表达式库GNU Regex Library&PCRE/PCRE++
坚持是世界上每个有所建树的人共有的品质！
12-16 957
一、GNU Regex Library 1. 什么是GNU正则表达式库(GNU Regex Library) ?GNU正则表达式库是glibc(GNU C Library)的一部分，它提供与POSIX标准兼容的正则表达式匹配的接口。这里是其主页：http://www.gnu.org/s/libc/manual/html_node/Regular-Expressions.html下载该库点这里：g
groktoregex:将 logstash grok 别名转换为正则表达式
06-06
GrokToRegex 是一个简单的命令行实用程序，可将已知的 grok 别名转换为其相应的正则表达式值。 安装 要安装 GrokToRegex，请使用 go get go get github.com/jrxfive/groktoregex 用法 一旦构建 groktoregex 只需要...
korg：根据可重用模式组合正则表达式（Python Logstash grok克隆）
02-04
korg是ruby logstash grok正则表达式模式的python端口。 快速开始 Logstash带有100多种内置模式，用于构造非结构化数据。 在处理诸如apache，linux，haproxy，aws等之类的日志数据时，绝对应该利用此优势。 但是，...
Pygrok：Python实现的Grok正则表达式库
在Python中，正则表达式可以通过内置的`re`模块来实现，而pygrok库则在此基础上进行了扩展，提供了更高级的功能。 4. “开源”意味着这个软件的源代码对所有人都是公开的，任何人都可以查看、修改和分发这些代码。...
hyperscan 安装测试以及性能分析报告
02-23
2015年10月19日，intel将它的高速正则表达式匹配引擎hyperscan开源了，版本4.0，基于BSD许可。这个基于自动机（Automata)的引擎经过了多年开发（2008年起），经过不断优化与完善，效率非常之高，虽然没有pcre等对正则语法支持全面，但非常适用于网络设备。用户可以在网络设备数据面(Data Plane)使用hyperscan进行规则匹配，实现高性能DPI/lPS/IDS等应用。之前开源的dpdk，搭配hyperscan，简直是双剑合璧。 从此之后，DPI/IPS/IDS/NGFW开发人员可以把精力更多地放在业务上，而不需要苦于优化报文转发(dpdk之功）和规则匹配(hyperscan之力)的性能。对于DPI来说，从此之后，也许特征/规则才是核心价值，软件本身已经不重要了。而基于dpdk的软件产品，凭借其不输太多的性能、较低的成本、较低的开发门槛和灵活性，也对相同功能的硬件产品发起了挑战。
grok:流行的grok库的Rust端口，用于文本和日志处理
05-06
ok grok库使您可以快速解析潜在的非结构化数据并将其匹配为结构化结果。 在分析各种日志文件时，它特别有用。 这个版本主要是一个移植，而则从原始的获得了灵感。 用法 将此添加到您的Cargo.toml ： [ dependencies ] grok = " 1.2 " 这到你的板条箱根： extern crate grok; 这是一个简单的示例，该示例存储模式，对其进行编译，然后匹配其上的一行： extern crate grok; use grok :: Grok; fn main () { // Instantiate Grok let mut grok = Grok :: default (); // Insert a definition which might be a regex or an alias grok. insert_d
groker库：解析Go语言中的grok模式
1. **Grok模式解析器**: groker是一个用Go语言编写的库，专注于解析符合grok模式的文本数据。 2. **标准输入**: 它支持从标准输入（stdin）读取数据，这使得它能够与Unix/Linux环境中的管道命令结合使用。 3. **JSON...
常用的正则表达式类库
muli`慕李
09-26 537
常用的正则表达式类库 一、校验数字的js正则表达式 1 数字：^[0-9]*$ 2 n位的数字：^d{n}$ 3 至少n位的数字：^d{n,}$ 4 m-n位的数字：^d{m,n}$ 5 零和非零开头的数字：^(0|[1-9][0-9]*)$ 6 非零开头的最多带两位小数的数字：^([1-9][0-9]*)+(.[0-9]{1,2})?$ 7 带1-2位小数的正数或负数：^(-)?d+(.d{1,2})?$ 8 正数、负数、和小数：^(-|+)?d+(.d+)?$ 9 有两位小数的正实数：^[0-9]+(.
flier/gohs go hyperscan 编译报错
guoguangwu的专栏
06-15 946
hyperscan
探索高效正则表达式处理利器：Xindoo.regex
gitblog_00064的博客
04-02 318
探索高效正则表达式处理利器：Xindoo.regex regexJava实现的正则表达式引擎项目地址:https://gitcode.com/gh_mirrors/regex4/regex 在编程世界里，正则表达式（Regex）是数据提取、文本验证和字符串操作的强大工具。然而，它的复杂性和学习曲线常常让开发者望而却步。这就是项目的出现目的，它是一款开源的Java库，旨在简化正则表达式的使用，并提...
从PCRE到Hyperscan
weixin_37097605的博客
04-21 835
点击蓝字关注DPDK开源社区 Hyperscan作为高性能的正则表达式匹配库，使用者通常乐意将其与传统的正则表达式匹配库进行比较。“为什么要用Hypersc...
应用安全系列之十二：正则表达式注入
jimmyleeee的专栏
03-07 3018
本系列文章主旨在于介绍一些漏洞类型产生的基本原理，探索最基础的解决问题的措施，不排除有些语言或者系统提供的安全的API可以更好地更直接地解决问题，也不排除可以严格地输入验证来解决。 如果有不妥之处，希望可以留言指出。谢谢！ ...
Logstash：你喜欢 Grok 吗？
Elastic 中国社区官方博客
03-27 776
Grok（动词）直觉地或同理心地理解（某事）。解析日志数据时最常见的任务之一是将原始文本行分解为一组其他工具可以操作的结构化字段。如果你使用的是 Elastic Stack，则可以利用 Elasticsearch 的聚合和 Kibana 的可视化，从日志中提取的信息（如 IP 地址、时间戳和域特定数据）中回答业务和运营问题。对于 Logstash，此解构工作由执行，这是一个过滤器插件，可帮助你描述日志格式的结构。有，它们抽象了诸如和等概念。简单吧？是的！太棒了！我们在这里完成了吗？不！因为...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值