shell 三剑客-grep

原创已于 2025-12-06 16:51:36 修改 · 318 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#shell

于 2024-06-19 16:52:53 首次发布

shell 专栏收录该内容

4 篇文章

订阅专栏

shell 三剑客是指在Linux/Unix系统下使用最广泛的命令行工具：grep、sed、awk
三剑客都是文本处理工具可用于搜索、替换和处理大量文本数据，三者通常是和正则表达式一起使用
正则表达式

\       转义符，将特殊字符进行转义，忽略其特殊意义
^       匹配行首，awk中，^则是匹配字符串的开始 
$       匹配行尾，awk中，$则是匹配字符串的结尾
^$      表示空行
.       匹配除换行符\n之外的任意单个字符
.*      匹配所有
[ ]     匹配包含在[字符]之中的任意一个字符
[^ ]    匹配[^字符]之外的任意一个字符
[ - ]   匹配[]中指定范围内的任意一个字符
?       匹配之前的项1次或者0次
+       匹配之前的项1次或者多次
*       匹配之前的项0次或者多次, .*
()      匹配表达式，创建一个用于匹配的子串 
{ n }   匹配之前的项n次，n是可以为0的正整数
{n,}    之前的项至少需要匹配n次
{n,m}   指定之前的项至少匹配n次，最多匹配m次，n<=m
|       交替匹配|两边的任意一项ab(c|d)匹配abc或abd

三剑客–grep
grep 擅长查找过滤
可以通过grep --help 查看
常用参数


-i：忽略大小写
-l：只输出包含匹配字符的文件名
-n：显示匹配行及行号
-v：显示不包含匹配文本的所有行
-o：只显示匹配字符串的部分
-c：只输出匹配行的计数
-r：递归查询
-h：不显示匹配行所在文件名
-H：同时显示行和文件名
-A<显示列数>或--after-context=<显示列数>：显示匹配行下?行
-B<显示列数>或--before-context=<显示列数>：显示匹配行上?行
-C<显示列数>或--context=<显示列数>或-<显示列数>：显示匹配行前后?行
-E：扩展正则表达式
-F：固定字符串搜索
-G：基本正则表达式
-q：静默模式，不输出任何信息，通过echo $?查看是否匹配到指定字符串
-s：不显示不存在或无匹配文本的错误信息
 -w, --word-regexp         强制 PATTERN 仅完全匹配字词
 -x, --line-regexp         强制 PATTERN 仅完全匹配一行
 -z, 一个 0 字节的数据行，但不是空行

1.过滤以#开头的行和空白行
egrep 是grep 的变体默认使用扩展正则表达式，使egrep 处理复杂模式更灵活
grep 默认使用基础表达式 -E 默认使用基础正则表达式
grep 通常更快

egrep -v -n "^#|^$" 文件名称

在这里插入图片描述

grep -E -v -n "^#|^$" 文件名称

在这里插入图片描述
匹配任意字符，过滤空行

grep -n "." 文件名称

在这里插入图片描述
找到.sh文件查找包含for关键的文件显示行数
.表示当前目录
xargs 能够捕获一个命令的输出，然后传递给另外一个命令。由于很多命令不支持|管道来传递参数，而日常工作中有有这个必要，所以就有了 xargs 命令

 find . -name "*.sh" | xargs grep  -in "for*"

在这里插入图片描述
查找重复连续出现2次以上，必须是连续哦

 grep -Eni "(重复){2,}" 03.sh

在这里插入图片描述
错误语法：缺少(),如果是查找多个字符，需要（），单个字符可不需要

多个关键字组合使用时
表达and关系，使用 .*
表达or关系，使用 |

#两个关键字的顺序不是固定的，可以是乱序的，如果要固定就只写"测试.*重复" 或者"重复.*测试"
 grep -Eni "测试.*重复|重复.*测试" 03.sh

在这里插入图片描述

 grep -Eni "echo|参数" 03.sh

在这里插入图片描述
查看最近有哪些IP以root身份登陆过我们的机器，统计每个IP登陆的次数。我们的思路如下：

1.通过last列出最近登陆系统的所有用户。
2.过滤出以root登陆系统的IP。
3.过滤出远程登陆的IP。
4.统计出每个IP登陆的次数。
uniq -c 统计出现的次数

#注意：'{print $3}' 使用的单引号
last |grep "root" |awk '{print $3}'|sort | uniq -c

在这里插入图片描述

创建文件

vi a.txt
#插入一下内容
{"@t":"2025-12-04T09:46:02.9324170Z","@mt":" FetchUserInfoAsync commId:1600024648_1786598449_1764839025 pageNumber:0 PageSize:100 请求结果：12","@tr":"1ef016ec10ddd88d35fab4f374cf59fc","@sp":"899e490d4a0a2ce2","response":{"Total":12,"UserList":[{"RoomStr":"1786598449","UserId":"37e831b7-0f46-400c-81a2-739b1366fe99","JoinTs":1764839025,"LeaveTs":1764841562,"DeviceType":"Windows/Edge 142.0.0","SdkVersion":"5.8.3.0","ClientIp":"180.194.142.5","Finished":false,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561187","JoinTs":1764839099,"LeaveTs":1764840777,"DeviceType":"ADY-AL10","SdkVersion":"12.6.0.18891","ClientIp":"112.97.66.98","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561493","JoinTs":1764840576,"LeaveTs":1764840579,"DeviceType":"SCMR-W09","SdkVersion":"12.6.0.18891","ClientIp":"114.92.4.20","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561491","JoinTs":1764840675,"LeaveTs":1764840731,"DeviceType":"iPhone18,4","SdkVersion":"12.6.0.18894","ClientIp":"223.104.250.14","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561187","JoinTs":1764840778,"LeaveTs":1764840782,"DeviceType":"ADY-AL10","SdkVersion":"12.6.0.18891","ClientIp":"112.97.66.98","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561187","JoinTs":1764840863,"LeaveTs":1764841562,"DeviceType":"ADY-AL10","SdkVersion":"12.6.0.18891","ClientIp":"112.97.66.98","Finished":false,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020558269","JoinTs":1764840998,"LeaveTs":1764841012,"DeviceType":"NIC-AN00","SdkVersion":"12.6.0.18891","ClientIp":"39.149.94.243","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020558269","JoinTs":1764841048,"LeaveTs":1764841053,"DeviceType":"NIC-AN00","SdkVersion":"12.6.0.18891","ClientIp":"39.149.94.243","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020558269","JoinTs":1764841093,"LeaveTs":1764841103,"DeviceType":"NIC-AN00","SdkVersion":"12.6.0.18891","ClientIp":"39.149.94.243","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020560828","JoinTs":1764841239,"LeaveTs":1764841562,"DeviceType":"M2102J2SC","SdkVersion":"12.6.0.18891","ClientIp":"120.242.52.230","Finished":false,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561509","JoinTs":1764841416,"LeaveTs":1764841418,"DeviceType":"MI 9","SdkVersion":"12.6.0.18891","ClientIp":"111.41.203.38","Finished":true,"IsStream":false,"$type":"UserInformation"},{"RoomStr":"1786598449","UserId":"1020561509","JoinTs":1764841426,"LeaveTs":1764841537,"DeviceType":"MI 9","SdkVersion":"12.6.0.18891","ClientIp":"111.41.203.38","Finished":true,"IsStream":false,"$type":"UserInformation"}],"RequestId":"227e096c-dea3-434c-9579-a1e3a778abd6","IsStream":false,"$type":"DescribeUserInfoResponse"},"request":{"CommId":"1600024648_1786598449_1764839025","StartTime":1764839025,"EndTime":1764841561,"SdkAppId":1600024648,"UserIds":null,"PageNumber":0,"PageSize":100,"IsStream":false,"$type":"DescribeUserInfoRequest"},"ActionId":"433f25c2-aff2-4d55-a83e-9ca0ed24976c","ActionName":"SalesSystem.Api.Controllers.InnerSerivces.BackgroundServices.JobManageTask.JobManageTaskController.EditSLSVLiveStreamingOneDayAsync (SalesSystem.Api)","RequestId":"0HNHJ37LT9B9A:00000001","RequestPath":"/api/JobManageTask/EditSLSVLiveStreamingOneDay","ConnectionId":"0HNHJ37LT9B9A","Acadsoc-Trace-ID":"a28dbf9443b24f9fb98a7165bb7abf39","RequestUrl":"http://localhost:3001/api/JobManageTask/EditSLSVLiveStreamingOneDay","RemoteIP":"::1","Application":"SalesSystem.Api"}
{"@t":"2025-12-04T09:46:03.3624378Z","@mt":" FetchUserInfoAsync commId:1600024648_1786598449_1764839025 pageNumber:1 PageSize:100 请求结果：0","@tr":"1ef016ec10ddd88d35fab4f374cf59fc","@sp":"899e490d4a0a2ce2","response":{"Total":0,"UserList":[],"RequestId":"2be44e7a-c678-4957-badb-3dbc3d45ee5e","IsStream":false,"$type":"DescribeUserInfoResponse"},"request":{"CommId":"1600024648_1786598449_1764839025","StartTime":1764839025,"EndTime":1764841561,"SdkAppId":1600024648,"UserIds":null,"PageNumber":1,"PageSize":100,"IsStream":false,"$type":"DescribeUserInfoRequest"},"ActionId":"433f25c2-aff2-4d55-a83e-9ca0ed24976c","ActionName":"SalesSystem.Api.Controllers.InnerSerivces.BackgroundServices.JobManageTask.JobManageTaskController.EditSLSVLiveStreamingOneDayAsync (SalesSystem.Api)","RequestId":"0HNHJ37LT9B9A:00000001","RequestPath":"/api/JobManageTask/EditSLSVLiveStreamingOneDay","ConnectionId":"0HNHJ37LT9B9A","Acadsoc-Trace-ID":"a28dbf9443b24f9fb98a7165bb7abf39","RequestUrl":"http://localhost:3001/api/JobManageTask/EditSLSVLiveStreamingOneDay","RemoteIP":"::1","Application":"SalesSystem.Api"}
{"@t":"2025-12-04T09:46:04.1118131Z","@mt":"插入SLSV2LiveStreamingUserFeatchTime 表 CommId：1600024648_1786598449_1764839025 i:1","@tr":"1ef016ec10ddd88d35fab4f374cf59fc","@sp":"899e490d4a0a2ce2","ActionId":"433f25c2-aff2-4d55-a83e-9ca0ed24976c","ActionName":"SalesSystem.Api.Controllers.InnerSerivces.BackgroundServices.JobManageTask.JobManageTaskController.EditSLSVLiveStreamingOneDayAsync (SalesSystem.Api)","RequestId":"0HNHJ37LT9B9A:00000001","RequestPath":"/api/JobManageTask/EditSLSVLiveStreamingOneDay","ConnectionId":"0HNHJ37LT9B9A","Acadsoc-Trace-ID":"a28dbf9443b24f9fb98a7165bb7abf39","RequestUrl":"http://localhost:3001/api/JobManageTask/EditSLSVLiveStreamingOneDay","RemoteIP":"::1","Application":"SalesSystem.Api"}
{"@t":"2025-12-04T09:46:04.4060113Z","@mt":" FetchUserInfoAsync commId:1600024648_1683274356_1764838242 pageNumber:0 PageSize:100 请求结果：1","@tr":"1ef016ec10ddd88d35fab4f374cf59fc","@sp":"899e490d4a0a2ce2","response":{"Total":1,"UserList":[{"RoomStr":"1683274356","UserId":"1020283704","JoinTs":1764838242,"LeaveTs":1764838380,"DeviceType":"iPad7,11","SdkVersion":"12.6.0.18894","ClientIp":"183.193.126.18","Finished":true,"IsStream":false,"$type":"UserInformation"}],"RequestId":"ec6368b9-d426-47ce-bf0e-f811b5bf3001","IsStream":false,"$type":"DescribeUserInfoResponse"},"request":{"CommId":"1600024648_1683274356_1764838242","StartTime":1764838242,"EndTime":1764838380,"SdkAppId":1600024648,"UserIds":null,"PageNumber":0,"PageSize":100,"IsStream":false,"$type":"DescribeUserInfoRequest"},"ActionId":"433f25c2-aff2-4d55-a83e-9ca0ed24976c","ActionName":"SalesSystem.Api.Controllers.InnerSerivces.BackgroundServices.JobManageTask.JobManageTaskController.EditSLSVLiveStreamingOneDayAsync (SalesSystem.Api)","RequestId":"0HNHJ37LT9B9A:00000001","RequestPath":"/api/JobManageTask/EditSLSVLiveStreamingOneDay","ConnectionId":"0HNHJ37LT9B9A","Acadsoc-Trace-ID":"a28dbf9443b24f9fb98a7165bb7abf39","RequestUrl":"http://localhost:3001/api/JobManageTask/EditSLSVLiveStreamingOneDay","RemoteIP":"::1","Application":"SalesSystem.Api"}
{"@t":"2025-12-04T09:46:04.7004011Z","@mt":" FetchUserInfoAsync commId:1600024648_1683274356_1764838242 pageNumber:1 PageSize:100 请求结果：0","@tr":"1ef016ec10ddd88d35fab4f374cf59fc","@sp":"899e490d4a0a2ce2","response":{"Total":0,"UserList":[],"RequestId":"e960f302-a407-4c57-ad17-13bf835fa8be","IsStream":false,"$type":"DescribeUserInfoResponse"},"request":{"CommId":"1600024648_1683274356_1764838242","StartTime":1764838242,"EndTime":1764838380,"SdkAppId":1600024648,"UserIds":null,"PageNumber":1,"PageSize":100,"IsStream":false,"$type":"DescribeUserInfoRequest"},"ActionId":"433f25c2-aff2-4d55-a83e-9ca0ed24976c","ActionName":"SalesSystem.Api.Controllers.InnerSerivces.BackgroundServices.JobManageTask.JobManageTaskController.EditSLSVLiveStreamingOneDayAsync (SalesSystem.Api)","RequestId":"0HNHJ37LT9B9A:00000001","RequestPath":"/api/JobManageTask/EditSLSVLiveStreamingOneDay","ConnectionId":"0HNHJ37LT9B9A","Acadsoc-Trace-ID":"a28dbf9443b24f9fb98a7165bb7abf39","RequestUrl":"http://localhost:3001/api/JobManageTask/EditSLSVLiveStreamingOneDay","RemoteIP":"::1","Application":"SalesSystem.Api"}

统计"UserList":[] 中 RoomStr出现的次数

#\[[^]]*\] \[ 和 \] 是指的 a.txt 中 UserList的[]
#[^]]* 匹配任意多个非 ] 字符
# 匹配 "UserList":[ 开头 [^]]* 匹配任意多个非 ] 字符 匹配到第一个 ] 结束
#-o 只显示匹配字符串的部分
#wc -l 统计数量
 grep -o '"UserList":\[[^]]*\]' a.txt | grep -o '"RoomStr":"[^"]*"' | wc -l

grep -o '"UserList":\[[^]]*\]' a.txt | grep -o '"RoomStr":"[^"]*"'

在这里插入图片描述

# 没有加-o 的效果
 grep -o '"UserList":\[[^]]*\]' a.txt | grep  '"RoomStr":"[^"]*"

在这里插入图片描述