linux awk--根据A列去重后按照B列统计

最新推荐文章于 2024-04-28 05:21:36 发布

转载最新推荐文章于 2024-04-28 05:21:36 发布 · 495 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/dforward2017/p/6957638.html

文章标签：

#awk #shell #操作系统

本文介绍了一种通过awk命令处理日志文件的方法，用于统计特定格式的日志中各项指标出现的频次，并提供了示例代码帮助理解如何操作。

日志统计源格式

Ingest null xy.pull.yximgs.com gifshow gL4C7H60iB4 null 15.06 30014 2125237
Ingest null xy01.pull.yximgs.com gifshow BsXEEB9bRvs null 25.12 30014 4252273
Ingest null pl7.live.panda.tv live_panda 71f647c4111cbba3535af4f4be151bf5 null 24.76 30014 6781568
Ingest null xy01.pull.yximgs.com gifshow Af5c51X-Ck4 null 30.99 30014 5176675
Ingest null hdl61.kascend.com chushou_live 5ee009dbc328473dac83a876181de84f null 23.99 30014 7734469
Ingest null hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f_500 null 19.99 30014 2331870
Ingest null xy0-flv.live.huajiao.com live_huajiao_v2 _LC_XY0_non_6719303814968062921570386_SX null 14.93 30014 4622499
Ingest null xy.pull.yximgs.com gifshow mrp-BUuDYEY null 15.06 30014 2268941
Play 119.124.90.93 xy.pull.yximgs.com gifshow vEKKsXtl6t8 183.57.36.130 14.78 12718 848808
Play 183.9.62.204 xy.pull.yximgs.com gifshow XeyZRLLQx58 183.57.36.130 14.96 26878 1710173
Play 14.20.129.136 hdl61.kascend.com chushou_live 5ee009dbc328473dac83a876181de84f 183.57.36.130 23.99 30016 7744293
Ingest null pl7.live.panda.tv live_panda 95044a4c38a5095cbbd62330aba73edb null 24.92 30012 6460547
Ingest null xl.live-play.acgvideo.com live-xl live_10372345_5240864 null 15.03 30012 2188106
Ingest null hdl61.kascend.com chushou_live 28d34e8c781740809bf6b08c0f453791 null 24.08 20768 2946198
Ingest null xy.pull.yximgs.com gifshow 8AxZti4_g-0 null 15.03 23752 1611642
Play 120.84.9.142 get.xycdn.kuwo.cn voicelive 181232893 183.57.36.130 4.48 23207 588222
Ingest null xy0-flv.live.huajiao.com live_huajiao_v2 _LC_XY0_non_2247003114968179501127237_SX null 9.78 30152 5128732
Ingest null get.xycdn.kuwo.cn voicelive 143003104 null 14.99 30016 1828485
Play 119.124.167.151 xy.pull.yximgs.com gifshow 4YJ5s6QiY6c 183.57.36.130 16.15 19075 1453976
Play 219.134.216.144 xy.pull.yximgs.com gifshow GIjOLc3AyIA 183.57.36.130 12.39 30017 2070905
Play 221.4.53.132 xy0-flv.live.huajiao.com live_huajiao_v2 _LC_XY0_non_11426331514968197561722639_OX 183.57.36.130 19.96 30017 2913820
Play 113.78.163.89 hdl61.kascend.com chushou_live c14273b7586144c6ba8bd09d48a3f637 183.57.36.130 24.11 40855 11563397
Play 124.225.62.60 pl7.live.panda.tv live_panda 76f18bde738262df150f85d4209fe4d1 183.57.36.130 29.92 30016 1621267
Play 223.198.122.72 pl8.live.panda.tv live_panda 043bb44c767506f79ce04b8876121ad2 183.57.36.130 29.98 30016 7271052
Play 180.218.27.14 xy0-flv.live.huajiao.com live_huajiao_v2 _LC_XY0_non_2035149914968188441266158_SX 183.57.36.130 18.89 30016 3998343
Play 183.31.235.209 pl7.live.panda.tv live_panda fe69a6688e80f9c309518efa8f702177_mid 183.57.36.130 19.99 30015 4113324
Play 116.22.55.249 pl7.live.panda.tv live_panda 76f18bde738262df150f85d4209fe4d1 183.57.36.130 29.92 30015 1610893
Play 183.3.187.155 xl.live-play.acgvideo.com live-xl live_6639802_2352392 183.57.36.130 29.85 30015 6040526
Play 113.65.20.229 xy01.pull.yximgs.com gifshow BsXEEB9bRvs 183.57.36.130 24.69 30016 4140644
Play 113.70.167.175 xy.pull.yximgs.com gifshow 4jAdN5vQrfg 183.57.36.130 14.93 30016 2113514
Play 113.80.107.174 xy.pull.yximgs.com gifshow iB5LSMc06lc 183.57.36.130 14.19 30016 1950748
Play 14.220.122.155 xy.pull.yximgs.com gifshow Yz_0bDnDE_A 183.57.36.130 15.03 30016 2033218
Play 58.62.183.33 xy01.pull.yximgs.com gifshow iMzAzFIX3RY 183.57.36.130 24.92 30018 3397743
Play 125.91.209.70 xl.live-play.acgvideo.com live-xl live_6639802_2352392 183.57.36.130 30.12 30018 6070742
Ingest null hdl-xycdn.zaizhibo.tv broadcast 11-1403575984129-1496820109012-328080386 null 23.75 30016 1401046
Ingest null pl7.live.panda.tv live_panda de2961e1e6f4207ec8f7c7a19bdf1807 null 60.17 30016 25233180
Play 119.120.168.209 xy.pr.live.fanxing.com live fx_hifi_989254 183.57.36.130 15.20 32098 5691670
Play 183.5.242.93 get.xycdn.kuwo.cn voicelive 265433231 183.57.36.130 15.80 40696 2647456
Play 113.91.35.162 get.xycdn.kuwo.cn voicelive 356897384 183.57.36.130 15.12 30018 3519213
Play 14.159.205.246 rtmp-xy.1234tv.com live 202304 183.57.36.130 14.99 30018 2104592
Play 14.159.207.203 rtmp-xy.1234tv.com live 202304 183.57.36.130 14.99 30018 2104592
Play 58.60.32.236 get.xycdn.kuwo.cn voicelive 310928779 183.57.36.130 14.99 30018 3282643
Play 183.38.63.8 xy.pr.live.fanxing.com live fx_hifi_110183147 183.57.36.130 15.46 30018 4754930
Play 183.56.76.216 xy.pr.live.fanxing.com live fx_hifi_342014692 183.57.36.130 16.59 35090 6005991
Play 125.91.146.247 xy.pr.live.fanxing.com live fx_hifi_23241362 183.57.36.130 17.52 36136 5878886
Play 119.29.216.22 xy.pr.live.fanxing.com live fx_hifi_33144330 183.57.36.130 14.26 30012 4453837
Play 27.38.142.68 get.xycdn.kuwo.cn voicelive 94888275 183.57.36.130 14.99 30012 1841845
Play 125.94.168.146 xy.pr.live.fanxing.com live fx_hifi_244430965 183.57.36.130 15.26 30012 4775051
Play 14.211.38.39 xy.pull.yximgs.com gifshow XOpjA_pJuTo 183.57.36.130 14.59 19050 1236052
Ingest null get.xycdn.kuwo.cn voicelive 71164097 null 20.68 30851 2599993
Ingest null hdl61.kascend.com chushou_live ddb0b31d310447b99db1fc2c5b55e98d null 23.76 16749 2262949
Play 183.8.245.30 hdl61.kascend.com chushou_live c5d371b86560428597086b1226f80924 183.57.36.130 24.15 17601 3180722
Ingest null xy.pull.yximgs.com gifshow sudeC1l0Sbo null 12.83 12625 696740
Ingest null xy0-flv.live.huajiao.com live_huajiao_v2 _LC_XY0_non_11426331514968197561722639_OX null 20.02 30014 2892840
Ingest null xy.pull.yximgs.com gifshow DLFtgQBvR8s null 14.99 30014 2181646
Play 183.13.201.87 xy0-flv.live.huajiao.com live_huajiao_v2 _LC_XY0_non_5631867814968201171661343_OX 183.57.36.130 16.59 31773 5548671
Play 219.136.198.74 pl8.live.panda.tv live_panda c9febad12e6c3cd81803bd23ee9ad32c 183.57.36.130 30.08 30015 6106360
Ingest null get.xycdn.kuwo.cn voicelive 364020074 null 14.99 30014 2795944
Ingest null xy.pr.live.fanxing.com live fx_hifi_188555749 null 15.36 30014 4790986
Ingest null xy.pr.live.fanxing.com live fx_hifi_51910260 null 15.13 30014 4643364
Ingest null xy.pr.live.fanxing.com live fx_hifi_70439820 null 14.56 30014 4484272
Ingest null xy.pr.live.fanxing.com live fx_hifi_365997602 null 12.93 30014 3989497
Ingest null xy.pr.live.fanxing.com live fx_hifi_375424828 null 15.43 30016 4742712
Ingest null xy.pr.live.fanxing.com live fx_hifi_121955046 null 15.23 30016 4657126
Play 119.130.228.183 pl8.live.panda.tv live_panda 8dd539552902c2ec70382e90bf67057d 183.57.36.130 29.99 30015 5849047
Play 183.39.234.40 xl.live-play.acgvideo.com live-xl live_6639802_2352392 183.57.36.130 30.05 30015 6084780
Play 171.107.52.37 pl7.live.panda.tv live_panda de2961e1e6f4207ec8f7c7a19bdf1807_mid 183.57.36.130 20.06 30015 5375624
Play 116.5.44.67 pl8.live.panda.tv live_panda 8db3b56f66234f09adb2c970f52f0af3 183.57.36.130 25.09 30015 6553960
Play 119.123.228.91 xy01.pull.yximgs.com gifshow 0tQunl6g20Q 183.57.36.130 24.78 13114 119025
Play 112.66.38.240 pl8.live.panda.tv live_panda 8db3b56f66234f09adb2c970f52f0af3_mid 183.57.36.130 19.92 30019 4336090
Play 14.126.123.212 pl8.live.panda.tv live_panda 043bb44c767506f79ce04b8876121ad2 183.57.36.130 29.98 30019 7184628
Play 222.216.227.135 pl8.live.panda.tv live_panda 043bb44c767506f79ce04b8876121ad2_mid 183.57.36.130 19.99 30019 4073912
Play 61.145.228.58 xl.live-play.acgvideo.com live-xl live_6639802_2352392 183.57.36.130 30.05 30019 6061246
Play 58.59.203.164 pl8.live.panda.tv live_panda 8db3b56f66234f09adb2c970f52f0af3 183.57.36.130 24.98 36351 10031046
Play 116.7.65.9 hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f 183.57.36.130 23.72 30019 8258224
Play 183.25.194.197 hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f 183.57.36.130 23.72 30019 8258224
Ingest null xy.pull.yximgs.com gifshow aU1WdiqTLA0 null 14.99 30014 2011369
Ingest null xy.pull.yximgs.com gifshow rV7Cvi6uIKw null 15.03 30014 2185128
Play 183.6.27.25 pl7.live.panda.tv live_panda 7d9bdfd8beca4be796bc4b757503decd_mid 183.57.36.130 19.99 30018 4302418
Play 121.8.239.194 pl8.live.panda.tv live_panda 93d8751b0ea555f2f045ab1f554266a9 183.57.36.130 24.99 30018 8032893
Play 119.126.206.250 pl7.live.panda.tv live_panda 1658ee9eca24c19208aff4f627d581f3 183.57.36.130 20.19 30018 4198084
Play 59.41.156.116 hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f 183.57.36.130 23.75 30018 8268678
Play 14.153.236.25 xy.pull.yximgs.com gifshow 6wx_iqrZFEA 183.57.36.130 16.17 37285 2887937
Ingest null pl7.live.panda.tv live_panda 321558d5085c720e0550ccdf10b06b0a null 17.83 18560 3996806
Play 14.153.149.73 xy.pr.live.fanxing.com live fx_hifi_365117734 183.57.36.130 15.52 12567 1990202
Ingest null get.xycdn.kuwo.cn voicelive 310928779 null 14.99 30016 3300641
Play 14.18.236.187 xy.pf.live.fanxing.com live fx_hifi_109939548 183.57.36.130 14.86 30018 4563947
Play 183.32.164.132 hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f 183.57.36.130 23.75 30016 8173126
Play 113.87.220.98 hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f 183.57.36.130 23.75 30018 8169413
Play 119.137.204.141 xy.pull.yximgs.com gifshow iB5LSMc06lc 183.57.36.130 14.29 30013 1926590
Play 14.153.14.103 pl7.live.panda.tv live_panda 9efab81ece0d102aef49dd2cd5140562 183.57.36.130 25.12 30013 6561119
Play 183.38.248.13 pl7.live.panda.tv live_panda 7d9bdfd8beca4be796bc4b757503decd 183.57.36.130 29.98 30017 8060412
Ingest null xy.pull.yximgs.com gifshow Yz_0bDnDE_A null 14.96 30015 2030986
Ingest null hdl61.kascend.com chushou_live a2335f8bc29346beaaf72b9c484c6e3f null 23.75 30015 8200290
Play 183.36.69.17 xy.pr.live.fanxing.com live fx_hifi_989254 183.57.36.130 15.09 30227 5354041
Play 14.114.95.251 xy.pr.live.fanxing.com live fx_hifi_92254313 183.57.36.130 14.75 39873 5943764
Play 14.16.8.90 xy.pr.live.fanxing.com live fx_hifi_366867429 183.57.36.130 15.02 30018 4646029
Play 220.173.207.114 get.xycdn.kuwo.cn voicelive 181232893 183.57.36.130 14.99 30018 2038412
Play 61.141.165.64 rtmp-xy.1234tv.com live 203010 183.57.36.130 14.99 30018 2701260
Play 113.72.9.144 xy.pr.live.fanxing.com live fx_hifi_365997602 183.57.36.130 12.96 30018 4035485
Play 116.16.97.125 xy.pr.live.fanxing.com live fx_hifi_56186331 183.57.36.130 15.29 30012 4725117
Play 183.39.91.196 xy.pr.live.fanxing.com live fx_hifi_332206706 183.57.36.130 15.49 30012 4780414
Play 119.120.180.135 xy.pr.live.fanxing.com live fx_hifi_231639064 183.57.36.130 15.66 30012 4878931
Play 119.120.36.117 xy.pr.live.fanxing.com live fx_hifi_302900128 183.57.36.130 15.16 30012 4681747

awk命令实现

awk '!stream[$5]++{S[$1]++}END{for(key in S) print key,S[key]}' filename

awk '{if(!stream[$5]++)S[$1]++}END{for(key in S) print key,S[key]}' filename

按照第五列去重后按照第一列统计
如果多列去重如3、4、5列可按如下调整

awk '!stream[$3" "$4" "$5]++{S[$1]++}END{for(key in S) print key,S[key]}' filename

转载于:https://www.cnblogs.com/dforward2017/p/6957638.html