运维常用命令

• redis模糊匹配批量删除键值
  redis-cli KEYS "pattern" | xargs redis-cli DEL

• 查询日志访问次数排行
  tail access.log -n 20000|awk '{print $7}'|sort|uniq -dc|sort -n

• 按文件扩展名grep字符串
  grep -r --include=\*.py User

• 分析LOG日志游戏包下载次数
  cat chn_app_rec.log|awk '{if ($1=="2017-02-12" && $5==2000973 && $8==2110000) print $0}'|wc -l

• 分析log日志某个url指定日期访问次数
  cat access.log |awk '{if ($7 ~ /rec\/app\/detail\/xmqy2110000.html/ && $4 ~ /12\/Feb/) print $0}' > /tmp/2-12-rec.log

• 分析日志某个url 并且返回码是500的统计
  cat access.log |awk '{if ($7 ~ /api\/wechat/ && $9 != "200") print $1}'|sort|uniq -c|sort -nr -k 1

• 分析nginx access log 的URL请求总数， 请求总时间，请求平均时间

  tail /data/log/nginx/api.mobi.ouwan.com.access.log -n 5000 | awk '/[0-9]\.[0-9]{3}$/{print  $NF, $7}'|cut -d "?" -f 1|awk '{url[$2]+=$1;url_count[$2]++}; END {for (k in url) print "url: " k, "time_consume: "url[k], "request_count: " url_count[k], "avg_time_consume: " url[k]/url_count[k]}'

  输出内容

  url: /essay/nativeEssayByHostId/ time_consume: 0.257 request_count: 3 avg_time_consume: 0.0856667
  url: /test/user/nativeFollowCommunityList/ time_consume: 1.361 request_count: 15 avg_time_consume: 0.0907333
  url: /mission/nativeGetCouponReward/ time_consume: 0.208 request_count: 4 avg_time_consume: 0.052
  url: /mission/nativeMissionSignin/ time_consume: 7.021 request_count: 43 avg_time_consume: 0.163279
  url: /user/nativeMyInfo/ time_consume: 89.307 request_count: 1127 avg_time_consume: 0.0792431
  url: /test/user/nativeMyInfo/ time_consume: 28.502 request_count: 15 avg_time_consume: 1.90013

• 获取socket链接状态统计
  ss -ant|awk '{s[$1]++} END {for (k in s) print k, s[k] }'

  CLOSE-WAIT 4
  ESTAB 130
  TIME-WAIT 64
  LISTEN 64
  SYN-SENT 3
  State 1

• awk 显示行数
  awk '{print NR " == " $0}' /etc/passwd

uniq 是去除相邻重复的两行， 所以用uniq之前先sort一下，把重复的都放到相邻位置
uniq
-c 统计重复行出现次数
-n 按数字大小排序
-r 倒序排列
-k 以第几列排序

    统计的IP
    次数  IP
     81 101.226.62.17
     51 101.226.233.156
     50 101.226.233.146
     49 101.226.62.79
     44 101.226.233.141
     42 101.226.233.151
     42 101.226.233.143
      .....

• 快速生成100M的文件 填入一个10MB的随机比特流数据
  head -c 10M < /dev/urandom > /var/log/log-file

• grep 的and 和 or

# 同时有这个搜索条件的行数
grep -E 'pattern1.*pattern2' filename
cat chn_app_rec.log|grep -E '2017-02-13.*2000973.*2104006'

# 同时搜索p1&&p2 || p1&&p3
grep -E 'pattern1.*pattern2|pattern1.*pattern3' filename
cat chn_app_rec.log|grep -E '2017-02-13.*2000973.*2098000|2017-02-15.*2000973.*2098000'


# or 用法1
grep 'pattern1\|pattern2' filename
# or 用法2
grep -E 'pattern1|pattern2' filename
# or 用法3
grep -e pattern1 -e pattern2 filename

# and 用法
grep -E 'pattern1.*pattern2' filename
grep -E 'pattern1.*pattern2|pattern2.*pattern1' filename

grep 链接 http://www.thegeekstuff.com/2011/10/grep-or-and-not-operators/

• 使用grep 查找文件名

  find ./ |xargs grep keyword

• 查看进程gid （按格式输出ps内容）
  ps ax o user,pid,%cpu,%mem,vsz,rss,tty,stat,start,time,comm,group,gid
ps ax o user,gid,pgid,pid,group,start,time,command|grep owan_web
  ps o 格式详解： http://blog.youkuaiyun.com/shuanghujushi/article/details/51814447