linux uniq学习

最新推荐文章于 2025-04-15 20:44:49 发布

majinbo111

最新推荐文章于 2025-04-15 20:44:49 发布

阅读量148

点赞数

文章标签： linux 运维

本文链接：https://blog.youkuaiyun.com/u011944141/article/details/121070309

版权

博客介绍了如何使用`uniq`命令进行行处理，包括去重、显示重复行、计数以及结合`sort`进行排序等操作。通过实例展示了在不同场景下如何有效地使用这些功能来处理文本文件中的重复行问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

uniq help

Usage: uniq [OPTION]... [INPUT [OUTPUT]]
Filter adjacent matching lines from INPUT (or standard input),
writing to OUTPUT (or standard output).

With no options, matching lines are merged to the first occurrence.

Mandatory arguments to long options are mandatory for short options too.
  -c, --count           prefix lines by the number of occurrences
  -d, --repeated        only print duplicate lines
  -D, --all-repeated[=delimit-method]  print all duplicate lines
                        delimit-method={none(default),prepend,separate}
                        Delimiting is done with blank lines.
  -f, --skip-fields=N   avoid comparing the first N fields
  -i, --ignore-case     ignore differences in case when comparing
  -s, --skip-chars=N    avoid comparing the first N characters
  -u, --unique          only print unique lines
  -z, --zero-terminated  end lines with 0 byte, not newline
  -w, --check-chars=N   compare no more than N characters in lines
      --help     display this help and exit
      --version  output version information and exit

A field is a run of blanks (usually spaces and/or TABs), then non-blank
characters.  Fields are skipped before chars.

Note: 'uniq' does not detect repeated lines unless they are adjacent.
You may want to sort the input first, or use `sort -u' without `uniq'.
Also, comparisons honor the rules specified by `LC_COLLATE'.

去重

cat test1
test 30
test 30
test 30
Hello 95
Hello 95
Hello 95
Hello 95
Linux 85
Linux 85
unix 66

uniq test
test 30
Hello 95
Linux 85
unix 66

仅显示重复的行

uniq -d test
test 30
Hello 95
Linux 85

去重+计数

uniq test -c
3 test 30
4 Hello 95
2 Linux 85
1 unix 66

去重+计数+排序

uniq test -c | sort -nr
4 Hello 95
3 test 30
2 Linux 85
1 unix 66

去重+计数+排序+top1

 uniq test -c | sort -nr | head -n 1
 4 Hello 95

当重复的行并不相邻时，uniq 命令是不起作用的，如下：

cat test2
test 30
test 30
Hello 95
unix 66
Hello 95
test 30
Hello 95
Linux 85
Hello 95
Linux 85

uniq -c test2
2 test 30
 1 Hello 95
 1 unix 66
 1 Hello 95
 1 test 30
 1 Hello 95
 1 Linux 85
 1 Hello 95
 1 Linux 85

改进

sort test2 | uniq -c
4 Hello 95
2 Linux 85
3 test 30
1 unix 66

# 在文件中找出重复的行
sort test2 | uniq -d
Hello 95
Linux 85
test 30