Linux 文本去重——uniq

最新推荐文章于 2024-07-16 14:40:54 发布

Schuyler_yuan

最新推荐文章于 2024-07-16 14:40:54 发布

阅读量836

点赞数

CC 4.0 BY-SA版权

分类专栏： # Linux Shell 文章标签： linux 运维服务器

本文链接：https://blog.youkuaiyun.com/sinat_34715587/article/details/121347820

Linux Shell 专栏收录该内容

56 篇文章

订阅专栏

uniq命令用于从排序输入中删除或显示重复的行。常见选项包括-u（只显示唯一行）、-c（计数重复行）、-d（仅显示重复行）和-f（忽略字段比较）。在实际使用时，通常需要先用sort对输入进行排序以确保正确处理。了解这些选项可以帮助更有效地处理文件中的重复数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

uniq

用于移除或发现文件中重复的条目。

注意：此命令对连续的重复行才能正确统计，使用时最好搭配sort命令。

常用选项

-c --count

-d --repeated

-D --all-repeated

-u --unique

-w --check-chars

-s --skip-chars

-f --skip-fields

uniq_help

$ uniq --help

Usage: uniq [OPTION]... [INPUT [OUTPUT]]
Filter adjacent matching lines from INPUT (or standard input),
writing to OUTPUT (or standard output).

With no options, matching lines are merged to the first occurrence.

Mandatory arguments to long options are mandatory for short options too.
  -c, --count           prefix lines by the number of occurrences
  -d, --repeated        only print duplicate lines, one for each group
  -D, --all-repeated[=METHOD]  print all duplicate lines
                          groups can be delimited with an empty line
                          METHOD={none(default),prepend,separate}
  -f, --skip-fields=N   avoid comparing the first N fields
      --group[=METHOD]  show all items, separating groups with an empty line
                          METHOD={separate(default),prepend,append,both}
  -i, --ignore-case     ignore differences in case when comparing
  -s, --skip-chars=N    avoid comparing the first N characters
  -u, --unique          only print unique lines
  -z, --zero-terminated  end lines with 0 byte, not newline
  -w, --check-chars=N   compare no more than N characters in lines
      --help     display this help and exit
      --version  output version information and exit

A field is a run of blanks (usually spaces and/or TABs), then non-blank
characters.  Fields are skipped before chars.

Note: 'uniq' does not detect repeated lines unless they are adjacent.
You may want to sort the input first, or use 'sort -u' without 'uniq'.
Also, comparisons honor the rules specified by 'LC_COLLATE'.

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Report uniq translation bugs to <http://translationproject.org/team/>
For complete documentation, run: info coreutils 'uniq invocation'

Have Fun