Python---去除txt文件中重复的行数

1.前言：

        采用python中set()的概念，通过遍历原始文档中的元素，并将其添加到set()中，然后根据set()的性质来判断新的元素是否要被添加到新的文档中去。最终生成的新的文档即满足所需。

2.代码实现：

#coding:utf-8

readDir = "./original_file.txt"
writeDir = "./new_file.txt"
outfile=open(writeDir,"w")
f = open(readDir,"r")

lines_seen = set()  # Build an unordered collection of unique elements.

for line in f:
    line = line.strip('\n')
    if line not in lines_seen:
        outfile.write(line+ '\n')
        lines_seen.add(line)