工作中遇到一些难题1_5

1.如何批量统计目标文件夹下各子文件夹下及其文件数？

(5000关键词爬取筛选)

2.根据python dict字典的唯一性,对某文件夹下不含info.txt的.txt中的某段特定位置的内容进行查重,并获取重复的文本内容及文件夹名称

(字典那里不知道怎么正确书写,贴出代码,欢迎一起探讨)

#coding:utf-8
import os

def printPath(path):
li_dirname = {}

for dirpath,dirnames,filenames in os.walk(path):
for filename in filenames:
if filename.endswith('.txt'):
if os.path.basename(filename) == 'info.txt':
continue

with open(os.path.join(dirpath,filename),'r') as f:
for line in f.readlines():
li = line.strip().split('\t')[-1]
if li_dirname.get(li):
li_dirname[li] = []
li_dirname[li].append(os.path.join(dirpath,filename))

for li in li_dirname:
# print key,value
if len(li_dirname[li]) > 1:
print li,li_dirname[li]
with open('./new.txt','w') as f:
f.writelines(li +'\t'+li_dirname[li])

if __name__ == '__main__':
printPath('a')

转载于:https://www.cnblogs.com/liuqi-beijing/p/6228390.html