大csv文件排序，c++ vs python

最新推荐文章于 2024-04-29 13:51:20 发布

made_in_chn

最新推荐文章于 2024-04-29 13:51:20 发布

阅读量1.2k

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/made_in_chn/article/details/8476494

本文通过对比Python和C++对大型CSV文件进行排序的时间消耗，展示了C++在处理大文件排序时的性能优势。Python在排序大文件时耗时较长，而C++在相同任务下完成速度更快，适合处理此类需求。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

先生成csv文件，按照name,score1,score2来

这个用python写比较简单：

gen_random_csv.py

#!/usr/bin/python                                                                                                                                             

import csv
import random


LETTERS = 'abcdefghijklmnopqrspuvwxyz'


def GenRandomName():
    arr = []
    cnt = random.randint(5, 20)
    while cnt > 0:
        arr.append(random.choice(LETTERS))
        cnt -= 1
    return ''.join(arr)


def GenRandomScore():
    return random.uniform(0, 1000)


def GenRandomFile(output_fn, total_line):
    print 'Processing:', output_fn
    with open(output_fn, 'wb') as csvfile:
        writer = csv.writer(csvfile)
        while total_line > 0:
            name = GenRandomName()
            score1 = GenRandomScore()
            score2 = GenRandomScore()
            writer.writerow([name, score1, score2])
            total_line -= 1
    print 'Save to:', output_fn


GenRandomFile('test_big2.csv', 1000000)
GenRandomFile('test_big3.csv', 2000000)
GenRandomFile('test_big4.csv', 4000000)
GenRandomFile('test_big5.csv', 6000000)
GenRandomFile('test_big6.csv', 8000000)

python gen_random_csv.py

wc -l *.csv

获得：

1000000 test_big2.csv
2000000 test_big3.csv
4000000 test_big4.csv
6000000 test_big5.csv
8000000 test_big6.csv

python的排序：

#!/usr/bin/python                                                                                                                                             

import csv
import time


def Sort1(reader, writer):
    ts1 = time.time()

    arr = []
    for row in reader:
	arr.append(row)

    ts2 = time.time()
    print ' == 2 == ', ts2

    arr.sort(key=lambda tup: tup[1], reverse=True)

    ts3 = time.time()
    print ' == 3 == ', ts3

    for line in arr:
        writer.writerow(line)

    ts4 = time.time()
    arr = []
    return (ts1, ts2, ts3, ts4)


def SortFile1(fn):
    print 'Start process