python分类统计字符中空格,快速/高效地计算Python中空格分隔的字符串列表

Given the input:

x = ['foo bar', 'bar blah', 'black sheep']

I could do this to get the count of each word in the list of space delimited string:

from itertools import chain

from collections import Counter

c = Counter(chain(*map(str.split, x)))

Or I could simple iterate through and get:

c = Counter()

for sent in x:

for word in sent.split():

c[word]+=1

[out]:

Counter({'bar': 2, 'sheep': 1, 'blah': 1, 'foo': 1, 'black': 1})

The question is which is more efficient if the input list of string is extremely huge? Are there other ways to achieve the same counter object?

Imagine it's a text file object that has billions of lines with 10-20 words each.

解决方案

The answer to your question is profiling.

Following are some profiling tools:

print time.time() in strategic places. (or use Unix time)

heapy tracks all objects inside Python’s memory (good for memory leaks)

For long-running systems, use dowser: allows live objects introspection (web browser interface)

examine Python bytecode with dis