python列表去重

Suppose you have a list in python that looks like this:

['a','b','a']
# or like this:
[1,2,2,2,3,4,5,6,6,6,6]

and you want to remove all duplicates so you get this result:

['a','b']
# or
[1,2,3,4,5,6]

How do you do that? ...the fastest way? I wrote a couple of alternative implementations and did a quick benchmark loop on the various implementations to find out which way was the fastest. (I haven't looked at memory usage). The slowest function was 78 times slower than the fastest function.

However, there's one very important difference between the various functions. Some are order preserving and some are not. For example, in an order preserving function, apart from the duplicates, the order is guaranteed to be the same as it was inputted. Eg, uniqify([1,2,2,3])==[1,2,3]

Here are the functions:

def f1(seq):
   # not order preserving
   set = {}
   map(set.__setitem__, seq, [])
   return set.keys()

def f2(seq): 
   # order preserving
   checked = []
   for e in seq:
       if e not in checked:
           checked.append(e)
   return checked

def f3(seq):
   # Not order preserving
   keys = {}
   for e in seq:
       keys[e] = 1
   return keys.keys()

def f4(seq): 
   # order preserving
   noDupes = []
   [noDupes.append(i) for i in seq if not noDupes.count(i)]
   return noDupes

def f5(seq, idfun=None): 
   # order preserving
   if idfun is None:
       def idfun(x): return x
   seen = {}
   result = []
   for item in seq:
       marker = idfun(item)
       # in old Python versions:
       # if seen.has_key(marker)
       # but in new ones:
       if marker in seen: continue
       seen[marker] = 1
       result.append(item)
   return result

def f6(seq):
   # Not order preserving    
   set = Set(seq)
   return list(set)

And what you've all been waiting for (if you're still reading). Here are the results:

* f2 13.24
* f4 11.73
* f5 0.37
f1 0.18
f3 0.17
f6 0.19

(* order preserving)

Clearly f5 is the "best" solution. Not only is it really really fast; it's also order preserving and supports an optional transform function which makes it possible to do this:

>>> a=list('ABeeE')
>>> f5(a)
['A','B','e','E']
>>> f5(a, lambda x: x.lower())
['A','B','e'] 

Download the benchmark script here

UPDATE

From the comments I've now added a couple of more functions to the benchmark. Some which don't support uniqify a list of objects that can't be hashed unless passed with a special hashing method. So see all the functions download the file

Here are the new results:

* f5 10.1
* f5b 9.99
* f8 6.49
* f10 6.57
* f11 6.6
f1 4.28
f3 3.55
f6 4.03
f7 2.59
f9 2.58

(f2 and f4) were too slow for this testdata.

### 回答1: 可以使用 set() 函数将列表转换为集合,因为集合具有的功能,然后再将集合转换回列表即可复元素。示例代码如下: ``` lst = [1, 2, 3, 2, 4, 1, 5, 3] lst = list(set(lst)) print(lst) ``` 输出结果为: ``` [1, 2, 3, 4, 5] ``` ### 回答2: Python 列表可以使用以下几种方法: 1. 使用集合(set)方法:将列表转换为集合,由于集合中的元素是唯一的,复的元素会被自动除,然后再将集合转换回列表。例如: ```python lst = [1, 2, 2, 3, 4, 4, 5] lst = list(set(lst)) print(lst) ``` 输出结果为:[1, 2, 3, 4, 5] 2. 使用列表推导式方法:通过列表推导式的方式创建一个列表,只包含原列表中的唯一元素。例如: ```python lst = [1, 2, 2, 3, 4, 4, 5] lst = [x for i, x in enumerate(lst) if x not in lst[:i]] print(lst) ``` 输出结果为:[1, 2, 3, 4, 5] 3. 使用循环方法:通过循环遍历原列表中的元素,将不复的元素添加到一个列表中。例如: ```python lst = [1, 2, 2, 3, 4, 4, 5] new_lst = [] for x in lst: if x not in new_lst: new_lst.append(x) print(new_lst) ``` 输出结果为:[1, 2, 3, 4, 5] 以上是三种常见的方法,可以根据实际情况选择合适的方法。 ### 回答3: Python 列表可以通过多种方式实现。以下是其中几种常见的方法: 方法一:使用set()函数 利用set()函数可以将列表转换为集合,由于集合的特性是元素唯一,转换后的集合将自动复元素,然后再将集合转换回列表。 示例代码: ``` lst = [1, 2, 3, 3, 4, 4, 5] lst = list(set(lst)) print(lst) # [1, 2, 3, 4, 5] ``` 方法二:使用列表推导式 使用列表推导式可以遍历列表,同时只添加未出现过的元素。 示例代码: ``` lst = [1, 2, 3, 3, 4, 4, 5] lst = list(set([x for x in lst])) print(lst) # [1, 2, 3, 4, 5] ``` 方法三:使用循环遍历 通过使用for循环遍历列表,判断元素是否已经存在于新列表中,如果不存在则添加到新列表中。 示例代码: ``` lst = [1, 2, 3, 3, 4, 4, 5] new_lst = [] for x in lst: if x not in new_lst: new_lst.append(x) print(new_lst) # [1, 2, 3, 4, 5] ``` 这些方法都可以有效地,并根据实际需求选择适合的方法来处理列表中的复元素。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值