Why is truth() faster than bool()? Part I

Why is truth() faster than bool()? Part I

At TellApart, we use Python to code our real-time bidding server. Real-time bidding means we have to respond in a few milliseconds to requests from ad exchanges, or else we’re, well, out of business. So it’s critical that we know how to use all of our tools effectively.

There are faster languages than Python, but if you’re writing Python you can still speed things up without making anything less readable. Even where performance differences are small it pays to know what you’re doing. The knowledge may come in handy someday when you discover that an apparently simple piece of code has a disproportionate impact on execution time, perhaps only because it’s in a block that gets executed zillions of times.

With that introduction, here’s a survey of some tips we’ve assembled to write performant Python code on a line-by-line basis.

Today we’ll learn:

1. How to use timeit
2. Why you should prefer tuples instead of lists
3. Where and why you should use generator expressions instead of list expressions
4. The right way to instantiate a set()
5. Why not to use dict.keys()

1) Using timeit

The timeit module is the best way to evaluate the performance of small code snippets.

The most common way to use timeit:

timeit.Timer(“code to time”, “code to initialize”).timeit(number of times to run)

You can also use a lambda expression:

timeit.Timer(lambda: code to execute).timeit(number of times to run)

If you need to use a function from somewhere else you can import it in the “code to initialize” section; to import a function from the current context, you can do like so:

def foo():

   ...

timeit.Timer(“foo()”, “from __main__ import foo”).timeit(number of times to run)

We’re going to use timeit a lot in the examples below. Let’s get started.

2) Tuples are faster than lists

Use tuples instead of lists where possible, especially if you know you’re not going to modify it or you just need a dummy “empty list” object.

Here’s the difference between empty ones:

>>> timeit.Timer("[]", "").timeit(10000000)

0.39651203155517578

>>> timeit.Timer("()", "").timeit(10000000)

0.25010514259338379

Look what happens when we start adding members:

>>> timeit.Timer("[1,2,3,4,5]").timeit(1000000)

0.15485715866088867

>>> timeit.Timer("(1,2,3,4,5)").timeit(1000000)

0.024662017822265625 

The advantage of using tuple is huge.

3) Use generator expressions where possible

One of the coolest features of Python is the list expression.

Instead of:

alist = []

for i in xrange(10):

   alist.append(i)

We can just write:

alist = [i for i in xrange(10)]

However, in many situations, you can just use a generator expression. A generator expression is written the same as a list expression in parentheses rather than in brackets:

agen = (i for i in xrange(10))

Generator expressions differ from list expressions in that where a list expression creates an entire list object in memory, a generator expression just emits one value at a time to some piece of code that consumes an iterable. If you’re just going to loop over the collection of values once, it makes sense to use a generator expression rather than constructing a list in memory only to immediately throw it away.

This is the case in many situations where one might be tempted to use a list expression.

set([i for i in xrange(10)])

dict([(k, v + 1) for k, v in another_dict.iteritems()])

can be rewritten as:

set(i for i in xrange(10))

dict((k, v + 1) for k, v in another_dict.iteritems())

In the second set of cases intermediate mutable lists won’t be constructed only to be looped over once and then thrown away.

You can also use generator expressions in counting and conditionals. Say, for example, you wanted to count the number of objects in a collection that have the attribute ‘composition’ equal to ‘polyester’.

polyester_shirt_count = len([shirt for shirt in shirts if shirt.composition==’polyester’])

In the above case, a list object will be created, the length will be taken from it, and then it will be thrown away. However, it can be rewritten using a generator expression:

polyester_shirt_count = sum(shirt.composition==’polyester’ for shirt in shirts)

This works because bool is a subclass of int in Python and True is just an alias of 1.

You can also use generator expressions with the useful functions any() and all().

Suppose you want to do something if any object in your collection has the ‘status’ attribute set to ‘lost’. One way you could do it is this:

if len([shirt for shirt in shirts if shirt.status==’lost’]) > 0:

  ...

However, using any():

if any(shirt.status==’lost’ for shirt in shirts):

  ...

any() is smart and will stop executing at the first True value it finds, so in addition to not needing the intermediate list in memory it also doesn’t even need to loop over the entire list.

Or, to see if all your shirts are lost, you could write something like:

if len([shirt for shirt in shirts if shirt.status==’lost’]) == len(shirts):

  ...

Rewritten using all():

if all(shirt.status==’lost’ for shirt in shirts):

  ...

Similarly, all() is optimized to stop iteration at the first False value it finds. any() and all() become much more powerful with generators.

Finally, we can also replace a common pattern using a generator expression.

How many times have you done this?

max_length = None

for egg in missing_eggs:

  if max_length is None or egg.length > max_length:

    max_length = egg.length

You can replace this ugly code with the following:

max_length = max(egg.length for egg in missing_eggs)

What about this slightly more complex example?

max_length = largest_egg = None

for egg in missing_eggs:

  if max_length is None or egg.length > max_length:

    max_length = egg.length

    largest_egg = egg

max() (as well as min()) have a little-known ‘key’ parameter, so in this case you don’t need a list or generator expression at all:

largest_egg = max(missing_eggs, key=lambda t: t.length)

4) Working with set

Don’t pass an empty list to set to make an empty new one.

Instead of:

set([])

Just do:

set()

Also if you just need an empty immutable set, an empty tuple will usually suffice and is much faster than using a set() that you won’t actually modify. Most methods of set objects like difference(), union() etc will accept any iterable as parameters and don’t actually need to be passed other set objects.

Also, even though this construction looks a little more readable:

set([1, 3, 4, 5])

it is slower than:

set((1, 3, 4, 5))

Because, as noted above, tuples are faster than lists.

5) Don’t use dict.keys()

If you call .keys() on a dict, it will return a copy of the dict’s keys as a new list. There are actually very few cases where you would want a new, mutable copy of the dicts keys in a separate list object.

If you just want to iterate over these keys once, this list will be created, used once, and then discarded.

In that case, instead you could call iterkeys(): this will return an iterator that will yield one key at a time, and not construct a new temporary list containing all the keys. (Just as you can usually use itervalues() instead of values()). However, if you just iterate over the dict itself without calling any of the above methods, it will automatically behave the same way as if you had called iterkeys(), and it saves Python a few tiny lookups:

>>> animal_population = {‘ewok’: 60, ‘dinosaur’: 350, ‘sea lion’: 93}

>>> for k in animal_population:

>>>  print k

ewok

dinosaur

sea lion

One exception where you must iterate over a copy is if you want to add or remove keys from the dict while looping over it, as below:

dinosaurs = {‘larry’: Dinosaur(), ‘curly sue’: Dinosaur()}

for name in dinosaurs.keys():

  dinosaurs[name].mechanize()

  dinosaurs[‘mecha ‘ + name] = dinosaurs[name]

  del dinosaurs[name]

In this case you must loop over a copy of the set of keys: you will get an exception (“RuntimeError: dictionary changed size during iteration”) if you try to iterate over the dict or iterkeys() and add or remove a key from the dict. (You can still change the value assigned to a key, however).

If you want a copy of the dict’s keys as a set, you also don’t need to call keys(), since the set constructor accepts an iterable it will automatically create itself from the equivalent of iterkeys(), as above when we looped directly over the dict:

set(adict)

Or any other function that accepts an iterable parameter.

Coda

That’s all for now. In part two of this series we’ll finally answer the question in the title of this post: Why is truth() faster than bool()?

From:
http://tellaparteng.tumblr.com/post/57005427993/why-is-truth-faster-than-bool-part-i?goback=%2Egde_25827_member_263947480