The world’s leading publication for data science, AI, and ML professionals.

Many Articles Tell You Python Tricks, But Few Tell You Why

Three common Python tricks make your program faster, I will explain the mechanisms

Building Hash Table
Building Hash Table
Image by Annette from Pixabay
Image by Annette from Pixabay

Just had a simple search and it is very easy to get many articles trying to tell us many Python tricks. They are either more "Pythonic" or make our program faster. There is nothing wrong with these articles because most of the tricks are quite useful. In fact, I have written many articles of this type myself.

However, there is often criticism for this type of article because there is no trick that applies to all scenarios. That is also true. In my opinion, it is more important to understand the reason why these tricks are there, so we can understand when to use and when not to use them.

In this article, I’ll pick up three of them and provide a detailed explanation of the mechanisms under the hood.

1. Joining Strings Faster

Image by 浩一 萩原 from Pixabay
Image by 浩一 萩原 from Pixabay

How do you usually join strings together?

For example, suppose we have a list of strings that need to be joined together.

strs = ['Life', 'is', 'short,', 'I', 'use', 'Python']

Of course, the most intuitive way of doing this will be looping the list and joining all the substrings with white spaces using the + operator.

def join_strs(strs):
    result = ''
    for s in strs:
        result += ' ' + s
    return result[1:]

join_strs(strs)

In the above code, we simply define an empty string and keep appending a whitespace and a substring from the list to this string. Eventually, we return the result string starting from the 2nd character so that the leading whitespace will be trimmed.

However, in this case, we have a much better way to achieve it. That is using the join() function as follows.

def join_strs_better(strs):
    return ' '.join(strs)

join_strs_better(strs)

It’s not only more readable but also much faster. See the performance comparison below.

Why using the join() function will be faster?

Firstly, let’s have a look at the steps of the first method that uses a for-loop and the + operator.

  1. For each loop, the string is found in the list
  2. The Python executor interprets the expression result += ' ' + s and apply for a memory address for the white space ' '.
  3. Then, the executor realises that the white space needs to be joined with a string, so it will apply for the memory address for the string s, which is "Life" for the first loop.
  4. For every loop, the executor will need to apply for memory address twice, one for the white spaces and the other one for the string
  5. There are 12 times memory allocations

Then, let’s have a look at the steps of using the join() function.

  1. The executor will count how many strings are in the list. There are 6.
  2. It means that the string that is used to join the strings in the list will need to be repeated 6–1=5 times.
  3. It knows that there are a total of 11 memory spaces are needed, so all of these will be applied at once and be allocated upfront.
  4. Put the strings in order, and return the result.

Therefore, it is obvious that the major difference is that the number of times for memory allocation is the main reason for the performance improvement.

If you’re after a more comprehensive explanation with diagrams for illustration, please check out one of my previous articles here.

Do Not Use "+" to Join Strings in Python

Some Extra Remarks

Just in case someone may argue, I do not recommend using the join() function for ALL the strings joining scenarios. Suppose we just have two strings to be joined in a data analysis script, please simply do something like my_str = "A" + "B". This is actually more readable. There won’t be much value from the performance perspective, too. Unless we are joining strings thousands of times.

Also, if your substrings are not in a collection, do not create a list just because you want to use the join() function. In fact, the overhead of initialising a list is much more than the benefit we can get from using the join() function.

2. Creating Lists Faster

Image by Leslie Zambrano from Pixabay
Image by Leslie Zambrano from Pixabay

In Python, List is no doubt the most commonly used data structure for its flexibility. However, there is not only one way to create an empty list.

Some argue that using list() is more readable, and the literal way like [] just blindly pursuing so-called "Pythonic". In fact, these two ways of creating lists are different mechanisms, and the latter is faster.

my_list_1 = list()
my_list_2 = []

Why using the literal one is faster?

How come these are different? We can leverage the build-in module dis to disassemble them into byte code,

from dis import dis

dis("[]")
dis("list()")

When we use [], the bytecode shows that there are only two steps.

  1. BUILD_LIST – Quite self-explained, just build a Python List
  2. RETURN_VALUE – Return the value

Very simple, right? When the Python interpreter sees the expression [] it just knows that it needs to build a list. So, it is very straightforward.

How about list()?

  1. LOAD_NAME – Try to find the object "list"
  2. CALL_FUNCTION – Call the "list" function to build the Python List
  3. RETURN_VALUE – Return the value

Every time we use something with a name, the Python interpreter will search for the name in the existing variables. The order is Local Scope -> Enclosing Scope -> Global Scope -> Built-in Scope. This search will definitely need some time, and this contribute to the flow performance to a large extent.

If you are interested in a more detailed explanation of the mechanism, please check out the article below.

No, [] And list() Are Different In Python

3. Use "Set" But Not "List"

Image by Gina Janosch from Pixabay
Image by Gina Janosch from Pixabay

As mentioned earlier, the List is very commonly used in Python. However, have you ever thought about if we should use something else? We know that Python List has an order for its items, but what if order does not matter? For example, sometimes, we just need a "container" of something, and the Python Set would satisfy all our requirements. In such cases, the performance of a set will be much better than a list.

Let’s define a list and a set with exactly the same values.

my_list = list(range(100))
my_set = set(range(100))

So, both of them will have from 0 to 99, a total of 100 integers. The only difference is the data structure type. Now, let’s suppose we want to check if the number "99" is in the container and compare the performance.

It can be seen that the performance of a list can vary depending on how far it needs to scan up. In the above example, if we are testing if "0" is in the list, the performance is about to be the same as a set. If we are testing "50" which is in the middle, the performance is getting worse. If we are testing the last element, it will be much slower than any other scenario.

On the other hand, the performance of a set tends to be pretty stable. No matter which number we are testing, the performance doesn’t change.

Why is the performance of a Set better than a List?

The short answer is that the data structures of a list and a set are completely different. While Python Set is implemented using Hash Table, Python List is a typical Array.

What Hash Table look like? In general, the value will be fed into a hash function, this hash function will convert the original value into another value which will be used as the index of the hash table.

Verify "Value 3" existing
Verify "Value 3" existing

When we are trying to find a value, it will be fed into the function again. So, we get the index directly and it will be used to get the corresponding value from the hash table immediately.

Values in a Python List
Values in a Python List

Now, let’s have a look at the Python List, which is an Array-typed data structure. The items in a list are linked together in a fixed order.

Therefore, when we are looking for a certain value, it has to start from the beginning of the list and check each item until the target time is found (or reach the end if not matched).

So, the luckiest scenario is that the first item is exactly what we are looking for, and the result will be returned immediately. The worst case is that the item we are looking for is the last one. This also explained why the performance varied when we were trying to find 0, 50 and 99 in the list.

Summary

Image by Theodor Moise from Pixabay
Image by Theodor Moise from Pixabay

In this article, I have picked three of the popular Python tricks and tried my best to explain the reasons behind the scene. Why it works and why the performance is better. This will also help us to decide when to use these tricks and when it is not really necessary.

This series will be continuing, and more "tricks" will be unveiled as the future works. Welcome to subscribe so you won’t miss them.

Unless otherwise noted all images are by the author


Related Articles

Some areas of this page may shift around if you resize the browser window. Be sure to check heading and document order.