The world’s leading publication for data science, AI, and ML professionals.

The Power of Decorators

Let's build a strict type checking system and an awesome logging tool in Python from scratch

Image by Rushenb from Wikimedia Commons
Image by Rushenb from Wikimedia Commons

Imagine you are the lead developer on some project. One day your boss tells you that you and your team have to make sure that all the functions and methods have logging and type checking protocols implemented and it needs to be tested by the end of the week.

You know that the project consists of 100’s of callable objects: functions, classes, generators, etc., and here you are, being asked by your boss when you can have this enormous task done and implemented.

You take a sip of your coffee, look at your boss, and say: "In a few hours"…

Why are you so confident?

Because you know about decorators. If this task was given to a more junior programmer/data scientist that didn’t know about this rather advanced feature in Python, it could have taken weeks if not months to gain the amount of flexibility necessary.

But you know advanced Python, and you get things done.

In the next couple of sections, we will simulate this situation together and I will show you what decorators really are and how we can use them for such a task.

I will even build a small module that you can copy and tweak to your own needs in your project.

Functions

Python’s functions are extremely powerful because as the first-class citizens that they are, they can be stored in variables and called later, they can be passed to other functions and they can create an environment (called a local scope) that one can use to do intermediate calculations. That is, you can define functions inside other functions.

In Python, we also have anonymous functions called lambda’s that we can use as functional tools in expressions such as map, filter, etc.

Today, we will be looking at how to use some of these concepts to change code dynamically and to improve your already written code without having to rewrite every function in your project.

In order to understand decorators, we need to understand functions in Python.

Note that these are not mathematical functions because in some cases for a given function the same input does not always ensure the same output that is, a function can have side-effects i.e. the function could be dependent on the outside scope.

Let us create a basic function in Python with the intent of modifying it later.

The function _multimes returns num self-concatenations of the msg string.

So for instance _multimes("HA", 10) will return "HAHAHAHAHAHAHAHAHAHA".

This is actually a very nice example of one of the problems with dynamic typing. This function will run and produce the desired output if it is fed the right parameters. However, if by accident, one inputs an integer or float as the msg argument, then all hell breaks loose.

Obviously, in production, such a vulnerability is not acceptable.

Our boss just told us to get rid of such potential bombs by logging and type-checking every function in the codebase.

Decorators

The idea of a decorator is to create a scope for manipulating a function or class in a controlled way. We do this in practice by defining a function inside another function.

Consider the following piece of code.

When we run this file, we get the output:

**************************************************

So what exactly is happening here?

On line 22, we define a new function called wrapped by passing in the function object _multimes to the function logging.

Okay, but what does logging do?

Glad you asked! logging creates a scope in terms of a nested function inside it called wrapper. Inside wrapper we execute the function passed in as an argument and store the result. In this case, _multimes.

We look out for type errors and in the unlikely scenario that the program throws such an error at us, we log the function name and its arguments in a file.

We could choose to ignore the error but since we are type checking fanatics, we raise TypeError and crash our program deliberately.

If wrapper doesn’t crash, then it returns the result of the function that we passed in.

Here comes the tricky bit: the outer function logging returns the wrapper function object, and not the value wrapper returns.

Now, when we assign _logging(multimes) to the object wrapped and we call wrapped on line 24, what is really happening is the following:

  • Since wrapped is really equal to wrapper inside logging by assignment, we store ‘*’ and 50 in the args tuple and pass it as parameters to the function wrapper.
  • Then we pass these parameters to _multimes and the return value is stored in the variable result.
  • Since there is no type error, the return value of wrapped is the above string that we called result which we then print to the console.

The above code can be written a little more elegantly by the following:

The ‘@’ symbol is syntactic sugar for the equivalent code above and we call such a function like logging, a decorator.

There is one problem though, we can’t be sure that functions in Python throw type errors every time a wrong type is passed. In fact, Python’s dynamic typing system makes sure that we only get an error if we try to do something illegal with an object such as calling a method that it doesn’t have.

This is a very flexible feature of Python and for some projects and scripts, it is very nice, but if you want to build something solid, well-tested and maintainable, you would need to do some type-checking.

Luckily for us, in later versions of Python, it comes with a type inference system. Let’s see how to write the above code with this implementation.

To our disappointment, when we run this code, we get… 100!!!

Python refuses to let go of that dynamical flexibility. We could try to crash it manually of course.

This would actually work, but we don’t want to write that in all our functions taking into account each type of each parameter of each function. It is simply too repetitive and stupid.

Instead, we want to capture Python’s type inference system automatically by implementing that in a decorator.

Let’s create a _typechecker decorator.

Let’s test it on our _multimes function.

This will crash with a TypeError.

It works! Now we just need to combine it with the logging decorator however there are some improvements that we could do at this point. It doesn’t really make sense to log functions if there is only one instance of a logging function and then a crash.

Now that we have handled the type checking, we could simply change the logging decorator to log all functions and then let the _typechecker decorator do the rest.

We will do this by changing the logging decorator and then nest the decorator by writing them on top of each other.

We still want to log the failed functions though. Consider the following code.

The program of course throws a TypeError, and the log.txt file looks like this:

Function: wrapper succeeded with args: ('3', 50, {})
Function: wrapper succeeded with args: ('Kasper', 10, {})
Function: wrapper succeeded with args: ('0', 0, {})
Function: wrapper succeeded with args: ('Medium', 1, {})
Function: wrapper failed with args: (-1, 100, {})

Hmmm not exactly what we had hoped for, huh?

It’s great that we can put the decorators on top of each other to get them chained, but why don’t they recognize the function name?

I will leave the answer as an exercise for the reader, but I will fix the issue here with some best practice Python.

It turns out that the library functools have seen this coming and we can borrow a decorator from functools called wraps.

Let’s take a look at the log.txt file again.

Function: mul_times succeeded with args: ('3', 50, {})
Function: mul_times succeeded with args: ('Kasper', 10, {})
Function: mul_times succeeded with args: ('0', 0, {})
Function: mul_times succeeded with args: ('Medium', 1, {})
Function: mul_times failed with args: (-1, 100, {})

Pretty good.

But we can make yet another improvement to our logging decorator. Right now It only logs a failed function if the error is TypeError and it doesn’t log the time.

We want it to be more flexible.

Finally, let’s raise a more understandable error in the _typechecker decorator.

This will throw a ZeroDivisionError and if we check the log file we see the following output.

Cool, but we have mostly methods in our code not that many functions. How do we decorate those?

Decorating Methods in Python

Well, this is almost the same. In fact, for most applications of decorators, you don’t have to change anything. However, we are manipulating the callable’s arguments and in a method, we always have a reference of an instance of the class as an argument (unless it is a static method).

This is slightly annoying, but it will not stop us. In the _typechecker decorator, we will simply not check the self parameter. We need to write a disclaimer though that this will only work if we name the instance argument "self". This is not required in Python although I have not seen it named anything else.

So I think it is an okay assumption to make.

At this point, we note a small vulnerability in the code. In the _typechecker decorator, we do a look-up in a dictionary. If we wanted to decorate a function with only partial or no type declarations, we would get a KeyError.

I don’t want a KeyError…

You don’t want a KeyError…

Nobody wants a KeyError!!!

Let’s fix this so that we can decorate any function or method regardless of type declarations and then let’s test it.

Full code up to this point with the above fix implemented:

The output from this code is:

This is great. But what if you have a monster class with 100s of methods that need to be type-checked and logged?

Here’s where class decorators really shine!

We are going to create a class decorator called togging (a truncation of type-checking and logging I guess) that will decorate a class and automatically decorate all the methods with _typechecker and logging.

It is also about time to split up the code a little bit. Let us make this into a module that we can use in another file or project.

Our module named decorators now becomes the following module:

Now, in another file, we import the togging class decorator and tests it. Note that when we import the module decorators, the first thing that happens is that the file is decorators.py actually gets executed, so the log file is deleted. This is a good thing because you don’t want to clutter your log file with multiple runs.

The output from this file in the log.txt:

Even the init method got decorated!

The final thing to improve on is yet again the logging decorator. You see, saving the log every time a function or method is called, is fine in some circumstances, however, if you are calling a function or method in a loop or an apply method, it will slow down the program significantly.

It would certainly be nice if we could choose either to save the log or to just print it out to the console. The I/O action of printing actually also slows down the program but not nearly as much as FS I/O of opening-writing-closing a file.

We need to be able to pass an argument to the logging decorator specifying if we want to save the log or just print it.

We do this by triple-nesting functions. It might look a little bit complicated at first, but you should really just think of the function definition as creating a scene or scope for the inner logic.

As a finishing touch, we will time the functions as well and put it in the log.

The following module is the last one. Feel free to copy it, modify it to fit your needs and use it in your own work or project.

What we have here is a lightweight module that gives us strict type-checking in the sense that if you decorate your function or method containing typed parameters in the definition with the type-checker decorator and if the function is fed an argument with a conflicting type, then the program will crash, which is what we wanted, remember?

If you want to leave some or all of the parameters unchecked, that is fine. You simply don’t specify a type in the function definition. The decorator will figure this out on its own.

If you want, you can then stack the logging decorator on top but you can also just use that on its own. It works completely fine by itself.

Let us test this yet again in another file.

The output from this program is the following which was printed to the terminal/console because of the togging decorator.

and in the log.txt file we have:


Now back to our story:

Our boss told us to log and type-check (if possible) any function or method in our huge project. Now you have a module that does just that. You simply decorate any class with togging and decorate any function definition with the logging and _typechecker decorators. Just be aware that the order of decoration does matter here.

I hope that you found this article useful and that you can see how powerful decorators can be. We didn’t even talk about caching, stateful decorators and the connection to metaclasses. I will do that in future articles.

Decorators are used all over the place in the standard library and I think it is one of the most interesting and useful features of Python.

If you have any questions, comments, corrections or concerns, please write me on LinkedIn.

Kasper Müller – Senior Consultant, Data and Analytics, FS, Technology Consulting – EY | LinkedIn

Happy togging.


Related Articles