Python is an object-oriented Programming language. In fact, it is often said that everything in Python is an object.
So in order to fully understand Python, you really need to understand the underlying structures like objects and classes to be able to understand what is really happening under the hood.
This article is about the concept of metaprogramming in Python and we will see how to manipulate classes and the instantiation of them.
Metaprogramming can be loosely defined as code that manipulates code.
An object is of course an instantiation of a class and as the curious beings that we are, when we hear that in Python, everything is an object, a natural question pops up in the back of our minds (right?), namely, are classes then also objects? and if so, what instantiates them?
Let’s find out.
Classes and Objects
The first thing we need to discuss is the relationship between objects and classes.
Say we have the following class
We create an empty class containing no attributes and no methods. Then we instantiate it in the object kasper, at last, we ask which class it’s an instantiation of via the type function.
This code will return
<class '__main__.Person'>
indicating that the class that instantiated the object kasper was Person and it was defined in the same module that you asked for the type of kasper.
Everything in Python is an object so we could do the same for the built-in types like
which will print out
<class 'str'>
So the string ‘Medium’ is an object instantiated by the class str.
#
Okay, but what is then the type of a class? Well, let’s try that out.
The output is
<class 'type'>
<class 'type'>
<class 'type'>
Interesting. It looks like the type of a class is type.
That’s because type is what is known as a metaclass. A metaclass instantiates classes just like classes instantiate objects.
This is all good but how can you use it?
Well, since a metaclass creates a class in certain steps, it might be useful if we could go in and manipulate that process and thereby make our own custom metaclass that would instantiate classes differently than type does.
This concept of changing the behavior by manipulating the code before execution is well-known from decorators.
Decorators in Python makes us capable of manipulating functions before calling them, and in fact, as we will see in a moment, the two have a lot in common.
Class instantiations
The first thing to understand when we talk about metaclasses is how a class in Python is built by the default metaclass type.
When instantiating a class, Python grabs some variables like the name of the class and the base classes that the class inherits from and basically forms a dictionary by calling a dunder method on type called prepare.
Then the body of the class gets executed in order to populate that dictionary with attributes and methods.
The last step is to give the name, the base classes and the above dictionary as parameters to the class type. It looks like the following:
Person = type('Person', (Base1, Base2), cls_dict)
The object Person is now a class that inherits from its parents Base1 and Base2 and has attributes and methods specified by the cls_dict.
If we were to recreate the empty class called Person, then we could do that dynamically like this
Person = type('Person', (), {})
This is equivalent to the above definition where we used the keyword pass to make an empty class.
How do we create custom metaclasses?
Consider the following example:
Here we create a custom metaclass called custom where we overwrite type‘s dunder method new.
Inside new we collect the methods from the dictionary passed to the new method in order to count them and check if there are at most two methods in the class. If not we will throw an error even before the class is instantiated.
The method new returns a class when called and we get the chance to manipulate the parameters of the class before instantiating it.
When we are done, we simply call type‘s new method to get the class and return it.
Consider this:
This will return
WOOOF!
No problem. But if we try to run the following:
it will crash with the following message
TypeError: More than two methods
and Decorators
Say that we have a bunch of classes that all inherit from the same base class, and we want to do some automatic debugging on all the methods of all these classes.
We also want to avoid code duplication following the DRY principle (Don’t Repeat Yourself). What is the best solution to this problem?
Obviously, decorators come to mind and indeed the concept of decorators where we can do all kinds of thing before and after the execution of a function (or class) is a nice debugging tool to have at your disposal and a massive DRY optimizer.
Let’s create a custom debugger that can tell us how many parameters are passed to the methods in a class.
This is of course just a toy example but it will do the trick in this article and one could imagine that we would write a logging decorator that would write to a log or some other actual useful decorator.
The output of this code is
arguments has 3 arguments
16
Pretty nice. The print statement in the wrapper function gets executed before the function arguments gets called.
Note that the @ syntax above is just syntactic sugar. What’s really going on behind the scenes is that you pass the function as a parameter to the decorator when the function is called.
If we want this decorator on each method in a class, one way is of course to decorate all methods one by one. However, as the lazy programmers that we are, we would hate to repeat ourselves so there is a small trick for this.
Consider the following custom class decorator:
Now we can define a class with that decorator on top to get all our methods (except class methods and static methods) debugged in one fellow sweep.
The output is
walk has 1 arguments
miaaw has 1 arguments
This is all great but recall from the problem statement above that we have a lot of classes inheriting from the same base class.
Can we somehow use this inheritance in order to save all those lines of code where we need to put decorators?
Well, yes. We could use a custom metaclass.
Let’s say that the common base class that all our classes inherit from is called Base. Then we could create the following metaclass that Base then could inherit from.
All our classes that inherit from Base will now have all their methods debugged specified by the _debugfunction decorator.
When Should I Use a Metaclass?
Metaclasses are rarely used in practice because there are a lot of other solutions to most problems where one could use metaclasses. For instance, decorators give us a general powerful tool which in most cases is enough.
However, I think it is important to know the inner workings of Python so that you really understand what’s going on under the hood and know when to use the right tools for the job.
Now you know about metaclasses and when it makes sense to use them. If you have a lot of classes that all have a common base class, then a metaclass would make sure to permeate through inheritance all the way down the ancestry tree.
A rule of thumb is this: it makes sense to use metaclasses if you have a scenario in which you need to change how a class is instantiated.
Decorators don’t quite give you just that because the class actually is instantiated before you do anything with a decorator.
I hope you found this article useful.
As always, if you have any questions, comments or concerns, feel free to reach out on LinkedIn.
Kasper Müller – Senior Consultant, Data and Analytics, FS, Technology Consulting – EY | LinkedIn