Metaprogramming is a collection of programming techniques which focus on ability of programs to introspect themselves, understand their own code and modify themselves. Such approach to programming gives programmers a lot of power and flexibility. Without metaprogramming techniques, we probably wouldnt have modern programming frameworks, or those frameworks would be way less expressive.
This article is an excerpt from the book, Expert Python Programming, Fourth Edition by Michal Jaworski and Tarek Ziade A book that expresses many years of professional experience in building all kinds of applications with Python, from small system scripts done in a couple of hours to very large applications written by dozens of developers over several years.
Metaclass is a Python feature that is considered by many as one of the most difficult things to understand in this language and thus avoided by a great number of developers. In reality, it is not as complicated as it sounds once you understand a few basic concepts. As a reward, knowing how to use metaclasses grants you the ability to do things that are not possible without them.
Metaclass is a type (class) that defines other types (classes). The most important thing to know in order to understand how they work is that classes (so types that define object structure and behavior) are objects too. So, if they are objects, then they have an associated class. The basic type of every class definition is simply the built-in type class (see Figure 1).
Figure 1: How classes are typed
In Python, it is possible to substitute the metaclass for a class object with youy own type. Usually, the new metaclass is still the subclass of the type class (refer to Figure 2) because not doing so would make the resulting classes highly incompatible with other classes in terms of inheritance:
Figure 2: Usual implementation of custom metaclasses
Let’s take a look at the general syntaxes for metaclasses in the next section.
The general syntax
The call to the built-in type() class can be used as a dynamic equivalent of the class statement. The following is an example of a class definition with the type() call:
def method(self):
return 1
MyClass = type(‘MyClass’, (object,), {‘method’: method})
This is equivalent to the explicit definition of the class with the class keyword:
class MyClass:
def method(self):
return 1
Every class that’s created with the class statement implicitly uses type as its metaclass. This default behavior can be changed by providing the metaclass keyword argument to the class statement, as follows:
class ClassWithAMetaclass(metaclass=type):
pass
The value that’s provided as a metaclass argument is usually another class object, but it can be any other callable that accepts the same arguments as the type class and is expected to return another class object. The call signature of metaclass is type(name, bases, namespace) and the meaning of the arguments are as follows:
- name: This is the name of the class that will be stored in the __name__ attribute
- bases: This is the list of parent classes that will become the __bases__ attribute and will be used to construct the MRO of a newly created class
- namespace: This is a namespace (mapping) with definitions for the class body that will become the __dict__ attribute
One way of thinking about metaclasses is the __new__() method, but at a higher level of class definition.
Despite the fact that functions that explicitly call type() can be used in place of metaclasses, the usual approach is to use a different class that inherits from type for this purpose. The common template for a metaclass is as follows:
class Metaclass(type):
def __new__(mcs, name, bases, namespace):
return super().__new__(mcs, name, bases, namespace)
@classmethod
def __prepare__(mcs, name, bases, **kwargs):
return super().__prepare__(name, bases, **kwargs)
def __init__(cls, name, bases, namespace, **kwargs):
super().__init__(name, bases, namespace)
def __call__(cls, *args, **kwargs):
return super().__call__(*args, **kwargs)
The name, bases, and namespace arguments have the same meaning as in the type() call we explained earlier, but each of these four methods is invoked at the different stage of class lifecycle:
- __new__(mcs, name, bases, namespace): This is responsible for the actual creation of the class object in the same way as it does for ordinary classes. The first positional argument is a metaclass object. In the preceding example, it would simply be a Metaclass. Note that mcs is the popular naming convention for this argument.
- __prepare__(mcs, name, bases, **kwargs): This creates an empty namespace object. By default, it returns an empty dict instance, but it can be overridden to return any other dict subclass instance. Note that it does not accept namespace as an argument because, before calling it, the namespace does not exist yet. Example usage of that method will be explained later in the Metaclass usage section.
- __init__(cls, name, bases, namespace, **kwargs): This is not seen popularly in metaclass implementations but has the same meaning as in ordinary classes. It can perform additional class object initialization once it is created with __new__(). The first positional argument is now named cls by convention to mark that this is already a created class object (metaclass instance) and not a metaclass object. When __init__() is called, the class has been already constructed and so the __init__() method can do less than the __new__() method. Implementing such a method is very similar to using class decorators, but the main difference is that __init__() will be called for every subclass, while class decorators are not called for subclasses.
- __call__(cls, *args, **kwargs): This is called when an instance of a metaclass is called. The instance of a metaclass is a class object (refer to Figure 1); it is invoked when you create new instances of a class. This can be used to override the default way of how class instances are created and initialized.
Each of the preceding methods can accept additional extra keyword arguments, all of which are represented by **kwargs. These arguments can be passed to the metaclass object using extra keyword arguments in the class definition in the form of the following code:
class Klass(metaclass=Metaclass, extra=”value”):
pass
This amount of information can be overwhelming at the beginning without proper examples, so let’s trace the creation of metaclasses, classes, and instances with some print() calls:
class RevealingMeta(type):
def __new__(mcs, name, bases, namespace, **kwargs):
print(mcs, “__new__ called”)
return super().__new__(mcs, name, bases, namespace)
@classmethod
def __prepare__(mcs, name, bases, **kwargs):
print(mcs, “__prepare__ called”)
return super().__prepare__(name, bases, **kwargs)
def __init__(cls, name, bases, namespace, **kwargs):
print(cls, “__init__ called”)
super().__init__(name, bases, namespace)
def __call__(cls, *args, **kwargs):
print(cls, “__call__ called”)
return super().__call__(*args, **kwargs)
Using RevealingMeta as a metaclass to create a new class definition will give the following output in the Python interactive session:
>>> class RevealingClass(metaclass=RevealingMeta):
… def __new__(cls):
… print(cls, “__new__ called”)
… return super().__new__(cls)
… def __init__(self):
… print(self, “__init__ called”)
… super().__init__()
…
<class ‘RevealingMeta’> __prepare__ called
<class ‘RevealingMeta’> __new__ called
<class ‘RevealingClass’> __init__ called
And when you try to create actual instance of RevealingClass you can get following output:
>>> instance = RevealingClass()
<class ‘RevealingClass’> __call__ called
<class ‘RevealingClass’> __new__ called
<RevealingClass object at 0x1032b9fd0> __init__ called
Let’s take a look at the new Python 3 syntax for metaclasses.
Metaclass Usage and Applications
Metaclasses are great tool for doing unusual and sometimes wonky things. They give a lot of flexibility and power in modifying typical class behaviour. So, it is hard to tell what the common examples of their usage are. It would be easier to say that most usages of metaclasses are pretty uncommon.
For instance lets take a look at the __preprare__() method of every object type. It is responsible for preparing the namespace of class attributes. The default type for class namespace is plain dictionary. For years the canonical example of __prepare__() method was providing an collections.OrderedDict instance as a class namespace. Preserving order of attributes in class namespace allowed for things like repeatable object representation and serialization. But since Python 3.7 dictionaries are guaranteed to preserve key insertion order so that use case is gone. But it doesnt mean that we cant play with namespaces.
Lets imagine a following problem: we have a large Python code base that was developed over dozens of years and the majority of the code was written way before anyone in the team cared about coding standards. We may have for instance classes mixing camelCase and snake_case as method naming convention. If we care about consistency, we would be forced to spend a tremendous amount effort to refactor the whole code base into either of naming conventions. Or we could just use some clever metaclass that could be added on top of existing classes that would allow for calling methods in both of ways. We could write new code using new calling convention (preferably snake_case) while leaving old code untouched and waiting for gradual update.
Thats the example of situation when the __prepare__() could be used! Lets start by writing a dict subclass that automatically interpolates camelCase names into snake_case keys:
from typing import Any
import inflection
class CaseInterpolationDict(dict):
def __setitem__(self, key: str, value: Any):
super().__setitem__(key, value)
super().__setitem__(inflection.underscore(key), value)
Note: To save some work we use the inflection module that is not a part of standard library. is able to convert strings between various string cases. You can download it from PyPI using pip:
pip install inflection
Our CaseInterpolationDict class works almost like an ordinary dict type but whenever it stores new value it saves it under two keys: original one and one converted to snake_case. Note that we used dict type as a parent class instead of recommended collections.UserDict. This is because we will use this class in metaclass __prepare__() method and Python requires namespaces to be dict instances.
Now its time to write actual metaclass that will override the class namespace type. It will be surprisingly short:
class CaseInterpolatedMeta(type):
@classmethod
def __prepare__(mcs, name, bases):
return CaseInterpolationDict()
Since we are set up, we can now use the CaseInterpolatedMeta metaclass to create a dummy class with few methods that uses camelCase naming convention:
class User(metaclass=CaseInterpolatedMeta):
def __init__(self, firstName: str, lastName: str):
self.firstName = firstName
self.lastName = lastName
def getDisplayName(self):
return f”{self.firstName} {self.lastName}”
def greetUser(self):
return f”Hello {self.getDisplayName()}!”
Lets save all that code in case_user.py file and start interactive session to see how User class behaves:
>>> from case_class import User
The first important thing to notice is the contents of the User.__dict__ attribute:
>>> User.__dict__
mappingproxy({
‘__module__’: ‘case_class’,
‘__init__’: <function case_class.User.__init__(self, firstName: str, lastName: str)>,
‘getDisplayName’: <function case_class.User.getDisplayName(self)>,
‘get_display_name’: <function case_class.User.getDisplayName(self)>,
‘greetUser’: <function case_class.User.greetUser(self)>,
‘greet_user’: <function case_class.User.greetUser(self)>,
‘__dict__’: <attribute ‘__dict__’ of ‘User’ objects>,
‘__weakref__’: <attribute ‘__weakref__’ of ‘User’ objects>,
‘__doc__’: None
})
The first thing that catches the eye is the fact that methods got duplicated. That was exactly what we wanted to achieve. The second important thing is the fact that User.__dict__ is of mappingproxy type. Thats because Python always copies contents of the namespace object to new dict when creating final class object. The mapping proxy also allows to proxy access to superclasses within the class MRO.
So, lets see if our solution works by invoking all of its methods:
>>> user = User(“John”, “Doe”)
>>> user.getDisplayName()
‘John Doe’
>>> user.get_display_name()
‘John Doe’
>>> user.greetUser()
‘Hello John Doe!’
>>> user.greet_user()
‘Hello John Doe!’
It works! We could call all the snake_case methods even though we havent defined them. For unaware developer that could look like almost like a magic!
However, this is kind of magic that should be used very carefully. Remember that what you have just seen is a toy example. The real purpose of it what to show what is possible with metaclasses and just few lines of code. Learn more in the book Expert Python Programming, Fourth Edition by Michal Jaworski and Tarek Ziadé.
Summary
In this article, we were first introduced to meta programming and eventually to the complex world of metaclasses. We explored the general syntax and practical usage of metaclasses. In the book, we further delve into advanced concepts of metaclasses pitfalls and the usage of __init__subclass__() method as alternative to metaclasses.
About the Authors
MichaÅ Jaworski has more than 10 years of professional experience in writing software using various programming languages. MichaÅ has spent most of his career writing high-performance and distributed backend services for web applications. He has served in various roles at multiple companies: from an ordinary software engineer to lead software architect. His beloved language of choice has always been Python.
Tarek Ziadé is a software engineer, located in Burgundy, France. He works at Elastic, building tools for developers. Before Elastic, he worked at Mozilla for 10 years, and he founded a French Python User group, called AFPy. Tarek has also written several articles about Python for various magazines, and a few books in French and English.
