Dynamically Altering Classes in Python

The Very Basics

Everything in Python is an object. Don't believe me?

In [1]: type(42)
Out[1]: <class 'int'>

Suffice to say that more complicated things are also classes. Derive from object to declare your own classes, and then create an instance:

class Cat(object):
    # Add a property
    @property
    def legs(self):
        return 4
    # Add a method
    def meow(self):
        print('meow')

cat_1 = Cat()

cat_1.meow()
# meow
cat_1.legs
# 4

Running dir() on will show you what methods and properties are on the object:

In [2]: dir(cat_1)
Out[2]:
['legs', 'meow']

Adding methods dynamically

One of the best things about Python is its delightfully adult attitude towards runtime mutability: if you have ever found yourself asking "can I add a method to an object after initializing it?", you should know not only that the answer is "yes", but that Python also makes it easy (and relatively safe) to do so by using the MethodType method from the types module:

import types

def pounce(*args):
    print("pounce!")

setattr(cat_1, 'pounce', types.MethodType(pounce, cat_1))

In [3]: dir(cat_1)
Out[3]:
[ 'legs', 'meow', 'pounce']

It's important to note that the new method only applies to the current instance of the class:

# cat_2 is not affected by the changes made to cat_1
cat_2 = Cat()

In [4]: dir(cat_2)
Out[4]:
['legs', 'meow']

But what if you want to add a method dynamically to all instances of the class?

Enter metaprogramming

Remember how we said that everything in Python is an object? Take a look at this:

In [5]: type(Cat)
Out[5]: <class 'type'>

If you are like me, it'll will take a while to internalize the message being conveyed here, which is that the Cat class is an instance of the type metaclass. And as we saw in the last section, anything that is an instance can be manipulated at runtime:

setattr(Cat, 'pounce', pounce)

Checking in on cat_1, we see an unsurprising result -- we did after all previously bind the pounce method to it:

In [6]: dir(cat_1)
Out[6]:
[ 'legs', 'meow', 'pounce']

cat_2 on the other hand never had a pounce method attached to it, but now shows one inherited from the class definition:

In [7]: dir(cat_2)
Out[7]:
[ 'legs', 'meow', 'pounce']

In [8]: cat_2.pounce()
pounce!

Lovely.

Footgun Alert

A few warnings for those of you who, like me, are occasionally taken unawares by things that others perhaps wouldn't be surprised by.

Firstly, class implementations of methods can be overridden on a per-instance basis:

def alternate_pounce(*args):
    print('alternate pounce!')

setattr(cat_2, 'pounce', alternate_pounce)

In [9]: cat_2.pounce()
alternate_pounce!

In [10]: cat_1.pounce()
pounce!

Secondly, any changes made at the class level do not override local changes at the instance level:

def alternate_pounce_2(*args):
    print('alternate pounce 2!')

setattr(Cat, 'pounce', alternate_pounce_2)

# We overrode cat_2's 'pounce' method with alternate_pounce and
# changing the class definition of 'pounce' isn't visible from cat_2
In [11]: cat_2.pounce()
alternate pounce!

# A newly instantiated object shows the new 'pounce' output, however
cat_3 = Cat()

ln [12]: cat_3.pounce()
alternate pounce 2!

Finally, changes to the class last as long as the process is alive. This particular footgun can cause unintuitive behavior when using (and mutating) the class in unit tests that are not run in separate processes. You may think that the class is going to start out fresh, but really the definition of the class at the beginning of a test is what it was at the end of the last test that was run. Spare yourself hours of debugging test order and unittest internals, and reset the class in your test class' setUp and tearDown methods.

Why does this exist?

Every Python metaprogramming article seems to include this quote from Tim Peters:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why)

Under most circumstances, altering code while it is running increases its complexity and makes reasoning about it more difficult. There are, however, certain circumstances where it is warranted: while developing the database persistence portion of OpenARC, for example, it was necessary to map dynamically changing relationships between database objects. Metaprogramming fit the bill in that very limited use case, and I happily used it.

Knowing your options is good. Don't let black magic scare you.