bill's mumbles

Arbitrary Computation with Python Types

Introduction

For a project I’m doing at work, I wanted to get a better understanding of some of Pydantic’s underlying mechanics related to generics. This led me down a rabbit hole and I was left wondering if I could do arbitrary computation with python types. That is, could I ergonomically do computations using types directly while never explicitly instantiating a class.

Implementation

For this exercise I’m going to look at classes that represent integers and implement the ability to add these classes to produce a new class. To do this I’m going to leverage __class_getitem__. As background, in Python, one can implement the method __getitem__(self, key) to define what indexing into an object using [] does. For example:

Note, this and the following examples were all executed in an ipython shell using python 3.10.13.

from typing import Any

class Indexable:
  def __getitem__(self, key: Any) -> Any:
    return f'my_{key}'

indexable = Indexable()
indexable[1]  # 'my_1'
indexable["computer"]  # 'my_computer'

__class_getitem__ is an analogous method for indexing classes and lets one implement using [] on a class. Usually (in all cases I’ve seen), the argument inside of [] is a class and is used for defining generics.

from typing import TypeVar, Generic

T = TypeVar('T')
class MyClass(Generic[T]): ...

Now one can instantiate classes via MyClass[int](), MyClass[str](), etc.

However, there is nothing restricting one to only pass classes to []. One could pass arbitrary data. Combining this with the ability to dynamically make classes on the fly, one could generate a new class for each integer. For example:

from types import new_class
from typing import Any

class Integer:

    integer_type_cache: dict[int, type] = {}

    def __class_getitem__(cls, integer: int) -> type:
        if integer not in Integer.integer_type_cache:
            neg = "Neg" if integer < 0 else ""
            Integer.integer_type_cache[integer] = new_class(
                f"Integer{neg}{abs(integer)}",
                bases=(),
                kwds=None,
                exec_body=lambda ns: ns.update(
                    {
                        "INTEGER": integer,
                    }
                ),
            )
        return Integer.integer_type_cache[integer]

# Create an integer type
three = Integer[3]
three  # <class 'types.Integer3'>
type(three)  # type

# We can instantiate one if we want
three()  # <types.Integer3 object at 0x1067fd700>

If we want to add we could define a function like:

def addIntegers(a: type, b: type) -> type:
    # I'm loose with types in this function. I'm only passing in classes
    # generated by Integer[N] to this method.
    return Integer[a.INTEGER + b.INTEGER]  # type: ignore[misc, attr-defined]

addIntegers(Integer[1],  Integer[2]) == three  # True

We can now add classes to get new classes! We could make this more ergonomic by overloading the + sign to add these classes. Overloading + is done by implementing __add__ on the class/type of an instance. However, the type of our dynamically generated types, say Integer3 is type, which is not modifiable. However, we can set a metaclass on the dynamically created type. Doing this will give us a hook to define __add__ since the type of Integer3 will be our metaclass. Since Integer3 is a class, it is still a type.

from types import new_class
from typing import cast


class MetaInteger(type):
    INTEGER: int

    def __add__(cls: "MetaInteger", other: "MetaInteger") -> "MetaInteger":
        # Using [] on a class without using Generics will make mypy complain
        return Integer[cls.INTEGER + other.INTEGER] # type: ignore[misc]

class Integer:

    integer_type_cache: dict[int, MetaInteger] = {}

    def __class_getitem__(cls, integer: int) -> MetaInteger:
        if integer not in Integer.integer_type_cache:
            neg = "Neg" if integer < 0 else ""
            # new_class returns a type object. We cast to a MetaInteger
            # since we are explicitly setting this as the metaclass and
            # MetaInteger is a subclass of type.
            Integer.integer_type_cache[integer] = cast(MetaInteger, new_class(
                f"Integer{neg}{abs(integer)}",
                bases=(),
                kwds={"metaclass": MetaInteger},
                exec_body=lambda ns: ns.update(
                    {
                        "INTEGER": integer,
                    }
                ),
            ))
        return Integer.integer_type_cache[integer]


# Adding Integers
Integer[3] + Integer[4] == Integer[7]  # True, adding works!
isinstance(Integer[7], type)  # True, it's a type
type(Integer[7]) # __main__.MetaInteger, MetaInteger is a type subclass
MetaInteger.mro(MetaInteger)  # [__main__.MetaInteger, type, object], the metaclass type hierarchy.

Conclusion

This is clearly ridiculous. By using [] to instantiate new types we can store arbitrary data on a type that is as ergonomic as instantiating a class with (). We can also put arbitrary methods on the class as well as overload operators using a metaclass. We’ve made metaclasses analogous to classes, classes analogous to instances, and instances (eg Integer[3]()), well they’re a new thing at the bottom of the hierarchy. This gives us the ability to do arbitrary computation while never explicitly instantiating a python class.

Would one ever want to do this? No, this is tomfoolery. But, I couldn’t resist trying to see if this was possible.

Published

Category

python

Tags

Contact