Customizing Instantiation with __new__#

The magic method we use most is __init__(). But __init__() isn’t the first thing called — __new__() runs before it. By default, __new__() creates the instance and passes it as the first argument to __init__(), which receives the new instance and returns it. By overriding __init__() we can automatically add attributes to newly created instances; by overriding __new__() we can customize the instance creation process itself. For example, here’s the singleton pattern:

1
class A:
2
    _instance = None
3
    _initialized = False
4

5
    def __new__(cls, *args, **kwargs):
6
        if not cls._instance:
7
            cls._instance = super().__new__(cls)
8
        return cls._instance
9

10
    def __init__(self, test):
11
        if not self._initialized:
12
            self._initialized = True
13
            self.test = test
14

15
a1 = A(test=10)
16
a2 = A(test=20)
17

18
print(a1 is a2)  # Output: True
19
print(a2.test)  # Output: 10

Because we’ve overridden __new__(), we must explicitly call the parent’s instantiation logic — that is, super().__new__(cls) (the parent is object, the root of all classes). We also must return an instance; otherwise the rest of the machinery never runs and instantiation is cut short. And if we returned an instance of some other class here, that’s the instance the rest of the process would use.

Also, since __init__() runs on every instantiation, we need to limit how many times it takes effect — otherwise later instantiations would overwrite the existing singleton’s state, unless that’s the behavior you want.

There’s one more magic method tied to an instance’s lifecycle: __del__(). I don’t recommend customizing it — the official Python docs advise against it too — and I won’t cover it here. Instead of customizing __del__, just write a close() method.

Custom String Representations#

When we print an object we defined, the output looks like <__main__.Book object at 0x00000230AA7EA060>:

1
class Book:
2
    def __init__(self, title, author, isbn, stock):
3
        self.title = title
4
        self.author = author
5
        self.isbn = isbn
6
        self.stock = stock
7

8
fluent_python = Book("Fluent Python", "Luciano Ramalho", "978-1492056355", 3)
9
print(fluent_python)  # Output: <__main__.Book object at 0x00000230AA7EA060>

If you want an easy-to-read string representation of the object, it might seem like the only option is a get_book_info method — but is it really?

We can define an object’s string representation with the magic method __str__():

1
class Book:
2
    def __init__(self, title, author, isbn, stock):
3
        self.title = title
4
        self.author = author
5
        self.isbn = isbn
6
        self.stock = stock
7

8
    def __str__(self):
9
        return f'"{self.title}" by {self.author} (ISBN: {self.isbn}) - Stock: {self.stock}'
10

11
fluent_python = Book("Fluent Python", "Luciano Ramalho", "978-1492056355", 3)
12
print(fluent_python)  # Output: "Fluent Python" by Luciano Ramalho (ISBN: 978-1492056355) - Stock: 3

Much clearer now, isn’t it? But there’s a catch: from this output you can’t tell what attributes the class has or what values they hold — in short, it’s not formal enough. That’s when you define a __repr__():

1
class Book:
2
    def __init__(self, title, author, isbn, stock):
3
        self.title = title
4
        self.author = author
5
        self.isbn = isbn
6
        self.stock = stock
7

8
    def __str__(self):
9
        return f'"{self.title}" by {self.author} (ISBN: {self.isbn}) - Stock: {self.stock}'
10

11
    def __repr__(self):
12
        return f"Book(title={self.title}, author={self.author}, isbn={self.isbn}, stock={self.stock})"
13

14
fluent_python = Book("Fluent Python", "Luciano Ramalho", "978-1492056355", 3)
15
print(fluent_python)  # "Fluent Python" by Luciano Ramalho (ISBN: 978-1492056355) - Stock: 3
16
print(f"{fluent_python!s}")  # "Fluent Python" by Luciano Ramalho (ISBN: 978-1492056355) - Stock: 3
17
print(f"{fluent_python!r}")  # Book(title=Fluent Python, author=Luciano Ramalho, isbn=978-1492056355, stock=3)

When nothing is specified, printing shows what __str__ defines. When you explicitly ask for the formal representation, you get what __repr__ defines.

Comparisons with Magic Methods#

The magic methods used for comparisons are:

• __eq__(self, other): defines equality behavior (==).
• __ne__(self, other): defines inequality behavior (!=).
• __lt__(self, other): defines less-than behavior (<).
• __le__(self, other): defines less-than-or-equal behavior (<=).
• __gt__(self, other): defines greater-than behavior (>).
• __ge__(self, other): defines greater-than-or-equal behavior (>=).

Here’s an example: a class that represents a word. You might want to compare words in lexicographic (alphabetical) order — the default string comparison already does that — but you might also want to compare by other criteria, like word length or syllable count. In this example we’ll compare by length. Here’s the implementation:

1
class Word(str):
2

3
    def __new__(cls, word):
4
        # Note that we have to use __new__. This is because str is an immutable
5
        # type, so we have to initialize the value at creation time
6
        if ' ' in word:
7
            print("Value contains spaces. Truncating to first space.")
8
            word = word[:word.index(' ')]  # the word is all characters before the first space
9
        return str.__new__(cls, word)
10

11
    def __gt__(self, other):
12
        return len(self) > len(other)
13
    def __lt__(self, other):
14
        return len(self) < len(other)
15
    def __ge__(self, other):
16
        return len(self) >= len(other)
17
    def __le__(self, other):
18
        return len(self) <= len(other)
19

20
word1 = Word('hello')
21
word2 = Word('world')
22
word3 = Word('python')
23

24
print(word1 > word2)  # Output: False
25
print(word1 < word3)  # Output: True
26
print(word2 >= word1) # Output: True

Note that we didn’t define __eq__() and __ne__(). That’s deliberate: defining them would produce some strange results (for example, Word('foo') == Word('bar') would return True). So we leave them out and fall back on str’s built-in comparison.

Context Managers#

Context managers make sure resources are cleaned up properly after use. We can implement one with the magic methods __enter__() and __exit__():

1
class DatabaseConnection:
2
    def __init__(self, database_name):
3
        self.database_name = database_name
4
        self.connection = None
5

6
    def __enter__(self):
7
        print(f"Connecting to database: {self.database_name}")
8
        self.connection = self.database_name
9
        return self.connection
10

11
    def __exit__(self, exc_type, exc_value, traceback):
12
        print(f"Closing database connection: {self.database_name}")
13

14
with DatabaseConnection("user database") as conn:
15
    print(f"Using connection: {conn}")
16
    print("Performing database operations...")

with runs whatever __enter__() defines when entering the block and binds __enter__()’s return value to the target in the as clause (if there is one). When leaving the block, whatever __exit__() defines runs.

For async code, just switch to __aenter__() and __aexit__().

Optimizing Attribute Access with slots#

By default, Python objects store attributes in a dictionary. The upside is that you can add attributes dynamically; the downside is that it can use more memory and lookups are relatively slow. Defining __slots__ prevents the class from creating a __dict__ and a __weakref__.

To use __slots__, define an attribute named __slots__ in the class and give it the allowed attribute names:

1
class Book:
2
    __slots__ = ("title", "author", "isbn", "stock")
3

4
    def __init__(self, title, author, isbn, stock):
5
        self.title = title
6
        self.author = author
7
        self.isbn = isbn
8
        self.stock = stock

Note that without __weakref__, this class doesn’t support weak references. If you need weak reference support, declare __weakref__ in __slots__ manually.

With inheritance, if the parent class defines __slots__, the child class must define its own too — otherwise child instances will have a __dict__ and __weakref__.

Iterators and Generators#

You may have noticed that most container objects work with the for statement:

1
for char in "abc":
2
    print(char)

Behind the scenes, the for statement calls iter() on the container object. That function returns an iterator object with a __next__() method, which accesses the container’s elements one at a time. When there are no more elements, __next__() raises StopIteration to signal the loop to end. You can call __next__() via next(). Here’s the equivalent of for char in "abc" spelled out:

1
>>> s = 'abc'
2
>>> it = iter(s)
3
>>> next(it)
4
'a'
5
>>> next(it)
6
'b'
7
>>> next(it)
8
'c'
9
>>> next(it)
10
Traceback (most recent call last):
11
  File "<stdin>", line 1, in <module>
12
StopIteration

To make one of our own classes iterable, we just implement the above. Define __iter__() to return an object with a __next__() method; if the class already defines __next__(), just return self:

1
class Reverse:
2
    """Iterator for looping over a sequence backwards."""
3
    def __init__(self, data):
4
        self.data = data
5
        self.index = len(data)
6

7
    def __iter__(self):
8
        return self
9

10
    def __next__(self):
11
        if self.index == 0:
12
            raise StopIteration
13
        self.index = self.index - 1
14
        return self.data[self.index]

Now for a side note: is there an easier way to implement iteration? Yes — generators. Generators look like regular functions, but they use yield when they want to return data. They automatically create __iter__() and __next__(), and each call to next() on a generator resumes execution from where it left off. When all elements have been produced, it automatically raises StopIteration. With generators, creating an iterator can be as easy as writing a regular function:

1
def reverse(data):
2
    for index in range(len(data)-1, -1, -1):
3
        yield data[index]

Introduction#

Let’s start with “Hello, World!”:

1
print("Hello, World!".__class__)  # Output: <class 'str'>
2
print(str.__class__)  # Output: <class 'type'>
3
print(str.__bases__)  # Output: (<class 'object'>,)
4

5
print("Hello, World!".__bases__)  # Raise: AttributeError: 'str' object has no attribute '__bases__'. Did you mean: '__class__'?

In Python, everything is an object. Even basic data types are objects created by built-in classes (which is exactly why you can call methods directly on a string). As the code shows, the string "Hello, World!" is an instance of the built-in class str, and the string instance itself has no base classes. Turning to the built-in class str, it’s an instance of the type class, and its base class is object. Since instances are what you get when a class is instantiated, the built-in class str must itself be instantiated by type.

1
class Book:
2
    pass
3

4
print(Book.__class__)  # <class 'type'>
5
print(Book.__bases__)  # (<class 'object'>,)

The same holds for our custom classes: they inherit from object and are instances of type.

The Python Object System — From the Ground Up#

NOTE

This section is largely based on “Python Types and Objects” by Shalabh Chaturvedi, reorganized and adapted by me (Raven95676) for educational purposes. Since the original author retains all rights, this section is not released under the CC BY-NC-SA 4.0 license. If the original copyright holder believes this section infringes on their copyright, please contact me (Raven95676) to have the relevant content removed.

The images referenced in this section are quite old; please read any occurrence of <type 'object'> in them as <class 'object'>.

We’re going to build the Python object system from scratch, starting at the very beginning — a blank sheet of paper.

Before drawing anything, let’s introduce the relationships. We connect objects with two kinds of relationships:

• Subclass-parent class relationship (inheritance): one object is a specialized version of another — “a catgirl is a kind of fantasy creature”.
• Type-instance relationship (instantiation): one object is a concrete instance of another — “Nova the catgirl is an instance of catgirl”.

Inheritance is drawn as a solid line from child to parent; instantiation is a dashed line from instance to type, as in the diagram below:

Let’s start our exploration at the root of everything:

1
print(object.__class__)  # Output: <class 'type'>
2
print(object.__bases__)  # Output: ()
3

4
print(type.__class__)  # Output: <class 'type'>
5
print(type.__bases__)  # Output: (<class 'object'>,)

object is instantiated by type and has no parent class. That makes object the starting point of inheritance. type is instantiated by itself, and its parent is object.

Let’s put these findings on the diagram:

Now let’s bring in the built-in data types list, dict, and tuple:

1
print(list.__class__)  # Output: <class 'type'>
2
print(list.__bases__)  # Output: (<class 'object'>,)
3
print(tuple.__class__)  # Output: <class 'type'>
4
print(tuple.__bases__)  # Output: (<class 'object'>,)
5
print(dict.__class__)  # Output: <class 'type'>
6
print(dict.__bases__)  # Output: (<class 'object'>,)

They all inherit from object and are all instantiated by type. Now let’s instantiate a list and see what happens:

1
mylist = [1,2,3]
2
print(mylist.__class__)  # Output: <class 'list'>
3
print(mylist.__bases__)  # Raise: AttributeError: 'list' object has no attribute '__bases__'. Did you mean: '__class__'?

mylist is instantiated by list and has no parent class.

Let’s add these findings to the diagram:

Now let’s define our own class based on the existing object class (if you don’t specify a base, the class inherits from object by default), then instantiate it:

1
class C:
2
    pass
3

4
obj = object()
5
cobj = C()
6

7

8
print(C.__class__)  # Output: <class 'type'>
9
print(C.__bases__)  # Output: (<class 'object'>,)
10
print(obj.__class__)  # Output: <class 'object'>
11
print(cobj.__class__)  # Output: <class '__main__.C'>

Let’s put these findings on the diagram:

In fact, what each of these three boxes represents is now plain to see:

The first box is the class of all classes — we call it the metaclass.

The second box is both the class of the third box and an instance of the first box — we call it the type object.

The third box is an instance of the second box — we simply call it the instance.

The metaclass instantiates type objects, and type objects instantiate classes. In other words, the creation of every new object ultimately boils down to instantiation.

So the following two snippets are essentially equivalent:

1
class MyClass:
2
    pass
3

4
MyClass = type("MyClass", (), {})

Using Metaclasses#

So now we——

We know: by default, classes are built with type.

We know: classes can be inherited from.

So can we inherit from type, create a new metaclass, and use that new metaclass to create classes? Of course we can — that’s the very core of metaclasses.

Congratulations — we’re finally ready to learn how to use metaclasses.

A basic metaclass typically inherits from type and overrides its __new__() or __init__():

1
class Meta(type):
2
    def __new__(cls, clsname: str, bases: tuple, clsdict: dict):
3
        return super().__new__(cls, clsname, bases, clsdict)
4

5
    def __init__(self, clsname: str, bases: tuple, clsdict: dict):
6
        super().__init__(clsname, bases, clsdict)

Here, clsname is the class name and will become the new class’s __name__ attribute. bases is the tuple of base classes and will become the __bases__ attribute; if it’s empty, object is added. clsdict is the dict of attribute and method definitions from the class body; it may be copied or wrapped before becoming the __dict__ attribute.

Now let’s give this metaclass a job: it will reject any class definition containing method names with mixed case:

1
class NoMixedCaseMeta(type):
2
    def __new__(cls, clsname, bases, clsdict):
3
        for name in clsdict:
4
            if name.lower() != name:
5
                raise TypeError('Bad attribute name: ' + name)
6
        return super().__new__(cls, clsname, bases, clsdict)

How do you apply a metaclass? By specifying the metaclass parameter:

1
class A(metaclass=NoMixedCaseMeta):
2
    def foo_bar(self):
3
        pass
4

5
class B(metaclass=NoMixedCaseMeta):
6
    def fooBar(self):  # Raise: TypeError: Bad attribute name: fooBar
7
        pass

From our discussion above, we know classes are instantiated by a metaclass, which defaults to type. So specifying metaclass is equivalent to going from:

1
A = type('A', (), {'foo_bar': lambda self: None})
2
B = type('B', (), {'fooBar': lambda self: None})

to:

1
A = NoMixedCaseMeta('A', (), {'foo_bar': lambda self: None})
2
B = NoMixedCaseMeta('B', (), {'fooBar': lambda self: None})  # Raise: TypeError: Bad attribute name: fooBar

Writing Your Own Decorators#

Generally speaking, a decorator is a function that takes another function as an argument and returns a closure with extra behavior added (hence the name “decorator”). Suppose we need functions that generate email content:

1
def welcome_message():
2
    print("Welcome to our team.")

Formal emails are expected to follow certain etiquette. But the added formatting and content are fixed — if every email-generating function copy-pasted the same boilerplate, a future change to the email format would force you to edit every function. To avoid that hassle, we can write a decorator:

1
def email_decorator(func):
2
    def wrapper():
3
        print("Hello,")
4
        func()
5
        print("Best regards,\nRaven")
6
    return wrapper

Let’s apply it:

1
@email_decorator
2
def welcome_message():
3
    print("Welcome to our team.")
4

5
welcome_message()

When we call welcome_message, what actually runs is the closure wrapper: it prints the greeting, calls the original function, and finally prints the signature.

In fact, the code above with @ is equivalent to this:

1
welcome_message = email_decorator(welcome_message)

@ is convenient syntactic sugar — decorators work perfectly fine without it.

Right now welcome_message takes no arguments. What if we want to extend it — say, to accept a department name? You might try this:

1
@email_decorator
2
def welcome_message(department):
3
    print(f"Welcome to our {department} team.")
4

5
welcome_message()  # Raise: TypeError: email_decorator.<locals>.wrapper() takes 0 positional arguments but 1 was given

But that raises a TypeError. Look closely: func() inside our decorator takes no arguments — that’s the root of the problem. To make the decorator more general, add *args and **kwargs so it accepts any number of arguments (though you can also add exactly the parameters you need):

1
def email_decorator(func):
2
    def wrapper(*args, **kwargs):
3
        print("Hello,")
4
        func(*args, **kwargs)
5
        print("Best regards,\nRaven")
6

7
    return wrapper

You may have noticed that this decorator returns nothing. But the decorated function may need to return a value — in that case the decorator has to “catch” that value and return it. Let’s use a decorator that measures a function’s execution time as an example:

1
import time
2

3
def timer_decorator(func):
4
    def wrapper(*args, **kwargs):
5
        start_time = time.time()
6
        result = func(*args, **kwargs)
7
        end_time = time.time()
8
        print(f"Function {func.__name__} took {end_time - start_time:.4f} seconds")
9
        return result
10
    return wrapper
11

12
@timer_decorator
13
def calc(n):
14
    def fibonacci(x):
15
        if x <= 1:
16
            return x
17
        return fibonacci(x - 1) + fibonacci(x - 2)
18

19
    result = fibonacci(n)
20

21
    return result
22

23
calc(35)

A detail many people miss: when you try to get metadata of the original function — like __name__ — you get wrapper instead of the intuitive calc. That’s expected, because wrapper is what’s actually being called. But sometimes you genuinely need the original function’s metadata. That’s where the wraps decorator from the functools module comes in:

1
import time
2

3
from functools import wraps
4

5

6
def timer_decorator(func):
7
    @wraps(func)
8
    def wrapper(*args, **kwargs):
9
        start_time = time.perf_counter()
10
        result = func(*args, **kwargs)
11
        end_time = time.perf_counter()
12
        print(f"Function {func.__name__} took {end_time - start_time:.4f} seconds")
13
        return result
14

15
    return wrapper

Back to our email decorator. Sometimes the decorator itself needs to take parameters — say, to customize the sender. In that case we add another layer on top of the existing decorator structure: a function called a decorator factory, which takes the parameters and “produces” a decorator:

1
from functools import wraps
2

3

4
def email_decorator(from_who):
5
    def decorator(func):
6
        @wraps(func)
7
        def wrapper(*args, **kwargs):
8
            print("Hello,")
9
            func(*args, **kwargs)
10
            print(f"Best regards,\n{from_who}")
11

12
        return wrapper
13

14
    return decorator
15

16

17
@email_decorator(from_who="Raven")
18
def welcome_message(department):
19
    print(f"Welcome to our {department} team.")
20

21

22
welcome_message("Development")

Why do we need a decorator factory? Can’t we just pass arguments directly? No — because the @ syntax only passes the decorated function to the decorator. If we want the decorator itself to accept arguments, passing them directly won’t work. The @ syntax above is equivalent to:

1
decorated_welcome = email_decorator(from_who="Raven")(welcome_message)

Here, email_decorator(from_who="Raven") runs first and returns a decorator function, which then receives welcome_message and produces the final decorated function.

Sometimes one decorator isn’t enough. In that case you simply stack them. One thing to keep in mind: decorators are applied from the bottom up, so don’t get the order wrong:

1
from functools import wraps
2

3

4
def email_decorator(from_who):
5
    def decorator(func):
6
        @wraps(func)
7
        def wrapper(*args, **kwargs):
8
            print("Hello,")
9
            func(*args, **kwargs)
10
            print(f"Best regards,\n{from_who}")
11

12
        return wrapper
13

14
    return decorator
15

16
def add_studio_info(func):
17
    @wraps(func)
18
    def wrapper(*args, **kwargs):
19
        result = func(*args, **kwargs)
20
        print("---")
21
        print("Raven-Studio")
22
        return result
23
    return wrapper
24

25
@add_studio_info
26
@email_decorator(from_who="Raven")
27
def welcome_message(department):
28
    print(f"Welcome to our {department} team.")
29

30

31
welcome_message("Development")

Actually, the order is easy to understand. The @ usage here is equivalent to:

1
welcome_message = add_studio_info(email_decorator(from_who="Raven")(welcome_message))

Besides functions, classes can also be used as decorators. This is very useful when you need to manage state or organize complex logic together, and it typically requires overriding two magic methods, __init__() and __call__():

1
from functools import wraps
2

3

4
class EmailDecorator:
5
    def __init__(self, from_who):
6
        self.from_who = from_who
7

8
    def __call__(self, func):
9
        @wraps(func)
10
        def wrapper(*args, **kwargs):
11
            print("Hello,")
12
            func(*args, **kwargs)
13
            print(f"Best regards,\n{self.from_who}")
14

15
        return wrapper
16

17

18
@EmailDecorator(from_who="Raven")
19
def welcome_message(department):
20
    print(f"Welcome to our {department} team.")
21

22

23
welcome_message("Development")

__init__() receives the decorator’s arguments; __call__() receives the function being decorated and returns the wrapped function.

Of course, decorators can also decorate classes — we’ll cover that together in the “Metaclass Alternatives” section below.

Using the lru_cache Decorator#

Now let’s turn back to the Fibonacci calculation function:

1
import time
2

3
from functools import wraps, lru_cache
4

5

6
def timer_decorator(func):
7
    @wraps(func)
8
    def wrapper(*args, **kwargs):
9
        start_time = time.perf_counter()
10
        result = func(*args, **kwargs)
11
        end_time = time.perf_counter()
12
        print(f"Function {func.__name__} took {end_time - start_time:.4f} seconds")
13
        return result
14
    return wrapper
15

16
@timer_decorator
17
def calc(n):
18
    @lru_cache(maxsize=4096)
19
    def fibonacci(x):
20
        if x <= 1:
21
            return x
22
        return fibonacci(x - 1) + fibonacci(x - 2)
23

24
    result = fibonacci(n)
25

26
    return result
27

28
calc(50)