Python's variable scoping and closure mechanisms are generally intuitive, but when you combine late-binding closures, default argument binding, generator expressions, and in-place list mutations (+=), subtle and surprising behavior can emerge.

The Core Issue

Consider a loop that creates dynamically generated lambda functions using standard scope captures and default arguments:

def create_transformers():
    data = [1, 2, 3]
    transformers = {}

    for i in range(3):
        # step=i attempts to capture 'i' at definition time
        transformers[f"key_{i}"] = lambda x, step=i: [x + step + j for j in data]
        # In-place mutation of the referenced object
        data += [i]

    return transformers

At first glance, setting step=i as a default parameter appears to capture the value of i at definition time, while data remains a reference captured by the closure. However, depending on whether you mutate data in place or reassign it, and whether you use a list comprehension or a generator expression inside the closure, execution results can vary dramatically.

1. How Default Arguments vs. Closure Lookups Work in Bytecode

In Python, default arguments (such as step=i) are evaluated at function definition time when the MAKE_FUNCTION instruction executes. The calculated default values are stored in the function object's __defaults__ tuple.

In contrast, non-default free variables (like data) inside a lambda or inner function are captured as cell variables (via PyCellObject). In CPython bytecode:

  • In-scope variable assignment creates cell reference via STORE_DEREF.
  • Inside the inner function, data is resolved dynamically at runtime using LOAD_DEREF.

2. In-Place Mutation (+=) vs. Re-assignment (=)

The distinction between data += [i] and data = data + [i] is crucial to understanding this behavior:

  • In-Place Mutation (data += [i]): Calls operator.iadd() (or __iadd__), modifying the existing list object in memory. The cell object created for closure binding continues pointing to this exact same list instance. All created lambdas read from and write to the same mutable container.
  • Re-assignment (data = data + [i]): Evaluates the right-hand side to create a brand new list object in memory, then updates the cell variable in the enclosing scope to reference the new list object via STORE_DEREF.

3. Why Generator Expressions Behave Differently

When you replace the list comprehension with a generator expression inside the closure:

transformers[f"key_{i}"] = lambda x, step=i: sum(x + step + j for j in (d for d in data))

Python treats generator expressions as anonymous inner functions. Crucially, CPython evaluates the outermost iterable expression immediately upon generator creation, while evaluation of inner clauses is deferred until the generator is actually consumed (e.g., by sum() or a loop).

When calling the lambda later, the closure cell for data is accessed. Because data += [i] modified the original object in place, the iterator consumes elements from the fully mutated list. However, if deferred generator creation interacts with re-bound scope cells, timing delays cause differences in which exact object reference gets evaluated when the frame runs.

4. CPython Frame Execution Mechanics (Python 3.11+)

In Python 3.11 and newer, CPython's bytecode interpreter (PyEval_EvalFrameDefault) features adaptive bytecode optimizations. Despite speed improvements, frame evaluation rules for variables remain consistent:

  • step is loaded locally from fast locals (via LOAD_FAST), reading from the pre-computed __defaults__ tuple passed during call time.
  • data is retrieved from closure storage using LOAD_DEREF at execution time, obtaining whatever object reference currently resides in the shared cell.

How to Avoid Scope Leaks and Mutation Bugs

To write clear, predictable, and maintainable code without scope lookup surprises, consider these approaches:

Option A: Pass Data Explicitly as a Default Argument

If you want to snapshot the current state of data at function definition time, bind it explicitly as a default argument by creating a copy:

transformers[f"key_{i}"] = lambda x, step=i, current_data=list(data): [x + step + j for j in current_data]

Option B: Use functools.partial

Using functools.partial provides explicit argument binding, avoiding closure-related ambiguities entirely:

from functools import partial

def helper(x, step, data_snapshot):
    return [x + step + j for j in data_snapshot]

transformers[f"key_{i}"] = partial(helper, step=i, data_snapshot=list(data))

Summary

Dynamic closures combined with mutated state in loop scopes can introduce subtle bugs. Understanding that default parameters bind at definition time while closure variables bind late at call time—alongside knowing how __iadd__ mutates in-place—allows you to structure clean, bug-free functional patterns in Python.