Understanding asyncio.shield() and the Orphaned Task Problem

When building asynchronous web APIs (like FastAPI or Sanic) that make expensive calls to Large Language Models (LLMs), handling client disconnects gracefully is critical. If a user cancels an HTTP request halfway through a 5-second generation, you often want the background process to finish anyway so you can cache the response or log analytics rather than wasting work.

Python's asyncio.shield() is commonly recommended for this pattern. However, while asyncio.shield() prevents the inner task from being cancelled when the caller task receives a cancellation request, it introduces a subtle problem: the inner task becomes unobserved once the caller exits.

Why asyncio.shield() Isn't Enough on Its Own

When a client disconnects, asyncio.shield() raises a CancelledError inside your request handler. The inner task continues running in the background, but because the request handler frame is popped off the stack, nothing is left awaiting the inner task.

This causes two primary issues:

  • Unretrieved Exception Warnings: If the shielded task fails (e.g., the LLM API throws a 500 Internal Server Error), no caller is around to catch the exception. Python will log a Task exception was never retrieved message late in execution.
  • Garbage Collection (GC) Risk: In modern Python (3.11+), tasks created with asyncio.create_task() that lack a strong reference outside the cancelled scope can be garbage collected before completion.

The Correct Pattern: Self-Contained Execution & Task Tracking

To safely let an LLM call survive cancellation while ensuring its output is logged or cached and its errors are handled, you need to apply two key design rules:

  1. Encapsulate side-effects: Move caching, completion logging, and exception handling inside the background task itself (or inside a dedicated worker/wrapper) instead of relying on the web handler to process the return value.
  2. Maintain strong references: Keep background tasks inside a global context or set with an add_done_callback to prevent premature garbage collection and swallow/log background exceptions properly.

Complete Working Solution

import asyncio
import logging

# Global set to retain strong references to background tasks
background_tasks = set()

async def _execute_and_cache_llm(prompt: str) -> str:
    """Encapsulates the LLM call along with error handling and caching."""
    try:
        print(f"Calling LLM for prompt: '{prompt}'")
        await asyncio.sleep(2)  # Stand-in for actual LLM API call
        result = f"Generated response for: {prompt}"
        
        # Caching happens inside the task itself, independent of caller state
        print(f"LLM call finished. Caching result for: '{prompt}'")
        return result
    except Exception as exc:
        # Ensure failures are logged/handled even if the HTTP caller disconnected
        logging.error(f"LLM call failed for prompt '{prompt}': {exc}")
        raise

async def handle_request(prompt: str) -> str:
    # Create the background task explicitly
    task = asyncio.create_task(_execute_and_cache_llm(prompt))
    
    # Preserve a strong reference to prevent Garbage Collection
    background_tasks.add(task)
    task.add_done_callback(background_tasks.discard)
    
    try:
        # Shield the task from outer cancellation
        return await asyncio.shield(task)
    except asyncio.CancelledError:
        print("Client disconnected! Request handler cancelled, but LLM call continues.")
        raise

Key Takeaways

  • Decouple Result Processing: Don't rely on the function awaiting asyncio.shield() to perform cleanup or caching. Handle those operations inside the shielded task.
  • Prevent GC Drops: Always add long-running background tasks to a set via background_tasks.add(task) and clean them up using task.add_done_callback(background_tasks.discard).
  • Clean Exception Management: Wrapping the shielded logic in a try...except block ensures you never encounter unretrieved exception warnings if the background service experiences downtime.