Why Python's asyncio.TaskGroup Hangs on Cancelled Tasks (And How to Fix It)
Introduction to Python's asyncio.TaskGroup
Python 3.11 introduced asyncio.TaskGroup, bringing structured concurrency to the standard library. With TaskGroup, if any subtask fails with an unhandled exception, the group automatically cancels all other running tasks and raises an ExceptionGroup. This clean fail-fast behavior simplifies managing concurrent asynchronous tasks like LLM API calls, batch web requests, or database operations.
However, developers often encounter a confusing issue: instead of failing fast and raising an exception group immediately, the entire TaskGroup hangs until tasks finish their retry delays or complete unexpectedly. In this article, we'll explore why this happens and how to write safe retry logic that plays nicely with structured concurrency.
Why Does TaskGroup Hang?
To understand why asyncio.TaskGroup hangs, we need to look at how structured concurrency works under the hood. When a task inside a TaskGroup encounters an exception or triggers cancellation, two main rules apply:
- TaskGroup waits for all child tasks to terminate: Before leaving the
async with TaskGroup()block and raising anExceptionGroup, the task group must wait for every single managed task to reach a finished state (either completed, failed, or cancelled). - Cancellation is requested, not forced: When a task fails,
TaskGroupsends anasyncio.CancelledErrorto all other tasks. However, Python cannot forcefully terminate anasynctask mid-execution. A task must yield control (usually at anawaitpoint) to receive and process the cancellation request.
Deconstructing the Problematic Code
Consider what happens in a retry loop like the one below when an error occurs:
async def call_llm(prompt: str):
try:
await asyncio.sleep(0.1)
if prompt == "bad-prompt":
raise ValueError(f"API rejected: {prompt}")
return f"response to {prompt}"
except Exception as e:
print(f"retrying {prompt} after {e}")
await asyncio.sleep(5) # Backoff before retry
return f"response to {prompt}"
Two issues happen depending on where the error occurs:
- Swallowing the Original Error: The
bad-prompttask catches its ownValueErrorin theexcept Exceptionblock. Because it catches the error internally,TaskGroupnever learns thatbad-promptfailed! Instead,bad-promptsleeps for 5 seconds and returns a dummy string, hiding the failure completely. - Swallowing or Delaying Cancellation: If a task does raise an unhandled exception to the
TaskGroup, the group cancels sibling tasks by throwingasyncio.CancelledErrorinto them. Whileasyncio.CancelledErrorinherits fromBaseException(since Python 3.8) to preventexcept Exceptionfrom swallowing it, anyexcept BaseExceptionor async operations inside teardown/cleanup blocks can delay task termination, causingTaskGroupto wait.
How to Write Safe Retry Workers for TaskGroup
To ensure your concurrent workers allow TaskGroup to cancel cleanly and fail fast, follow these best practices.
1. Catch Only Specific Exceptions
Never catch generic Exception or BaseException in retry handlers. Only catch expected, transient operational errors (e.g., HTTP status errors, connection timeouts).
import asyncio
import httpx
async def call_llm_safe(prompt: str):
# Catch specific network/API exceptions only
try:
# Imagine this calls an actual LLM client
if prompt == "bad-prompt":
raise httpx.HTTPStatusError("400 Bad Request", request=None, response=None)
return f"response to {prompt}"
except httpx.HTTPStatusError as e:
# Handle or re-raise if non-retryable
raise e
2. Always Propagate CancelledError Explicitly
If you must use broad exception handling for unexpected errors, make sure asyncio.CancelledError is explicitly caught and re-raised immediately:
async def worker(prompt: str):
try:
return await perform_request(prompt)
except asyncio.CancelledError:
# Always allow cancellation to propagate immediately!
raise
except Exception as e:
# Handle transient application errors here
print(f"Error processing {prompt}: {e}")
raise
3. Use an Async-Aware Retry Library Like Tenacity
Instead of manually writing try/except retry loops with asyncio.sleep, use a robust library like tenacity, which handles asyncio.CancelledError correctly by default:
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type(httpx.RequestError) # Don't retry on CancelledError or unexpected errors
)
async def call_llm_tenacity(prompt: str):
# Perform API call here
pass
Summary
When asyncio.TaskGroup appears to hang, it is usually waiting for child tasks that have caught cancellation, caught their own internal exceptions, or engaged in lengthy await delays inside exception handlers. By catching specific exception types, allowing asyncio.CancelledError to propagate without delay, and using robust retry policies, you can build responsive, resilient asynchronous applications in Python.