Why Generators Changed How I Handle Large Files in PHP

Early in my career, I’d reach for file() or fread() to process logs, CSVs, or exports without thinking much about memory. It worked fine — until it didn’t. One day, a 2GB log file brought our staging server to its knees because we loaded it all into memory at once. That’s when I started using PHP generators, and honestly, they’ve become one of my go-to patterns for anything involving sequential data.

Generators let you iterate over data without loading everything into memory upfront. Instead of returning an array, a generator function yields values one at a time, pausing execution until the next value is requested. This is perfect for reading files line by line, processing database cursors, or handling API pagination.

Real-World Scenario: Processing a Massive CSV Upload

Imagine you’re building a feature where users upload product inventories — CSVs with hundreds of thousands of rows. Each row needs validation, transformation, and saving to the database. If you load the entire file into an array, you risk hitting memory limits (memory_limit in php.ini), especially on shared hosting or containerized environments with tight constraints.

Using a generator, you can process each row as it’s read, keep memory usage flat, and still maintain clean, readable code.

The Code: A Reusable CSV Generator Function

Using the Generator in Practice


 $row) {
        // Skip header row if needed
        if ($rowIndex === 0) {
            continue;
        }
        
        // Validate and process each field
        [$sku, $name, $price, $stock] = $row;
        
        if (empty($sku) || !is_numeric($price) || $price < 0) {
            logError("Invalid data at line {$rowIndex + 1}: SKU={$sku}, Price={$price}");
            continue;
        }
        
        // Save to database (using prepared statements, of course)
        $stmt = $pdo->prepare('INSERT INTO products (sku, name, price, stock) VALUES (?, ?, ?, ?)');
        $stmt->execute([$sku, $name, (float)$price, (int)$stock]);
    }
    
    echo "Processing complete.";
} catch (Exception $e) {
    // Handle file access or unexpected errors
    error_log("CSV processing failed: " . $e->getMessage());
    throw new RuntimeException("Failed to process upload", 0, $e);
}

Pro tip: Wrap your generator in a try/catch at the call site. That way, you can distinguish between I/O errors (like missing files) and data validation issues.

Why This Approach Works So Well

  • Memory efficiency: Whether the file is 10 KB or 10 GB, memory usage stays low because only one line is in memory at a time.
  • Clean separation of concerns: The generator handles I/O; the calling code handles business logic. This makes both easier to test and maintain.
  • Lazy evaluation: Processing stops as soon as the loop breaks — useful if you’re looking for a match and can exit early.
  • Composable: You can pipe generators together. For example, filter rows, then map them to objects, all without intermediate arrays.

When Not to Use Generators

Generators aren’t a silver bullet. If you need random access, sorting, or multiple passes over the data, you’ll still need to materialize it into an array. But for sequential processing — which is surprisingly common in web apps — generators are often the better choice.

I’ve used this pattern for processing Apache logs, validating JSONL exports, and even reading large configuration files incrementally. Once you start thinking in streams instead of bulks, you’ll find more places where generators simplify your code and protect your servers.