Why Generators Changed How I Process Data in PHP

I used to dread tasks involving large CSV files or database exports. Loading everything into memory with file() or fetching all rows at once often led to fatal errors: Allowed memory size exhausted. It wasn’t just frustrating — it was a blocker in production pipelines. Then I started using PHP generators, and suddenly, processing gigabytes of data became manageable, even on modest servers.

Generators let you iterate over data without loading it all into memory. Instead of returning an array, a generator function yields values one at a time, preserving state between iterations. This is perfect for reading logs, parsing feeds, or exporting reports where you don’t need random access — just sequential processing.

Real-World Scenario: Nightly Sales Report Generation

At my last job, we generated daily sales reports from a MySQL table with over 10 million rows. The old approach used SELECT * and fetchAll(), which consumed 2+ GB of RAM and regularly crashed the PHP worker. We needed a way to stream results, calculate aggregates, and write output incrementally.

Here’s how we refactored it using a generator:

prepare(
        'SELECT product_id, quantity, price FROM sales WHERE sale_date = :date'
    );
    $stmt->execute([':date' => $date]);

    // Fetch rows one at a time to keep memory usage low
    while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
        yield $row; // Pause here, resume on next iteration
    }
}

// Usage: Process each sale without loading all data
$pdo = new PDO('mysql:host=localhost;dbname=reports', 'user', 'pass');
$totalRevenue = 0.0;
$totalItems = 0;

foreach (getSalesGenerator($pdo, '2024-05-20') as $sale) {
    $totalRevenue += $sale['quantity'] * $sale['price'];
    $totalItems += $sale['quantity'];
}

echo "Report: {$totalItems} items sold, \$${number_format($totalRevenue, 2)} revenue\n";
?>



Why This Beats Alternatives


    

  • Memory efficiency: Only one row lives in memory at a time, regardless of dataset size.
    • 

  • Clean separation: The data-fetching logic is isolated, making the processing loop easy to read and test.
    • 

  • Composable: You can pipe generators — filter, transform, or limit results using other generator functions.
    • 

  • No magic: It’s built into PHP since 5.5, requires no extensions, and works with any iterable source.
    • 





I’ve found generators especially valuable in CLI scripts, queue workers, and data migration tools — anywhere you’re processing streams of information.




    

        
        
        
    

    
When Not to Use Them


Generators aren’t a free lunch. Avoid them when you need:

    

  • Random access to data (e.g., sorting or jumping to page 100)
    • 

  • Multiple passes over the same data (unless you rewind or recreate the generator)
    • 

  • Array-specific functions like array_map() or usort() without converting to an array first (which defeats the purpose)
    • 



In those cases, reconsider whether you truly need to process everything at once — or if a database-level solution (like aggregation or cursors) might be better.


Final Thoughts


Generators taught me to think differently about data flow in PHP. Instead of asking, "How do I get all the data?", I now ask, "What do I need to do with each piece?" That shift has made my scripts more reliable, scalable, and easier to maintain. If you’re working with large datasets and haven’t tried generators yet, start small — replace a while ($row = $stmt->fetch()) loop with a yield-based function. You’ll likely wonder how you ever worked without them.