Why Generators Changed How I Handle Large Datasets

I used to dread processing large CSV files or API responses in PHP. Loading everything into an array meant watching memory usage spike, sometimes crashing scripts on modest servers. Then I started using generators, and it completely changed how I approach data-heavy tasks. They let you process data one piece at a time, keeping memory usage low while maintaining clean, readable code.

Real-World Scenario: Processing User Export Files

At my last job, we needed to validate and transform user data exports before importing them into our CRM. These files could easily reach 50,000+ rows. The old approach loaded the entire file into memory, performed validation, then transformed each record. It worked for small files but failed regularly with larger ones.

With generators, we now process each row as we read it — validate, transform, and send it off — without ever holding more than one row in memory at a time.

How Generators Work in PHP

A generator is a function that uses yield instead of return. When called, it returns a Generator object that can be iterated over. Each time yield is hit, the function pauses, returns a value, and resumes from exactly that point on the next iteration.

This lazy evaluation means values are produced only when needed — perfect for streaming data.

Production-Ready Example: Reading and Validating a CSV


 Yields validated row data with line number
 */
function processUserCsv(string $filePath): Generator
{
    $handle = fopen($filePath, 'r');
    if (!$handle) {
        throw new RuntimeException("Cannot open file: {$filePath}");
    }

    // Skip header if present
    $header = fgetcsv($handle);
    $lineNumber = 2; // Start at line 2 (after header)

    while (($row = fgetcsv($handle)) !== false) {
        // Validate required fields
        if (count($row) < 3 || empty($row[0]) || empty($row[1])) {
            error_log("Skipping invalid row at line {$lineNumber}: missing required fields");
            $lineNumber++;
            continue;
        }

        // Transform and yield clean data
        yield [
            'email'    => strtolower(trim($row[0])),
            'name'     => trim($row[1]),
            'age'      => (int)$row[2],
            '_line'    => $lineNumber
        ];

        $lineNumber++;
    }

    fclose($handle);
}

// Usage: Process without loading entire file into memory
try {
    foreach (processUserCsv('uploads/users_export.csv') as $user) {
        // Send to API, save to DB, etc. — one user at a time
        if (!isValidEmail($user['email'])) {
            error_log("Invalid email: {$user['email']} (line {$user['_line']})")
            continue;
        }

        saveUserToCrm($user); // Imagine this sends data via API
    }
} catch (Exception $e) {
    error_log("Processing failed: " . $e->getMessage());
    // Handle gracefully — maybe alert admin
}
?>

Why This Approach Beats Alternatives

  • Memory efficiency: Whether the file has 10 rows or 100,000, memory usage stays nearly constant.
  • Clean separation of concerns: The generator handles reading and validation; the calling code focuses on what to do with each valid record.
  • Early error handling: You can catch issues early and log context (like line numbers) without breaking the whole process.
  • Composable: You can chain generators — for example, one that reads CSV, another that validates, another that transforms.

Generators aren’t about speed — they’re about scalability. If your script runs fine on small data but fails on production-sized inputs, generators are often the fix.

When Not to Use Generators

They’re not a silver bullet. If you need to sort, randomly access, or repeatedly iterate over the data, generators add complexity without benefit. In those cases, arrays are still appropriate — just be mindful of size.

But for streaming data — logs, files, API paginated responses — generators are one of PHP’s most underused tools for writing robust, scalable code.

Final Thoughts

I keep a snippet like this in my toolkit because it solves a problem I hit all the time: processing data that’s too big to fit comfortably in memory. It’s helped me avoid midnight server alerts and made our data pipelines far more reliable. If you’re working with large datasets in PHP and haven’t tried generators yet, give them a shot. Start small — maybe just replace one fgetcsv loop — and see how it changes your approach.