When I Needed to Process a Huge CSV Without Killing Memory

When I first needed to parse a multi‑gigabyte CSV export from a legacy system, the naïve approach of reading the whole file into an array quickly exhausted the server’s RAM. I realized I needed a way to work with the data piece by piece, keeping only a small slice in memory at any moment. Ruby’s Enumerator::Lazy turned out to be the perfect tool for that job.

Why Lazy Enumeration Matters

Ruby’s enumerable methods (map, select, reject, …) are eager: they traverse the entire collection and return a new array before the next step can start. With large inputs this creates intermediate arrays that balloon memory usage. A lazy enumerator, on the other hand, defers work until a value is actually requested. Each element flows through the chain one at a time, so memory stays constant regardless of input size.

Think of a lazy chain as a pipeline: data enters at the source, each stage processes a single item and passes it forward, and the consumer pulls items only when needed. This mirrors how Unix utilities like grep or awk operate on streams.

Real‑World Scenario: Generating a Summary Report

Imagine a nightly job that reads a CSV of user activity logs, filters out bots, aggregates actions per hour, and writes a compact JSON summary. The source file can be 5 GB, but the summary is only a few kilobytes. Loading the file outright would be wasteful and risky.

Below is a production‑ready snippet that streams the CSV, applies filters, performs a simple aggregation, and outputs the result. Comments explain each step.


require 'csv'

# Path to the massive CSV file
log_file = 'user_activity_2024-09-24.csv'

# We'll store hourly counts in a hash: { hour_string => count }
hourly_counts = Hash.new(0)

# Open the file and create a lazy CSV parser
lazy_csv = CSV.foreach(log_file, headers: true).lazy

# Process each record without loading the whole file
lazy_csv.each do |row|
  # Skip rows where the user_agent indicates a known bot
  next if row['user_agent'].to_s.match?(/(bot|crawler|spider)/i)

  # Parse the timestamp; assume it's in ISO 8601 format
  timestamp = Time.parse(row['timestamp'] rescue nil)
  next unless timestamp   # Skip malformed lines

  # Build a key like "2024-09-24 14" for the hour
  hour_key = timestamp.strftime('%Y-%m-%d %H')

  # Increment the counter for this hour
  hourly_counts[hour_key] += 1
end

# At this point the enumeration is finished; we have the aggregated counts
# Convert to a sorted array for deterministic output
summary = hourly_counts.sort.map { |hour, count| { hour: hour, events: count } }

# Write the summary as JSON (pretty‑printed for readability)
require 'json'
File.write('activity_summary.json', JSON.pretty_generate(summary))
puts 'Summary written to activity_summary.json'

What Makes This Approach Robust

  • Constant memory footprint: The lazy enumerator yields one CSV row at a time; the hash grows only with the number of distinct hours (usually 24).
  • Early filtering: Bot rows are discarded immediately, saving work downstream.
  • Clear separation of concerns: Each step (filter, parse, aggregate) is a simple, readable block.
  • Easy to extend: Adding another transformation (e.g., weighting events) is as easy as inserting another .map or .select in the chain.

Potential Pitfalls and How to Avoid Them

While lazy enumerators are powerful, they are not a silver bullet.

  • Side effects inside the chain: If you rely on mutating external state inside a lazy block, ensure the mutation is idempotent or that you understand the order of execution. In the example we only update a hash, which is safe because each row is processed exactly once.
  • Early termination: Methods like find or take on a lazy enumerator will stop the chain as soon as the condition is met, which can be useful but also surprising if you expect the whole input to be consumed.
  • I/O bottlenecks: The lazy parser still reads from disk; if the disk is slow, consider using buffered reads or memory‑mapped files. However, the memory advantage remains.

When to Reach for This Technique

Use Enumerator::Lazy whenever you:

  1. Read a potentially unbounded stream (files, network sockets, command output).
  2. Need to apply a series of transformations that each produce a collection of the same size as the input.
  3. Want to keep memory usage low and avoid building large intermediate arrays.

In daily Rails work I often replace Model.all.map { … } with Model.find_each.lazy.map { … } when processing thousands of records, and the same principle applies to any enumerable source.

Final Thoughts

The elegance of lazy enumeration lies in its simplicity: you write code that looks exactly like an eager pipeline, yet it behaves like a streaming process. By embracing laziness you gain scalability without sacrificing readability—a trade‑off that pays off every time you handle data that exceeds available memory.

If you find yourself reaching for each_with_object or reduce on a massive array, pause and ask whether a lazy chain could do the same job with far less RAM.