The Problem

In many Ruby applications we eventually need to read a file that is larger than the available RAM — think multi‑gigabyte logs, CSV exports from legacy systems, or streaming data feeds. The naïve approach of reading the whole file into an array or string works fine for small inputs, but it quickly leads to out‑of‑memory errors and degraded performance.

Why Enumerator::Lazy Helps

Ruby’s Enumerable module already gives us powerful iteration tools (map, select, reduce, etc.). Those methods are eager: they process the entire collection and return a new array before the next step can start. When the collection is a file stream, that means we first load every line into memory.

Enumerator::Lazy changes the game by making each step return a lazy enumerator that only computes values as they are needed. The pipeline stays pull‑based: each element travels through the chain one at a time, keeping memory usage constant regardless of input size.

Real‑World Scenario: Extracting Error Lines from a Massive Log

Imagine a nightly job that scans a web‑server access log (~5 GB) looking for lines that contain the word "ERROR" and then writes those lines to a separate file for the ops team. The log is line‑oriented, so we can process it sequentially.


require 'enumerator'   # Enumerator::Lazy lives here, but it's loaded automatically in Ruby 2.0+

log_path   = '/var/log/app/access.log'
error_path = '/var/log/app/errors.log'

File.open(error_path, 'w') do |out_file|
  File.foreach(log_path).lazy do |line|
    # Keep only lines that contain the word ERROR (case‑insensitive)
    line if line.downcase.include?('error')
  end.each do |error_line|
    out_file.puts(error_line)
  end
end

puts "Finished extracting error lines."

Why we use File.foreach instead of File.readlines: foreach returns an enumerator that yields one line at a time without loading the whole file into memory. Pairing it with lazy ensures that the subsequent select‑like filter also works lazily.

How the Lazy Chain Works

  1. File.foreach(log_path) creates an enumerator over the file’s lines.
  2. Calling .lazy wraps that enumerator in a Enumerator::Lazy object.
  3. The block passed to lazy acts like select: for each line we either yield the line (when it matches) or yield nothing (by returning nil). In Ruby, returning a falsy value from the block inside lazy effectively filters it out.
  4. Finally, .each drives the pipeline, pulling matching lines one by one and writing them to the output file.

Because each step only holds a single line in memory at any moment, the memory footprint stays low — typically a few kilobytes — regardless of the log’s size.

Alternatives and When Not to Use Lazy

Lazy evaluation shines when:

  • The data source is large or infinite (e.g., reading from a socket, generating a sequence).
  • You chain multiple transformations (map, filter, take, etc.) and want to avoid intermediate arrays.
  • You can stop early (take, detect) and don’t need to process the whole collection.

If the dataset is tiny and you need random access or multiple passes, an eager array may be simpler and faster because there’s no overhead of creating lazy objects.

Production‑Ready Tips

  1. Handle errors gracefully. Wrap the file‑opening code in a begin..rescue block to catch Errno::ENOENT or permission issues, and log them appropriately.
  2. Respect encoding. If your log might contain non‑UTF‑8 bytes, open the file with the proper encoding: File.foreach(path, encoding: 'UTF-8').
  3. Close resources. Using File.open with a block (as shown) guarantees the file handle is closed even if an exception occurs.
  4. Consider buffered writes. For very high‑volume outputs, you might wrap the output file in a buffered writer or use StringIO to batch writes.
  5. Test with a small sample. Before running on the full log, verify your logic on a few hundred lines to ensure the filter works as expected.

Wrapping Up

Ruby’s Enumerator::Lazy is a modest‑sized tool that can save you from costly memory spikes and make your code feel more functional. By turning an eager iteration into a pull‑based pipeline, you get the readability of high‑level enumerables with the efficiency of a stream processor. The next time you face a large file or an unbounded data source, give lazy enumerators a try — your future self (and your ops team) will thank you.