Introduction

When I first hit a production job that streamed millions of log lines into a CSV, the naive each loop blew the memory ceiling and the worker got OOM‑killed. The fix turned out to be a single method call: lazy. In this article I’ll walk through why Enumerable#lazy matters, how to wield it safely, and a concrete pattern I use every week.

The Problem: Eager Evaluation Eats RAM

Ruby’s enumerable methods (map, select, flat_map, …) are eager. They build an intermediate array at each step. For a 200 MB log file that translates into several copies of the data in memory — enough to crash a modest container.

Rule of thumb: if you chain more than two transformations on a large collection, reach for lazy before you hit the first to_a.

Introducing Enumerable#lazy

Calling lazy on any enumerable returns a Enumerator::Lazy instance. Subsequent calls to map, select, reject, flat_map, take, drop, and friends become lazy — they yield one element at a time, passing it down the pipeline without materializing intermediate arrays.

# Eager – builds three arrays
lines.map(&:strip).select { |l| l.start_with?('ERROR') }.map(&:upcase)

# Lazy – one element flows through the whole chain
lines.lazy.map(&:strip).select { |l| l.start_with?('ERROR') }.map(&:upcase).force

The only eager operation is the terminal call — force, to_a, each, or any method that consumes the enumerator.

Real‑World Example: Streaming a 5 GB CSV into a Database

Imagine a nightly job that ingests a massive CSV, filters rows, normalizes timestamps, and bulk‑inserts into Postgres. Below is the pattern I ship to production.

require 'csv'
require 'pg'

def import_huge_csv(path, conn)
  # Open the file once, iterate lazily
  CSV.foreach(path, headers: true, liberal_parsing: true).lazy
    # 1️⃣ Keep only rows we care about
    .select { |row| row['status'] == 'completed' }
    # 2️⃣ Normalize timestamp without allocating a new Hash for every column
    .map do |row|
      row['created_at'] = Time.iso8601(row['created_at']).utc.iso8601
      row
    end
    # 3️⃣ Batch 1 000 rows at a time to keep transaction size sane
    .each_slice(1000)
    .each do |batch|
      conn.transaction do
        batch.each do |row|
          conn.exec_params(
            'INSERT INTO events (id, created_at, payload) VALUES ($1,$2,$3)',
            [row['id'], row['created_at'], row.to_json]
          )
        end
      end
    end
end

Key points:

  • CSV.foreach already streams line‑by‑line; adding .lazy makes the subsequent select/map lazy as well.
  • each_slice is eager on the slice size but still processes one slice at a time, so memory stays bounded.
  • The transaction wraps each batch, giving us atomicity without a single gigantic transaction.

Pitfalls & Tips

  1. Don’t forget a terminal call. A lazy chain that never gets consumed does nothing — great for debugging, disastrous in production.
  2. Watch out for count, size, or first on a lazy enumerator. They force evaluation of the entire pipeline up to that point.
  3. Combine with Enumerator::Producer for custom sources. If you pull from an API with pagination, wrap the pagination logic in a producer and then .lazy the rest.
  4. Profile before you lazy‑ify everything. For small collections the overhead of the lazy wrapper outweighs the benefit.

Wrapping Up

Lazy enumerators aren’t magic, but they’re a lightweight lever that turns a memory‑hungry pipeline into a streaming one with a single method call. I keep a tiny helper in my toolbox:

module LazyHelper
  def self.stream(enum, &block)
    enum.lazy.instance_eval(&block).force
  end
end

Now any LazyHelper.stream(csv) { select { … }.map { … }.each_slice(500) } reads like a declarative data‑flow description while staying O(1) in memory. Give it a spin on your next big‑data task — you’ll wonder how you ever lived without it.