Why Lazy Enumeration Matters

When you start pulling millions of rows from a database, reading a massive log file, or consuming a paginated API, loading everything into an array is a quick way to run out of memory. Ruby’s Enumerable#lazy gives you a way to build a pipeline that evaluates one element at a time, keeping the memory footprint tiny while still letting you chain map, select, reject, and friends.

A Real‑World Scenario: Streaming JSON Logs

Imagine a service that writes JSON‑lines logs to a rotating file. Each line is a complete JSON object, but the file can grow to several gigabytes. You need to extract only the error level entries, parse the message field, and feed them into an alerting system. Loading the whole file into an array would crush the process; a lazy pipeline processes line‑by‑line and never holds more than a few kilobytes.

Building the Pipeline

# log_processor.rb
require 'json'

# Open the file in read mode; each line is a JSON object.
File.open('application.log', 'r') do |file|
  # Create a lazy enumerator over each line.
  file.each_line.lazy
    # Parse JSON only when needed.
    .map { |line| JSON.parse(line) }
    # Keep only error‑level entries.
    .select { |entry| entry['level'] == 'error' }
    # Transform to a simpler hash for the alerting service.
    .map { |entry| { timestamp: entry['ts'], message: entry['msg'] } }
    # Take the first 100 errors (or drop .first to process all).
    .first(100)
    # Finally, send each to the alerting endpoint.
    .each { |alert| AlertingService.notify(alert) }
end

Notice the .first(100) call. Because the chain is lazy, Ruby stops reading the file as soon as it has collected 100 matching entries. If you omit it, the pipeline will stream through the entire file without ever materialising a giant array.

Under the Hood: How lazy Works

Enumerable#lazy returns an instance of Enumerator::Lazy. Each method you call on it (map, select, etc.) returns another lazy enumerator that remembers the transformation but does not execute it. Only when you invoke a terminal method — each, to_a, first, force — does Ruby pull elements one by one through the whole chain.

Key point: Lazy enumerators are composable. You can build a library of reusable transformation steps and combine them at call‑site without worrying about intermediate allocations.

When Not to Use Lazy

  • Small collections – the overhead of lazy enumerators outweighs the benefit.
  • Multiple passes – if you need to iterate over the same data several times, materialising once (e.g., to_a) is faster.
  • Side‑effects in the chain – lazy evaluation delays side‑effects, which can be surprising if you rely on ordering.

Performance Tips

  1. Batch I/O: When reading from a socket or HTTP stream, wrap the source in IO.each_line or Net::HTTP.get_stream and call lazy on that enumerator.
  2. Avoid to_a early: Only call force or to_a at the very end of the pipeline.
  3. Use chunk_while or slice_before for grouping without loading groups into memory.

Extending the Pattern: Custom Lazy Enumerables

You can expose a lazy API from your own classes by returning Enumerator::Lazy.new(self) { |yielder, *values| ... }. This lets callers treat your object like any other enumerable while you control the pull‑based iteration.

class PaginatedApi
  include Enumerable

  def initialize(client, path)
    @client = client
    @path   = path
  end

  def each(&block)
    return enum_for(:each) unless block_given?
    page = 1
    loop do
      response = @client.get(@path, query: { page: page })
      break if response.body.empty?
      response.body.each(&block)
      page += 1
    end
  end

  # Enable lazy out of the box
  def lazy
    super
  end
end

# Usage
api = PaginatedApi.new(HttpClient.new, '/events')
api.lazy.select { |e| e['type'] == 'purchase' }.first(10).each { |e| puts e }

By delegating to super in lazy, the class automatically gains all lazy combinators without extra code.

Wrapping Up

Lazy enumeration is one of those Ruby features that feels like a secret weapon once you’ve been bitten by an OutOfMemoryError. It lets you write expressive, functional‑style code while staying friendly to the GC. Next time you face a fire‑hose of data — logs, CSVs, paginated APIs — reach for .lazy first; your future self (and your production servers) will thank you.