Why I Reach for pathlib Almost Every Time

When I first started writing Python scripts that touched the file system, I reached for os.path out of habit. It works, but the code quickly becomes a nesting of function calls that obscures intent. A few years ago I switched to pathlib.Path and haven’t looked back. The object‑oriented API lets me treat paths as first‑class values, chain operations naturally, and write code that reads like a short story rather than a puzzle.

A Real‑World Scenario: Aggregating Daily Log Files

Imagine a service that writes a log file each day into a nested directory structure like logs/2024/09/25/service.log. At the end of the month I need to:

  1. Find every .log file for the current month.
  2. Read each file, extract lines that contain the word ERROR, and write them to a consolidated report.
  3. After processing, compress the original logs into a .tar.gz archive and move the archive to an archive/ folder.

Doing this with raw string manipulation is error‑prone, especially when dealing with different operating systems. pathlib removes most of that friction.

Code Walkthrough

Below is a production‑ready snippet that accomplishes the three steps. I’ve added comments to explain the reasoning behind each block.


from pathlib import Path
import tarfile
import gzip
import shutil

def main() -> None:
    # Base directory where logs are stored
    base_log_dir = Path('logs')
    # Directory where the final report will be placed
    report_dir = Path('reports')
    report_dir.mkdir(parents=True, exist_ok=True)
    
    # Determine the year and month we want to process
    from datetime import datetime
    now = datetime.now()
    year_month = f"{now.year:04d}/{now.month:02d}"
    
    # 1️⃣ Find all .log files for the current month
    # Using rglob lets us walk the tree recursively without extra loops.
    log_files = list(base_log_dir.rglob(f"{year_month}/*.log"))
    if not log_files:
        print(f"No log files found for {year_month}")
        return
    
    # 2️⃣ Build the error report
    report_path = report_dir / f"errors_{now:%Y%m}.txt"
    with report_path.open('w', encoding='utf-8') as report_file:
        for log_path in log_files:
            # Open each log file lazily; we only keep one line in memory at a time.
            with log_path.open('r', encoding='utf-8') as f:
                for line in f:
                    if 'ERROR' in line:
                        # Prefix each line with the source file for traceability.
                        report_file.write(f"{log_path}: {line}")
    
    # 3️⃣ Archive the processed logs
    archive_dir = Path('archive')
    archive_dir.mkdir(parents=True, exist_ok=True)
    archive_name = archive_dir / f"logs_{now:%Y%m}.tar"
    
    # Create an uncompressed tar first; we’ll compress it with gzip afterward.
    with tarfile.open(archive_name, 'w') as tar:
        for log_path in log_files:
            # arcname strips the leading 'logs/' so the archive contains a clean layout.
            tar.add(log_path, arcname=log_path.relative_to(base_log_dir))
    
    # Compress the tar file using gzip (produces .tar.gz)
    gzipped_name = archive_name.with_suffix(archive_name.suffix + '.gz')
    with open(archive_name, 'rb') as f_in:
        with gzip.open(gzipped_name, 'wb') as f_out:
            shutil.copyfileobj(f_in, f_out)
    
    # Remove the intermediate tar file to save space.
    archive_name.unlink()
    
    # Optional: remove the original log files after successful archiving.
    for log_path in log_files:
        log_path.unlink()
    
    print(f"Report written to {report_path}")
    print(f"Logs archived to {gzipped_name}")

if __name__ == '__main__':
    main()

Why This Approach Works Well

The power of pathlib shows up in three places.

  • Readable traversal: base_log_dir.rglob(f"{year_month}/*.log") expresses the intent “find all log files under this year/month subtree” in a single line. No nested os.walk loops, no manual string concatenation.
  • Chainable operations: Each Path object knows how to open itself, compute relatives, and produce new paths. For example, log_path.relative_to(base_log_dir) gives us a clean archive entry without fiddling with os.path.relpath.
  • Cross‑platform safety: Because Path abstracts away the separator, the same script runs on Windows, Linux, and macOS without worrying about backslashes versus forward slashes.

Beyond readability, the object‑oriented design encourages a functional style. I can pass a Path to another function, let it open the file, and return a new Path for the output—all without exposing low‑level OS details.

A Few Practical Tips

  1. Prefer Path.open() over the built‑in open when you already have a Path instance; it guarantees the correct encoding handling and keeps the API uniform.
  2. When you need to create parent directories, call parent.mkdir(parents=True, exist_ok=True) on the target file’s parent. This one‑liner replaces the older os.makedirs pattern.
  3. If you’re dealing with a huge number of files, consider using Path.iterdir() with a generator expression instead of materializing a list with list(rglob(...)). This keeps memory usage low.

Remember: the goal isn’t to replace every os.path call blindly. Use pathlib when you’re constructing, inspecting, or moving files. For low‑level descriptor operations (like os.fsync), the os module is still the right tool.

Wrapping Up

Adopting pathlib changed how I think about file system code. It turned a series of error‑prone string manipulations into a set of clear, composable steps. The next time you reach for os.path.join or os.walk, pause and ask whether a Path object could make the intention clearer. In my experience, the answer is almost always yes.