The Problem: Removing Duplicate Consecutive Lines Except the Last One

When processing log files or structured text data, you often run into scenarios where a specific pattern (like a timestamp or a heartbeat message) repeats multiple times consecutively. Frequently, you only care about the final occurrence of that pattern before the log moves on to other important information.

For example, given a file like this:

important information 1
important information 2
2026-07-10T11:12:31.013620-04:00     <-- delete
2026-07-10T11:13:34.058824-04:00     <-- delete
2026-07-10T11:14:37.116946-04:00     <-- delete
2026-07-10T11:15:09.647148-04:00     <-- keep
important information 3

You want to filter out the intermediate timestamps, preserving only the last timestamp in each block of consecutive timestamps.

If you tried using a reverse-pipeline with tac and awk like this:

tac file.txt | awk '/2026-/ {if (f) next; f=1}1' | tac

You probably noticed that it deleted all timestamps in the file except for the very last one at the bottom. This happens because the flag f is set once and never reset, causing the script to skip timestamps across the entire file rather than block-by-block. Let's look at the best ways to solve this problem.

Solution 1: The Efficient, Single-Pass awk Solution (Recommended)

The cleanest and most efficient way to solve this is to process the file forward in a single pass without reversing it using tac. We can achieve this by buffering the matching lines in awk.

awk '/2026-/ { last=$0; next } { if (last != "") { print last; last="" } print } END { if (last != "") print last }' file.txt

How It Works:

  • /2026-/ { last=$0; next }: If the line matches the pattern, we store it in the variable last and skip to the next line. This continuously overwrites last, meaning we only ever hold onto the most recent matching line.
  • { if (last != "") { print last; last="" } print }: If the current line does not match the pattern, we check if we have a buffered line in last. If we do, we print it (since it was the last of the consecutive run) and reset the buffer. Then, we print the current non-matching line.
  • END { if (last != "") print last }: If the file ends with a matching pattern, this block ensures the final buffered line is printed.

Solution 2: Fixing the tac + awk Pipeline

If you prefer using the reverse-processing method with tac, you can easily fix your original command. The key is to reset the flag f back to 0 whenever a non-matching line is encountered.

tac file.txt | awk '!/2026-/ {f=0} /2026-/ {if (f) next; f=1} 1' | tac

How It Works:

  • !/2026-/ {f=0}: When awk reads a line that does not contain the pattern (going backwards), we reset the flag f to 0. This marks the end of a consecutive block.
  • /2026-/ {if (f) next; f=1}: When we hit a pattern, if f is already 1, we skip it. Otherwise, we set f=1 and let it print. Because we are reading backwards, this guarantees that only the first timestamp we encounter in each block (which is the last one chronologically) is kept.
  • 1: A shorthand in awk to print the current line.

Conclusion

While both methods yield the exact same output, Solution 1 (Single-Pass Awk) is generally preferred because it avoids reading the file multiple times and doesn't require the overhead of reversing the file twice. It is faster, uses less memory, and is highly portable across different UNIX environments.