Understanding the Unexpected Perl Regex Failure on Binary Data

When working with binary data in Perl—such as data read from shared memory, packed byte arrays, or binary files—you often need to untaint the data before using it under Perl's security checks (-T flag). A common pattern developers try for unconditional untainting is matching the string length using a regular expression like /^(.{$size})$/.

However, you might run into confusing edge cases where a string of exact length $size fails to match /^(.{$size})$/, even though length($data) == $size evaluates to true. If this happens intermittently depending on the input data, you are likely encountering a classic Perl regex quirk: the dot (.) metacharacter does not match newline characters by default.

The Root Cause: The 0x0A (Newline) Byte

In Perl regular expressions, the dot (.) matches any character except a newline (\n, ASCII/hex 0x0A). When dealing with text, this behavior is usually desired. But when dealing with raw binary data produced by functions like pack('H*', ...), your data will inevitably contain the byte 0x0A whenever that hex byte appears.

Consider the hex sample from the problem statement:

'020002000a0000000200000000000000000000541ab2913f000020cae4b95e4000'

Notice the byte sequence 0a early in the string. When packed into binary format, this byte becomes \x0A (newline). Because the regex /^(.{$size})$/ stops matching upon encountering \n, the pattern fails to match the full length of the binary string.

The Solution: Enable Single-Line Mode (/s Modifier)

To tell Perl's regex engine that the dot (.) metacharacter should match every character including newlines, you must add the /s modifier (single-line mode, or dot-all mode).

Instead of:

my $RE = qr/^(.{$size})$/a;

You should use the /s modifier:

my $RE = qr/^(.{$size})$/sa;

Updated Code Example

Here is the corrected untaint snippet using the /s flag:

#!/usr/bin/perl -T
use strict;
use warnings;
use Carp;

sub untaint(\$$;$) {
    my ($val_ref, $RE, $opt) = @_;

    return $$val_ref = $1 if ($$val_ref =~ $RE);
    croak "untaint: $$val_ref is tainted\n" unless ($#_ > 1);
    if (defined $opt && ref($opt) eq 'CODE') {
        $opt->($val_ref, $RE);
    } else {
        $$val_ref = $opt;
    }
    $$val_ref;
}

my @data = (
    '020002000a0000000200000000000000000000541ab2913f000020cae4b95e4000',
    '1234',
);

foreach (@data) {
    my $size = length($_) / 2;
    my $data = pack('H*', $_);

    # Using /s allows '.' to match newlines (0x0A bytes)
    untaint($data, qr/^(.{$size})$/sa,
            sub ($$) {
                my $val_ref = $_[0];
                carp "failed to untaint $size bytes\n";
            });
    print "Successfully untainted $size bytes!\n";
}

Alternative Solutions for Untainting Binary Strings

If your sole intention is to untaint binary data unconditionally without relying on character semantics, consider these alternative approaches:

  • Use ^([\s\S]{$size})$: Matching character or non-character classes avoids the single-line flag requirement entirely.
  • Use unpack / substr trick: Using substr($data, 0) combined with pattern capture like ^(\C*)$ (in older Perl) or capturing directly via /^([\x00-\xFF]*)$/ explicitly defines byte ranges.

Summary

When working with binary data in Perl regular expressions, always remember that binary payloads can contain 0x0A (newline) bytes. Adding the /s modifier ensures the dot wildcard matches across all byte values, preventing unexplained regex match failures when untainting data.