When working with high-performance Unicode processing libraries like simdutf, you might notice an interesting asymmetry in function signatures. While UTF-16 features explicit endianness variants such as validate_utf16le() and validate_utf16be(), UTF-32 only provides a single validation function: validate_utf32().

If you are handling cross-platform data or streaming UTF-32 strings across different CPU architectures, this raises an important question: How does validate_utf32() handle endianness? Is it platform-dependent, auto-detected, or hardcoded to Little Endian?

The Short Answer

In simdutf, validate_utf32() assumes input data is in the native host endianness of the system executing the code. Because almost all modern desktop, server, and mobile processors (x86_64, ARM64) operate in Little Endian mode, UTF-32 strings are effectively treated as UTF-32LE by default on these systems.

Clearing Up a Common Misconception: Integer Literals in C++

When inspecting the implementation of simdutf scalar fallback code, you might see checks like this:

simdutf_warn_unused simdutf_constexpr23 bool validate(InputPtr data, size_t len) noexcept {
  uint64_t pos = 0;
  for (; pos < len; pos++) {
    uint32_t word = data[pos];
    if (word > 0x10FFFF || (word >= 0xD800 && word <= 0xDFFF)) {
      return false;
    }
  }
  return true;
}

It is easy to misinterpret hexadecimal values like 0x10FFFF as Big Endian representations. However, in C and C++, hexadecimal literals represent abstract numerical values, not physical byte sequences in memory.

When the CPU executes uint32_t word = data[pos], it reads 4 bytes from memory into a CPU register using the host architecture's byte order. Numerical comparisons like word > 0x10FFFF evaluate the abstract value stored in that register regardless of how bytes were laid out in RAM.

How simdutf Processes UTF-32 Strings

  • No Auto-Detection: simdutf does not automatically scan for Byte Order Marks (BOM) to infer endianness during validation.
  • Native Endian Expectation: validate_utf32() expects raw char32_t or uint32_t code units to already match the CPU's native byte order.
  • Data Interchange Context: UTF-32 is rarely used for network transmission or disk storage (where UTF-8 dominates). It is primarily used as an in-memory representation, where strings natively match host architecture.

What If You Have Cross-Endian UTF-32 Data?

If you receive UTF-32BE data on a Little Endian machine (or UTF-32LE on a Big Endian system), passing it directly to validate_utf32() will produce incorrect results because byte values will be inverted when loaded into registers.

To safely validate cross-endian UTF-32 data, you must byte-swap the 32-bit words into host byte order before validating:

#include <simdutf.h>
#include <vector>
#include <bit> // C++20 for std::byteswap

// Example: Validating Big Endian UTF-32 on a Little Endian host
bool validate_utf32be_on_le_host(const uint32_t* utf32be_data, size_t size) {
    std::vector<uint32_t> host_endian_data(size);
    
    for (size_t i = 0; i < size; ++i) {
        // Convert from BE to native host order
        host_endian_data[i] = std::byteswap(utf32be_data[i]);
    }

    return simdutf::validate_utf32(reinterpret_cast<const char32_t*>(host_endian_data.data()), size);
}

Summary

The absence of validate_utf32le() and validate_utf32be() functions in simdutf is intentional. Because UTF-32 is used almost exclusively for internal in-memory character manipulation rather than serialization, validate_utf32() targets the platform's native endianness directly for maximum performance.