The Problem: Strange Characters When Zipping Strings

In C++23, the introduction of std::views::zip made it incredibly easy to iterate over multiple ranges simultaneously. However, if you try to zip two instances of std::string_view containing non-ASCII characters (such as Cyrillic, accented characters, or emojis) and print them character-by-character, you will likely encounter highly corrupted output full of replacement characters ().

Consider the following code snippet:

#include <iostream>
#include <string_view>
#include <ranges>

int main() {
    std::string_view str_1 = "ТРОЙНИК Д20 БЕЛЫЙ ";
    std::string_view str_2 = "ТРОЙНИК 16-16-16(ДЛЯ М/П) ";

    // This prints perfectly:
    for(char ch : str_1)
        std::cout << ch;
    std::cout << '\n';

    // This also prints perfectly:
    for(char ch : str_2)
        std::cout << ch;
    std::cout << '\n';

    // This produces garbage output!
    for(const auto [lhs, rhs] : std::views::zip(str_1, str_2)) {
        std::cout << lhs << rhs;
    }
}

Why does iterating over the strings separately work flawlessly, while zipping them together results in a garbled mess?

The Root Cause: UTF-8 and Byte Interleaving

The issue does not lie within a compiler bug or a defect in std::views::zip. Instead, it is a fundamental consequence of how UTF-8 multi-byte encoding interacts with byte-level iteration.

In C++, a standard char represents a single 8-bit byte. While basic English ASCII characters fit into a single byte, Cyrillic characters (like Т, Р, О) require two bytes in UTF-8.

1. Sequential Iteration (Why it works)

When you iterate over str_1 sequentially, you output its raw bytes one by one. Even though the loop processes them individually, they arrive at the terminal in their exact original sequence. The terminal detects the valid UTF-8 byte sequences (e.g., 0xD0 0xD2 for Т) and renders the correct glyph.

2. Zipped Iteration (Why it breaks)

When you use std::views::zip, the view pairs the first byte of str_1 with the first byte of str_2, then the second byte of each, and so on.

Let's look at what happens to the first character Т (which is represented by the bytes 0xD0 0xA2 in UTF-8) in both strings:

  • str_1: Byte 1 = 0xD0, Byte 2 = 0xA2
  • str_2: Byte 1 = 0xD0, Byte 2 = 0xA2

When zipped, the loop executes as follows:

  1. Prints lhs (Byte 1 of str_1: 0xD0)
  2. Prints rhs (Byte 1 of str_2: 0xD0)
  3. Prints lhs (Byte 2 of str_1: 0xA2)
  4. Prints rhs (Byte 2 of str_2: 0xA2)

Instead of sending the coherent sequence 0xD0 0xA2 0xD0 0xA2 (which represents ТТ), the console receives 0xD0 0xD0 0xA2 0xA2. This is an invalid UTF-8 sequence, causing the terminal to render replacement characters ().

How to Fix It

To safely zip and compare multi-byte strings character-by-character, you must iterate over Unicode code points rather than raw bytes.

Solution 1: Use UTF-32 (std::u32string_view)

UTF-32 uses a fixed 32 bits (4 bytes) per character. This guarantees that every element in the view represents a complete, individual Unicode character (code point), making it perfectly safe to zip.

#include <iostream>
#include <string_view>
#include <ranges>
#include <locale>
#include <codecvt>

int main() {
    // Note the 'U' prefix for UTF-32 literals
    std::u32string_view str_1 = U"ТРОЙНИК Д20 БЕЛЫЙ ";
    std::u32string_view str_2 = U"ТРОЙНИК 16-16-16(ДЛЯ М/П) ";

    // Setup locale for UTF-8 output
    std::locale::global(std::locale(""));
    std::wcout.imbue(std::locale());

    // We must convert UTF-32 code points back to UTF-8 or use wide streams to print
    for (const auto [lhs, rhs] : std::views::zip(str_1, str_2)) {
        // Convert char32_t to a printable format or handle comparison directly
        std::cout << "Comparing: " << static_cast<uint32_t>(lhs) 
                  << " with " << static_cast<uint32_t>(rhs) << '\n';
    }
}

Solution 2: Use a Unicode-Aware Library (Recommended)

For production-grade applications, converting raw strings to UTF-32 can be memory-intensive. Instead, use a lightweight Unicode library like UTF8-CPP or Boost.Locale to create code-point iterators over your UTF-8 std::string_view.

Using utf8::iterator ensures that your zip loop advances by whole UTF-8 characters rather than single bytes, preserving the integrity of your text during comparison and manipulation.

Summary

C++ std::string_view and standard string types are byte-oriented, not character-oriented. When using C++23 ranges like std::views::zip on UTF-8 strings, you interleave their raw bytes, corrupting multi-byte characters. To prevent this, always decode your UTF-8 strings into code points (using UTF-32 or a Unicode library) before performing element-wise operations.

Block zip operations.