This switches the backing format of PyStr from Box<str> to Box<Wtf8>. From the motivation I wrote in a doc comment:

An implementation of WTF-8, a utf8-compatible encoding that allows for unpaired surrogate codepoints. This implementation additionally allows for paired surrogates that are nonetheless treated as two separate codepoints.

RustPython uses this because CPython internally uses a variant of UCS-1/2/4 as its string storage, which treats each u8/u16/u32 value (depending on the highest codepoint value in the string) as simply integers, unlike UTF-8 or UTF-16 where some characters are encoded using multi-byte sequences. CPython additionally doesn't disallow the use of surrogates in strs (which in UTF-16 pair together to represent codepoints with a value higher than u16::MAX) and in fact takes quite extensive advantage of the fact that they're allowed. The surrogateescape codec-error handler uses them to represent byte sequences which are invalid in the given codec (e.g. bytes with their high bit set in ASCII or UTF-8) by mapping them into the surrogate range. surrogateescape is the default error handler in Python for interacting with the filesystem, and thus if RustPython is to properly support surrogateescape, its strs must be able to represent surrogates.

We use WTF-8 over something more similar to CPython's string implementation because of its compatibility with UTF-8, meaning that in the case where a string has no surrogates, it can be viewed as a UTF-8 Rust str without needing any copies or re-encoding.

This implementation is mostly copied from the WTF-8 implentation in the Rust standard library, which is used as the backing for OsStr on Windows targets. As previously mentioned, however, it is modified to not join two surrogates into one codepoint when concatenating strings, in order to match CPython's behavior.

Method

I opted not to change the signature of PyStr::as_str(), since that would mean a truly massive diff. Instead, I have it panic, and my hope is that we can get to a place with this where tests pass, and then we can deal with the rest of the as_str calls incrementally.

Status

Currently panicking in cformat code. Is there still a reason Parser et al need to be a separate repo if ruff isn't using it anymore? It'd be really convenient to merge it into the monorepo.