Skip to content

A best-fit substitute made the strict UTF-8 converter lossy on Windows - #1410

Open
xroche wants to merge 5 commits into
masterfrom
nobestfit-wc
Open

A best-fit substitute made the strict UTF-8 converter lossy on Windows#1410
xroche wants to merge 5 commits into
masterfrom
nobestfit-wc

Conversation

@xroche

@xroche xroche commented Aug 24, 2026

Copy link
Copy Markdown
Owner

On Windows, WideCharToMultiByte substitutes a lookalike character when the target codepage lacks a code point, and does not report that substitution through lpUsedDefaultChar. U+00A5 comes out of CP932 as 0x5C, so hts_convertStringFromUTF8Strict saw no loss and handed back a path separator where the document wrote a yen sign. Both WideCharToMultiByte calls now pass WC_NO_BEST_FIT_CHARS, gated on cp_reports_default_char. That predicate already picks out the codepages needing dwFlags == 0, so it needed no change. The yen becomes the codepage's default character, usedDefault is set, and strict returns NULL.

Setting the flag is not quite enough on its own. Sizing with it reports the length of the substitute, but the converting pass still wants room for the form it replaced, so a buffer sized from the flagged pass is refused: U+00B5 sizes as 1 byte on CP932 while its best-fit takes 2. The exact-length guard then rejected a conversion that had succeeded, and hts_convertStringFromUTF8 returned NULL for ten Latin-1 code points. The buffer is now sized for the larger of the two passes and the written length is taken from the conversion. The Windows CI leg is what caught this; the numbers above are measured there, not inferred. The max() is defensive: per code point the flagged size is never larger than the best-fit size, since no codepage we know of has a default character longer than a best-fit substitute, so the fsize > bsize arm is believed unreachable and the bug fixed here is the reverse, fsize < bsize, which the flagged pass reports and the converting pass cannot fit.

The non-strict callers change too (hts_convertStringFromUTF8, hts_convertStringUTF8ToSystem, hence WinHTTrack's ANSI entry points). Approximations that used to pass through (folding full-width to half-width, or dropping an accent) now come out as '?'. This is intended, but the change is visible well beyond the dangerous cases. It also brings the Windows build to where the iconv build already stood: failing instead of approximating.

The new -#test=nobestfit self-test asserts the best-fit mappings and the two sizing lengths themselves before it tests the engine, so it cannot pass vacuously if any of them ever changes. U+00A5 alone would not have found the sizing bug: it is one byte either way. The syscharset oracle needed the flag as well, or it disagrees with the engine on any ACP that best-fits. Found by the httrack-windows session while finishing its newlang.cpp UTF-8 work.

xroche and others added 4 commits August 24, 2026 20:54
WideCharToMultiByte substitutes a lookalike for a code point the codepage
lacks and does not report it through lpUsedDefaultChar, so U+00A5 came back
from CP932 as 0x5C with no loss flagged: a path separator, from a page that
wrote a yen sign. Pass WC_NO_BEST_FIT_CHARS to both calls, gated on the
predicate that already gates lpUsedDefaultChar, since the codepages rejecting
that pointer are the same ones requiring dwFlags == 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
U+00B5 best-fits to two CP932 bytes where the substitute is one, so the
sizing and converting calls disagree on the length unless both carry
WC_NO_BEST_FIT_CHARS. U+00A5 alone could not see that: it is one byte either
way. Pin the substitute to the codepage's own default char rather than a
literal '?', and cover the gate's other arm, where a non-zero dwFlags would
make UTF-7 fail outright.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The Windows leg caught this: a sizing pass does not substitute, so with
WC_NO_BEST_FIT_CHARS it still reports the best-fit length while the conversion
writes the shorter default character. U+00B5 is two CP932 bytes and its
substitute one, so the exact-length guard rejected a conversion that had
succeeded, and hts_convertStringFromUTF8 returned NULL for ten Latin-1 code
points. Size for the larger of the two passes and trust the length written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The sizing pass does substitute: with the flag it reports 1 byte for U+00B5 on
CP932, where the best-fit takes 2. What fails is the converting pass in that
1-byte buffer. Replace the throwaway diagnostic with the assertion, so the
platform contract the fix rests on is pinned rather than described.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
…o 932

Every st_nobestfit assertion converted a single code point, so a build that
drops the usize max() in favor of the flagged-only size, or that gates the
WC_NO_BEST_FIT_CHARS flag on wsize==1 instead of the codepage, still passed.
Add "yen+micro+hira+ASCII" and assert both the strict (NULL, lossy) and
non-strict (byte-exact default-char substitution) paths through it.

Also derive the codepage from the charset name instead of repeating the 932
literal, split the GetCPInfo call out of its assertion, and cut the buffer-
sizing comment to one line per house style.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant