A best-fit substitute made the strict UTF-8 converter lossy on Windows - #1410
Open
xroche wants to merge 5 commits into
Open
A best-fit substitute made the strict UTF-8 converter lossy on Windows#1410xroche wants to merge 5 commits into
xroche wants to merge 5 commits into
Conversation
WideCharToMultiByte substitutes a lookalike for a code point the codepage lacks and does not report it through lpUsedDefaultChar, so U+00A5 came back from CP932 as 0x5C with no loss flagged: a path separator, from a page that wrote a yen sign. Pass WC_NO_BEST_FIT_CHARS to both calls, gated on the predicate that already gates lpUsedDefaultChar, since the codepages rejecting that pointer are the same ones requiring dwFlags == 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
U+00B5 best-fits to two CP932 bytes where the substitute is one, so the sizing and converting calls disagree on the length unless both carry WC_NO_BEST_FIT_CHARS. U+00A5 alone could not see that: it is one byte either way. Pin the substitute to the codepage's own default char rather than a literal '?', and cover the gate's other arm, where a non-zero dwFlags would make UTF-7 fail outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The Windows leg caught this: a sizing pass does not substitute, so with WC_NO_BEST_FIT_CHARS it still reports the best-fit length while the conversion writes the shorter default character. U+00B5 is two CP932 bytes and its substitute one, so the exact-length guard rejected a conversion that had succeeded, and hts_convertStringFromUTF8 returned NULL for ten Latin-1 code points. Size for the larger of the two passes and trust the length written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The sizing pass does substitute: with the flag it reports 1 byte for U+00B5 on CP932, where the best-fit takes 2. What fails is the converting pass in that 1-byte buffer. Replace the throwaway diagnostic with the assertion, so the platform contract the fix rests on is pinned rather than described. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
enabled auto-merge (squash)
August 24, 2026 19:53
…o 932 Every st_nobestfit assertion converted a single code point, so a build that drops the usize max() in favor of the flagged-only size, or that gates the WC_NO_BEST_FIT_CHARS flag on wsize==1 instead of the codepage, still passed. Add "yen+micro+hira+ASCII" and assert both the strict (NULL, lossy) and non-strict (byte-exact default-char substitution) paths through it. Also derive the codepage from the charset name instead of repeating the 932 literal, split the GetCPInfo call out of its assertion, and cut the buffer- sizing comment to one line per house style. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Windows,
WideCharToMultiBytesubstitutes a lookalike character when the target codepage lacks a code point, and does not report that substitution throughlpUsedDefaultChar. U+00A5 comes out of CP932 as0x5C, sohts_convertStringFromUTF8Strictsaw no loss and handed back a path separator where the document wrote a yen sign. BothWideCharToMultiBytecalls now passWC_NO_BEST_FIT_CHARS, gated oncp_reports_default_char. That predicate already picks out the codepages needingdwFlags == 0, so it needed no change. The yen becomes the codepage's default character,usedDefaultis set, and strict returns NULL.Setting the flag is not quite enough on its own. Sizing with it reports the length of the substitute, but the converting pass still wants room for the form it replaced, so a buffer sized from the flagged pass is refused: U+00B5 sizes as 1 byte on CP932 while its best-fit takes 2. The exact-length guard then rejected a conversion that had succeeded, and
hts_convertStringFromUTF8returned NULL for ten Latin-1 code points. The buffer is now sized for the larger of the two passes and the written length is taken from the conversion. The Windows CI leg is what caught this; the numbers above are measured there, not inferred. The max() is defensive: per code point the flagged size is never larger than the best-fit size, since no codepage we know of has a default character longer than a best-fit substitute, so thefsize > bsizearm is believed unreachable and the bug fixed here is the reverse,fsize < bsize, which the flagged pass reports and the converting pass cannot fit.The non-strict callers change too (
hts_convertStringFromUTF8,hts_convertStringUTF8ToSystem, hence WinHTTrack's ANSI entry points). Approximations that used to pass through (folding full-width to half-width, or dropping an accent) now come out as'?'. This is intended, but the change is visible well beyond the dangerous cases. It also brings the Windows build to where the iconv build already stood: failing instead of approximating.The new
-#test=nobestfitself-test asserts the best-fit mappings and the two sizing lengths themselves before it tests the engine, so it cannot pass vacuously if any of them ever changes. U+00A5 alone would not have found the sizing bug: it is one byte either way. Thesyscharsetoracle needed the flag as well, or it disagrees with the engine on any ACP that best-fits. Found by the httrack-windows session while finishing itsnewlang.cppUTF-8 work.