Read a kana voicing mark as the script of the kana before it (#596) - #598
Merged
Merged
Conversation
…urce order (#596) rules.md#W4's Accepted clause said a name the 间隔号 divides keeps its source order. That holds under the default order only: the divider stands the script override down, so under name_order=FAMILY_FIRST 威廉·莎士比亚 reads family 威廉 like any other name. The clause now says declared order and carries a family-first example pinning it. Same correction at the sites that described the parser's behavior (AGENTS.md, concepts.rst, usage.rst, a test_assign comment). Sites describing how a transcription is WRITTEN, or a parse under the default order (migrate.rst's HumanName example, case-row notes), were read and left as they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The four kana voicing marks U+3099-U+309C sit in the hiragana block, and NFC composes one only where a precomposed voiced kana exists. So ア゙イ (ア + U+3099), ン゙ and every spacing ゛/゜ kept a hiragana-block character inside a katakana word, the word took the kana license, and a wholly-katakana name read family-first -- flipping its neighbour too (マイケル ア゙イ read family マイケル). Present since 2.1.0. Classification now deletes, from its NFC copy only, every mark with a character before it, so the word reads as its base's script. Token text is untouched. A str.translate table, not a pattern: C-level, and no new module-level regex for test_regex_sync to declare. Declined, measured on patched copies: dropping the marks from the hiragana range fixes katakana but leaves a marked word in no script, so あ゙い たろう and 山田 ア゙イ lose their Japanese reading; narrowing to the two combining marks leaves ア゛イ タロウ family-first. NFD text is unaffected by any option, since classification composes first. Three fix(#596) case rows (the movers), two parity rows guarding the declined alternative, parametrized unit pins on effective_script, a W4 example and W Background sentence, a decisions.md#W4 entry, fix(#596) ledger rules at 2.1.0/2.2.0/2.3.0 (1.4.0 and 2.0.0 had no license to misfire), and a release-log bullet. The gate exits 0 at every baseline. Closes #596. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review: the first cut deleted a voicing mark after ANY character, so 김゙민준 classified as hangul and the surname division, which cuts the raw text, split the mark off its base -- given 김, family ゙민준, unreported, where 2.3.0 left the word whole and reported it (parser_for(ZH) cut 王゙小明 the same way). A mark voices the kana it follows, so it is now dropped only after a kana; after anything else it keeps the table's answer, as before. The common path stays C-level (frozenset.isdisjoint); only a token holding a mark walks it. A parity row pins 김゙민준, with a #449 ledger rule at 2.1.0 and 2.2.0 for its report, new in 2.3. Docs review: usage.rst's "Forms the script carries" still said form 7 needs no name_order -- a declared FAMILY_FIRST reverses it, as W4 now says. decisions.md's #596 entry claimed nothing moves at 1.4.0/2.0.0; the two kana parity rows do there, under the native-script rule. It now also records that #594's case against an NFKC fold has lost its W4 half: #594's own recompute finds no role differing on this tree. rules.md's W Background and the code comments say "after a kana". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 김゙민준 moves in its report at 2.0.0 too, not only 2.1.0/2.2.0: the #449 rule joins the 2.0.0 ledger (seven #449 rules there now), and decisions.md's MEASURED sentence says so; 1.4.0 has no report and is parity. - The hangul row is tolerated: rules.md's W Background leaves a mark after a non-kana unpromised, and a contract row contradicted it. It moves to the radar corpus; decisions.md says "watches", not "pins". - Stale "follows a character" in the three fix(#596) ledger comments and "every voicing mark" in _policy.py now say "after a kana". - The 147 -> 153 native-script claim bumps carry dated comments, and the new #449 rule has its _MUST_NOT_MATCH near-misses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #598 +/- ##
=======================================
Coverage 98.98% 98.98%
=======================================
Files 45 45
Lines 4220 4235 +15
=======================================
+ Hits 4177 4192 +15
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A katakana name typed with a separate voicing mark —
ア゙イ タロウ(ア + combining U+3099), or the spacing゛/゜— read family-first. The voicing marks U+3099–U+309C sit in the hiragana block, and NFC composes one only where a precomposed voiced kana exists. Soア゙イstayed katakana plus a hiragana-block character, took the kana license, and turned the name around, along with its neighbour (マイケル ア゙イread familyマイケル). The bug has been there since 2.1.0.Fix. Script classification drops, from its NFC copy only, every voicing mark that stands after a kana, so the word reads as the script of the kana before the mark. Token text is untouched. A mark after anything else (hangul, Han, Latin, or nothing) voices no kana and keeps the table's answer, as before. The common path is one
frozenset.isdisjointcheck in C; only a token that holds a mark walks it in Python.ア゙イ タロウア゙イア゙イ, familyタロウア゛イ タロウア゛イア゛イマイケル ア゙イマイケルマイケル, familyア゙イあ゙い たろう,山田 ア゙イDeclined, measured on patched copies of the tree:
あ゙い たろうand山田 ア゙イlose their Japanese reading.ア゛イ タロウfamily-first.NFD-typed text, the issue's worry, is unaffected by every option, because classification composes first. A mark with nothing before it (
゙アイ) still reads as hiragana. That's degenerate input, and rules.md's W Background now leaves it unpromised.Also in this PR: the W4 interpunct wording. A name the 间隔号 divides keeps the declared order, not its source order: under
FAMILY_FIRST,威廉·莎士比亚reads family 威廉. rules.md#W4 now says so, with afamily-firstexample. The same correction is in AGENTS.md, concepts.rst and usage.rst, including the "Forms the script carries" passage, which the issue's list missed.Commits
docs:the W4 wording (docs only).fix(#596)ledger rules and the release log.김゙민준then classified as hangul, and the surname split, which cuts the raw text, stranded the mark (given김, family゙민준). The fix narrows the drop to marks after a kana. It also fixes two doc findings: the usage.rst form-7 passage, and a wrong 1.4.0/2.0.0 claim in decisions.md.김゙민준added at 2.0.0; that row markedtoleratedso it agrees with "unpromised"; stale wording; dated claim comments.Verification
fix(#596)rows move.김゙민준moves in its parse("Andrew") reports no ambiguity, but given-or-family is a guess #449 report only.Closes #596. #597 (halfwidth corner brackets) follows as its own PR.
🤖 Generated with Claude Code