Fix four segmentation rules and the empty language tag

[egonelbre]

Five commits, each one rule. The segmenter ones come from cases where our line and grapheme breaks disagreed with UAX #14 and UAX #29 on input that the Unicode test files do not cover.

• LB19, LB19a and LB30 compared the previous character against the raw runes even though the class they were given already had LB9 applied. A combining mark between an ideograph and a quote got the wrong treatment on both sides. U+6F22 U+201D U+0308 U+6F22 now breaks before the last ideograph, and U+6F22 U+0308 U+201C U+6F22 breaks before the quote. LineBreakTest.txt has no EastAsian, CM, QU triples, so both are regression tests.
• The same rules, and the numeric prefix and decimal mark rules, look ahead at the next base character. They now skip attached combining marks and ZWJ there too, and resolve SA marks with LB1, so opening quotes with marks in East Asian text stop prohibiting the break.
• LB25 lost track of a number after closing punctuation. In 1)2% the digit after the parenthesis started no new numeric sequence, so a break before the percent sign was allowed.
• GB11 never matched when two pictographs sit next to each other. In 😀👩‍💻 the second pictograph did not open a sequence, so the ZWJ joined nothing and the emoji split.
• NewLangID("") returned (0, true) because the first table entry is an empty sentinel. It now returns false.

[egonelbre]

PS: Here I'm less confident in my own review of the code as I do not have the necessary in-depth knowledge.

[benoitkugler]

Hello,

Thank you for these changes. I've checked the three last commit and I do agree with the fixes and the test results.

I've included two changes to make the tests a bit clearer (to me at least).

I have not yet grasped the meaning of the two first commits. Could you extract these two out of this PR, so that we can already merge the 3 lasts (pending the test updates) ?

[benoitkugler]

Could you simplify along these lines :

func TestAdjacentPictographicSequences(t *testing.T) {
	for _, test := range []struct {
		text      string
		graphemes []int
	}{
		{"😀👩\u200d💻", []int{1, 4}},        // only one break after the first emoji
		{"😀👩\u0301\u200d💻", []int{1, 5}},  // only one break after the first emoji
		{"😀👩\u200d💻\u200d👩", []int{1, 6}}, // only one break after the first emoji
	} {
		var seg Segmenter
		seg.Init([]rune(test.text))
		got := collectGraphemes(&seg)
		tu.Assert(t, reflect.DeepEqual(got, test.graphemes))
	}
}

[egonelbre]

done

[benoitkugler]

And here :

func TestNumericSequenceAfterClosingPunctuation(t *testing.T) {
	for _, text := range []string{"1)2%", "1]2$", "1)2\u0301%", "1)2,3%"} {
		var seg Segmenter
		seg.Init([]rune(text))
		got := collectLineBreaks(&seg)
		tu.Assert(t, reflect.DeepEqual(got, []int{len([]rune(text))})) // no breaks
	}
}

[egonelbre]

done

[benoitkugler]

Thank you !