tommy/libpostal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[tokenization] Including full-width numbers in numeric tokens

Browse Source
This commit is contained in:
Al
2016-06-14 01:28:25 +02:00
parent 02d40c23a6
commit b5d4dd6f37
2 changed files with 196721 additions and 196612 deletions
Download Patch File Download Diff File Expand all files Collapse all files

393328
src/scanner.c
View File

File diff suppressed because it is too large Load Diff

5
src/scanner.re
View File

@@ -117,8 +117,9 @@ hebrew_word_double_quote = ({hebrew_letter_chars}+{double_quote})+{hebrew_letter
// WB11 and WB12 (modified slightly)
possible_numeric_chars = ({numeric_chars}|{letter}|{non_breaking_dash}|{hyphen}|{mid_num_letter_chars}|{single_quote});
number = ({non_breaking_dash}?(({numeric_chars}+({mid_number_chars}|{mid_num_letter_chars})*)*){numeric_chars}+);
numeric = (({non_breaking_dash}?|((({numeric_chars}|{letter})+{possible_numeric_chars}*)*)){numeric_chars}+({possible_numeric_chars}*({numeric_chars}|{letter})+)*);
number = ({non_breaking_dash}?((({numeric_chars}|{number_or_digit_chars})+({mid_number_chars}|{mid_num_letter_chars})*)*)({numeric_chars}|{number_or_digit_chars})+);
numeric = (({non_breaking_dash}?|((({numeric_chars}|{letter})+{possible_numeric_chars}*)*)){numeric_chars}+({possible_numeric_chars}*({numeric_chars}|{letter})+)*)|({number_or_digit_chars}+);
// WB13
katakana = {katakana_chars}+;