tommy/libpostal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[normalize/api] exposing normalize_string_languages and normalized_tokens_languages to the API for pre-normalizing numeric expressions at tokenization time

Browse Source
This commit is contained in:
Al
2018-02-22 18:47:36 -05:00
parent 0edb897143
commit c5bb9d8daa
3 changed files with 22 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
src/normalize.h
View File

@@ -68,7 +68,10 @@ As well as normalizations for individual string tokens:
char *normalize_string_utf8(char *str, uint64_t options);
char *normalize_string_utf8_languages(char *str, uint64_t options, size_t num_languages, char **languages);
char *normalize_string_latin(char *str, size_t len, uint64_t options);
char *normalize_string_latin_languages(char *str, size_t len, uint64_t options, size_t num_languages, char **languages);
// Takes NORMALIZE_TOKEN_* options
void add_normalized_token(char_array *array, char *str, token_t token, uint64_t options);