[similarity] adding a stopword-aware acronym alignment method for matching U.N. with United Nations, Museum of Modern Art with MoMA, as well as things like University of California - Los Angeles with UCLA. All of these should work across languages, including non-Latin character sets like Cyrllic (but not ideograms as the concept doesn't make as much sense there). Skipping tokens like "of" or "the" depends only on the stopwords dictionary being defined for a given language.

This commit is contained in:
Al
2017-12-04 15:21:09 -05:00
parent 252d5a0f37
commit cfa5b1ce42
6 changed files with 181 additions and 2 deletions

View File

@@ -91,6 +91,7 @@ uint32_array *unicode_codepoints(const char *str);
bool unicode_equals(uint32_array *u1_array, uint32_array *u2_array);
bool utf8_is_hyphen(int32_t ch);
bool utf8_is_period(int32_t ch);
bool utf8_is_letter(int cat);
bool utf8_is_number(int cat);
bool utf8_is_digit(int cat);