Al
|
b34e578366
|
[similarity] using new sequence alignment breakdown by operation to tell if any two words are an abbreviation. The loose variant requires that the alignment covers all characters in the shortest string, which matches things like Services vs. Svc, whereas the strict variant requires that either the shorter string is a prefix of the longer one (Inc and Incorporated) or that the two strings share both a prefix and a suffix (Dept and Department). Both variants require that the strings share at least the first letter in common.
|
2017-11-11 04:02:28 -05:00 |
|
Al
|
751873e56b
|
[similarity] a *NEW* sequence alignment algorithm which builds on Smith-Waterman-Gotoh with affine gap penalties. Like Smith-Waterman, it performs a local alignment, and like the cost-only version of Gotoh's improvement, it needs O(mn) time and O(m) space (where m is the length of the longer string). However, this version of the algorithm stores and returns a breakdown of the number and specific types of edits it makes (matches, mismatches, gap opens, gap extensions, and transpositions) rather than rolling them up into a single cost, and without needing to return/compute the full alignment as in Needleman-Wunsch or Hirschberg's variant
|
2017-11-11 03:07:39 -05:00 |
|
Al
|
665b780422
|
[utils] adding unicode_equals function in string_utils for testing equality of unicode char arrays
|
2017-11-11 02:45:41 -05:00 |
|
Al
|
669e52b329
|
[build] adding --no-same-owner explicitly when untarring the data files for #267
|
2017-11-01 20:05:36 -04:00 |
|
Al
|
bc9f11d6e3
|
[similarity] exposing unicode versions of Damerau-Levenshtein and Jaro-Winkler distances
|
2017-10-28 02:45:48 -04:00 |
|
Al
|
2d6079b06f
|
[expand] added search_address_dictionaries_substring to support the new use case (i.e. returns "does this substring in the trie?" regardless of if it's stored under the special prefixes/suffixes namespaces)
|
2017-10-28 02:40:14 -04:00 |
|
Al
|
053dca82ba
|
[expand] adding a normalization for a single non-acronym internal period where there's an expansion at the prefix/suffix (for #218 and https://github.com/openvenues/libpostal/issues/216#issuecomment-306617824). Helps in cases like "St.Michaels" or "Jln.Utara" without needing to specify concatenated prefix phrases for every possibility
|
2017-10-28 02:38:15 -04:00 |
|
Al
|
6d430f7e9b
|
[utils] adding functions for finding the next index of a full stop/period charater in a string
|
2017-10-27 04:07:28 -04:00 |
|
Al
|
e38e57b8e8
|
[numex] fixing edge case where something like "IV Michael" could cause a partial Roman numeral to get added for the MI portion of "Michael"
|
2017-10-27 04:04:12 -04:00 |
|
Al
|
e8ae3bbbaf
|
[similarity] using NULL-terminated varargs in double metaphone instead of specifying the number of arguments, should be more maintainable
|
2017-10-23 15:20:04 -04:00 |
|
Al
|
5c0ecf8963
|
[dedupe] Jaccard similarity
|
2017-10-21 10:34:12 -04:00 |
|
Al
|
4ccc2a9e9f
|
[fix] making string args const in string_similarity module
|
2017-10-21 02:45:22 -04:00 |
|
Al
|
5c927e780f
|
[expand] adding ability to expand Roman numerals with ordinal suffixes like IXe in French
|
2017-10-20 02:51:26 -04:00 |
|
Al
|
b7eda37e44
|
[utils] adding utf8_is_digit to string_utils.h
|
2017-10-20 02:46:00 -04:00 |
|
Al
|
1fbc238b60
|
[numex] adding functions to parse and validate a Roman numeral
|
2017-10-20 02:45:32 -04:00 |
|
Al
|
1c5afcafd2
|
[phrases] when skipping/ignoring hyphens in trie search, make sure that the new longer phrase ends at a word boundary (space, hyphen, end of string, etc.)
|
2017-10-20 02:43:39 -04:00 |
|
Al
|
9d2a111286
|
[numex] when parsing numex, bail on rules in whole_tokens_only languages if there are contiguous rules with no right context rules (example: something that wouldn't make sense like VL in Latin)
|
2017-10-20 02:34:30 -04:00 |
|
Al
|
bd477976d1
|
[similarity] string similarity measures for Damerau-Levenshtein and Jaro-Winkler distances. Both operate on unicode points internally for lengths, etc. instead of byte strings and the Levenshtein distance uses only one array instead of needing to store the full matrix of transitions.
|
2017-10-19 04:51:33 -04:00 |
|
Al
|
245aa226e0
|
[utils] function to create an array of uint32_t codepoints from a UTF-8 string, a few bug fixes to string_utils
|
2017-10-19 04:48:50 -04:00 |
|
Al
|
c61007388b
|
[similarity] bug fixes and additional French, Spanish, Italian, and Slavic phonetics
|
2017-10-18 13:31:35 -04:00 |
|
Al
|
3a3aca8490
|
[similarity] adding basic double metaphone implementation
|
2017-10-18 03:59:05 -04:00 |
|
Al
|
09fbb02042
|
[utils] adding utf8_equal_ignore_separators to string utils
|
2017-10-14 01:36:56 -04:00 |
|
Al
|
f8a808e254
|
[utils] adding utf8_len function for strings, and utf8_is_digit
|
2017-10-12 11:16:53 -04:00 |
|
Al
|
448ca6a61a
|
[merge] merging commit from v1.1
|
2017-10-12 01:41:04 -04:00 |
|
Travis
|
bb277fb326
|
[auto][ci skip] Adding data files from Travis build #268
|
2017-10-10 18:58:10 +00:00 |
|
Al
|
c984dca459
|
[fix] removing log error for sequences of length 0
|
2017-09-19 23:20:03 -04:00 |
|
Al
|
0c6af2b74c
|
[fix] normalize canonical strings (after expanding abbreviations, concatenated suffixes, etc.) with Latin-ASCII, Latin-ASCII-Simple or simple UTF-8 normalization depending on the options
|
2017-08-03 14:08:05 -06:00 |
|
Al
|
97044f5a8b
|
[fix] 32-bit safety in numex table loading
|
2017-07-20 17:55:43 -04:00 |
|
Iestyn Pryce
|
b96a687182
|
Merge https://github.com/openvenues/libpostal
|
2017-05-29 18:23:03 +01:00 |
|
Travis
|
8dd84b71ba
|
[auto][ci skip] Adding data files from Travis build #250
|
2017-05-24 05:05:06 +00:00 |
|
Iestyn Pryce
|
87cf7b5bca
|
Add portable way of formatting khint_t type (from klib)
|
2017-05-21 11:58:37 +01:00 |
|
Iestyn Pryce
|
d8239a9cc4
|
Revert format regression introduced in ecd07b18c1
|
2017-05-21 11:14:21 +01:00 |
|
Iestyn Pryce
|
73d27caeb9
|
Fix log_* formats which expect long long uint but receive uint64_t.
|
2017-05-21 10:57:20 +01:00 |
|
Iestyn Pryce
|
6aa3cb61fd
|
Fix log_* formats which expect long long int but receive int64_t.
|
2017-05-21 10:29:34 +01:00 |
|
Iestyn Pryce
|
ecd07b18c1
|
Fix log_* formats which expect size_t but receive uint32_t.
|
2017-05-19 22:31:56 +01:00 |
|
Iestyn Pryce
|
87a76bf967
|
Fix log_{debug,info} formats which expect size_t but receive int.
|
2017-05-17 22:40:53 +01:00 |
|
Iestyn Pryce
|
f34fc56fec
|
Fix log_debug formats which expect unsigned int but receive size_t
|
2017-05-14 17:48:26 +01:00 |
|
Al
|
a7e67c4967
|
[fix] adding maximum number of permutations for libpostal_expand_address to consider (n=100 for both the inner and outer loop, so max strings=10000), fixes #200
|
2017-05-13 14:11:08 -04:00 |
|
Al
|
5780a08b48
|
[fix] check that possible ordinal suffix also has non-zero digit length before normalizing
|
2017-05-12 15:48:20 -04:00 |
|
Al
|
cea3ced533
|
[fix] open files in binary format for #69
|
2017-05-03 17:34:38 -04:00 |
|
Al
|
6ea2273263
|
[fix] terminate the char_array if input token is zero-length in add_normalized_token
|
2017-04-28 11:25:07 -04:00 |
|
Al
|
278679b7fb
|
[fix] in tokenized trie_search, in the case of a partial failed match, reset to the root node before rolling the pointer back to phrase start + 1
|
2017-04-21 13:51:07 -04:00 |
|
Travis
|
074b6ff802
|
[auto][ci skip] Adding data files from Travis build #231
|
2017-04-20 02:39:39 +00:00 |
|
Travis
|
4762ff2638
|
[auto][ci skip] Adding data files from Travis build #228
|
2017-04-20 00:51:42 +00:00 |
|
Al
|
f3adde746e
|
[numex] adding ability to handle handle the degree symbol in numex parsing since it's technically a separate token
|
2017-04-19 20:18:21 -04:00 |
|
Oliver Keyes
|
35821f975e
|
Remove unused variable
What it says on the tin!
|
2017-04-18 21:25:00 -07:00 |
|
Al
|
f3cf119e58
|
[build] Makefile changes to support moving numeric expression parsing to normalize.c
|
2017-04-18 21:41:24 -04:00 |
|
Al
|
cddc368533
|
[numex] adding one form of normalization which strips ordinal suffixes so {96th, Ninety-sixth} => 96. This is an additional form of normalization, so there's still one form where the suffixes are kept. One case that's still not handled is something like "IXe Arrondissement"
|
2017-04-18 21:39:54 -04:00 |
|
Al
|
92051863ba
|
[numex] adding ordinal suffixes themselves to the numex trie so they can be removed from strings
|
2017-04-18 17:20:02 -04:00 |
|
Al Barrentine
|
63ac3cf921
|
Merge pull request #183 from openvenues/cdn
Hosting model files and training data on CloudFront CDN
|
2017-04-17 14:39:35 -04:00 |
|