Al
|
9eef46adee
|
[expand] in cases like "Avenue D" where there are two phrases, one is ambiguous (and canonical) but not necessarily edge-ignorable (pre/post-directional), allow deletion of the other token (so "Avenue" in this case). Also allows skipping in cases where the language classifier may predict a second language with some small probability, such as French for a short string like "Avenue D" (in addition to English). If the token was ignorable in the highest probability language, ignore it in both.
|
2017-12-17 17:24:27 -05:00 |
|
Al
|
3f7abd5b24
|
[expand] adding a method that allows hash/equality comparisons of addresses like "100 Main" with "100 S Main St." or units like "Apt 101" vs. "#101". Instead of expanding the phrase abbreviations, this version tries its best to delete all but the root words in a string for a specific component. It's probably not perfect, but does handle a number of edge cases related to pre/post directionals in English e.g. "E St" will have a root word of simply "E", "Avenue E" => "E", etc. Also handles a variety of cases where the phrase could be a thoroughfare type but is really a root word such as "Park Pl" or the famous "Avenue Rd". This can be used for near dupe hashing to catch possible dupes for later analysis. Note that it will normalize "St Marks Pl" and "St Marks Ave" to the same thing, which is sometimes warranted (if the user typed the wrong thoroughfare), but can also be reconciled at deduping time.
|
2017-12-17 15:48:11 -05:00 |
|
Al
|
d0364ab6fb
|
[expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation
|
2017-12-17 03:14:00 -05:00 |
|
Travis
|
e1d89b5d62
|
[auto][ci skip] Adding data files from Travis build #323
|
2017-12-12 14:57:29 +00:00 |
|
Al
|
8968a6c966
|
[expand] moving expand to its own module so the internal methods can be exposed, calling from libpostal.c
|
2017-12-08 16:26:13 -05:00 |
|
Al
|
e4e84f0147
|
[utils] adding unicode_common_prefix/unicode_common_suffix, string_hyphen_prefix_len and string_hyphen_suffix_len to string_utils
|
2017-12-08 14:28:30 -05:00 |
|
Al
|
55ba627c3c
|
[similarity] needed to add utf8proc_category and invert the indices for counting transposes in affine gap
|
2017-12-08 14:27:23 -05:00 |
|
Al
|
cfa5b1ce42
|
[similarity] adding a stopword-aware acronym alignment method for matching U.N. with United Nations, Museum of Modern Art with MoMA, as well as things like University of California - Los Angeles with UCLA. All of these should work across languages, including non-Latin character sets like Cyrllic (but not ideograms as the concept doesn't make as much sense there). Skipping tokens like "of" or "the" depends only on the stopwords dictionary being defined for a given language.
|
2017-12-04 15:21:44 -05:00 |
|
Al
|
e27f5f1d70
|
[api] adding LIBPOSTAL_EXPORT to some of the new public API functions in this branch
|
2017-11-29 18:10:10 -05:00 |
|
Al
|
ec4d683d1b
|
Merge branch 'master' into lieu_api
|
2017-11-29 15:49:52 -05:00 |
|
Al
|
9e837f7209
|
[fix] deleting comment, this is not a header-only implementation
|
2017-11-29 12:26:36 -05:00 |
|
Al
|
26e4ef08bc
|
[fix] conditional compilation for strndup and drand48 for Windows, using config.h
|
2017-11-29 12:21:13 -05:00 |
|
Al
|
1a64ad682b
|
[merge] merging in the Ohio expansion numex changes from master
|
2017-11-29 11:51:43 -05:00 |
|
Al Barrentine
|
18eb5ef9ee
|
Merge pull request #272 from AeroXuk/master
Windows support via AppVeyor
|
2017-11-28 21:35:46 -05:00 |
|
AeroXuk
|
19ae97d527
|
Adding include config.h to strndup.c so that the function is not compiled and doesn't cause errors when the system has its own implementation.
|
2017-11-27 23:40:46 +00:00 |
|
AeroXuk
|
9090811826
|
Modifed the libpostal API to add an extra function libpostal_parser_print_features to toggle debugging info. Updated address_parser app to use the new function.
|
2017-11-27 19:20:37 +00:00 |
|
AeroXuk
|
69e0d5d963
|
Updated linenoise to be MSys2/MinGW compatible. Updated address_parser app to use the defined libpostal api and not include internal components directly. Removed windows src Makefile as it is now the same as the standard one.
|
2017-11-27 01:42:25 +00:00 |
|
AeroXuk
|
26ac9ab5c2
|
Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.
|
2017-11-25 04:35:28 +00:00 |
|
Travis
|
15b3758be8
|
[auto][ci skip] Adding data files from Travis build #284
|
2017-11-24 22:29:45 +00:00 |
|
Al
|
ef098fd2e7
|
[numex] implementing the numex concat_only_if_number left context, which helps in the case of e.g. Columbus, OH in #271
|
2017-11-24 15:42:50 -05:00 |
|
Al
|
c276cf1529
|
[numex] adding a new type of left context for numeric expressions called conat_only_if_number (for something like "oh" which can be "Columbus, OH" or something like "Twenty-One Oh One"
|
2017-11-24 15:36:53 -05:00 |
|
AeroXuk
|
f0246e7333
|
Fix bug in strndup fix for windows. Move all includes out of headers and into code for strndup.h and move it to be the last include.
|
2017-11-23 19:11:25 +00:00 |
|
AeroXuk
|
f07ab765cb
|
Adding the export marker to all functions used in tests.
|
2017-11-20 20:58:37 +00:00 |
|
AeroXuk
|
ad682b7592
|
Altered Makefile to include strndup.c on the other programs which require it. For the windows version of the Makefile, commented out address_parser lines as it has dependencies on includes we don't have.
|
2017-11-20 20:24:11 +00:00 |
|
AeroXuk
|
2d3b420d35
|
Merging changes from AeroXuk/libpostal_windows.
|
2017-11-19 12:44:38 +00:00 |
|
Travis
|
7d6e648fc3
|
[auto][ci skip] Adding data files from Travis build #271
|
2017-11-17 19:36:25 +00:00 |
|
Al
|
fbf88aee88
|
[similarity] adding possible abbreviation functions to header, making everything const char *
|
2017-11-12 04:48:26 -05:00 |
|
Al
|
b34e578366
|
[similarity] using new sequence alignment breakdown by operation to tell if any two words are an abbreviation. The loose variant requires that the alignment covers all characters in the shortest string, which matches things like Services vs. Svc, whereas the strict variant requires that either the shorter string is a prefix of the longer one (Inc and Incorporated) or that the two strings share both a prefix and a suffix (Dept and Department). Both variants require that the strings share at least the first letter in common.
|
2017-11-11 04:02:28 -05:00 |
|
Al
|
751873e56b
|
[similarity] a *NEW* sequence alignment algorithm which builds on Smith-Waterman-Gotoh with affine gap penalties. Like Smith-Waterman, it performs a local alignment, and like the cost-only version of Gotoh's improvement, it needs O(mn) time and O(m) space (where m is the length of the longer string). However, this version of the algorithm stores and returns a breakdown of the number and specific types of edits it makes (matches, mismatches, gap opens, gap extensions, and transpositions) rather than rolling them up into a single cost, and without needing to return/compute the full alignment as in Needleman-Wunsch or Hirschberg's variant
|
2017-11-11 03:07:39 -05:00 |
|
Al
|
665b780422
|
[utils] adding unicode_equals function in string_utils for testing equality of unicode char arrays
|
2017-11-11 02:45:41 -05:00 |
|
Al
|
669e52b329
|
[build] adding --no-same-owner explicitly when untarring the data files for #267
|
2017-11-01 20:05:36 -04:00 |
|
Al
|
bc9f11d6e3
|
[similarity] exposing unicode versions of Damerau-Levenshtein and Jaro-Winkler distances
|
2017-10-28 02:45:48 -04:00 |
|
Al
|
2d6079b06f
|
[expand] added search_address_dictionaries_substring to support the new use case (i.e. returns "does this substring in the trie?" regardless of if it's stored under the special prefixes/suffixes namespaces)
|
2017-10-28 02:40:14 -04:00 |
|
Al
|
053dca82ba
|
[expand] adding a normalization for a single non-acronym internal period where there's an expansion at the prefix/suffix (for #218 and https://github.com/openvenues/libpostal/issues/216#issuecomment-306617824). Helps in cases like "St.Michaels" or "Jln.Utara" without needing to specify concatenated prefix phrases for every possibility
|
2017-10-28 02:38:15 -04:00 |
|
Al
|
6d430f7e9b
|
[utils] adding functions for finding the next index of a full stop/period charater in a string
|
2017-10-27 04:07:28 -04:00 |
|
Al
|
e38e57b8e8
|
[numex] fixing edge case where something like "IV Michael" could cause a partial Roman numeral to get added for the MI portion of "Michael"
|
2017-10-27 04:04:12 -04:00 |
|
Al
|
e8ae3bbbaf
|
[similarity] using NULL-terminated varargs in double metaphone instead of specifying the number of arguments, should be more maintainable
|
2017-10-23 15:20:04 -04:00 |
|
Al
|
5c0ecf8963
|
[dedupe] Jaccard similarity
|
2017-10-21 10:34:12 -04:00 |
|
Al
|
4ccc2a9e9f
|
[fix] making string args const in string_similarity module
|
2017-10-21 02:45:22 -04:00 |
|
Al
|
5c927e780f
|
[expand] adding ability to expand Roman numerals with ordinal suffixes like IXe in French
|
2017-10-20 02:51:26 -04:00 |
|
Al
|
b7eda37e44
|
[utils] adding utf8_is_digit to string_utils.h
|
2017-10-20 02:46:00 -04:00 |
|
Al
|
1fbc238b60
|
[numex] adding functions to parse and validate a Roman numeral
|
2017-10-20 02:45:32 -04:00 |
|
Al
|
1c5afcafd2
|
[phrases] when skipping/ignoring hyphens in trie search, make sure that the new longer phrase ends at a word boundary (space, hyphen, end of string, etc.)
|
2017-10-20 02:43:39 -04:00 |
|
Al
|
9d2a111286
|
[numex] when parsing numex, bail on rules in whole_tokens_only languages if there are contiguous rules with no right context rules (example: something that wouldn't make sense like VL in Latin)
|
2017-10-20 02:34:30 -04:00 |
|
Al
|
bd477976d1
|
[similarity] string similarity measures for Damerau-Levenshtein and Jaro-Winkler distances. Both operate on unicode points internally for lengths, etc. instead of byte strings and the Levenshtein distance uses only one array instead of needing to store the full matrix of transitions.
|
2017-10-19 04:51:33 -04:00 |
|
Al
|
245aa226e0
|
[utils] function to create an array of uint32_t codepoints from a UTF-8 string, a few bug fixes to string_utils
|
2017-10-19 04:48:50 -04:00 |
|
Al
|
c61007388b
|
[similarity] bug fixes and additional French, Spanish, Italian, and Slavic phonetics
|
2017-10-18 13:31:35 -04:00 |
|
Al
|
3a3aca8490
|
[similarity] adding basic double metaphone implementation
|
2017-10-18 03:59:05 -04:00 |
|
Al
|
09fbb02042
|
[utils] adding utf8_equal_ignore_separators to string utils
|
2017-10-14 01:36:56 -04:00 |
|
Al
|
f8a808e254
|
[utils] adding utf8_len function for strings, and utf8_is_digit
|
2017-10-12 11:16:53 -04:00 |
|