4356174630[similarity] adding a match count in Soft-TFIDF to allow answering questions about subsets i.e. the set of tokens in "Park Pl" contain the set of tokens in "Park". Setting Jaro-Winkler minimum length of 4 chars on, more specific option name for possible abbeviation detection
Al
2018-01-06 03:50:03 -05:00
0cb488ecea[test] different test case for expansion, male names are overemphasized as it is
Al
2018-01-02 19:01:50 -05:00
513fdd775fMerge pull request #297 from oschwald/greg/fix-more-leaks
Al Barrentine
2018-01-02 18:50:38 -05:00
2f6749fe03Fix segfault in expand_alternative_phrase_option
Gregory Oschwald
2018-01-02 13:28:51 -08:00
18cc0e37e6Only create parser response when it is used
Gregory Oschwald
2018-01-02 11:56:02 -08:00
95ea873498Fix memory leaks in test_trie
Gregory Oschwald
2018-01-02 10:38:12 -08:00
999de2bf6aAdd missing char_array_destroy when numex is invalid
Gregory Oschwald
2018-01-02 10:24:39 -08:00
d33b6693b9[test] adding Suite/Ste tests for root expansion bugfix
Al
2018-01-02 03:50:23 -05:00
071aee0e85[fix] in root expansions, removing phrases that are invalid for the given components if there are other ignorable components
Al
2018-01-02 03:49:52 -05:00
d8a0a344cdMerge branch 'master' of ssh://github.com/openvenues/libpostal
Al
2017-12-31 19:20:42 -05:00
7651a7b9b9[fix] fixing a couple of warnings in dedupe/near_dupe
Al
2017-12-31 19:20:17 -05:00
95ea250cb1[auto][ci skip] Adding data files from Travis build #330
Travis
2017-12-31 22:30:34 +00:00
8a917d8594Merge pull request #294 from openvenues/lieu_api
Al Barrentine
2017-12-31 17:00:03 -05:00
3bdb8c8630[similarity] max out the Jaro-Winkler shared prefix at 4 characters in accordance with Winkler's paper
Al
2017-12-31 13:22:00 -05:00
4e32565746[dedupe] fixing toponym matching for city-equivalents, adding the LIBPOSTAL_ADDRESS_ANY component in each function call so it can be removed as needed.
Al
2017-12-30 18:05:23 -05:00
34c3ee7f7a[fix] update to struct
Al
2017-12-30 03:24:39 -05:00
34fe7ec305[expand] adding a few of the address phrase checks to the expand header
Al
2017-12-30 02:34:06 -05:00
668e467967[dedupe/test] checking for NULL in near_dupe test program
Al
2017-12-30 02:33:33 -05:00
3263c84b32[api] using uint32_t for geohash precision option
Al
2017-12-30 02:32:35 -05:00
434bbd4dc2[fix] removing unused vars
Al
2017-12-30 02:31:43 -05:00
86d5eca521[api] checking for NULL responses in the cstring_array methods before converting them to char arrays
Al
2017-12-30 02:31:25 -05:00
c48c2b778c[dedupe] fixes to near dupe hashing, geohash lengths, cutting off name hashing at 50 unique tokens, fixing memory leaks, checking for valid geo components and returning NULL if one of the required fields isn't present
Al
2017-12-30 02:28:38 -05:00
6dff154a99[api] adding APIs for getting default options and using a consistent naming convention
Al
2017-12-29 17:48:05 -05:00
53543be5a5[build] adding new source files to Makefile for the lieu APIs
Al
2017-12-29 17:46:35 -05:00
8495cda1eb[api] adding pairwise-dupe functions/structs to the public header
Al
2017-12-29 13:48:54 -05:00
cadf52d19f[fix] making a few internal functions static
Al
2017-12-29 04:50:08 -05:00
098babfdee[dedupe] adding the core pairwise deduping module which ties together most of the work on this branch. Includes simple phrase-aware exact deduping methods, with per-component variations as to whether e.g. a root expansion match counts as an exact duplicate or not (in a secondary unit, "No. 2" and "Apt 2" can be considered an exact match in English whereas we wouldn't want to make that kind of assumption for street e.g. "Park Ave" and "Park Pl"). The API is fairly low-level at present, and may require a few calls. Notably, we leave the TFIDF scores or other weighting schemes to the client. Since each component gets its own dupe classification, it leaves the door open for doing more specific checks around e.g. compound house numbers/ranges in the future.
Al
2017-12-29 04:48:00 -05:00
1f1412c120[api] adding libpostal_place_languages method to public API for classifying languages consistently from components (may need to make several calls using the same languages and don't necessarily want the language classifier to be run on house numbers when we already know the languages from e.g. the street name - this provides a simple window into the language classifier focused on the entire address/record
Al
2017-12-29 03:32:41 -05:00
1d1ce10fad[similarity] adding a string array version of Jaccard similarity that creates the string sets internally for convenience
Al
2017-12-29 03:08:48 -05:00
c5ad080fb0[similarity] moving stopword tokens array to a separate function in acronym token alignments
Al
2017-12-29 02:42:22 -05:00
f1e6886536[similarity/dedupe] adding options for acronym alignments and address phrase matches in Soft-TFIDF. Acronym alignments will give higher similarity to NYU vs. "New York University" whereas phrase matches would match known phrases that share the same canonical like "Cty Rd" vs. "C.R." vs. "County Road" within the Soft-TFIDF similarity calculation.
Al
2017-12-29 02:38:48 -05:00
24a77ea03f[fix] another valgrind error in counting transposes in our counting affine gap implementation (mixed indices)
Al
2017-12-29 00:04:24 -05:00
cabdbfccd2[fix] using same order in root expansions
Al
2017-12-28 23:55:41 -05:00
8fd4242eb8[fix] bug in Jaro distance
Al
2017-12-28 23:54:10 -05:00
b90c3dab4b[similarity/dedupe] adding Soft-TFIDF implementation with several different fallback qualifiers for the max-sim function (Damerau-Levenshtein and libpostal's new bucketed affine gap method for detecting abbreviations), but keeping Jaro-Winkler as the secondary similarity function in the final distance metric. Overall this should results in higher similarity values when one of the tokens may not quite match the pure secondary threshold in terms of Jaro-Winkler but may match on one of the other criteria.
Al
2017-12-28 04:34:25 -05:00
33bb90d94b[utils] adding place.h header, which converts parser-like output into an object that can be used for comparisons. Currently single-value, but could use cstring_arrays for fields instead.
Al
2017-12-27 22:13:04 -05:00
d731339811[expand] fixing case where too many permutations were getting added for longer strings due to the new-ish ordinal suffix handling, using string_tree_num_tokens instead of string_tree_num_strings throughout to check for previously added words, using new is_likely_roman_numeral API
Al
2017-12-27 21:48:54 -05:00
b4fdc51bf9[numex] changing is_roman_numeral to is_likely_roman_numeral to get rid of most of the false positives like \"La\" in Spanish which could be L(=50) + the ordinal suffix \"a\", but in practice it never means that. For Roman numerals that are shorter than two characters (whether on their own like "DC" or "MD", or attached to a potential ordinal suffix like \"Ce\" in French), will be ignored unless they're composed of more likely, smaller, Roman numerals: I, V, and X, so VI, IX, etc. are expanded as Roman numerals but LI is not.
Al
2017-12-27 19:27:23 -05:00
b17b2bdcc4[dictionaries] adding hill/hills to synonyms lists in English. In general any ambiguous street types that can also be part of a core street name can also be stored in synonyms
Al
2017-12-27 19:12:01 -05:00
bb9f6a4c6bMerge branch 'master' into lieu_api
Al
2017-12-26 10:47:29 -05:00
1e7cc23b81Merge pull request #292 from oschwald/greg/fix-leak
Al Barrentine
2017-12-26 10:44:27 -05:00
1bb6278446Fix leak of normalized value in early return
Gregory Oschwald
2017-12-25 19:33:07 -08:00
2afcd74779[test] adding E Ctr St tests
Al
2017-12-25 01:38:50 -05:00
152761fcbc[expand] adding improvements to root expansions (using possible phrase roots even if they're abbreviated e.g. "E Ctr St", adding special valid components check for root expansions beyond what's stored in the build address dictionaries), removing spaces before checking unique strings, only splitting numeric from alpha in the case of non-ordinals, using cstring_array internally and char ** in the public API
Al
2017-12-25 01:37:29 -05:00
b4ce042f80[dictionaries] removing ave/avens/aves from ambiguous
Al
2017-12-25 00:29:52 -05:00
a3f39be0d4[fix] reverting gazetteer changes as it would affect the parser features as well and require retraining
Al
2017-12-24 23:51:35 -05:00
acbebc9ecf[build] adding new source files for near dupe hashing and the command-line program to the Makefile
Al
2017-12-24 12:44:10 -05:00
f3a626463a[api] adding API functions for near dupe hashes to the public header
Al
2017-12-24 12:43:28 -05:00
8b75c44026[dedupe] adding a test program for near dupe hashing that simply prints out the results. Automated tests in the works
Al
2017-12-24 12:41:44 -05:00
acfdb50d7c[dedupe] adding near-dupe hashing function, which can be thought of as the blocking function in record linkage or as a form of locally sensitive hashing in general document deduping. The goal is, if two addresses/names are the same, they should share at least one hash. These hashes can also be used as an inverted index (DB, ES, hashtable, etc.). Uses the double metaphone for name words in Latin script (otherwise each individual token, and sequences of two tokens in the case of ideograms for e.g. Chinese, Japanese, Korean, etc.)
Al
2017-12-24 02:43:46 -05:00
6c6e5062e0[gazetteers] removing stopwords, etc. from numeric type components, adding street type expansions to name components
Al
2017-12-24 02:19:41 -05:00
c78566c241[utils] adding cstring_array_extend and string_tree_clear
Al
2017-12-24 01:46:20 -05:00
4e3d868bd0[parser] adding label constants to address_parser header
Al
2017-12-24 01:45:50 -05:00
3e554b8033[dictionaries] adding ambiguous expansions in English
Al
2017-12-24 00:53:24 -05:00
03c89bcf3c[dictionaries] adding "a" to English stopwords, "service" and "services" to English place names
Al
2017-12-23 19:49:05 -05:00
1fd5433bc5[dictionaries] adding associates/association to company types
Al
2017-12-23 19:36:03 -05:00
7d42c94b19[dictionaries] adding "for" to English stopword dictionaries
Al
2017-12-23 18:14:14 -05:00
dfc9064b0f[dictionaries] adding Stores to place names dictionary
Al
2017-12-23 18:13:46 -05:00
e432243256[dictionaries] adding "7 11" as a name for 7-eleven, even though it's completely numeric. Only affects the house/name component in deduping, so should be fine
Al
2017-12-19 15:11:47 -05:00
d03ce4e058[expand] remove blank expansions and strip spaces
Al
2017-12-18 18:17:16 -05:00
66b32ee377Merge pull request #289 from AeroXuk/FixWindowsLog
Al Barrentine
2017-12-18 17:42:33 -05:00
f6157224edRemoving console colors for Windows builds.
AeroXuk
2017-12-18 18:33:50 +00:00
ff3c7ab3b6[test] PO box expansion tests
Al
2017-12-17 22:12:37 -05:00
f63a9cc579[expand] adding number phrases as ignorable in PO boxes
Al
2017-12-17 22:12:12 -05:00
27f4eb2721[test] unit expansion tests
Al
2017-12-17 22:01:30 -05:00
f7326e52f6[test] level expansion tests
Al
2017-12-17 22:00:03 -05:00
727469b736[expand] no longer delete phrases in cases like "PH 1" for units, where there's a phrase that can accompany numbered units and thus be ignored similar to "Apt 1" but that phrase may also be a qualifier (i.e. Apt 1 and Penthouse 1 are not the same)
Al
2017-12-17 21:57:21 -05:00
1d22da603f[test] house number expansion tests
Al
2017-12-17 20:17:28 -05:00
bfdb6b8f87[test] adding header to fix warning
Al
2017-12-17 20:17:01 -05:00
26a6d9684d[test] adding tests for root-only expansions. Mostly English tests for the moment to deal with the various edge cases, but is also important for Spanish where "Calle" is so common that it's often omitted, same with French and "rue", etc.
Al
2017-12-17 20:00:48 -05:00
a1db4d7734[expand/normalize] the split_alpha_from_numeric option now applies to both e.g. A1 and 1A since we now strip out ordinal suffixes prior to normalization
Al
2017-12-17 19:53:11 -05:00
8b2a4d1ecf[api] adding libpostal_expand_address_root to the public API. This will attempt to delete tokens that can be safely ignored. It's deterministic and rule-based, but is informed by libpostal's fairly comprehensive dictionaries, and should work relatively well across languages for deduping purposes.
Al
2017-12-17 17:46:26 -05:00
9eef46adee[expand] in cases like "Avenue D" where there are two phrases, one is ambiguous (and canonical) but not necessarily edge-ignorable (pre/post-directional), allow deletion of the other token (so "Avenue" in this case). Also allows skipping in cases where the language classifier may predict a second language with some small probability, such as French for a short string like "Avenue D" (in addition to English). If the token was ignorable in the highest probability language, ignore it in both.
Al
2017-12-17 17:22:37 -05:00
3f7abd5b24[expand] adding a method that allows hash/equality comparisons of addresses like "100 Main" with "100 S Main St." or units like "Apt 101" vs. "#101". Instead of expanding the phrase abbreviations, this version tries its best to delete all but the root words in a string for a specific component. It's probably not perfect, but does handle a number of edge cases related to pre/post directionals in English e.g. "E St" will have a root word of simply "E", "Avenue E" => "E", etc. Also handles a variety of cases where the phrase could be a thoroughfare type but is really a root word such as "Park Pl" or the famous "Avenue Rd". This can be used for near dupe hashing to catch possible dupes for later analysis. Note that it will normalize "St Marks Pl" and "St Marks Ave" to the same thing, which is sometimes warranted (if the user typed the wrong thoroughfare), but can also be reconciled at deduping time.
Al
2017-12-17 15:48:07 -05:00
d0364ab6fb[expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation
Al
2017-12-17 03:14:00 -05:00
272ee3b965[dictionaries] adding a few more ambiguous expansions in English
Al
2017-12-17 03:11:46 -05:00
e1d89b5d62[auto][ci skip] Adding data files from Travis build #323
Travis
2017-12-12 14:57:29 +00:00
22ee778c8cMerge pull request #285 from dmvianna/master
Al Barrentine
2017-12-12 09:44:49 -05:00
daa9cb2896melbourne city toponym
Daniel Vianna
2017-12-12 09:33:44 +11:00
8968a6c966[expand] moving expand to its own module so the internal methods can be exposed, calling from libpostal.c
Al
2017-12-08 16:26:00 -05:00
e4e84f0147[utils] adding unicode_common_prefix/unicode_common_suffix, string_hyphen_prefix_len and string_hyphen_suffix_len to string_utils
Al
2017-12-08 14:28:30 -05:00
55ba627c3c[similarity] needed to add utf8proc_category and invert the indices for counting transposes in affine gap
Al
2017-12-08 14:27:23 -05:00
cfa5b1ce42[similarity] adding a stopword-aware acronym alignment method for matching U.N. with United Nations, Museum of Modern Art with MoMA, as well as things like University of California - Los Angeles with UCLA. All of these should work across languages, including non-Latin character sets like Cyrllic (but not ideograms as the concept doesn't make as much sense there). Skipping tokens like "of" or "the" depends only on the stopwords dictionary being defined for a given language.
Al
2017-12-04 15:21:09 -05:00
825408feffMerge pull request #283 from AeroXuk/AppVeyor_32bit
Al Barrentine
2017-12-01 15:10:12 -05:00
e740970d5aMerging upstream changes from openvenues/libpostal. Tweeked AppVeyor to package artifacts after setting /LARGEADDRESSAWARE on 32-bit build.
AeroXuk
2017-12-01 12:52:52 +00:00
252d5a0f37Merge branch 'master' into lieu_api
Al
2017-11-30 23:10:17 -05:00
363e13263eAdding make -j4 and setting appveyor to only build master branch.
AeroXuk
2017-12-01 00:22:35 +00:00
8b2f91477eMerge pull request #282 from openvenues/faster_windows_build
Al Barrentine
2017-11-30 19:13:06 -05:00
cf56da98f7[build] only build master on Appveyor so PRs don't trigger multiple builds
Al
2017-11-30 17:39:24 -05:00
02d049b8d9[build] also trying make -j4 to the Windows build. Set an option on the Appveyor side that will hopefully not build twice on pull requests
Al
2017-11-30 17:18:04 -05:00
133dce6f2cMerge pull request #280 from openvenues/faster_builds
Al Barrentine
2017-11-30 14:37:18 -05:00
f207a4680d[docs/build] adding make -j4 as the default for make, including in the Travis/Appveyor builds, should make build times faster
Al
2017-11-30 14:09:13 -05:00
e27f5f1d70[api] adding LIBPOSTAL_EXPORT to some of the new public API functions in this branch
Al
2017-11-29 18:10:10 -05:00
e35d6bf392Setup AppVeyor Build Matrix for 32 & 64 bit builds.
AeroXuk
2017-11-29 22:53:44 +00:00
ec4d683d1bMerge branch 'master' into lieu_api
Al
2017-11-29 15:49:52 -05:00
2cb0d146e5[docs][ci skip] adding a section for Windows installation, shoutouts to @BenK10 and @AeroXuk
Al
2017-11-29 13:12:50 -05:00
1c42df5ca9Merge pull request #279 from openvenues/drand48_fix
Al Barrentine
2017-11-29 13:04:24 -05:00
9e837f7209[fix] deleting comment, this is not a header-only implementation
Al
2017-11-29 12:26:36 -05:00
26e4ef08bc[fix] conditional compilation for strndup and drand48 for Windows, using config.h
Al
2017-11-29 12:21:13 -05:00
1a64ad682b[merge] merging in the Ohio expansion numex changes from master
Al
2017-11-29 11:51:25 -05:00
Al