Al
|
c29557c16b
|
[expand] adding another check in root expansions, making sure we don't ignore the unmodified ambiguous phrase
|
2018-01-08 19:03:50 -05:00 |
|
Al
|
e6edf54adb
|
[dedupe] adding a near-dupe hash for the entire name without spaces.
|
2018-01-08 19:02:19 -05:00 |
|
Al
|
66aee0fffa
|
[expand] make street type dictionaries ignorable for venue names as well (many company names mention their address, so sort of have to apply the same rules)
|
2018-01-07 01:39:10 -05:00 |
|
Al
|
e935f2a036
|
[fix] need to calculate max Jaro-Winkler for other methods, so only test whether we should use it after we've cycled through all the tokens
|
2018-01-06 03:59:34 -05:00 |
|
Al
|
179e6581e5
|
[dedupe] for fuzzy street duplicates, using the likely dupe classification regardless of similarity if one set of tokens is fuzzily-contained within the other (the set of matches allows for words matching Jaro-Winkler/Levenshtein similarity, being out of order, etc. but not acronyms or the more flexible abbreviation detection, just abbreviations out of libpostal's dictionaries provided that the abbreviations use share the same canonical phrase with the aligned phrase in the other string)
|
2018-01-06 03:55:20 -05:00 |
|
Al
|
4356174630
|
[similarity] adding a match count in Soft-TFIDF to allow answering questions about subsets i.e. the set of tokens in "Park Pl" contain the set of tokens in "Park". Setting Jaro-Winkler minimum length of 4 chars on, more specific option name for possible abbeviation detection
|
2018-01-06 03:50:03 -05:00 |
|
Al
|
0cb488ecea
|
[test] different test case for expansion, male names are overemphasized as it is
|
2018-01-02 19:01:58 -05:00 |
|
Al Barrentine
|
513fdd775f
|
Merge pull request #297 from oschwald/greg/fix-more-leaks
Fix several more memory leaks and a segfault
|
2018-01-02 18:50:38 -05:00 |
|
Gregory Oschwald
|
2f6749fe03
|
Fix segfault in expand_alternative_phrase_option
string_tree_get_alternative can return NULL
|
2018-01-02 13:28:51 -08:00 |
|
Gregory Oschwald
|
18cc0e37e6
|
Only create parser response when it is used
Previously, an unused response would not be freed, causing a leak.
|
2018-01-02 11:56:02 -08:00 |
|
Gregory Oschwald
|
95ea873498
|
Fix memory leaks in test_trie
The primary motivation is to make the test suite run clean under
Valgrind so that we don't need to wade through unimportant failures.
|
2018-01-02 10:38:12 -08:00 |
|
Gregory Oschwald
|
999de2bf6a
|
Add missing char_array_destroy when numex is invalid
|
2018-01-02 10:24:39 -08:00 |
|
Al
|
d33b6693b9
|
[test] adding Suite/Ste tests for root expansion bugfix
|
2018-01-02 03:50:23 -05:00 |
|
Al
|
071aee0e85
|
[fix] in root expansions, removing phrases that are invalid for the given components if there are other ignorable components
|
2018-01-02 03:49:52 -05:00 |
|
Al
|
d8a0a344cd
|
Merge branch 'master' of ssh://github.com/openvenues/libpostal
|
2017-12-31 19:20:42 -05:00 |
|
Al
|
7651a7b9b9
|
[fix] fixing a couple of warnings in dedupe/near_dupe
|
2017-12-31 19:20:17 -05:00 |
|
Travis
|
95ea250cb1
|
[auto][ci skip] Adding data files from Travis build #330
|
2017-12-31 22:30:34 +00:00 |
|
Al Barrentine
|
8a917d8594
|
Merge pull request #294 from openvenues/lieu_api
Near-duplicate detection and address deduping
|
2017-12-31 17:00:03 -05:00 |
|
Al
|
3bdb8c8630
|
[similarity] max out the Jaro-Winkler shared prefix at 4 characters in accordance with Winkler's paper
|
2017-12-31 13:23:54 -05:00 |
|
Al
|
4e32565746
|
[dedupe] fixing toponym matching for city-equivalents, adding the LIBPOSTAL_ADDRESS_ANY component in each function call so it can be removed as needed.
|
2017-12-30 18:05:46 -05:00 |
|
Al
|
34c3ee7f7a
|
[fix] update to struct
|
2017-12-30 03:24:39 -05:00 |
|
Al
|
34fe7ec305
|
[expand] adding a few of the address phrase checks to the expand header
|
2017-12-30 02:34:06 -05:00 |
|
Al
|
668e467967
|
[dedupe/test] checking for NULL in near_dupe test program
|
2017-12-30 02:33:33 -05:00 |
|
Al
|
3263c84b32
|
[api] using uint32_t for geohash precision option
|
2017-12-30 02:32:35 -05:00 |
|
Al
|
434bbd4dc2
|
[fix] removing unused vars
|
2017-12-30 02:31:43 -05:00 |
|
Al
|
86d5eca521
|
[api] checking for NULL responses in the cstring_array methods before converting them to char arrays
|
2017-12-30 02:31:25 -05:00 |
|
Al
|
c48c2b778c
|
[dedupe] fixes to near dupe hashing, geohash lengths, cutting off name hashing at 50 unique tokens, fixing memory leaks, checking for valid geo components and returning NULL if one of the required fields isn't present
|
2017-12-30 02:28:38 -05:00 |
|
Al
|
6dff154a99
|
[api] adding APIs for getting default options and using a consistent naming convention
|
2017-12-29 17:48:54 -05:00 |
|
Al
|
53543be5a5
|
[build] adding new source files to Makefile for the lieu APIs
|
2017-12-29 17:46:35 -05:00 |
|
Al
|
8495cda1eb
|
[api] adding pairwise-dupe functions/structs to the public header
|
2017-12-29 13:48:54 -05:00 |
|
Al
|
cadf52d19f
|
[fix] making a few internal functions static
|
2017-12-29 04:50:08 -05:00 |
|
Al
|
098babfdee
|
[dedupe] adding the core pairwise deduping module which ties together most of the work on this branch. Includes simple phrase-aware exact deduping methods, with per-component variations as to whether e.g. a root expansion match counts as an exact duplicate or not (in a secondary unit, "No. 2" and "Apt 2" can be considered an exact match in English whereas we wouldn't want to make that kind of assumption for street e.g. "Park Ave" and "Park Pl"). The API is fairly low-level at present, and may require a few calls. Notably, we leave the TFIDF scores or other weighting schemes to the client. Since each component gets its own dupe classification, it leaves the door open for doing more specific checks around e.g. compound house numbers/ranges in the future.
|
2017-12-29 04:48:00 -05:00 |
|
Al
|
1f1412c120
|
[api] adding libpostal_place_languages method to public API for classifying languages consistently from components (may need to make several calls using the same languages and don't necessarily want the language classifier to be run on house numbers when we already know the languages from e.g. the street name - this provides a simple window into the language classifier focused on the entire address/record
|
2017-12-29 03:32:41 -05:00 |
|
Al
|
1d1ce10fad
|
[similarity] adding a string array version of Jaccard similarity that creates the string sets internally for convenience
|
2017-12-29 03:08:48 -05:00 |
|
Al
|
c5ad080fb0
|
[similarity] moving stopword tokens array to a separate function in acronym token alignments
|
2017-12-29 02:42:22 -05:00 |
|
Al
|
f1e6886536
|
[similarity/dedupe] adding options for acronym alignments and address phrase matches in Soft-TFIDF. Acronym alignments will give higher similarity to NYU vs. "New York University" whereas phrase matches would match known phrases that share the same canonical like "Cty Rd" vs. "C.R." vs. "County Road" within the Soft-TFIDF similarity calculation.
|
2017-12-29 02:39:49 -05:00 |
|
Al
|
24a77ea03f
|
[fix] another valgrind error in counting transposes in our counting affine gap implementation (mixed indices)
|
2017-12-29 00:04:24 -05:00 |
|
Al
|
cabdbfccd2
|
[fix] using same order in root expansions
|
2017-12-28 23:55:41 -05:00 |
|
Al
|
8fd4242eb8
|
[fix] bug in Jaro distance
|
2017-12-28 23:54:10 -05:00 |
|
Al
|
b90c3dab4b
|
[similarity/dedupe] adding Soft-TFIDF implementation with several different fallback qualifiers for the max-sim function (Damerau-Levenshtein and libpostal's new bucketed affine gap method for detecting abbreviations), but keeping Jaro-Winkler as the secondary similarity function in the final distance metric. Overall this should results in higher similarity values when one of the tokens may not quite match the pure secondary threshold in terms of Jaro-Winkler but may match on one of the other criteria.
|
2017-12-28 04:34:46 -05:00 |
|
Al
|
33bb90d94b
|
[utils] adding place.h header, which converts parser-like output into an object that can be used for comparisons. Currently single-value, but could use cstring_arrays for fields instead.
|
2017-12-27 22:13:04 -05:00 |
|
Al
|
d731339811
|
[expand] fixing case where too many permutations were getting added for longer strings due to the new-ish ordinal suffix handling, using string_tree_num_tokens instead of string_tree_num_strings throughout to check for previously added words, using new is_likely_roman_numeral API
|
2017-12-27 21:48:54 -05:00 |
|
Al
|
b4fdc51bf9
|
[numex] changing is_roman_numeral to is_likely_roman_numeral to get rid of most of the false positives like \"La\" in Spanish which could be L(=50) + the ordinal suffix \"a\", but in practice it never means that. For Roman numerals that are shorter than two characters (whether on their own like "DC" or "MD", or attached to a potential ordinal suffix like \"Ce\" in French), will be ignored unless they're composed of more likely, smaller, Roman numerals: I, V, and X, so VI, IX, etc. are expanded as Roman numerals but LI is not.
|
2017-12-27 19:38:02 -05:00 |
|
Al
|
b17b2bdcc4
|
[dictionaries] adding hill/hills to synonyms lists in English. In general any ambiguous street types that can also be part of a core street name can also be stored in synonyms
|
2017-12-27 19:12:01 -05:00 |
|
Al
|
bb9f6a4c6b
|
Merge branch 'master' into lieu_api
|
2017-12-26 10:47:29 -05:00 |
|
Al Barrentine
|
1e7cc23b81
|
Merge pull request #292 from oschwald/greg/fix-leak
Fix leak of normalized value in early return
|
2017-12-26 10:44:27 -05:00 |
|
Gregory Oschwald
|
1bb6278446
|
Fix leak of normalized value in early return
|
2017-12-25 19:33:07 -08:00 |
|
Al
|
2afcd74779
|
[test] adding E Ctr St tests
|
2017-12-25 01:38:50 -05:00 |
|
Al
|
152761fcbc
|
[expand] adding improvements to root expansions (using possible phrase roots even if they're abbreviated e.g. "E Ctr St", adding special valid components check for root expansions beyond what's stored in the build address dictionaries), removing spaces before checking unique strings, only splitting numeric from alpha in the case of non-ordinals, using cstring_array internally and char ** in the public API
|
2017-12-25 01:37:42 -05:00 |
|
Al
|
b4ce042f80
|
[dictionaries] removing ave/avens/aves from ambiguous
|
2017-12-25 00:29:52 -05:00 |
|