Al
|
7cb85aa23c
|
[dedupe] to make soft token similarity order invariant, we swap the order so the shorter token sequence comes first. In the case of a tie, pick the shorter full string length
|
2018-01-26 18:04:45 -05:00 |
|
Al
|
4aeb549054
|
[dedupe] with some term weighting schemes (especially information gain which will soon be the default in the lieu project), single letters may have very low weights such that they will be discarded, which can lead to false positives for things like "A & B" vs. "B & C", so add a simple heuristic to simply demote likely dupes to needs review when there's a positive symmetric difference (or whatever the set theory term is for when A - B and B - A are both non-empty)
|
2018-01-26 01:20:35 -05:00 |
|
Al
|
af5a5c3039
|
[dedupe] in the case of abbreviations and acronyms, where we use the higher of the two scores, calculate an offset to the norm of the other string's scores i.e. sincey we're replacing the score(s) in the lower-scoring vector with the higher one in the dot product for the numerator, do the same for the L2-norm product in the denominator. This way we don't accidentally inflate the similarity value simply because e.g. an acronym token was more rare than the same acronym spelled out as multiple individual letters (tend to be low-information/common tokens).
|
2018-01-25 16:32:47 -05:00 |
|
Al
|
13230824a2
|
[fix] Damerau-Levenshtein distance costing for transposes was off
|
2018-01-25 16:26:41 -05:00 |
|
Al
|
d0fe31d359
|
[dedupe] for strict abbreviations (defined as sharing a prefix and a suffix, and containing matches+gaps only by the subtotaling affine gap measure), using the greater of the two scores. This accounts for cases where the abbreviated version may have a much higher weight in one string than the non-abbreviated version does in the other. Same for acronym alignments. Making sure there's a common prefix in regular abbeviation detection Capping the Soft-TFIDF similarity at 1.0.
|
2018-01-25 14:23:18 -05:00 |
|
Al
|
b4cc7395a2
|
[fix] was missing some shorter tokens that are unicode equal in Soft-TFIDF
|
2018-01-25 04:23:35 -05:00 |
|
Al
|
c4aaee7dbf
|
[dedupe/similarity] also utilizing the L2 norm in similarity when acronyms are detected. Similarity in this case should be the acronym token's score * the L2 norm of the expanded tokens' scores in the longer string
|
2018-01-23 01:20:14 -05:00 |
|
Al
|
ccce4f793f
|
[optimization] for the FTRL and SGD optimizers, use the new *_array_sum_sq function to do L2 regularization, vs. the L2 norm which will use the linear algebra meaning
|
2018-01-22 01:56:32 -05:00 |
|
Al
|
eb3fb37ad4
|
[similarity/dedupe] normalizing by the product of the L2 norms in soft token similarity function, as in cosine similarity. Score vectors should be passed in unnormalized, and typically with unit length. Also, for aligned phrases that share the same canonical phrase, contribute the product of the two norms of the phrase vectors to the similarity's numerator (maximum value, as if each token in both strings had matched exactly). The previous version over-counted the importance of aligned multi-word phrases by doing a cross product, which could overshadow other more important terms.
|
2018-01-22 01:38:12 -05:00 |
|
Al
|
2883b57eb3
|
[fix] sqrt in l2 norm
|
2018-01-22 01:07:52 -05:00 |
|
Al
|
3c5713ef59
|
[fix] check for sub-acronyms with no stopwords in near-dupe hashing
|
2018-01-21 17:49:46 -05:00 |
|
Al
|
fa40a7e87c
|
[fix] initialize repeat_state_end in transliteration. Couldn't reproduce the conditional jumps in #308, but this seems to be where they're occurring, and it's generally good practice to initialize things
|
2018-01-21 16:24:19 -05:00 |
|
Al
|
e4c35a0911
|
Merge branch 'master' of ssh://github.com/openvenues/libpostal
|
2018-01-21 01:21:11 -05:00 |
|
Al
|
dc8bffd5a0
|
[fix] expansion test valgrind errors for #308
|
2018-01-21 01:20:40 -05:00 |
|
Al Barrentine
|
48ea0d7801
|
[docs][ci skip] Adding issue template
|
2018-01-20 02:29:23 -05:00 |
|
Al
|
984235e879
|
[fix] load transliteration for language classifier cli for #302
|
2018-01-19 14:24:08 -05:00 |
|
Al
|
7121642c62
|
[dedupe] fixing sub-acronym near-dupe hashes with punctuation, and making sure to add the current token after a new sub-acronym has been cut
|
2018-01-18 00:11:21 -05:00 |
|
Al
|
95e483e3ca
|
[fix] case-insensitive comparison of content-length header in data download script
|
2018-01-17 17:31:42 -05:00 |
|
Al Barrentine
|
3e3558e199
|
Merge pull request #303 from Maurice-Betzel/master
Added unofficial Java language binding
|
2018-01-17 13:17:30 -05:00 |
|
Maurice Betzel
|
46585c89e0
|
Added unofficial Java language binding
|
2018-01-17 11:00:34 +01:00 |
|
Maurice Betzel
|
ebc332a311
|
Added unofficially supported Java language binding
|
2018-01-17 10:57:09 +01:00 |
|
Travis
|
a4793f0f79
|
[auto][ci skip] Adding data files from Travis build #343
|
2018-01-16 09:30:46 +00:00 |
|
Al
|
ac350d90f6
|
[dictionaries] adding BYP to ambiguous expansions for the Black Youth Project
|
2018-01-16 03:40:36 -05:00 |
|
Al
|
03e5e25240
|
[dedupe] adding a near-dupe hash which takes into account existing acronyms which may have appeared in the string, either known acronyms as defined in the dictionaries like "HS" and includes the full token in the acronym. This feature is particularly useful for public schools or other cases where the canonical string may be used i.e. "Foo High School", "Foo HS" and "FHS". It also does the same thing other acronyms that are identified by the tokenizer from the internal period structure like A.B.C. Also now allowing mixed alpha-numeric tokens to use the double metaphone encoding as well, and for numeric tokens with script=Common (digits but may also contain hyphens, etc.), the full token is included as one of the words rather than quadgrams, which don't make sense for numerics.
|
2018-01-16 03:38:22 -05:00 |
|
Al
|
0286a2fef3
|
[expand] for root expansions, delete ambiguous tokens only when there's a non-numeric non-phrase token present. This applies to all name components, not for components where numerics can be the root (house numbers, units, streets, etc.)
|
2018-01-16 03:02:26 -05:00 |
|
Al
|
0ee18b4f6c
|
[dedupe] adding a function to acronyms module to detect existing/known acronyms like MS for middle school, HS for high school, etc. Forms like MS have to be deined in the dictionaries specifically but any acronym written like M.S. will be detected as such by the tokenizer
|
2018-01-15 23:47:16 -05:00 |
|
Al
|
133381f439
|
Merge branch 'master' of ssh://github.com/openvenues/libpostal
|
2018-01-14 19:40:44 -05:00 |
|
Al
|
c553fe81ee
|
[dedupe] using 4-grams with no edge disambiguation in near dupe hashing of names instead of full tokens (uses the double metaphone for Latin script, 2-grams for ideographic scripts and 4-gram unicode chars for other scripts like Arabic or Cyrllic). The fully concatenated name string with no whitespace + acronyms/subacronyms now also use double-metaphone in Latin script, and are split into 4-grams. Overall this reduces the number of keys, accounts for more misspellings as well as languages with longer words such as German, and various spacing/concatenations differences in general, while still being relatively selective. Most words in Latin scripts will resolve to less than 4 characters, so this mostly affects longer words with many consonants. 4-gram blocking tends to be what's used in the literature, and works well in practice on human and venue names. This is a slight departure from said literature in that we use 4-grams of the phonetic normalization for Latin scripts.
|
2018-01-14 19:12:28 -05:00 |
|
Travis
|
c58b16e745
|
[auto][ci skip] Adding data files from Travis build #341
|
2018-01-11 23:35:43 +00:00 |
|
Al Barrentine
|
b2aa2e4fdd
|
Merge pull request #299 from antimirov/master
Updating Ukrainian dictionaries, small fixes and additional variants.
|
2018-01-11 18:22:34 -05:00 |
|
Al
|
f5e41a1f57
|
[fix] logic in sub-acronym generation for near-dupe hashes
|
2018-01-11 13:15:19 -05:00 |
|
Al
|
6ba0403748
|
[dedupe] adding a near-dupe hash for acronyms both with and without stopwords. This will create basic acronyms for institutions like MoMA, UCLA, the NAACP, as well as human initials, etc. It also handles sub-acronyms, so when either at every other non-contiguous stopword (University of Texas at Austin) or punctuation (University of Texas, Austin), it cuts a new sub-acronym (so UT). All of the acronyms for Latin script use a double metaphone as well, so can potentially catch many cases. It does not handle all possible acronyms (e.g. where some of the letters are word-internal as in medical acronyms), but should do relatively well on many common variations.
|
2018-01-10 22:23:40 -05:00 |
|
Yevgen Antymyrov
|
b1f3760a81
|
Updating Ukrainian dictionaries, small fixes and additional variants.
|
2018-01-10 11:35:59 +01:00 |
|
Al
|
c29557c16b
|
[expand] adding another check in root expansions, making sure we don't ignore the unmodified ambiguous phrase
|
2018-01-08 19:03:50 -05:00 |
|
Al
|
e6edf54adb
|
[dedupe] adding a near-dupe hash for the entire name without spaces.
|
2018-01-08 19:02:19 -05:00 |
|
Al
|
66aee0fffa
|
[expand] make street type dictionaries ignorable for venue names as well (many company names mention their address, so sort of have to apply the same rules)
|
2018-01-07 01:39:10 -05:00 |
|
Al
|
e935f2a036
|
[fix] need to calculate max Jaro-Winkler for other methods, so only test whether we should use it after we've cycled through all the tokens
|
2018-01-06 03:59:34 -05:00 |
|
Al
|
179e6581e5
|
[dedupe] for fuzzy street duplicates, using the likely dupe classification regardless of similarity if one set of tokens is fuzzily-contained within the other (the set of matches allows for words matching Jaro-Winkler/Levenshtein similarity, being out of order, etc. but not acronyms or the more flexible abbreviation detection, just abbreviations out of libpostal's dictionaries provided that the abbreviations use share the same canonical phrase with the aligned phrase in the other string)
|
2018-01-06 03:55:20 -05:00 |
|
Al
|
4356174630
|
[similarity] adding a match count in Soft-TFIDF to allow answering questions about subsets i.e. the set of tokens in "Park Pl" contain the set of tokens in "Park". Setting Jaro-Winkler minimum length of 4 chars on, more specific option name for possible abbeviation detection
|
2018-01-06 03:50:03 -05:00 |
|
Al
|
0cb488ecea
|
[test] different test case for expansion, male names are overemphasized as it is
|
2018-01-02 19:01:58 -05:00 |
|
Al Barrentine
|
513fdd775f
|
Merge pull request #297 from oschwald/greg/fix-more-leaks
Fix several more memory leaks and a segfault
|
2018-01-02 18:50:38 -05:00 |
|
Gregory Oschwald
|
2f6749fe03
|
Fix segfault in expand_alternative_phrase_option
string_tree_get_alternative can return NULL
|
2018-01-02 13:28:51 -08:00 |
|
Gregory Oschwald
|
18cc0e37e6
|
Only create parser response when it is used
Previously, an unused response would not be freed, causing a leak.
|
2018-01-02 11:56:02 -08:00 |
|
Gregory Oschwald
|
95ea873498
|
Fix memory leaks in test_trie
The primary motivation is to make the test suite run clean under
Valgrind so that we don't need to wade through unimportant failures.
|
2018-01-02 10:38:12 -08:00 |
|
Gregory Oschwald
|
999de2bf6a
|
Add missing char_array_destroy when numex is invalid
|
2018-01-02 10:24:39 -08:00 |
|
Al
|
d33b6693b9
|
[test] adding Suite/Ste tests for root expansion bugfix
|
2018-01-02 03:50:23 -05:00 |
|
Al
|
071aee0e85
|
[fix] in root expansions, removing phrases that are invalid for the given components if there are other ignorable components
|
2018-01-02 03:49:52 -05:00 |
|
Al
|
d8a0a344cd
|
Merge branch 'master' of ssh://github.com/openvenues/libpostal
|
2017-12-31 19:20:42 -05:00 |
|
Al
|
7651a7b9b9
|
[fix] fixing a couple of warnings in dedupe/near_dupe
|
2017-12-31 19:20:17 -05:00 |
|
Travis
|
95ea250cb1
|
[auto][ci skip] Adding data files from Travis build #330
|
2017-12-31 22:30:34 +00:00 |
|