Yaroslav Veremenko
|
c0b770101a
|
Removed final period
|
2018-03-07 17:51:34 -07:00 |
|
Yaroslav Veremenko
|
fd2b864b7d
|
Updated street types for EN
|
2018-03-07 12:48:30 -07:00 |
|
Al
|
6575bdc339
|
[dedupe/test] adding remaining options to near_dupe_test
|
2018-02-24 00:34:58 -05:00 |
|
Al
|
835de327c3
|
[dedupe] for near-dupe hashing, remove whitespace from root expansions so something like "Ocean Walk Dr" and "Oceanwalk Dr" will have a chance of matching downstream
|
2018-02-24 00:34:09 -05:00 |
|
Al
|
283be99b44
|
[numex] helper function to retrieve ordinal suffix lengths from a tokenized string for use in deduping
|
2018-02-24 00:31:26 -05:00 |
|
Al
|
b2dcb18d7e
|
[dedupe] account for missing ordinal suffixes in Soft-TFIDF deduping i.e. to count 1st Place and 1 Plce as the same where there might be a misspelling and the phrase wouldn't match under exact expansions
|
2018-02-23 23:44:05 -05:00 |
|
Al
|
b03fbdd681
|
[dedupe] adding multi-word phrase alignments to deduping
|
2018-02-23 01:28:02 -05:00 |
|
Al
|
591891951d
|
[utils] adding utf8 case insensitive comparison
|
2018-02-23 01:22:58 -05:00 |
|
Al
|
2b4e7073c2
|
[similarity] adding a multi-word alignmnet algorithm for streets and names like "de la cruz" vs. "dela cruz" or "Oceanwalk Ter" vs. "Ocean Walk Ter"
|
2018-02-23 01:22:12 -05:00 |
|
Al
|
c5bb9d8daa
|
[normalize/api] exposing normalize_string_languages and normalized_tokens_languages to the API for pre-normalizing numeric expressions at tokenization time
|
2018-02-22 18:47:36 -05:00 |
|
Al
|
0edb897143
|
[docs][ci skip] adding definition for the street_names.txt dictionary
|
2018-02-22 02:25:27 -05:00 |
|
Travis
|
cf2156c0e2
|
[auto][ci skip] Adding data files from Travis build #375
|
2018-02-22 03:32:16 +00:00 |
|
Al
|
afd225048a
|
[fix] DICTIONARY_STREET_NAME applies only to the street component, unlike synonyms, which would apply to any compnoent. This makes street names a good place to add synonyms found in streets that are not exactly thoroughfare types i.e. could not be removed from the string and have it retain more or less the same meaning.
|
2018-02-21 22:17:04 -05:00 |
|
Al
|
0f20613c13
|
[expand] using street name dictionaries as a possible root component instead of having to pollute the synonyms dictionary, which also affects the parser and might be a better place for general purpose synonyms affecting all components.
|
2018-02-21 22:16:07 -05:00 |
|
Al
|
ab67e0864a
|
[dictionaries] adding new street_names.txt dictionary and movign all the synonyms to there, generating the new dictionary type in address_dictionaries.py
|
2018-02-21 22:13:51 -05:00 |
|
Al
|
cd7bf81522
|
[dictionaries] reverting synonyms.txt back to previous commit
|
2018-02-21 22:13:12 -05:00 |
|
Travis
|
8075ee6f49
|
[auto][ci skip] Adding data files from Travis build #374
|
2018-02-22 01:21:23 +00:00 |
|
Al
|
a7b7568159
|
[dictionaries] removing "Corner" from the synonyms dictionaries for #320
|
2018-02-21 19:55:57 -05:00 |
|
Al
|
09408b1075
|
[fix] for regular expansion, use gazetteer components or overrides
|
2018-02-15 18:55:37 -05:00 |
|
Al
|
c5429de4d7
|
[fix] no gazetteer changes yet, breaks the parser
|
2018-02-15 18:52:16 -05:00 |
|
Al
|
ee2bac66d3
|
[build] rebuild dictionaries when gazteeer_data.c changes as well
|
2018-02-15 18:18:01 -05:00 |
|
Al
|
78d621ac85
|
[fix] adding street type gazetteer to name component as well for things like "24th St Cheese Co"
|
2018-02-15 18:13:27 -05:00 |
|
Al
|
9c12a11fd7
|
[fix] check expansion address components for regular expansion, overrides for root expansion
|
2018-02-15 16:19:50 -05:00 |
|
Al
|
9390e638ae
|
[fix] for regular non-root expansion, check that components are valid (for near-dupe expansions or other cases where component options are passed in)
|
2018-02-08 13:49:24 -05:00 |
|
Al
|
2290b0991e
|
[fix] transliteration case where a context no match comes at the end of the string
|
2018-02-06 15:08:11 -05:00 |
|
Al
|
156c8bed40
|
[fix] check that second double metaphone alternative is not the empty string
|
2018-02-06 03:08:37 -05:00 |
|
Al
|
3a5c048419
|
[fix] in root expansions, if the current phrase has at least one valid expansion, and the current expansion is not valid, ignore it
|
2018-02-06 02:36:05 -05:00 |
|
Al
|
8b96173a32
|
[fix] max dictionaries is now 5, weird that that wasn't committed by the Travis build
|
2018-02-06 01:58:15 -05:00 |
|
Travis
|
a9dded3a18
|
[auto][ci skip] Adding data files from Travis build #367
|
2018-02-06 01:43:25 +00:00 |
|
Al
|
a67df14854
|
[fix] adding newline to trigger new build
|
2018-02-05 20:30:21 -05:00 |
|
Al
|
86fdaf7188
|
[fix] changing root expansion test for "E Broadway", which now returns "broadway" instead of "east", which is a better result anyway
|
2018-02-05 19:36:01 -05:00 |
|
Al
|
0ee3f6b294
|
Merge branch 'master' of ssh://github.com/openvenues/libpostal
|
2018-02-05 19:32:17 -05:00 |
|
Al Barrentine
|
a6628e918f
|
Merge pull request #309 from antoine-de/small_fix
Fix typo in python imports
|
2018-02-05 19:30:55 -05:00 |
|
Al
|
e3a252d4eb
|
[dictionaries] adding most of the street types to synonyms/place_names in English so e.g. "Spring" in "Spring St" will not be removed in root expansions
|
2018-02-05 18:21:25 -05:00 |
|
Al
|
9baac813f6
|
[dictionaries] add "a" and "b" to the ambiguous expansions for English, randomly weren't in there
|
2018-02-05 18:18:00 -05:00 |
|
Al
|
4d3619d493
|
[dedupe] moving name-only near-dupe hashes to a separate if block so they can be used in conjunction with name+address
|
2018-02-03 14:03:17 -05:00 |
|
Al
|
7cb85aa23c
|
[dedupe] to make soft token similarity order invariant, we swap the order so the shorter token sequence comes first. In the case of a tie, pick the shorter full string length
|
2018-01-26 18:04:45 -05:00 |
|
Al
|
4aeb549054
|
[dedupe] with some term weighting schemes (especially information gain which will soon be the default in the lieu project), single letters may have very low weights such that they will be discarded, which can lead to false positives for things like "A & B" vs. "B & C", so add a simple heuristic to simply demote likely dupes to needs review when there's a positive symmetric difference (or whatever the set theory term is for when A - B and B - A are both non-empty)
|
2018-01-26 01:20:35 -05:00 |
|
Al
|
af5a5c3039
|
[dedupe] in the case of abbreviations and acronyms, where we use the higher of the two scores, calculate an offset to the norm of the other string's scores i.e. sincey we're replacing the score(s) in the lower-scoring vector with the higher one in the dot product for the numerator, do the same for the L2-norm product in the denominator. This way we don't accidentally inflate the similarity value simply because e.g. an acronym token was more rare than the same acronym spelled out as multiple individual letters (tend to be low-information/common tokens).
|
2018-01-25 16:32:47 -05:00 |
|
Al
|
13230824a2
|
[fix] Damerau-Levenshtein distance costing for transposes was off
|
2018-01-25 16:26:41 -05:00 |
|
Al
|
d0fe31d359
|
[dedupe] for strict abbreviations (defined as sharing a prefix and a suffix, and containing matches+gaps only by the subtotaling affine gap measure), using the greater of the two scores. This accounts for cases where the abbreviated version may have a much higher weight in one string than the non-abbreviated version does in the other. Same for acronym alignments. Making sure there's a common prefix in regular abbeviation detection Capping the Soft-TFIDF similarity at 1.0.
|
2018-01-25 14:23:18 -05:00 |
|
Al
|
b4cc7395a2
|
[fix] was missing some shorter tokens that are unicode equal in Soft-TFIDF
|
2018-01-25 04:23:35 -05:00 |
|
Al
|
c4aaee7dbf
|
[dedupe/similarity] also utilizing the L2 norm in similarity when acronyms are detected. Similarity in this case should be the acronym token's score * the L2 norm of the expanded tokens' scores in the longer string
|
2018-01-23 01:20:14 -05:00 |
|
antoine-de
|
1c13bf3678
|
fix typo in python imports
|
2018-01-22 18:39:27 +01:00 |
|
Al
|
ccce4f793f
|
[optimization] for the FTRL and SGD optimizers, use the new *_array_sum_sq function to do L2 regularization, vs. the L2 norm which will use the linear algebra meaning
|
2018-01-22 01:56:32 -05:00 |
|
Al
|
eb3fb37ad4
|
[similarity/dedupe] normalizing by the product of the L2 norms in soft token similarity function, as in cosine similarity. Score vectors should be passed in unnormalized, and typically with unit length. Also, for aligned phrases that share the same canonical phrase, contribute the product of the two norms of the phrase vectors to the similarity's numerator (maximum value, as if each token in both strings had matched exactly). The previous version over-counted the importance of aligned multi-word phrases by doing a cross product, which could overshadow other more important terms.
|
2018-01-22 01:38:12 -05:00 |
|
Al
|
2883b57eb3
|
[fix] sqrt in l2 norm
|
2018-01-22 01:07:52 -05:00 |
|
Al
|
3c5713ef59
|
[fix] check for sub-acronyms with no stopwords in near-dupe hashing
|
2018-01-21 17:49:46 -05:00 |
|
Al
|
fa40a7e87c
|
[fix] initialize repeat_state_end in transliteration. Couldn't reproduce the conditional jumps in #308, but this seems to be where they're occurring, and it's generally good practice to initialize things
|
2018-01-21 16:24:19 -05:00 |
|
Al
|
e4c35a0911
|
Merge branch 'master' of ssh://github.com/openvenues/libpostal
|
2018-01-21 01:21:11 -05:00 |
|