Al
|
b2dcb18d7e
|
[dedupe] account for missing ordinal suffixes in Soft-TFIDF deduping i.e. to count 1st Place and 1 Plce as the same where there might be a misspelling and the phrase wouldn't match under exact expansions
|
2018-02-23 23:44:05 -05:00 |
|
Al
|
b03fbdd681
|
[dedupe] adding multi-word phrase alignments to deduping
|
2018-02-23 01:28:02 -05:00 |
|
Al
|
4aeb549054
|
[dedupe] with some term weighting schemes (especially information gain which will soon be the default in the lieu project), single letters may have very low weights such that they will be discarded, which can lead to false positives for things like "A & B" vs. "B & C", so add a simple heuristic to simply demote likely dupes to needs review when there's a positive symmetric difference (or whatever the set theory term is for when A - B and B - A are both non-empty)
|
2018-01-26 01:20:35 -05:00 |
|
Al
|
179e6581e5
|
[dedupe] for fuzzy street duplicates, using the likely dupe classification regardless of similarity if one set of tokens is fuzzily-contained within the other (the set of matches allows for words matching Jaro-Winkler/Levenshtein similarity, being out of order, etc. but not acronyms or the more flexible abbreviation detection, just abbreviations out of libpostal's dictionaries provided that the abbreviations use share the same canonical phrase with the aligned phrase in the other string)
|
2018-01-06 03:55:20 -05:00 |
|
Al
|
7651a7b9b9
|
[fix] fixing a couple of warnings in dedupe/near_dupe
|
2017-12-31 19:20:17 -05:00 |
|
Al
|
4e32565746
|
[dedupe] fixing toponym matching for city-equivalents, adding the LIBPOSTAL_ADDRESS_ANY component in each function call so it can be removed as needed.
|
2017-12-30 18:05:46 -05:00 |
|
Al
|
6dff154a99
|
[api] adding APIs for getting default options and using a consistent naming convention
|
2017-12-29 17:48:54 -05:00 |
|
Al
|
098babfdee
|
[dedupe] adding the core pairwise deduping module which ties together most of the work on this branch. Includes simple phrase-aware exact deduping methods, with per-component variations as to whether e.g. a root expansion match counts as an exact duplicate or not (in a secondary unit, "No. 2" and "Apt 2" can be considered an exact match in English whereas we wouldn't want to make that kind of assumption for street e.g. "Park Ave" and "Park Pl"). The API is fairly low-level at present, and may require a few calls. Notably, we leave the TFIDF scores or other weighting schemes to the client. Since each component gets its own dupe classification, it leaves the door open for doing more specific checks around e.g. compound house numbers/ranges in the future.
|
2017-12-29 04:48:00 -05:00 |
|