Al
|
acbebc9ecf
|
[build] adding new source files for near dupe hashing and the command-line program to the Makefile
|
2017-12-24 12:44:10 -05:00 |
|
Al
|
f3a626463a
|
[api] adding API functions for near dupe hashes to the public header
|
2017-12-24 12:43:28 -05:00 |
|
Al
|
8b75c44026
|
[dedupe] adding a test program for near dupe hashing that simply prints out the results. Automated tests in the works
|
2017-12-24 12:41:44 -05:00 |
|
Al
|
acfdb50d7c
|
[dedupe] adding near-dupe hashing function, which can be thought of as the blocking function in record linkage or as a form of locally sensitive hashing in general document deduping. The goal is, if two addresses/names are the same, they should share at least one hash. These hashes can also be used as an inverted index (DB, ES, hashtable, etc.). Uses the double metaphone for name words in Latin script (otherwise each individual token, and sequences of two tokens in the case of ideograms for e.g. Chinese, Japanese, Korean, etc.)
|
2017-12-24 02:47:45 -05:00 |
|
Al
|
6c6e5062e0
|
[gazetteers] removing stopwords, etc. from numeric type components, adding street type expansions to name components
|
2017-12-24 02:19:41 -05:00 |
|
Al
|
c78566c241
|
[utils] adding cstring_array_extend and string_tree_clear
|
2017-12-24 01:46:20 -05:00 |
|
Al
|
4e3d868bd0
|
[parser] adding label constants to address_parser header
|
2017-12-24 01:45:50 -05:00 |
|
Al
|
3e554b8033
|
[dictionaries] adding ambiguous expansions in English
|
2017-12-24 00:53:24 -05:00 |
|
Al
|
03c89bcf3c
|
[dictionaries] adding "a" to English stopwords, "service" and "services" to English place names
|
2017-12-23 19:49:05 -05:00 |
|
Al
|
1fd5433bc5
|
[dictionaries] adding associates/association to company types
|
2017-12-23 19:36:03 -05:00 |
|
Al
|
7d42c94b19
|
[dictionaries] adding "for" to English stopword dictionaries
|
2017-12-23 18:14:14 -05:00 |
|
Al
|
dfc9064b0f
|
[dictionaries] adding Stores to place names dictionary
|
2017-12-23 18:13:46 -05:00 |
|
Al
|
e432243256
|
[dictionaries] adding "7 11" as a name for 7-eleven, even though it's completely numeric. Only affects the house/name component in deduping, so should be fine
|
2017-12-19 15:11:47 -05:00 |
|
Al
|
d03ce4e058
|
[expand] remove blank expansions and strip spaces
|
2017-12-18 18:17:16 -05:00 |
|
Al Barrentine
|
66b32ee377
|
Merge pull request #289 from AeroXuk/FixWindowsLog
Removing console colors for Windows builds
|
2017-12-18 17:42:33 -05:00 |
|
AeroXuk
|
f6157224ed
|
Removing console colors for Windows builds.
|
2017-12-18 18:33:50 +00:00 |
|
Al
|
ff3c7ab3b6
|
[test] PO box expansion tests
|
2017-12-17 22:12:37 -05:00 |
|
Al
|
f63a9cc579
|
[expand] adding number phrases as ignorable in PO boxes
|
2017-12-17 22:12:12 -05:00 |
|
Al
|
27f4eb2721
|
[test] unit expansion tests
|
2017-12-17 22:01:30 -05:00 |
|
Al
|
f7326e52f6
|
[test] level expansion tests
|
2017-12-17 22:00:03 -05:00 |
|
Al
|
727469b736
|
[expand] no longer delete phrases in cases like "PH 1" for units, where there's a phrase that can accompany numbered units and thus be ignored similar to "Apt 1" but that phrase may also be a qualifier (i.e. Apt 1 and Penthouse 1 are not the same)
|
2017-12-17 21:57:25 -05:00 |
|
Al
|
1d22da603f
|
[test] house number expansion tests
|
2017-12-17 20:17:28 -05:00 |
|
Al
|
bfdb6b8f87
|
[test] adding header to fix warning
|
2017-12-17 20:17:01 -05:00 |
|
Al
|
26a6d9684d
|
[test] adding tests for root-only expansions. Mostly English tests for the moment to deal with the various edge cases, but is also important for Spanish where "Calle" is so common that it's often omitted, same with French and "rue", etc.
|
2017-12-17 20:00:48 -05:00 |
|
Al
|
a1db4d7734
|
[expand/normalize] the split_alpha_from_numeric option now applies to both e.g. A1 and 1A since we now strip out ordinal suffixes prior to normalization
|
2017-12-17 19:53:15 -05:00 |
|
Al
|
8b2a4d1ecf
|
[api] adding libpostal_expand_address_root to the public API. This will attempt to delete tokens that can be safely ignored. It's deterministic and rule-based, but is informed by libpostal's fairly comprehensive dictionaries, and should work relatively well across languages for deduping purposes.
|
2017-12-17 17:46:26 -05:00 |
|
Al
|
9eef46adee
|
[expand] in cases like "Avenue D" where there are two phrases, one is ambiguous (and canonical) but not necessarily edge-ignorable (pre/post-directional), allow deletion of the other token (so "Avenue" in this case). Also allows skipping in cases where the language classifier may predict a second language with some small probability, such as French for a short string like "Avenue D" (in addition to English). If the token was ignorable in the highest probability language, ignore it in both.
|
2017-12-17 17:24:27 -05:00 |
|
Al
|
3f7abd5b24
|
[expand] adding a method that allows hash/equality comparisons of addresses like "100 Main" with "100 S Main St." or units like "Apt 101" vs. "#101". Instead of expanding the phrase abbreviations, this version tries its best to delete all but the root words in a string for a specific component. It's probably not perfect, but does handle a number of edge cases related to pre/post directionals in English e.g. "E St" will have a root word of simply "E", "Avenue E" => "E", etc. Also handles a variety of cases where the phrase could be a thoroughfare type but is really a root word such as "Park Pl" or the famous "Avenue Rd". This can be used for near dupe hashing to catch possible dupes for later analysis. Note that it will normalize "St Marks Pl" and "St Marks Ave" to the same thing, which is sometimes warranted (if the user typed the wrong thoroughfare), but can also be reconciled at deduping time.
|
2017-12-17 15:48:11 -05:00 |
|
Al
|
d0364ab6fb
|
[expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation
|
2017-12-17 03:14:00 -05:00 |
|
Al
|
272ee3b965
|
[dictionaries] adding a few more ambiguous expansions in English
|
2017-12-17 03:11:46 -05:00 |
|
Travis
|
e1d89b5d62
|
[auto][ci skip] Adding data files from Travis build #323
|
2017-12-12 14:57:29 +00:00 |
|
Al Barrentine
|
22ee778c8c
|
Merge pull request #285 from dmvianna/master
melbourne city toponym
|
2017-12-12 09:44:49 -05:00 |
|
Daniel Vianna
|
daa9cb2896
|
melbourne city toponym
|
2017-12-12 09:33:44 +11:00 |
|
Al
|
8968a6c966
|
[expand] moving expand to its own module so the internal methods can be exposed, calling from libpostal.c
|
2017-12-08 16:26:13 -05:00 |
|
Al
|
e4e84f0147
|
[utils] adding unicode_common_prefix/unicode_common_suffix, string_hyphen_prefix_len and string_hyphen_suffix_len to string_utils
|
2017-12-08 14:28:30 -05:00 |
|
Al
|
55ba627c3c
|
[similarity] needed to add utf8proc_category and invert the indices for counting transposes in affine gap
|
2017-12-08 14:27:23 -05:00 |
|
Al
|
cfa5b1ce42
|
[similarity] adding a stopword-aware acronym alignment method for matching U.N. with United Nations, Museum of Modern Art with MoMA, as well as things like University of California - Los Angeles with UCLA. All of these should work across languages, including non-Latin character sets like Cyrllic (but not ideograms as the concept doesn't make as much sense there). Skipping tokens like "of" or "the" depends only on the stopwords dictionary being defined for a given language.
|
2017-12-04 15:21:44 -05:00 |
|
Al Barrentine
|
825408feff
|
Merge pull request #283 from AeroXuk/AppVeyor_32bit
Setup AppVeyor Build Matrix for 32 & 64 bit builds.
|
2017-12-01 15:10:12 -05:00 |
|
AeroXuk
|
e740970d5a
|
Merging upstream changes from openvenues/libpostal. Tweeked AppVeyor to package artifacts after setting /LARGEADDRESSAWARE on 32-bit build.
|
2017-12-01 12:52:52 +00:00 |
|
Al
|
252d5a0f37
|
Merge branch 'master' into lieu_api
|
2017-11-30 23:10:17 -05:00 |
|
AeroXuk
|
363e13263e
|
Adding make -j4 and setting appveyor to only build master branch.
|
2017-12-01 00:22:35 +00:00 |
|
Al Barrentine
|
8b2f91477e
|
Merge pull request #282 from openvenues/faster_windows_build
adding make -j4 to Windows build
|
2017-11-30 19:13:06 -05:00 |
|
Al
|
cf56da98f7
|
[build] only build master on Appveyor so PRs don't trigger multiple builds
|
2017-11-30 17:39:24 -05:00 |
|
Al
|
02d049b8d9
|
[build] also trying make -j4 to the Windows build. Set an option on the Appveyor side that will hopefully not build twice on pull requests
|
2017-11-30 17:18:04 -05:00 |
|
Al Barrentine
|
133dce6f2c
|
Merge pull request #280 from openvenues/faster_builds
make -j4 for all builds
|
2017-11-30 14:37:18 -05:00 |
|
Al
|
f207a4680d
|
[docs/build] adding make -j4 as the default for make, including in the Travis/Appveyor builds, should make build times faster
|
2017-11-30 14:09:13 -05:00 |
|
Al
|
e27f5f1d70
|
[api] adding LIBPOSTAL_EXPORT to some of the new public API functions in this branch
|
2017-11-29 18:10:10 -05:00 |
|
AeroXuk
|
e35d6bf392
|
Setup AppVeyor Build Matrix for 32 & 64 bit builds.
|
2017-11-29 22:53:44 +00:00 |
|
Al
|
ec4d683d1b
|
Merge branch 'master' into lieu_api
|
2017-11-29 15:49:52 -05:00 |
|
Al
|
2cb0d146e5
|
[docs][ci skip] adding a section for Windows installation, shoutouts to @BenK10 and @AeroXuk
|
2017-11-29 13:13:00 -05:00 |
|