tommy/libpostal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,122 Commits 2 Branches 0 Tags
1d22da603f5970ba796b041f4f76dcfd5e98f31c
Commit Graph

5122 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Al
1d22da603f [test] house number expansion tests 2017-12-17 20:17:28 -05:00
Al
bfdb6b8f87 [test] adding header to fix warning 2017-12-17 20:17:01 -05:00
Al
26a6d9684d [test] adding tests for root-only expansions. Mostly English tests for the moment to deal with the various edge cases, but is also important for Spanish where "Calle" is so common that it's often omitted, same with French and "rue", etc. 2017-12-17 20:00:48 -05:00
Al
a1db4d7734 [expand/normalize] the split_alpha_from_numeric option now applies to both e.g. A1 and 1A since we now strip out ordinal suffixes prior to normalization 2017-12-17 19:53:15 -05:00
Al
8b2a4d1ecf [api] adding libpostal_expand_address_root to the public API. This will attempt to delete tokens that can be safely ignored. It's deterministic and rule-based, but is informed by libpostal's fairly comprehensive dictionaries, and should work relatively well across languages for deduping purposes. 2017-12-17 17:46:26 -05:00
Al
9eef46adee [expand] in cases like "Avenue D" where there are two phrases, one is ambiguous (and canonical) but not necessarily edge-ignorable (pre/post-directional), allow deletion of the other token (so "Avenue" in this case). Also allows skipping in cases where the language classifier may predict a second language with some small probability, such as French for a short string like "Avenue D" (in addition to English). If the token was ignorable in the highest probability language, ignore it in both. 2017-12-17 17:24:27 -05:00
Al
3f7abd5b24 [expand] adding a method that allows hash/equality comparisons of addresses like "100 Main" with "100 S Main St." or units like "Apt 101" vs. "#101". Instead of expanding the phrase abbreviations, this version tries its best to delete all but the root words in a string for a specific component. It's probably not perfect, but does handle a number of edge cases related to pre/post directionals in English e.g. "E St" will have a root word of simply "E", "Avenue E" => "E", etc. Also handles a variety of cases where the phrase could be a thoroughfare type but is really a root word such as "Park Pl" or the famous "Avenue Rd". This can be used for near dupe hashing to catch possible dupes for later analysis. Note that it will normalize "St Marks Pl" and "St Marks Ave" to the same thing, which is sometimes warranted (if the user typed the wrong thoroughfare), but can also be reconciled at deduping time. 2017-12-17 15:48:11 -05:00
Al
d0364ab6fb [expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation 2017-12-17 03:14:00 -05:00
Al
272ee3b965 [dictionaries] adding a few more ambiguous expansions in English 2017-12-17 03:11:46 -05:00
Al
8968a6c966 [expand] moving expand to its own module so the internal methods can be exposed, calling from libpostal.c 2017-12-08 16:26:13 -05:00
Al
e4e84f0147 [utils] adding unicode_common_prefix/unicode_common_suffix, string_hyphen_prefix_len and string_hyphen_suffix_len to string_utils 2017-12-08 14:28:30 -05:00
Al
55ba627c3c [similarity] needed to add utf8proc_category and invert the indices for counting transposes in affine gap 2017-12-08 14:27:23 -05:00
Al
cfa5b1ce42 [similarity] adding a stopword-aware acronym alignment method for matching U.N. with United Nations, Museum of Modern Art with MoMA, as well as things like University of California - Los Angeles with UCLA. All of these should work across languages, including non-Latin character sets like Cyrllic (but not ideograms as the concept doesn't make as much sense there). Skipping tokens like "of" or "the" depends only on the stopwords dictionary being defined for a given language. 2017-12-04 15:21:44 -05:00
Al
252d5a0f37 Merge branch 'master' into lieu_api 2017-11-30 23:10:17 -05:00
Al Barrentine
8b2f91477e Merge pull request #282 from openvenues/faster_windows_build 
adding make -j4 to Windows build
 2017-11-30 19:13:06 -05:00
Al
cf56da98f7 [build] only build master on Appveyor so PRs don't trigger multiple builds 2017-11-30 17:39:24 -05:00
Al
02d049b8d9 [build] also trying make -j4 to the Windows build. Set an option on the Appveyor side that will hopefully not build twice on pull requests 2017-11-30 17:18:04 -05:00
Al Barrentine
133dce6f2c Merge pull request #280 from openvenues/faster_builds 
make -j4 for all builds
 2017-11-30 14:37:18 -05:00
Al
f207a4680d [docs/build] adding make -j4 as the default for make, including in the Travis/Appveyor builds, should make build times faster 2017-11-30 14:09:13 -05:00
Al
e27f5f1d70 [api] adding LIBPOSTAL_EXPORT to some of the new public API functions in this branch 2017-11-29 18:10:10 -05:00
Al
ec4d683d1b Merge branch 'master' into lieu_api 2017-11-29 15:49:52 -05:00
Al
2cb0d146e5 [docs][ci skip] adding a section for Windows installation, shoutouts to @BenK10 and @AeroXuk 2017-11-29 13:13:00 -05:00
Al Barrentine
1c42df5ca9 Merge pull request #279 from openvenues/drand48_fix 
fix Mac build/standardize conditional compilation of strndup and drand48 for Windows
 2017-11-29 13:04:24 -05:00
Al
9e837f7209 [fix] deleting comment, this is not a header-only implementation 2017-11-29 12:26:36 -05:00
Al
26e4ef08bc [fix] conditional compilation for strndup and drand48 for Windows, using config.h 2017-11-29 12:21:13 -05:00
Al
1a64ad682b [merge] merging in the Ohio expansion numex changes from master 2017-11-29 11:51:43 -05:00
Al Barrentine
18eb5ef9ee Merge pull request #272 from AeroXuk/master 
Windows support via AppVeyor
 2017-11-28 21:35:46 -05:00
AeroXuk
19ae97d527 Adding include config.h to strndup.c so that the function is not compiled and doesn't cause errors when the system has its own implementation. 2017-11-27 23:40:46 +00:00
AeroXuk
9090811826 Modifed the libpostal API to add an extra function libpostal_parser_print_features to toggle debugging info. Updated address_parser app to use the new function. 2017-11-27 19:20:37 +00:00
AeroXuk
69e0d5d963 Updated linenoise to be MSys2/MinGW compatible. Updated address_parser app to use the defined libpostal api and not include internal components directly. Removed windows src Makefile as it is now the same as the standard one. 2017-11-27 01:42:25 +00:00
AeroXuk
bb5535602a Adding libpostal.h to the AppVeyor package. 2017-11-25 10:13:14 +00:00
AeroXuk
26ac9ab5c2 Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal. 2017-11-25 04:35:28 +00:00
Travis
15b3758be8 [auto][ci skip] Adding data files from Travis build #284 2017-11-24 22:29:45 +00:00
Al Barrentine
7d001489ef Merge pull request #274 from openvenues/fix_oh_expansion 
Context-sensitive expansion of words like "oh" inside vs. outside numeric expressions
 2017-11-24 17:13:24 -05:00
Al
ebe7fc9be9 [test] missing paren in Columbus, OH test. Adding test for "oh" as part of a number in Nineteen oh one W El Segundo Blvd 2017-11-24 16:11:07 -05:00
Al
d7f22544b4 [test] adding an expansion test for the Columbus, OH case 2017-11-24 15:44:37 -05:00
Al
ef098fd2e7 [numex] implementing the numex concat_only_if_number left context, which helps in the case of e.g. Columbus, OH in #271 2017-11-24 15:42:50 -05:00
Al
c276cf1529 [numex] adding a new type of left context for numeric expressions called conat_only_if_number (for something like "oh" which can be "Columbus, OH" or something like "Twenty-One Oh One" 2017-11-24 15:36:53 -05:00
AeroXuk
f0246e7333 Fix bug in strndup fix for windows. Move all includes out of headers and into code for strndup.h and move it to be the last include. 2017-11-23 19:11:25 +00:00
AeroXuk
d205f4d2bb Adding artifacts to AppVeyor config. 2017-11-23 02:24:06 +00:00
AeroXuk
f07ab765cb Adding the export marker to all functions used in tests. 2017-11-20 20:58:37 +00:00
AeroXuk
ad682b7592 Altered Makefile to include strndup.c on the other programs which require it. For the windows version of the Makefile, commented out address_parser lines as it has dependencies on includes we don't have. 2017-11-20 20:24:11 +00:00
AeroXuk
dbf232b8f8 Fix bugs in AppVeyor config and build script. Added call to test script. 2017-11-19 13:35:08 +00:00
AeroXuk
2d3b420d35 Merging changes from AeroXuk/libpostal_windows. 2017-11-19 12:44:38 +00:00
Travis
7d6e648fc3 [auto][ci skip] Adding data files from Travis build #271 2017-11-17 19:36:25 +00:00
Al Barrentine
27b3e99515 Merge pull request #269 from Jeffrey04/ms-dictionary-expansion-1.0 
Ms dictionary expansion for 1.0
 2017-11-17 14:20:43 -05:00
jeffrey04
86c3105d44 new names with alternate spelling 2017-11-16 11:23:20 +08:00
jeffrey04
e9d2ab6400 reordered list of synonyms 2017-11-16 11:22:42 +08:00
jeffrey04
b3d306456f new synonyms 2017-11-16 11:22:14 +08:00
jeffrey04
0d76d190e1 updated street types 2017-11-16 11:21:39 +08:00