9542c8c435[build] adding libposatl_data.in which allows us to fetch data files from Github without using the API
Al
2018-12-13 13:35:01 -05:00
64f274b834Update README.md
Matt W
2018-11-23 14:28:49 -08:00
641395e811[build] configure now uses the in-repo version strings and builds the libpostal_data script from a template
Al
2018-11-18 11:53:46 -05:00
b94519122e[build] adding new in-repo version files for each of the data files so we don't have to use the Github API on Travis builds and a template version of the download script so that the version numbers can be configured based on those files.
Al
2018-11-18 11:44:28 -05:00
75d6dab4afAdde support for German office/house numbers with gebäude/geb prefix word.
Yevgen Antymyrov
2018-11-01 00:58:12 +01:00
714d69a792Fixing a small typo in Ukrainian transliterated word 'budynok' (house), was: budnyok.
Yevgen Antymyrov
2018-11-01 00:30:57 +01:00
363e83304afix memory leak in setup when datadir is invalid or setup failed
Edward Ding
2018-10-26 15:08:56 -07:00
e8569f9e26Add haskell-postal to unofficial language bindings
Fábián Tamás László
2018-08-14 12:19:59 +02:00
1681321452minor typo
Peter Johnson
2018-08-12 19:22:21 +02:00
216c947e37Add all missing libpostal_ exports to .def
Max Bolingbroke
2018-08-09 10:32:08 +01:00
1c4d4001de[docs] Update Readme.md on free text extraction
Ricardo Alanis
2018-05-28 13:04:36 -05:00
eae01d3e60Updating issue template to not use checkboxes
Al Barrentine
2018-05-24 16:55:03 -04:00
2890f739a3overrides shouldn't be nested in admin_level
Antoine D
2018-04-27 12:38:27 +02:00
87b92f381eadd missing "relation" tag
Antoine D
2018-04-27 12:37:45 +02:00
04a77d2e59[docs] fix minor typo
missinglink
2018-04-12 16:10:18 +02:00
b86c03ddcfAdd to Romanian dictionary
Catalin Marius Ciochina
2018-04-10 08:53:26 +03:00
2ee412c98eUpdate street_types.txt
Maciej Beręsewicz
2018-03-27 15:55:34 +02:00
43795a3d90Merge pull request #328 from openvenues/github_model_releases
Al Barrentine
2018-03-11 03:55:53 -04:00
027fbc5afc[fix] filename
Al
2018-03-11 03:26:37 -04:00
b4a0c79d64[fix] adding the tempfiles in the datadir, where the user must have permissions for this to work anyway
Al
2018-03-10 22:23:58 -05:00
15cb5f68ad[fix] making this work with sh...wondering if supporting the few shells that don't have bash is worth losing array functionality.
Al
2018-03-10 22:06:19 -05:00
270bae7a32[docs] adding the links to the Internet Archive versions of the training data as well as a file that contains the current parser training set date
Al
2018-03-10 19:33:24 -05:00
0c91379424[build/fix] using Github for the model releases rather than the Mapzen S3 buckets which are no longer working after the shutdown. It requires a little more effort to get the metadata, but downloads should still be just as fast since Github releases are on S3 as well. Note: still need to implement the upload piece, but this at least provides a model endpoint for users.
Al
2018-03-10 19:03:14 -05:00
fabd040860[auto][ci skip] Adding data files from Travis build #381
Travis
2018-03-08 01:50:54 +00:00
6d41863068Merge pull request #324 from veremenko-y/feature-update-street-types
Al Barrentine
2018-03-07 20:37:30 -05:00
c0b770101aRemoved final period
Yaroslav Veremenko
2018-03-07 17:51:34 -07:00
fd2b864b7dUpdated street types for EN
Yaroslav Veremenko
2018-03-07 12:48:30 -07:00
6575bdc339[dedupe/test] adding remaining options to near_dupe_test
Al
2018-02-24 00:34:58 -05:00
835de327c3[dedupe] for near-dupe hashing, remove whitespace from root expansions so something like "Ocean Walk Dr" and "Oceanwalk Dr" will have a chance of matching downstream
Al
2018-02-24 00:33:04 -05:00
283be99b44[numex] helper function to retrieve ordinal suffix lengths from a tokenized string for use in deduping
Al
2018-02-24 00:31:02 -05:00
b2dcb18d7e[dedupe] account for missing ordinal suffixes in Soft-TFIDF deduping i.e. to count 1st Place and 1 Plce as the same where there might be a misspelling and the phrase wouldn't match under exact expansions
Al
2018-02-23 23:42:49 -05:00
b03fbdd681[dedupe] adding multi-word phrase alignments to deduping
Al
2018-02-23 01:25:43 -05:00
591891951d[utils] adding utf8 case insensitive comparison
Al
2018-02-23 01:22:58 -05:00
2b4e7073c2[similarity] adding a multi-word alignmnet algorithm for streets and names like "de la cruz" vs. "dela cruz" or "Oceanwalk Ter" vs. "Ocean Walk Ter"
Al
2018-02-23 01:22:12 -05:00
c5bb9d8daa[normalize/api] exposing normalize_string_languages and normalized_tokens_languages to the API for pre-normalizing numeric expressions at tokenization time
Al
2018-02-22 18:47:36 -05:00
0edb897143[docs][ci skip] adding definition for the street_names.txt dictionary
Al
2018-02-22 02:25:27 -05:00
cf2156c0e2[auto][ci skip] Adding data files from Travis build #375
Travis
2018-02-22 03:32:16 +00:00
afd225048a[fix] DICTIONARY_STREET_NAME applies only to the street component, unlike synonyms, which would apply to any compnoent. This makes street names a good place to add synonyms found in streets that are not exactly thoroughfare types i.e. could not be removed from the string and have it retain more or less the same meaning.
Al
2018-02-21 22:17:01 -05:00
0f20613c13[expand] using street name dictionaries as a possible root component instead of having to pollute the synonyms dictionary, which also affects the parser and might be a better place for general purpose synonyms affecting all components.
Al
2018-02-21 22:16:04 -05:00
ab67e0864a[dictionaries] adding new street_names.txt dictionary and movign all the synonyms to there, generating the new dictionary type in address_dictionaries.py
Al
2018-02-21 22:13:38 -05:00
cd7bf81522[dictionaries] reverting synonyms.txt back to previous commit
Al
2018-02-21 22:13:05 -05:00
8075ee6f49[auto][ci skip] Adding data files from Travis build #374
Travis
2018-02-22 01:21:23 +00:00
a7b7568159[dictionaries] removing "Corner" from the synonyms dictionaries for #320
Al
2018-02-21 19:55:57 -05:00
09408b1075[fix] for regular expansion, use gazetteer components or overrides
Al
2018-02-15 18:55:37 -05:00
c5429de4d7[fix] no gazetteer changes yet, breaks the parser
Al
2018-02-15 18:52:16 -05:00
ee2bac66d3[build] rebuild dictionaries when gazteeer_data.c changes as well
Al
2018-02-15 18:18:01 -05:00
78d621ac85[fix] adding street type gazetteer to name component as well for things like "24th St Cheese Co"
Al
2018-02-15 18:13:27 -05:00
9c12a11fd7[fix] check expansion address components for regular expansion, overrides for root expansion
Al
2018-02-15 16:13:17 -05:00
9390e638ae[fix] for regular non-root expansion, check that components are valid (for near-dupe expansions or other cases where component options are passed in)
Al
2018-02-08 13:49:21 -05:00
2290b0991e[fix] transliteration case where a context no match comes at the end of the string
Al
2018-02-06 15:08:11 -05:00
156c8bed40[fix] check that second double metaphone alternative is not the empty string
Al
2018-02-06 03:08:37 -05:00
3a5c048419[fix] in root expansions, if the current phrase has at least one valid expansion, and the current expansion is not valid, ignore it
Al
2018-02-06 02:30:03 -05:00
8b96173a32[fix] max dictionaries is now 5, weird that that wasn't committed by the Travis build
Al
2018-02-06 01:58:15 -05:00
a9dded3a18[auto][ci skip] Adding data files from Travis build #367
Travis
2018-02-06 01:43:25 +00:00
a67df14854[fix] adding newline to trigger new build
Al
2018-02-05 20:30:21 -05:00
86fdaf7188[fix] changing root expansion test for "E Broadway", which now returns "broadway" instead of "east", which is a better result anyway
Al
2018-02-05 19:35:57 -05:00
0ee3f6b294Merge branch 'master' of ssh://github.com/openvenues/libpostal
Al
2018-02-05 19:32:17 -05:00
a6628e918fMerge pull request #309 from antoine-de/small_fix
Al Barrentine
2018-02-05 19:30:55 -05:00
e3a252d4eb[dictionaries] adding most of the street types to synonyms/place_names in English so e.g. "Spring" in "Spring St" will not be removed in root expansions
Al
2018-02-05 18:21:17 -05:00
9baac813f6[dictionaries] add "a" and "b" to the ambiguous expansions for English, randomly weren't in there
Al
2018-02-05 18:18:00 -05:00
4d3619d493[dedupe] moving name-only near-dupe hashes to a separate if block so they can be used in conjunction with name+address
Al
2018-02-03 14:03:17 -05:00
7cb85aa23c[dedupe] to make soft token similarity order invariant, we swap the order so the shorter token sequence comes first. In the case of a tie, pick the shorter full string length
Al
2018-01-26 18:04:45 -05:00
4aeb549054[dedupe] with some term weighting schemes (especially information gain which will soon be the default in the lieu project), single letters may have very low weights such that they will be discarded, which can lead to false positives for things like "A & B" vs. "B & C", so add a simple heuristic to simply demote likely dupes to needs review when there's a positive symmetric difference (or whatever the set theory term is for when A - B and B - A are both non-empty)
Al
2018-01-26 01:20:35 -05:00
af5a5c3039[dedupe] in the case of abbreviations and acronyms, where we use the higher of the two scores, calculate an offset to the norm of the other string's scores i.e. sincey we're replacing the score(s) in the lower-scoring vector with the higher one in the dot product for the numerator, do the same for the L2-norm product in the denominator. This way we don't accidentally inflate the similarity value simply because e.g. an acronym token was more rare than the same acronym spelled out as multiple individual letters (tend to be low-information/common tokens).
Al
2018-01-25 16:32:47 -05:00
13230824a2[fix] Damerau-Levenshtein distance costing for transposes was off
Al
2018-01-25 16:26:41 -05:00
d0fe31d359[dedupe] for strict abbreviations (defined as sharing a prefix and a suffix, and containing matches+gaps only by the subtotaling affine gap measure), using the greater of the two scores. This accounts for cases where the abbreviated version may have a much higher weight in one string than the non-abbreviated version does in the other. Same for acronym alignments. Making sure there's a common prefix in regular abbeviation detection Capping the Soft-TFIDF similarity at 1.0.
Al
2018-01-25 14:19:44 -05:00
b4cc7395a2[fix] was missing some shorter tokens that are unicode equal in Soft-TFIDF
Al
2018-01-25 04:23:35 -05:00
c4aaee7dbf[dedupe/similarity] also utilizing the L2 norm in similarity when acronyms are detected. Similarity in this case should be the acronym token's score * the L2 norm of the expanded tokens' scores in the longer string
Al
2018-01-23 01:20:14 -05:00
1c13bf3678fix typo in python imports
antoine-de
2018-01-22 18:39:27 +01:00
ccce4f793f[optimization] for the FTRL and SGD optimizers, use the new *_array_sum_sq function to do L2 regularization, vs. the L2 norm which will use the linear algebra meaning
Al
2018-01-22 01:56:32 -05:00
eb3fb37ad4[similarity/dedupe] normalizing by the product of the L2 norms in soft token similarity function, as in cosine similarity. Score vectors should be passed in unnormalized, and typically with unit length. Also, for aligned phrases that share the same canonical phrase, contribute the product of the two norms of the phrase vectors to the similarity's numerator (maximum value, as if each token in both strings had matched exactly). The previous version over-counted the importance of aligned multi-word phrases by doing a cross product, which could overshadow other more important terms.
Al
2018-01-22 01:38:12 -05:00
2883b57eb3[fix] sqrt in l2 norm
Al
2018-01-22 01:07:52 -05:00
3c5713ef59[fix] check for sub-acronyms with no stopwords in near-dupe hashing
Al
2018-01-21 17:49:46 -05:00
fa40a7e87c[fix] initialize repeat_state_end in transliteration. Couldn't reproduce the conditional jumps in #308, but this seems to be where they're occurring, and it's generally good practice to initialize things
Al
2018-01-21 16:24:19 -05:00
e4c35a0911Merge branch 'master' of ssh://github.com/openvenues/libpostal
Al
2018-01-21 01:21:11 -05:00
dc8bffd5a0[fix] expansion test valgrind errors for #308
Al
2018-01-21 01:20:30 -05:00
48ea0d7801[docs][ci skip] Adding issue template
Al Barrentine
2018-01-20 02:29:23 -05:00
984235e879[fix] load transliteration for language classifier cli for #302
Al
2018-01-19 14:24:08 -05:00
7121642c62[dedupe] fixing sub-acronym near-dupe hashes with punctuation, and making sure to add the current token after a new sub-acronym has been cut
Al
2018-01-17 18:56:21 -05:00
95e483e3ca[fix] case-insensitive comparison of content-length header in data download script
Al
2018-01-17 17:31:42 -05:00
3e3558e199Merge pull request #303 from Maurice-Betzel/master
Al Barrentine
2018-01-17 13:17:30 -05:00
46585c89e0Added unofficial Java language binding
Maurice Betzel
2018-01-17 11:00:34 +01:00
ebc332a311Added unofficially supported Java language binding
Maurice Betzel
2018-01-17 10:57:09 +01:00
a4793f0f79[auto][ci skip] Adding data files from Travis build #343
Travis
2018-01-16 09:30:46 +00:00
ac350d90f6[dictionaries] adding BYP to ambiguous expansions for the Black Youth Project
Al
2018-01-16 03:40:36 -05:00
03e5e25240[dedupe] adding a near-dupe hash which takes into account existing acronyms which may have appeared in the string, either known acronyms as defined in the dictionaries like "HS" and includes the full token in the acronym. This feature is particularly useful for public schools or other cases where the canonical string may be used i.e. "Foo High School", "Foo HS" and "FHS". It also does the same thing other acronyms that are identified by the tokenizer from the internal period structure like A.B.C. Also now allowing mixed alpha-numeric tokens to use the double metaphone encoding as well, and for numeric tokens with script=Common (digits but may also contain hyphens, etc.), the full token is included as one of the words rather than quadgrams, which don't make sense for numerics.
Al
2018-01-16 03:07:32 -05:00
0286a2fef3[expand] for root expansions, delete ambiguous tokens only when there's a non-numeric non-phrase token present. This applies to all name components, not for components where numerics can be the root (house numbers, units, streets, etc.)
Al
2018-01-16 03:00:12 -05:00
0ee18b4f6c[dedupe] adding a function to acronyms module to detect existing/known acronyms like MS for middle school, HS for high school, etc. Forms like MS have to be deined in the dictionaries specifically but any acronym written like M.S. will be detected as such by the tokenizer
Al
2018-01-15 23:46:49 -05:00
133381f439Merge branch 'master' of ssh://github.com/openvenues/libpostal
Al
2018-01-14 19:40:44 -05:00
c553fe81ee[dedupe] using 4-grams with no edge disambiguation in near dupe hashing of names instead of full tokens (uses the double metaphone for Latin script, 2-grams for ideographic scripts and 4-gram unicode chars for other scripts like Arabic or Cyrllic). The fully concatenated name string with no whitespace + acronyms/subacronyms now also use double-metaphone in Latin script, and are split into 4-grams. Overall this reduces the number of keys, accounts for more misspellings as well as languages with longer words such as German, and various spacing/concatenations differences in general, while still being relatively selective. Most words in Latin scripts will resolve to less than 4 characters, so this mostly affects longer words with many consonants. 4-gram blocking tends to be what's used in the literature, and works well in practice on human and venue names. This is a slight departure from said literature in that we use 4-grams of the phonetic normalization for Latin scripts.
Al
2018-01-14 19:11:50 -05:00
c58b16e745[auto][ci skip] Adding data files from Travis build #341
Travis
2018-01-11 23:35:43 +00:00
b2aa2e4fddMerge pull request #299 from antimirov/master
Al Barrentine
2018-01-11 18:22:34 -05:00
f5e41a1f57[fix] logic in sub-acronym generation for near-dupe hashes
Al
2018-01-11 13:15:19 -05:00
6ba0403748[dedupe] adding a near-dupe hash for acronyms both with and without stopwords. This will create basic acronyms for institutions like MoMA, UCLA, the NAACP, as well as human initials, etc. It also handles sub-acronyms, so when either at every other non-contiguous stopword (University of Texas at Austin) or punctuation (University of Texas, Austin), it cuts a new sub-acronym (so UT). All of the acronyms for Latin script use a double metaphone as well, so can potentially catch many cases. It does not handle all possible acronyms (e.g. where some of the letters are word-internal as in medical acronyms), but should do relatively well on many common variations.
Al
2018-01-10 22:23:40 -05:00
b1f3760a81Updating Ukrainian dictionaries, small fixes and additional variants.
Yevgen Antymyrov
2018-01-10 11:35:59 +01:00
c29557c16b[expand] adding another check in root expansions, making sure we don't ignore the unmodified ambiguous phrase
Al
2018-01-08 19:03:50 -05:00
e6edf54adb[dedupe] adding a near-dupe hash for the entire name without spaces.
Al
2018-01-08 19:02:19 -05:00
66aee0fffa[expand] make street type dictionaries ignorable for venue names as well (many company names mention their address, so sort of have to apply the same rules)
Al
2018-01-07 01:39:10 -05:00
e935f2a036[fix] need to calculate max Jaro-Winkler for other methods, so only test whether we should use it after we've cycled through all the tokens
Al
2018-01-06 03:59:34 -05:00
179e6581e5[dedupe] for fuzzy street duplicates, using the likely dupe classification regardless of similarity if one set of tokens is fuzzily-contained within the other (the set of matches allows for words matching Jaro-Winkler/Levenshtein similarity, being out of order, etc. but not acronyms or the more flexible abbreviation detection, just abbreviations out of libpostal's dictionaries provided that the abbreviations use share the same canonical phrase with the aligned phrase in the other string)
Al
2018-01-06 03:55:20 -05:00
Al