Kyrill Alyoshin
186f6c77cc
#501 . Fixed build issues on MSYS on Windows.
2020-11-14 19:26:55 +00:00
Kyrill Alyoshin
9fcf066e38
#511 . Fixed C compilation errors for the latest versions of Mac OS X
2020-08-26 15:28:54 -04:00
in-cloud-opensource
ed41f42c97
Fix a bug in file geohash/geohash.c
...
Add missing bit shifts in the function geohashstr_to_interleaved in order to be able to calculate correctly the
1) geo coordinates from a given geohash and
2) neigbor geohashes of a given geohash.
The incorrect calculation of neighbor geohashes e.g. has the following impact on the deduplication with lieu: Features are compared that should not be compared because they are too far away from each other. This often results in distances between duplicates that exceed by far the max. expected distance according to the used geohash precision.
2019-12-12 11:34:39 +01:00
Al Barrentine
95bf70eb35
Merge pull request #465 from luizvbo/lang_classifier_exposure
...
Exposes language classification functions
2019-12-06 05:03:33 -06:00
in-cloud-opensource
d6595aee9b
Fix segmentation fault in soft_tfidf.c
...
Initialize local arrays suffixes1 and suffixes2 in function soft_tfidf_similarity_with_phrases_and_acronyms explicitly with NULL in order to prevent segmentation faults due to undefined initial values if ordinal_suffixes1 and ordinal_suffixes2 are NULL.
2019-11-19 12:49:54 +01:00
Al
7828e568a7
[fix] remove hyphens option in libpostal expansions/token normalizations
2019-08-15 02:01:46 -04:00
Luiz Otavio V. B. Oliveira
0327150d2b
Exposes language classification functions
2019-06-14 14:31:12 +02:00
Al
42207769ef
[expand] adding a --root flag to the libpostal script for root-only expansion
2019-02-16 22:27:39 -05:00
Al
b7052caf6b
[dedupe] allow near-dupe hashes if only a small containing boundary is present (e.g. county/state district).
2019-02-16 22:26:13 -05:00
Al
3610ffaa05
[expand/dedupe] expansion with multiple languages (or multiple predicted languages) can sometimes produce weird string trees and thus either too many results or incorrect results, particularly for root expansions which we depend on for matching/deduping. Making one call per language identified. This may slightly affect performance on languages that are highly ambiguous (even that's doubtful, as libpostal usually identifies one or two languages with high accurracy and/or people are using a known geography) but should improve the results and was simpler implementation-wise than trying to use the single single string tree for multiple languages where, say, a two word phrase in one language might simply be token-space-token in another.
2019-02-16 22:20:36 -05:00
Al
8a022698ee
[fix] transliteration bug causing potential infinite loop, closes #351 , closes #378
2019-02-13 04:01:29 -05:00
Al
7a23d8922a
[build] data dir version handling
2019-02-12 14:10:02 -05:00
Al
fdb3b7e32e
[build] fixing multi-part downloads, which no longer know the Content-Length so no longer know the byte ange for the last chunk. No longer needs to use any file dates/timestamps (compatibility 🎉 ), only the version numbers that are checked in to the repo
2019-02-12 13:33:08 -05:00
Al
acebc951b6
[fix] != instead of -ne
2019-02-12 12:32:34 -05:00
Al
9542c8c435
[build] adding libposatl_data.in which allows us to fetch data files from Github without using the API
2018-12-13 13:35:01 -05:00
Al
b94519122e
[build] adding new in-repo version files for each of the data files so we don't have to use the Github API on Travis builds and a template version of the download script so that the version numbers can be configured based on those files.
2018-11-18 11:46:37 -05:00
Al
027fbc5afc
[fix] filename
2018-03-11 03:26:37 -04:00
Al
b4a0c79d64
[fix] adding the tempfiles in the datadir, where the user must have permissions for this to work anyway
2018-03-10 22:24:03 -05:00
Al
15cb5f68ad
[fix] making this work with sh...wondering if supporting the few shells that don't have bash is worth losing array functionality.
2018-03-10 22:12:46 -05:00
Al
0c91379424
[build/fix] using Github for the model releases rather than the Mapzen S3 buckets which are no longer working after the shutdown. It requires a little more effort to get the metadata, but downloads should still be just as fast since Github releases are on S3 as well. Note: still need to implement the upload piece, but this at least provides a model endpoint for users.
2018-03-10 19:03:30 -05:00
Travis
fabd040860
[auto][ci skip] Adding data files from Travis build #381
2018-03-08 01:50:54 +00:00
Al
6575bdc339
[dedupe/test] adding remaining options to near_dupe_test
2018-02-24 00:34:58 -05:00
Al
835de327c3
[dedupe] for near-dupe hashing, remove whitespace from root expansions so something like "Ocean Walk Dr" and "Oceanwalk Dr" will have a chance of matching downstream
2018-02-24 00:34:09 -05:00
Al
283be99b44
[numex] helper function to retrieve ordinal suffix lengths from a tokenized string for use in deduping
2018-02-24 00:31:26 -05:00
Al
b2dcb18d7e
[dedupe] account for missing ordinal suffixes in Soft-TFIDF deduping i.e. to count 1st Place and 1 Plce as the same where there might be a misspelling and the phrase wouldn't match under exact expansions
2018-02-23 23:44:05 -05:00
Al
b03fbdd681
[dedupe] adding multi-word phrase alignments to deduping
2018-02-23 01:28:02 -05:00
Al
591891951d
[utils] adding utf8 case insensitive comparison
2018-02-23 01:22:58 -05:00
Al
2b4e7073c2
[similarity] adding a multi-word alignmnet algorithm for streets and names like "de la cruz" vs. "dela cruz" or "Oceanwalk Ter" vs. "Ocean Walk Ter"
2018-02-23 01:22:12 -05:00
Al
c5bb9d8daa
[normalize/api] exposing normalize_string_languages and normalized_tokens_languages to the API for pre-normalizing numeric expressions at tokenization time
2018-02-22 18:47:36 -05:00
Travis
cf2156c0e2
[auto][ci skip] Adding data files from Travis build #375
2018-02-22 03:32:16 +00:00
Al
afd225048a
[fix] DICTIONARY_STREET_NAME applies only to the street component, unlike synonyms, which would apply to any compnoent. This makes street names a good place to add synonyms found in streets that are not exactly thoroughfare types i.e. could not be removed from the string and have it retain more or less the same meaning.
2018-02-21 22:17:04 -05:00
Al
0f20613c13
[expand] using street name dictionaries as a possible root component instead of having to pollute the synonyms dictionary, which also affects the parser and might be a better place for general purpose synonyms affecting all components.
2018-02-21 22:16:07 -05:00
Travis
8075ee6f49
[auto][ci skip] Adding data files from Travis build #374
2018-02-22 01:21:23 +00:00
Al
09408b1075
[fix] for regular expansion, use gazetteer components or overrides
2018-02-15 18:55:37 -05:00
Al
c5429de4d7
[fix] no gazetteer changes yet, breaks the parser
2018-02-15 18:52:16 -05:00
Al
78d621ac85
[fix] adding street type gazetteer to name component as well for things like "24th St Cheese Co"
2018-02-15 18:13:27 -05:00
Al
9c12a11fd7
[fix] check expansion address components for regular expansion, overrides for root expansion
2018-02-15 16:19:50 -05:00
Al
9390e638ae
[fix] for regular non-root expansion, check that components are valid (for near-dupe expansions or other cases where component options are passed in)
2018-02-08 13:49:24 -05:00
Al
2290b0991e
[fix] transliteration case where a context no match comes at the end of the string
2018-02-06 15:08:11 -05:00
Al
156c8bed40
[fix] check that second double metaphone alternative is not the empty string
2018-02-06 03:08:37 -05:00
Al
3a5c048419
[fix] in root expansions, if the current phrase has at least one valid expansion, and the current expansion is not valid, ignore it
2018-02-06 02:36:05 -05:00
Al
8b96173a32
[fix] max dictionaries is now 5, weird that that wasn't committed by the Travis build
2018-02-06 01:58:15 -05:00
Travis
a9dded3a18
[auto][ci skip] Adding data files from Travis build #367
2018-02-06 01:43:25 +00:00
Al
4d3619d493
[dedupe] moving name-only near-dupe hashes to a separate if block so they can be used in conjunction with name+address
2018-02-03 14:03:17 -05:00
Al
7cb85aa23c
[dedupe] to make soft token similarity order invariant, we swap the order so the shorter token sequence comes first. In the case of a tie, pick the shorter full string length
2018-01-26 18:04:45 -05:00
Al
4aeb549054
[dedupe] with some term weighting schemes (especially information gain which will soon be the default in the lieu project), single letters may have very low weights such that they will be discarded, which can lead to false positives for things like "A & B" vs. "B & C", so add a simple heuristic to simply demote likely dupes to needs review when there's a positive symmetric difference (or whatever the set theory term is for when A - B and B - A are both non-empty)
2018-01-26 01:20:35 -05:00
Al
af5a5c3039
[dedupe] in the case of abbreviations and acronyms, where we use the higher of the two scores, calculate an offset to the norm of the other string's scores i.e. sincey we're replacing the score(s) in the lower-scoring vector with the higher one in the dot product for the numerator, do the same for the L2-norm product in the denominator. This way we don't accidentally inflate the similarity value simply because e.g. an acronym token was more rare than the same acronym spelled out as multiple individual letters (tend to be low-information/common tokens).
2018-01-25 16:32:47 -05:00
Al
13230824a2
[fix] Damerau-Levenshtein distance costing for transposes was off
2018-01-25 16:26:41 -05:00
Al
d0fe31d359
[dedupe] for strict abbreviations (defined as sharing a prefix and a suffix, and containing matches+gaps only by the subtotaling affine gap measure), using the greater of the two scores. This accounts for cases where the abbreviated version may have a much higher weight in one string than the non-abbreviated version does in the other. Same for acronym alignments. Making sure there's a common prefix in regular abbeviation detection Capping the Soft-TFIDF similarity at 1.0.
2018-01-25 14:23:18 -05:00
Al
b4cc7395a2
[fix] was missing some shorter tokens that are unicode equal in Soft-TFIDF
2018-01-25 04:23:35 -05:00
