tommy/libpostal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,496 Commits 2 Branches 0 Tags
master
Commit Graph

1196 Commits

Author SHA1 Message Date
Al
8bc5270141 [fix] memcpy in geohash, closes #707 2025-08-25 00:17:31 -04:00
Al
0d05426c2d [rm] remove sse2neon.h, not used anymore and SIMDe is a better when ready to start using the clib dependencies 2025-07-16 22:47:05 -04:00
Al
bf3acdcb89 [fix] double-free on a non-lowercased transliterator name 2025-07-02 01:21:08 -04:00
Al
95e97c0585 [fix/utf8] reviewed and fixed all points where utf8proc_iterate is called and may return an error which can cause the iteration not to make forward progress. This includes fixing a bug where injecting invalid UTF-8 through a series of HTML-encoded codepoints can cause the C library to hang. Note: we're not fixing all the garbage encoding in the world, so if encoding is bad the output of expand_address may not be useful but it won't hang. Fixes #448 2025-07-02 00:10:49 -04:00
Al B
036af617bb Merge pull request #666 from Vedingrot/fix_uninitialized_var 
[fix] initialize next_id by zero in trie_new_from_cstring_array function
 2025-02-08 12:48:46 -05:00
Al B
fcbb13cad0 Merge pull request #391 from edding/fix_memory_leak 
fix memory leak in setup when datadir is invalid or setup failed
 2025-02-08 12:03:59 -05:00
Al
052d99de74 [fix] ifdef includes around stndup 2025-01-31 16:14:33 -05:00
Al
d491739b21 [fix] sparse_matrix_utils also had an incompatible pointer 2025-01-30 13:13:29 -05:00
Al
0540d7c7e3 [api/compat] PR #465 redefined the language classifier response struct in the API and was casting between incompatible pointer types. Using the exported struct throughout. 2025-01-30 01:45:18 -05:00
Egor Shestakov
aa712e60c1 [fix] initialize next_id by zero in trie_new_from_cstring_array function 2024-06-06 00:34:55 +07:00
Al
330bd2e158 [fix] Windows name 2023-08-18 22:37:50 -04:00
Al
e4982b733f [fix] memcpy in aligned vector/matrix resize needs to know the original size. Since this is an internal static function and does not affect client APIs, changing it to aligned_malloc, aligned_free, and aligned_resize, which takes the original pointer size as well as the new. On Windows it simply passes the pointer, new size, and alignment through _aligned_realloc, whereas on other platforms only the aligned_malloc is for new_size bytes and uses old_size bytes for memcpy 2023-08-18 13:39:36 -04:00
Al
7bdcf96c9d [memalign] no more realloc on aligned pointers, just do an aligned malloc and copy to it. Slower but safe and this is not called that often in practice, usually to resize larger matrices. 2023-07-06 16:01:02 -04:00
Al
57eaa414ce [revert] reverting the commits from #578, leaving header file in repo for the moment 2023-07-06 01:54:46 -04:00
Al
7a448b718d [crf] using 32 bytes for posix_memalign to align blocks of 4 doubles for remez algorithm to fix test which uses an odd-sized context 2023-07-05 21:02:41 -04:00
Davide Madrisan
dcb63d8768 Fix dash syntax error in libpostal_data 
Fix the syntax error reported by dash:

    ./src/libpostal_data: 39: [: ==: unexpected operatora

when the variable DATAMODEL is empty.

Signed-off-by: Davide Madrisan <davide.madrisan@gmail.com>
 2023-06-29 14:36:10 +02:00
Motiejus Jakštys
5d77298e88 avoid UB in bit shifts 
unsigned char* gets promoted to `int`, which cannot always be shifted by 24 bits.

Justine Tunney blogs about it here: https://justine.lol/endian.html

Example:

```deserialize.c
#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>

uint32_t file_deserialize_uint32_ok(unsigned char *buf) {
    return ((uint32_t)buf[0] << 24) | ((uint32_t)buf[1] << 16) | ((uint32_t)buf[2] << 8) | (uint32_t)buf[3];
}

uint32_t file_deserialize_uint32(unsigned char *buf) {
    return (buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3];
}

int main() {
    unsigned char arr[4] = {0xaa, 0xaa, 0xaa, 0xaa};

    printf("%d\n", file_deserialize_uint32_ok((unsigned char*)arr));
    printf("%d\n", file_deserialize_uint32((unsigned char*)arr));
}
```

Output:
```
$ clang-16 -fsanitize=undefined ./deserialize.c -o deserialize && ./deserialize
-1431655766
deserialize.c:10:20: runtime error: left shift of 170 by 24 places cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior deserialize.c:10:20 in 
-1431655766
```
 2023-06-23 12:16:35 +03:00
Oskar Thorbjornsson
00568da290 Modifying README and config parameter, based on code review. 2023-02-14 21:02:51 -08:00
Oskar Thorbjornsson
ec9e0e341f Enable downloading of Senzing data model. 2023-02-12 17:58:36 -08:00
Al B
544d510db0 Merge pull request #578 from reisub/sse2neon 
Use NEON on ARM hardware via sse2neon.h
 2022-07-10 20:00:17 -04:00
Dino Kovač
6064bc6c06 Use NEON on ARM hardware via sse2neon.h 
The autoconf changes were adapted from:
https://github.com/glennrp/libpng/blob/libpng16/configure.ac
 2022-06-16 15:49:01 +02:00
Al B
a97717f2b9 Merge pull request #567 from kmicklas/config-check 
Check HAVE_CONFIG_H in matrix.h
 2022-04-18 16:03:29 -04:00
Al
893745f09b [near_dupes] using quadgrams in Latin scripts as well for near dupe hashes 2022-03-25 14:05:03 -04:00
Al
26124ee72f [near_dupes] exposing name_word_hashes directly in the API 2022-03-25 14:04:26 -04:00
Al
0d8e4ec56d [fix] possible acronym for single toke phrases if it's a directional 2022-03-25 12:24:42 -04:00
Ken Micklas
b0c1c75209 Check HAVE_CONFIG_H in matrix.h 2021-12-08 09:33:32 -05:00
Kyrill Alyoshin
186f6c77cc #501. Fixed build issues on MSYS on Windows. 2020-11-14 19:26:55 +00:00
Kyrill Alyoshin
9fcf066e38 #511. Fixed C compilation errors for the latest versions of Mac OS X 2020-08-26 15:28:54 -04:00
in-cloud-opensource
ed41f42c97 Fix a bug in file geohash/geohash.c 
Add missing bit shifts in the function geohashstr_to_interleaved in order to be able to calculate correctly the  
    1) geo coordinates from a given geohash and 
    2) neigbor geohashes of a given geohash.

The incorrect calculation of neighbor geohashes e.g. has the following impact on the deduplication with lieu: Features are compared that should not be compared because they are too far away from each other. This often results in distances between duplicates that exceed by far the max. expected distance according to the used geohash precision.
 2019-12-12 11:34:39 +01:00
Al Barrentine
95bf70eb35 Merge pull request #465 from luizvbo/lang_classifier_exposure 
Exposes language classification functions
 2019-12-06 05:03:33 -06:00
in-cloud-opensource
d6595aee9b Fix segmentation fault in soft_tfidf.c 
Initialize local arrays suffixes1 and suffixes2 in function soft_tfidf_similarity_with_phrases_and_acronyms explicitly with NULL in order to prevent segmentation faults due to undefined initial values if ordinal_suffixes1 and ordinal_suffixes2 are NULL.
 2019-11-19 12:49:54 +01:00
Al
7828e568a7 [fix] remove hyphens option in libpostal expansions/token normalizations 2019-08-15 02:01:46 -04:00
Luiz Otavio V. B. Oliveira
0327150d2b Exposes language classification functions 2019-06-14 14:31:12 +02:00
Al
42207769ef [expand] adding a --root flag to the libpostal script for root-only expansion 2019-02-16 22:27:39 -05:00
Al
b7052caf6b [dedupe] allow near-dupe hashes if only a small containing boundary is present (e.g. county/state district). 2019-02-16 22:26:13 -05:00
Al
3610ffaa05 [expand/dedupe] expansion with multiple languages (or multiple predicted languages) can sometimes produce weird string trees and thus either too many results or incorrect results, particularly for root expansions which we depend on for matching/deduping. Making one call per language identified. This may slightly affect performance on languages that are highly ambiguous (even that's doubtful, as libpostal usually identifies one or two languages with high accurracy and/or people are using a known geography) but should improve the results and was simpler implementation-wise than trying to use the single single string tree for multiple languages where, say, a two word phrase in one language might simply be token-space-token in another. 2019-02-16 22:20:36 -05:00
Al
8a022698ee [fix] transliteration bug causing potential infinite loop, closes #351, closes #378 2019-02-13 04:01:29 -05:00
Al
7a23d8922a [build] data dir version handling 2019-02-12 14:10:02 -05:00
Al
fdb3b7e32e [build] fixing multi-part downloads, which no longer know the Content-Length so no longer know the byte ange for the last chunk. No longer needs to use any file dates/timestamps (compatibility 🎉), only the version numbers that are checked in to the repo 2019-02-12 13:33:08 -05:00
Al
acebc951b6 [fix] != instead of -ne 2019-02-12 12:32:34 -05:00
Al
9542c8c435 [build] adding libposatl_data.in which allows us to fetch data files from Github without using the API 2018-12-13 13:35:01 -05:00
Al
b94519122e [build] adding new in-repo version files for each of the data files so we don't have to use the Github API on Travis builds and a template version of the download script so that the version numbers can be configured based on those files. 2018-11-18 11:46:37 -05:00
Edward Ding
363e83304a fix memory leak in setup when datadir is invalid or setup failed 2018-10-26 16:07:57 -07:00
Al
027fbc5afc [fix] filename 2018-03-11 03:26:37 -04:00
Al
b4a0c79d64 [fix] adding the tempfiles in the datadir, where the user must have permissions for this to work anyway 2018-03-10 22:24:03 -05:00
Al
15cb5f68ad [fix] making this work with sh...wondering if supporting the few shells that don't have bash is worth losing array functionality. 2018-03-10 22:12:46 -05:00
Al
0c91379424 [build/fix] using Github for the model releases rather than the Mapzen S3 buckets which are no longer working after the shutdown. It requires a little more effort to get the metadata, but downloads should still be just as fast since Github releases are on S3 as well. Note: still need to implement the upload piece, but this at least provides a model endpoint for users. 2018-03-10 19:03:30 -05:00
Travis
fabd040860 [auto][ci skip] Adding data files from Travis build #381 2018-03-08 01:50:54 +00:00
Al
6575bdc339 [dedupe/test] adding remaining options to near_dupe_test 2018-02-24 00:34:58 -05:00
Al
835de327c3 [dedupe] for near-dupe hashing, remove whitespace from root expansions so something like "Ocean Walk Dr" and "Oceanwalk Dr" will have a chance of matching downstream 2018-02-24 00:34:09 -05:00