libpostal

cmp

[geodisambig] C msgpack serialization dependency

2015-04-12 15:14:01 -04:00

geohash

[fix] Removing C++ checks from all but the main API functions

2015-08-07 17:15:39 -04:00

klib

[fix] deleting comment, this is not a header-only implementation

2017-11-29 12:26:36 -05:00

linenoise

Updated linenoise to be MSys2/MinGW compatible. Updated address_parser app to use the defined libpostal api and not include internal components directly. Removed windows src Makefile as it is now the same as the standard one.

2017-11-27 01:42:25 +00:00

log

Removing console colors for Windows builds.

2017-12-18 18:33:50 +00:00

murmur

[geodisambig] adding MurmurHash3, used by the Bloom filter

2015-03-11 17:47:57 -04:00

sparkey

[build] pulling in change from parser-data that allows user to pass CFLAGS

2016-12-21 14:39:27 -05:00

utf8proc

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

acronyms.c

2017-12-04 15:21:44 -05:00

acronyms.h

2017-12-04 15:21:44 -05:00

address_dictionary_builder.c

[expansion] using easier-to-access data structure for address dictionaries

2016-11-27 00:56:48 -08:00

address_dictionary.c

[expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation

2017-12-17 03:14:00 -05:00

address_dictionary.h

[expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation

2017-12-17 03:14:00 -05:00

address_expansion_data.c

[auto][ci skip] Adding data files from Travis build #323

2017-12-12 14:57:29 +00:00

address_expansion_rule.h

[fix] Adding stdint.h include to most of the header files for portability

2015-08-07 02:43:44 -04:00

address_parser_cli.c

Modifed the libpostal API to add an extra function libpostal_parser_print_features to toggle debugging info. Updated address_parser app to use the new function.

2017-11-27 19:20:37 +00:00

address_parser_io.c

[fix] handle multiple separators (like parens used in https://www.openstreetmap.org/node/244081449 ). Creates bad trie entries otherwise, which affect more than just that toponym

2017-03-18 06:09:52 -04:00

address_parser_io.h

[parser] header changes for the data set struct

2016-12-09 13:37:45 -05:00

address_parser_test.c

[api] doing this now since we're bumping a major version. Using a libpostal prefixes for all public header functions and definitions

2017-03-31 03:35:51 -04:00

address_parser_train.c

Add portable way of formatting khint_t type (from klib)

2017-05-21 11:58:37 +01:00

address_parser.c

Fix leak of normalized value in early return

2017-12-25 19:33:07 -08:00

address_parser.h

[parser] adding label constants to address_parser header

2017-12-24 01:45:50 -05:00

averaged_perceptron_tagger.c

[parser] moving feature printing to averaged perceptron tagger, taking advantage of trie prefix-sharing in feature incorporating previous tags

2017-03-06 20:32:50 -05:00

averaged_perceptron_tagger.h

[parser] moving feature printing to averaged perceptron tagger, taking advantage of trie prefix-sharing in feature incorporating previous tags

2017-03-06 20:32:50 -05:00

averaged_perceptron_trainer.c

Fix log_* formats which expect long long uint but receive uint64_t.

2017-05-21 10:57:20 +01:00

averaged_perceptron_trainer.h

[parser] learning a sparser averaged perceptron model for the parser using the following method:

2017-03-06 22:28:33 -05:00

averaged_perceptron.c

[parser] using new zeros API in averaged_perceptron.c

2017-02-19 14:02:54 -08:00

averaged_perceptron.h

[parser] Averaged perceptron training

2015-09-10 10:26:24 -07:00

bench.c

[api] doing this now since we're bumping a major version. Using a libpostal prefixes for all public header functions and definitions

2017-03-31 03:35:51 -04:00

bloom.c

[fix] calloc instead of malloc when performing initialization on structs that may fail halfway and need to clean up while partially initialized (calloc will set all the bytes to zero so the member pointers are NULL instead of garbage memory)

2017-01-13 18:30:04 -05:00

bloom.h

[fix] Adding stdint.h include to most of the header files for portability

2015-08-07 02:43:44 -04:00

cartesian_product.c

[utils] cartesian product iterator for grid search during model selection

2017-04-03 00:15:31 -04:00

cartesian_product.h

[utils] cartesian product iterator for grid search during model selection

2017-04-03 00:15:31 -04:00

chunked_shuffle

[utils] adding a chunked shuffle as the concatenated file sizes may get larger than memory

2016-11-21 14:04:34 -05:00

collections.h

[utils] hash_get is no longer a string-only function, can be used for generic hashtables

2017-04-02 23:28:17 -04:00

constants.h

[constants] max country code length

2015-09-19 01:39:58 -04:00

crf_context.c

[openaddresses] adding Chesterfield, SC

2017-03-19 16:10:29 -04:00

crf_context.h

[parser/crf] adding the beginnings of a linear-chain Conditional Random Field

2017-03-10 01:10:22 -05:00

crf_trainer_averaged_perceptron.c

Fix log_* formats which expect long long uint but receive uint64_t.

2017-05-21 10:57:20 +01:00

crf_trainer_averaged_perceptron.h

[parser/crf] adding an initial training algorithm for CRFs, the averaged

2017-03-10 01:28:31 -05:00

crf_trainer.c

[parser/crf] adding crf_trainer, which can be thought of as a "base class" as much as that's possible in C, for creating trainers for the CRF. It doesn't deal with the weights or their representation, just provides an interface for keeping track of string features and label names, and holds the crf_context

2017-03-10 01:25:20 -05:00

crf_trainer.h

[utils] adding a function for checking if files exists (yay C), or at least the closest agreed-upon method for it (may return false if the user doesn't have permissions, but that's ok for our purposes here)

2017-03-10 13:39:52 -05:00

crf.c

[fix] removing log error for sequences of length 0

2017-09-19 23:20:03 -04:00

crf.h

[parser/crf] adding runtime CRF tagger, which can be loaded/used once trained. Currently only does Viterbi inference, can add top-N and/or sequence probabilities later

2017-03-10 02:06:45 -05:00

double_metaphone.c

2017-10-23 15:20:04 -04:00

double_metaphone.h

[similarity] adding basic double metaphone implementation

2017-10-18 03:59:05 -04:00

expand.c

[expand] adding improvements to root expansions (using possible phrase roots even if they're abbreviated e.g. "E Ctr St", adding special valid components check for root expansions beyond what's stored in the build address dictionaries), removing spaces before checking unique strings, only splitting numeric from alpha in the case of non-ordinals, using cstring_array internally and char ** in the public API

2017-12-25 01:37:42 -05:00

expand.h

[expand] adding improvements to root expansions (using possible phrase roots even if they're abbreviated e.g. "E Ctr St", adding special valid components check for root expansions beyond what's stored in the build address dictionaries), removing spaces before checking unique strings, only splitting numeric from alpha in the case of non-ordinals, using cstring_array internally and char ** in the public API

2017-12-25 01:37:42 -05:00

features.c

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

features.h

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

file_utils.c

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

file_utils.h

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

float_utils.c

[utils] adding non-branching sign functions

2017-04-02 13:41:57 -04:00

float_utils.h

[utils] adding non-branching sign functions

2017-04-02 13:41:57 -04:00

ftrl.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

ftrl.h

[optimization] implemented Google's FTRL-Proximal, adapted for the multiclass/multinomial case. It is L1 and L2 regularized, and should both encourage sparsity with the L1 penalty while being robust to collinearity of features due to the L2 penalty. Ref: https://research.google.com/pubs/archive/41159.pdf

2017-04-02 14:28:25 -04:00

gazetteer_data.c

[fix] reverting gazetteer changes as it would affect the parser features as well and require retraining

2017-12-24 23:51:35 -05:00

gazetteers.h

[dictionaries] Making new component for near/nearby prepositions

2016-07-21 17:04:57 -04:00

geo_disambiguation.c

[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)

2015-10-11 00:45:26 -05:00

geo_disambiguation.h

[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)

2015-10-11 00:45:26 -05:00

geodb_builder.c

[merge] merging master

2017-01-13 19:58:49 -05:00

geodb.c

[fix] calloc instead of malloc when performing initialization on structs that may fail halfway and need to clean up while partially initialized (calloc will set all the bytes to zero so the member pointers are NULL instead of garbage memory)

2017-01-13 18:30:04 -05:00

geodb.h

[geodb] Additional filenames and struct members in geodb.h

2015-10-09 15:37:10 -04:00

geonames_fields.h

[geonames] Re-generating postal/geonames fields headers

2015-07-08 17:02:59 -04:00

geonames.c

[fix] Check the return of malloc() in geonames.c

2016-02-25 14:53:31 -06:00

geonames.h

[geodb] Adding separate bitset for geonames place types and using NFC normalization instead of NFD (requires retraining)

2016-07-21 17:04:57 -04:00

graph_builder.c

[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead

2017-03-18 06:05:28 -04:00

graph_builder.h

[geodb] Additional filenames and struct members in geodb.h

2015-10-09 15:37:10 -04:00

graph.c

[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead

2017-03-18 06:05:28 -04:00

graph.h

[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead

2017-03-18 06:05:28 -04:00

jaccard.c

[dedupe] Jaccard similarity

2017-10-21 10:34:12 -04:00

jaccard.h

[dedupe] Jaccard similarity

2017-10-21 10:34:12 -04:00

json_encode.c

[api] Simple JSON encoding for strings, UTF-8 rather than Unicode

2015-12-17 12:25:05 -05:00

json_encode.h

[api] Simple JSON encoding for strings, UTF-8 rather than Unicode

2015-12-17 12:25:05 -05:00

language_classifier_cli.c

[language_classification] Making directory optional on language_classifier client/test program

2016-01-27 03:18:53 -05:00

language_classifier_io.c

[log] log the offending line if token count does not match in language_classifier_io

2017-04-02 23:47:07 -04:00

language_classifier_io.h

[language_classification] Automatic hyperparameter optimization using either the cross-validation set or two distinct subsets of the training set

2016-01-17 21:11:37 -05:00

language_classifier_test.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

language_classifier_train.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

language_classifier.c

[language_classification] Runtime language classifier can now use dense or sparse weights, with a different header signature for the sparse version (using old signature for the dense version, so backward-compatible)

2017-04-02 23:51:54 -04:00

language_classifier.h

[language_classification] Runtime language classifier can now use dense or sparse weights, with a different header signature for the sparse version (using old signature for the dense version, so backward-compatible)

2017-04-02 23:51:54 -04:00

language_features.c

[languages] adding replace_hyphens and split_alpha_from_numeric in language classifier input normalization

2017-04-02 23:32:24 -04:00

language_features.h

[language_classifier] Features for address languages classification, quadgrams for most languages, unigrams for ideographic characters, script for single-script languages like Thai, Hebrew, etc.

2016-01-09 03:42:57 -05:00

libpostal_config.h

[build] adding libpostal_setup_datadir, libpostal_setup_parser_datadir, libpostal_setup_language_classifier_datadir functions for configuring the datadir at runtime

2017-01-09 16:11:26 -05:00

libpostal_data

[build] adding --no-same-owner explicitly when untarring the data files for #267

2017-11-01 20:05:36 -04:00

libpostal.c

[api] adding API functions for near dupe hashes to the public header

2017-12-24 12:43:28 -05:00

libpostal.h

[api] adding API functions for near dupe hashes to the public header

2017-12-24 12:43:28 -05:00

logistic_regression_trainer.c

[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps

2017-04-05 14:18:13 -04:00

logistic_regression_trainer.h

[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps

2017-04-05 14:18:13 -04:00

logistic_regression.c

Revert format regression introduced in ecd07b18c1

2017-05-21 11:14:21 +01:00

logistic_regression.h

[classification] removing regularization update from gradient computation in logistic regression, as that's now handled by the optimizer

2017-04-02 14:32:14 -04:00

logistic.c

[math] using new matrix methods in softmax

2017-04-02 23:29:52 -04:00

logistic.h

[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available

2016-08-06 00:40:01 -04:00

main.c

[fix] expansion array destroy API in libpostal expand program

2017-04-02 23:55:04 -04:00

Makefile.am

[build] adding new source files for near dupe hashing and the command-line program to the Makefile

2017-12-24 12:44:10 -05:00

matrix.h

[fix] typo in compiler warning when no CBLAS found

2017-04-12 20:40:08 -04:00

minibatch.c

[fix] NULL check

2016-01-09 01:43:25 -05:00

minibatch.h

[features] Functions for dealing with minibatches

2016-01-08 00:48:11 -05:00

msgpack_utils.c

[utils] cstring_array_terminate, moving msgpack_utils to separate file

2015-07-25 18:41:02 -04:00

msgpack_utils.h

[fix] Adding stdint.h include to most of the header files for portability

2015-08-07 02:43:44 -04:00

near_dupe_test.c

[dedupe] adding a test program for near dupe hashing that simply prints out the results. Automated tests in the works

2017-12-24 12:41:44 -05:00

near_dupe.c

[dedupe] adding near-dupe hashing function, which can be thought of as the blocking function in record linkage or as a form of locally sensitive hashing in general document deduping. The goal is, if two addresses/names are the same, they should share at least one hash. These hashes can also be used as an inverted index (DB, ES, hashtable, etc.). Uses the double metaphone for name words in Latin script (otherwise each individual token, and sequences of two tokens in the case of ideograms for e.g. Chinese, Japanese, Korean, etc.)

2017-12-24 02:47:45 -05:00

near_dupe.h

[dedupe] adding near-dupe hashing function, which can be thought of as the blocking function in record linkage or as a form of locally sensitive hashing in general document deduping. The goal is, if two addresses/names are the same, they should share at least one hash. These hashes can also be used as an inverted index (DB, ES, hashtable, etc.). Uses the double metaphone for name words in Latin script (otherwise each individual token, and sequences of two tokens in the case of ideograms for e.g. Chinese, Japanese, Korean, etc.)

2017-12-24 02:47:45 -05:00

ngrams.c

[ngrams] changing args to ngrams

2016-12-21 18:09:45 -05:00

ngrams.h

[ngrams] changing args to ngrams

2016-12-21 18:09:45 -05:00

normalize.c

[expand/normalize] the split_alpha_from_numeric option now applies to both e.g. A1 and 1A since we now strip out ordinal suffixes prior to normalization

2017-12-17 19:53:15 -05:00

normalize.h

Fix bug in strndup fix for windows. Move all includes out of headers and into code for strndup.h and move it to be the last include.

2017-11-23 19:11:25 +00:00

numex_data.c

[auto][ci skip] Adding data files from Travis build #284

2017-11-24 22:29:45 +00:00

numex_rule.h

[numex] Separating rules from keys for Linux gcc compilation

2015-08-09 01:00:57 -04:00

numex_table_builder.c

[numex] adding ordinal suffixes themselves to the numex trie so they can be removed from strings

2017-04-18 17:20:02 -04:00

numex.c

[merge] merging in the Ohio expansion numex changes from master

2017-11-29 11:51:43 -05:00

numex.h

[merge] merging in the Ohio expansion numex changes from master

2017-11-29 11:51:43 -05:00

postal_fields.h

[geonames] Re-generating postal/geonames fields headers

2015-07-08 17:02:59 -04:00

regularization.c

[optimization] moving regularization methods to their own module

2017-04-03 00:16:30 -04:00

regularization.h

[optimization] moving regularization methods to their own module

2017-04-03 00:16:30 -04:00

scanner.c

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

scanner.h

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

scanner.re

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

shuffle.c

[utils] adding a function for checking if files exists (yay C), or at least the closest agreed-upon method for it (may return false if the user doesn't have permissions, but that's ok for our purposes here)

2017-03-10 13:39:52 -05:00

shuffle.h

[utils] adding default chunk size to shuffle.h

2017-04-02 13:51:45 -04:00

sparse_matrix_utils.c

[utils] sparse_matrix_add_unique_columns_alias, adds the actual column indices to hashtable/array and aliases those in the table from 1 to N (where N is the number of unique columns in this batch). This way it's compatible with smaller matrices of batch weights.

2017-04-02 13:48:46 -04:00

sparse_matrix_utils.h

[utils] sparse_matrix_add_unique_columns_alias, adds the actual column indices to hashtable/array and aliases those in the table from 1 to N (where N is the number of unique columns in this batch). This way it's compatible with smaller matrices of batch weights.

2017-04-02 13:48:46 -04:00

sparse_matrix.c

[merge] merging master

2017-01-13 19:58:49 -05:00

sparse_matrix.h

[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available

2016-08-06 00:40:01 -04:00

stochastic_gradient_descent.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

stochastic_gradient_descent.h

[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps

2017-04-05 14:18:13 -04:00

string_similarity.c

2017-12-08 14:27:23 -05:00

string_similarity.h

2017-11-12 04:48:26 -05:00

string_utils.c

[utils] adding cstring_array_extend and string_tree_clear

2017-12-24 01:46:20 -05:00

string_utils.h

[utils] adding cstring_array_extend and string_tree_clear

2017-12-24 01:46:20 -05:00

strndup.c

[fix] conditional compilation for strndup and drand48 for Windows, using config.h

2017-11-29 12:21:13 -05:00

strndup.h

[fix] conditional compilation for strndup and drand48 for Windows, using config.h

2017-11-29 12:21:13 -05:00

tagger.h

[fix] add #include guard to tagger.h

2017-04-13 13:02:03 -04:00

token_types.h

[merge] merging commit from v1.1

2017-10-12 01:41:04 -04:00

tokens.c

Fix bug in strndup fix for windows. Move all includes out of headers and into code for strndup.h and move it to be the last include.

2017-11-23 19:11:25 +00:00

tokens.h

[expand] adding improvements to root expansions (using possible phrase roots even if they're abbreviated e.g. "E Ctr St", adding special valid components check for root expansions beyond what's stored in the build address dictionaries), removing spaces before checking unique strings, only splitting numeric from alpha in the case of non-ordinals, using cstring_array internally and char ** in the public API

2017-12-25 01:37:42 -05:00

transliterate.c

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

transliterate.h

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

transliteration_data.c

[transliteration] regenerate transliteration data

2017-03-17 18:28:41 -04:00

transliteration_rule.h

[transliteration] Using revisit strings instead of keeping a backtrack count so we don't have to later map logical characters to the actual string, removing any duplicate keys in the table builder so that if any rules happen to overlap within a step, the first will take precedence

2015-05-29 16:54:05 -04:00

transliteration_scripts_data.c

[transliteration] regenerated transliteration_scripts_data.c

2017-01-02 13:52:48 -05:00

transliteration_table_builder.c

[logging] converting most of the steps in building the transliteration table to use debug logging

2017-01-02 00:41:11 -05:00

trie_search.c

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

trie_search.h

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

trie_utils.c

Fix log_{debug,info} formats which expect size_t but receive int.

2017-05-17 22:40:53 +01:00

trie_utils.h

[mv] Moving trie_new_from_hash to a module

2016-01-05 16:43:17 -05:00

trie.c

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

trie.h

Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.

2017-11-25 04:35:28 +00:00

unicode_script_types.h

[unicode] Regenerating unicode_script_data.c

2016-01-17 20:53:44 -05:00

unicode_scripts_data.c

[unicode] Regenerating unicode_script_data.c

2016-01-17 20:53:44 -05:00

unicode_scripts.c

[fix] declaring is_common_script function as static

2017-04-02 23:53:21 -04:00

unicode_scripts.h

[transliteration] Regenerating transliteration data files

2015-09-23 04:04:38 -04:00

vector_math.h

[math] adding mean, variance and standard deviation to generic vector functions

2017-04-02 23:29:15 -04:00

vector.h

[utils] new_fixed and resize_fixed in vector.h

2017-03-10 19:31:34 -05:00