libpostal

cmp

[geodisambig] C msgpack serialization dependency

2015-04-12 15:14:01 -04:00

geohash

[fix] Removing C++ checks from all but the main API functions

2015-08-07 17:15:39 -04:00

klib

Add portable way of formatting khint_t type (from klib)

2017-05-21 11:58:37 +01:00

linenoise

[cli] Adding antirez's linenoise for command-line interfaces

2015-11-29 11:28:31 -05:00

log

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

murmur

[geodisambig] adding MurmurHash3, used by the Bloom filter

2015-03-11 17:47:57 -04:00

sparkey

[build] pulling in change from parser-data that allows user to pass CFLAGS

2016-12-21 14:39:27 -05:00

utf8proc

[strings] reverting to utf8proc v1.3.1, as 2.0 and above can chop off certain sequences

2017-01-01 20:03:23 -05:00

address_dictionary_builder.c

[expansion] using easier-to-access data structure for address dictionaries

2016-11-27 00:56:48 -08:00

address_dictionary.c

[expand] added search_address_dictionaries_substring to support the new use case (i.e. returns "does this substring in the trie?" regardless of if it's stored under the special prefixes/suffixes namespaces)

2017-10-28 02:40:14 -04:00

address_dictionary.h

[expand] added search_address_dictionaries_substring to support the new use case (i.e. returns "does this substring in the trie?" regardless of if it's stored under the special prefixes/suffixes namespaces)

2017-10-28 02:40:14 -04:00

address_expansion_data.c

[auto][ci skip] Adding data files from Travis build #268

2017-10-10 18:58:10 +00:00

address_expansion_rule.h

[fix] Adding stdint.h include to most of the header files for portability

2015-08-07 02:43:44 -04:00

address_parser_cli.c

[parser] removing special commands other than .exit from address_parser_cli

2017-04-04 20:49:37 -04:00

address_parser_io.c

[fix] handle multiple separators (like parens used in https://www.openstreetmap.org/node/244081449 ). Creates bad trie entries otherwise, which affect more than just that toponym

2017-03-18 06:09:52 -04:00

address_parser_io.h

[parser] header changes for the data set struct

2016-12-09 13:37:45 -05:00

address_parser_test.c

[api] doing this now since we're bumping a major version. Using a libpostal prefixes for all public header functions and definitions

2017-03-31 03:35:51 -04:00

address_parser_train.c

Add portable way of formatting khint_t type (from klib)

2017-05-21 11:58:37 +01:00

address_parser.c

[fix] open files in binary format for #69

2017-05-03 17:34:38 -04:00

address_parser.h

[parser] storing address_parser_context on the parser struct itself so it doesn't have to be allocated every time

2017-04-04 20:40:55 -04:00

averaged_perceptron_tagger.c

[parser] moving feature printing to averaged perceptron tagger, taking advantage of trie prefix-sharing in feature incorporating previous tags

2017-03-06 20:32:50 -05:00

averaged_perceptron_tagger.h

[parser] moving feature printing to averaged perceptron tagger, taking advantage of trie prefix-sharing in feature incorporating previous tags

2017-03-06 20:32:50 -05:00

averaged_perceptron_trainer.c

Fix log_* formats which expect long long uint but receive uint64_t.

2017-05-21 10:57:20 +01:00

averaged_perceptron_trainer.h

[parser] learning a sparser averaged perceptron model for the parser using the following method:

2017-03-06 22:28:33 -05:00

averaged_perceptron.c

[parser] using new zeros API in averaged_perceptron.c

2017-02-19 14:02:54 -08:00

averaged_perceptron.h

[parser] Averaged perceptron training

2015-09-10 10:26:24 -07:00

bench.c

[api] doing this now since we're bumping a major version. Using a libpostal prefixes for all public header functions and definitions

2017-03-31 03:35:51 -04:00

bloom.c

[fix] calloc instead of malloc when performing initialization on structs that may fail halfway and need to clean up while partially initialized (calloc will set all the bytes to zero so the member pointers are NULL instead of garbage memory)

2017-01-13 18:30:04 -05:00

bloom.h

[fix] Adding stdint.h include to most of the header files for portability

2015-08-07 02:43:44 -04:00

cartesian_product.c

[utils] cartesian product iterator for grid search during model selection

2017-04-03 00:15:31 -04:00

cartesian_product.h

[utils] cartesian product iterator for grid search during model selection

2017-04-03 00:15:31 -04:00

chunked_shuffle

[utils] adding a chunked shuffle as the concatenated file sizes may get larger than memory

2016-11-21 14:04:34 -05:00

collections.h

[utils] hash_get is no longer a string-only function, can be used for generic hashtables

2017-04-02 23:28:17 -04:00

constants.h

[constants] max country code length

2015-09-19 01:39:58 -04:00

crf_context.c

[openaddresses] adding Chesterfield, SC

2017-03-19 16:10:29 -04:00

crf_context.h

[parser/crf] adding the beginnings of a linear-chain Conditional Random Field

2017-03-10 01:10:22 -05:00

crf_trainer_averaged_perceptron.c

Fix log_* formats which expect long long uint but receive uint64_t.

2017-05-21 10:57:20 +01:00

crf_trainer_averaged_perceptron.h

[parser/crf] adding an initial training algorithm for CRFs, the averaged

2017-03-10 01:28:31 -05:00

crf_trainer.c

[parser/crf] adding crf_trainer, which can be thought of as a "base class" as much as that's possible in C, for creating trainers for the CRF. It doesn't deal with the weights or their representation, just provides an interface for keeping track of string features and label names, and holds the crf_context

2017-03-10 01:25:20 -05:00

crf_trainer.h

[utils] adding a function for checking if files exists (yay C), or at least the closest agreed-upon method for it (may return false if the user doesn't have permissions, but that's ok for our purposes here)

2017-03-10 13:39:52 -05:00

crf.c

[fix] removing log error for sequences of length 0

2017-09-19 23:20:03 -04:00

crf.h

[parser/crf] adding runtime CRF tagger, which can be loaded/used once trained. Currently only does Viterbi inference, can add top-N and/or sequence probabilities later

2017-03-10 02:06:45 -05:00

double_metaphone.c

2017-10-23 15:20:04 -04:00

double_metaphone.h

[similarity] adding basic double metaphone implementation

2017-10-18 03:59:05 -04:00

features.c

[utils] char_array_add_vjoined to stay consistent (add_* methods NUL termiante)

2017-01-09 16:10:07 -05:00

features.h

[features] No copy versions of feature counts functions

2016-01-05 16:39:50 -05:00

file_utils.c

[utils] adding a function for checking if files exists (yay C), or at least the closest agreed-upon method for it (may return false if the user doesn't have permissions, but that's ok for our purposes here)

2017-03-10 13:39:52 -05:00

file_utils.h

[utils] adding file_exists to header

2017-03-10 19:30:46 -05:00

float_utils.c

[utils] adding non-branching sign functions

2017-04-02 13:41:57 -04:00

float_utils.h

[utils] adding non-branching sign functions

2017-04-02 13:41:57 -04:00

ftrl.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

ftrl.h

[optimization] implemented Google's FTRL-Proximal, adapted for the multiclass/multinomial case. It is L1 and L2 regularized, and should both encourage sparsity with the L1 penalty while being robust to collinearity of features due to the L2 penalty. Ref: https://research.google.com/pubs/archive/41159.pdf

2017-04-02 14:28:25 -04:00

gazetteer_data.c

[api] doing this now since we're bumping a major version. Using a libpostal prefixes for all public header functions and definitions

2017-03-31 03:35:51 -04:00

gazetteers.h

[dictionaries] Making new component for near/nearby prepositions

2016-07-21 17:04:57 -04:00

geo_disambiguation.c

[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)

2015-10-11 00:45:26 -05:00

geo_disambiguation.h

[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)

2015-10-11 00:45:26 -05:00

geodb_builder.c

[merge] merging master

2017-01-13 19:58:49 -05:00

geodb.c

[fix] calloc instead of malloc when performing initialization on structs that may fail halfway and need to clean up while partially initialized (calloc will set all the bytes to zero so the member pointers are NULL instead of garbage memory)

2017-01-13 18:30:04 -05:00

geodb.h

[geodb] Additional filenames and struct members in geodb.h

2015-10-09 15:37:10 -04:00

geonames_fields.h

[geonames] Re-generating postal/geonames fields headers

2015-07-08 17:02:59 -04:00

geonames.c

[fix] Check the return of malloc() in geonames.c

2016-02-25 14:53:31 -06:00

geonames.h

[geodb] Adding separate bitset for geonames place types and using NFC normalization instead of NFD (requires retraining)

2016-07-21 17:04:57 -04:00

graph_builder.c

[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead

2017-03-18 06:05:28 -04:00

graph_builder.h

[geodb] Additional filenames and struct members in geodb.h

2015-10-09 15:37:10 -04:00

graph.c

[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead

2017-03-18 06:05:28 -04:00

graph.h

[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead

2017-03-18 06:05:28 -04:00

jaccard.c

[dedupe] Jaccard similarity

2017-10-21 10:34:12 -04:00

jaccard.h

[dedupe] Jaccard similarity

2017-10-21 10:34:12 -04:00

json_encode.c

[api] Simple JSON encoding for strings, UTF-8 rather than Unicode

2015-12-17 12:25:05 -05:00

json_encode.h

[api] Simple JSON encoding for strings, UTF-8 rather than Unicode

2015-12-17 12:25:05 -05:00

language_classifier_cli.c

[language_classification] Making directory optional on language_classifier client/test program

2016-01-27 03:18:53 -05:00

language_classifier_io.c

[log] log the offending line if token count does not match in language_classifier_io

2017-04-02 23:47:07 -04:00

language_classifier_io.h

[language_classification] Automatic hyperparameter optimization using either the cross-validation set or two distinct subsets of the training set

2016-01-17 21:11:37 -05:00

language_classifier_test.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

language_classifier_train.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

language_classifier.c

[language_classification] Runtime language classifier can now use dense or sparse weights, with a different header signature for the sparse version (using old signature for the dense version, so backward-compatible)

2017-04-02 23:51:54 -04:00

language_classifier.h

[language_classification] Runtime language classifier can now use dense or sparse weights, with a different header signature for the sparse version (using old signature for the dense version, so backward-compatible)

2017-04-02 23:51:54 -04:00

language_features.c

[languages] adding replace_hyphens and split_alpha_from_numeric in language classifier input normalization

2017-04-02 23:32:24 -04:00

language_features.h

[language_classifier] Features for address languages classification, quadgrams for most languages, unigrams for ideographic characters, script for single-script languages like Thai, Hebrew, etc.

2016-01-09 03:42:57 -05:00

libpostal_config.h

[build] adding libpostal_setup_datadir, libpostal_setup_parser_datadir, libpostal_setup_language_classifier_datadir functions for configuring the datadir at runtime

2017-01-09 16:11:26 -05:00

libpostal_data

[data] deployed model files and training data to CloudFront for easier downloading around the world and in places like China where the Great Fire Wall may prevent large downloads from abroad. TTL is set to 0 so it still caches the files themselves but checks with origin for the If-Modified-Since headers, allowing the files to be updated dynamically

2017-04-17 14:11:44 -04:00

libpostal.c

[expand] adding a normalization for a single non-acronym internal period where there's an expansion at the prefix/suffix (for #218 and https://github.com/openvenues/libpostal/issues/216#issuecomment-306617824 ). Helps in cases like "St.Michaels" or "Jln.Utara" without needing to specify concatenated prefix phrases for every possibility

2017-10-28 02:38:15 -04:00

libpostal.h

[expand] adding a normalization for a single non-acronym internal period where there's an expansion at the prefix/suffix (for #218 and https://github.com/openvenues/libpostal/issues/216#issuecomment-306617824 ). Helps in cases like "St.Michaels" or "Jln.Utara" without needing to specify concatenated prefix phrases for every possibility

2017-10-28 02:38:15 -04:00

logistic_regression_trainer.c

[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps

2017-04-05 14:18:13 -04:00

logistic_regression_trainer.h

[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps

2017-04-05 14:18:13 -04:00

logistic_regression.c

Revert format regression introduced in ecd07b18c1

2017-05-21 11:14:21 +01:00

logistic_regression.h

[classification] removing regularization update from gradient computation in logistic regression, as that's now handled by the optimizer

2017-04-02 14:32:14 -04:00

logistic.c

[math] using new matrix methods in softmax

2017-04-02 23:29:52 -04:00

logistic.h

[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available

2016-08-06 00:40:01 -04:00

main.c

[fix] expansion array destroy API in libpostal expand program

2017-04-02 23:55:04 -04:00

Makefile.am

[build] Makefile changes to support moving numeric expression parsing to normalize.c

2017-04-18 21:41:24 -04:00

matrix.h

[fix] typo in compiler warning when no CBLAS found

2017-04-12 20:40:08 -04:00

minibatch.c

[fix] NULL check

2016-01-09 01:43:25 -05:00

minibatch.h

[features] Functions for dealing with minibatches

2016-01-08 00:48:11 -05:00

msgpack_utils.c

[utils] cstring_array_terminate, moving msgpack_utils to separate file

2015-07-25 18:41:02 -04:00

msgpack_utils.h

[fix] Adding stdint.h include to most of the header files for portability

2015-08-07 02:43:44 -04:00

ngrams.c

[ngrams] changing args to ngrams

2016-12-21 18:09:45 -05:00

ngrams.h

[ngrams] changing args to ngrams

2016-12-21 18:09:45 -05:00

normalize.c

[expand] adding a normalization for a single non-acronym internal period where there's an expansion at the prefix/suffix (for #218 and https://github.com/openvenues/libpostal/issues/216#issuecomment-306617824 ). Helps in cases like "St.Michaels" or "Jln.Utara" without needing to specify concatenated prefix phrases for every possibility

2017-10-28 02:38:15 -04:00

normalize.h

[expand] adding a normalization for a single non-acronym internal period where there's an expansion at the prefix/suffix (for #218 and https://github.com/openvenues/libpostal/issues/216#issuecomment-306617824 ). Helps in cases like "St.Michaels" or "Jln.Utara" without needing to specify concatenated prefix phrases for every possibility

2017-10-28 02:38:15 -04:00

numex_data.c

[auto][ci skip] Adding data files from Travis build #231

2017-04-20 02:39:39 +00:00

numex_rule.h

[numex] Separating rules from keys for Linux gcc compilation

2015-08-09 01:00:57 -04:00

numex_table_builder.c

[numex] adding ordinal suffixes themselves to the numex trie so they can be removed from strings

2017-04-18 17:20:02 -04:00

numex.c

[numex] fixing edge case where something like "IV Michael" could cause a partial Roman numeral to get added for the MI portion of "Michael"

2017-10-27 04:04:12 -04:00

numex.h

[numex] adding functions to parse and validate a Roman numeral

2017-10-20 02:45:32 -04:00

postal_fields.h

[geonames] Re-generating postal/geonames fields headers

2015-07-08 17:02:59 -04:00

regularization.c

[optimization] moving regularization methods to their own module

2017-04-03 00:16:30 -04:00

regularization.h

[optimization] moving regularization methods to their own module

2017-04-03 00:16:30 -04:00

scanner.c

[tokenization] Re-generating scanner

2016-07-21 17:04:57 -04:00

scanner.h

[fix] stdint include

2015-09-20 20:10:47 -04:00

scanner.re

[tokenization] Hyphens, etc. between non-ASCII digits (e.g. Unicode full-width numbers) should be single tokens

2016-07-21 17:04:57 -04:00

shuffle.c

[utils] adding a function for checking if files exists (yay C), or at least the closest agreed-upon method for it (may return false if the user doesn't have permissions, but that's ok for our purposes here)

2017-03-10 13:39:52 -05:00

shuffle.h

[utils] adding default chunk size to shuffle.h

2017-04-02 13:51:45 -04:00

sparse_matrix_utils.c

[utils] sparse_matrix_add_unique_columns_alias, adds the actual column indices to hashtable/array and aliases those in the table from 1 to N (where N is the number of unique columns in this batch). This way it's compatible with smaller matrices of batch weights.

2017-04-02 13:48:46 -04:00

sparse_matrix_utils.h

[utils] sparse_matrix_add_unique_columns_alias, adds the actual column indices to hashtable/array and aliases those in the table from 1 to N (where N is the number of unique columns in this batch). This way it's compatible with smaller matrices of batch weights.

2017-04-02 13:48:46 -04:00

sparse_matrix.c

[merge] merging master

2017-01-13 19:58:49 -05:00

sparse_matrix.h

[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available

2016-08-06 00:40:01 -04:00

stochastic_gradient_descent.c

Fix log_* formats which expect size_t but receive uint32_t.

2017-05-19 22:31:56 +01:00

stochastic_gradient_descent.h

[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps

2017-04-05 14:18:13 -04:00

string_similarity.c

2017-11-11 03:07:39 -05:00

string_similarity.h

2017-11-11 03:07:39 -05:00

string_utils.c

[utils] adding unicode_equals function in string_utils for testing equality of unicode char arrays

2017-11-11 02:45:41 -05:00

string_utils.h

[utils] adding unicode_equals function in string_utils for testing equality of unicode char arrays

2017-11-11 02:45:41 -05:00

tagger.h

[fix] add #include guard to tagger.h

2017-04-13 13:02:03 -04:00

token_types.h

[merge] merging commit from v1.1

2017-10-12 01:41:04 -04:00

tokens.c

[fix] calls and NULL checks

2016-12-12 11:37:27 -05:00

tokens.h

[merge] merging commit from v1.1

2017-10-12 01:41:04 -04:00

transliterate.c

[merge] merging master

2017-01-13 19:58:49 -05:00

transliterate.h

[merge] merging master

2017-01-13 19:58:49 -05:00

transliteration_data.c

[transliteration] regenerate transliteration data

2017-03-17 18:28:41 -04:00

transliteration_rule.h

[transliteration] Using revisit strings instead of keeping a backtrack count so we don't have to later map logical characters to the actual string, removing any duplicate keys in the table builder so that if any rules happen to overlap within a step, the first will take precedence

2015-05-29 16:54:05 -04:00

transliteration_scripts_data.c

[transliteration] regenerated transliteration_scripts_data.c

2017-01-02 13:52:48 -05:00

transliteration_table_builder.c

[logging] converting most of the steps in building the transliteration table to use debug logging

2017-01-02 00:41:11 -05:00

trie_search.c

[phrases] when skipping/ignoring hyphens in trie search, make sure that the new longer phrase ends at a word boundary (space, hyphen, end of string, etc.)

2017-10-20 02:43:39 -04:00

trie_search.h

[phrases] adding token_phrase_memberships to trie_search for reuse

2017-02-08 01:59:39 -05:00

trie_utils.c

Fix log_{debug,info} formats which expect size_t but receive int.

2017-05-17 22:40:53 +01:00

trie_utils.h

[mv] Moving trie_new_from_hash to a module

2016-01-05 16:43:17 -05:00

trie.c

[fix] open files in binary format for #69

2017-05-03 17:34:38 -04:00

trie.h

[fix] All file_read_uint64 calls that use stack variables read into a uint64_t not a size_t so as not to smash the stack under a 32-bit arch (issue #18 )

2016-02-29 22:36:00 -05:00

unicode_script_types.h

[unicode] Regenerating unicode_script_data.c

2016-01-17 20:53:44 -05:00

unicode_scripts_data.c

[unicode] Regenerating unicode_script_data.c

2016-01-17 20:53:44 -05:00

unicode_scripts.c

[fix] declaring is_common_script function as static

2017-04-02 23:53:21 -04:00

unicode_scripts.h

[transliteration] Regenerating transliteration data files

2015-09-23 04:04:38 -04:00

vector_math.h

[math] adding mean, variance and standard deviation to generic vector functions

2017-04-02 23:29:15 -04:00

vector.h

[utils] new_fixed and resize_fixed in vector.h

2017-03-10 19:31:34 -05:00