tommy/libpostal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3ff472c8cfa79ccac1ebac19635a5c1bfb40c58b
libpostal/src
History
Al 22c4e99ea0 [parser] As part of reading/tokenizing the address parser data set, 
several copies of the same training example will be generated.

1. with only lowercasing
2. with simple Latin-ASCII normalization (no umlauts, only things that
are common to all languages)
3. basic UTF-8 normalizations (accent stripping)
4. language-specific Latin-ASCII transliteration (e.g. ü => ue in German)

This will apply both on the initial passes when building the phrase
gazetteers and during each iteration of training. In this way, only the
most basic normalizations like lowercasing need to be done at runtime
and it's possible to use only minimal normalizations like lowercasing.

May have a small effect on randomization as examples are created in a
deterministic order. However, this should not lead to cycles since the
base examples are shuffled, thus still satisfying the random permutation
requirement of an online/stochastic learning algorithm.
2016-12-02 13:09:03 -05:00
..
cmp
[geodisambig] C msgpack serialization dependency
2015-04-12 15:14:01 -04:00
geohash
[fix] Removing C++ checks from all but the main API functions
2015-08-07 17:15:39 -04:00
klib
[fix] warnings in klib/ksort.h
2015-10-26 18:50:22 -04:00
linenoise
[cli] Adding antirez's linenoise for command-line interfaces
2015-11-29 11:28:31 -05:00
log
[logging] Wrapping logging statements in a do while (0) so the compiler always at least sees debug code
2015-10-12 13:42:10 -05:00
murmur
[geodisambig] adding MurmurHash3, used by the Bloom filter
2015-03-11 17:47:57 -04:00
sparkey
[fix] Ubuntu build
2015-12-28 17:19:50 -05:00
utf8proc
[unicode] Upgrading to latest utf8proc from JuliaLang (Unicode 8)
2015-12-23 19:33:09 -05:00
address_dictionary_builder.c
[expansion] using easier-to-access data structure for address dictionaries
2016-11-27 00:56:48 -08:00
address_dictionary.c
[expansion] using easier-to-access data structure for address dictionaries
2016-11-27 00:56:48 -08:00
address_dictionary.h
[expansion] using easier-to-access data structure for address dictionaries
2016-11-27 00:56:48 -08:00
address_expansion_data.c
[expansion] regenerated address_expansion_data.c
2016-11-28 10:47:15 -08:00
address_expansion_rule.h
[fix] Adding stdint.h include to most of the header files for portability
2015-08-07 02:43:44 -04:00
address_parser_cli.c
[parser/cli] Using NFC normalization on the output in the parser client (closes #30). Optional command-line arg for parser output dir, useful for spot-checking different experiments
2016-07-21 17:04:57 -04:00
address_parser_io.c
[parser] As part of reading/tokenizing the address parser data set,
2016-12-02 13:09:03 -05:00
address_parser_io.h
[parser] Adding address_parser_rewind to make multiple passes through the file when compiling the phrase tries
2016-07-28 17:13:58 -04:00
address_parser_test.c
[fix] loading transliteration module in address_parser_test.c as well
2016-07-21 17:04:57 -04:00
address_parser_train.c
[parser] stripping postal codes of phrases like CP (in Spanish) before adding them to the gazetteers, whether it's concatenated or a separate token. Adding a command-line argument for the number of iterations
2016-11-30 15:58:03 -08:00
address_parser.c
[parser] Adding country_region label to parser as a boundary component
2016-07-28 15:19:48 -04:00
address_parser.h
[parser] As part of reading/tokenizing the address parser data set,
2016-12-02 13:09:03 -05:00
averaged_perceptron_tagger.c
[fix] Labels in averaged perceptron tagger
2015-09-29 13:06:34 -04:00
averaged_perceptron_tagger.h
[doc] Averaged perceptron tagger
2015-09-23 19:37:55 -04:00
averaged_perceptron_trainer.c
[parser] fixing some uninitialized value issues during parser training
2016-11-30 15:42:09 -08:00
averaged_perceptron_trainer.h
[mv] Moving trie_new_from_hash to a module
2016-01-05 16:43:17 -05:00
averaged_perceptron.c
[parser] Averaged perceptron training
2015-09-10 10:26:24 -07:00
averaged_perceptron.h
[parser] Averaged perceptron training
2015-09-10 10:26:24 -07:00
bench.c
[api] Default options accessible through getters, not static structs
2016-02-15 17:34:00 -05:00
bloom.c
[fix] float comparison
2015-08-07 17:28:21 -04:00
bloom.h
[fix] Adding stdint.h include to most of the header files for portability
2015-08-07 02:43:44 -04:00
chunked_shuffle
[utils] adding a chunked shuffle as the concatenated file sizes may get larger than memory
2016-11-21 14:04:34 -05:00
collections.h
[math] double_array definition in collections.h to use new vectorized exp
2016-08-06 00:40:38 -04:00
constants.h
[constants] max country code length
2015-09-19 01:39:58 -04:00
features.c
[fix] only strdup when necessary in feature counting functions
2016-01-08 00:46:41 -05:00
features.h
[features] No copy versions of feature counts functions
2016-01-05 16:39:50 -05:00
file_utils.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
file_utils.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
float_utils.c
[math] Floating point equality with relative epsilon comparisons
2016-01-02 15:39:49 -05:00
float_utils.h
[math] Floating point equality with relative epsilon comparisons
2016-01-02 15:39:49 -05:00
gazetteer_data.c
[expansion] Prefix/suffix expansions by default can apply to ADDRESS_ANY but also inherit the types of any dictionary that lists their canonical form (so we can add suffixes without worrying about whether they're for streets or place names, etc.)
2016-07-21 17:04:57 -04:00
gazetteers.h
[dictionaries] Making new component for near/nearby prepositions
2016-07-21 17:04:57 -04:00
geo_disambiguation.c
[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)
2015-10-11 00:45:26 -05:00
geo_disambiguation.h
[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)
2015-10-11 00:45:26 -05:00
geodb_builder.c
[geodb] Adding separate bitset for geonames place types and using NFC normalization instead of NFD (requires retraining)
2016-07-21 17:04:57 -04:00
geodb.c
[fix] All file_read_uint64 calls that use stack variables read into a uint64_t not a size_t so as not to smash the stack under a 32-bit arch (issue #18)
2016-02-29 22:36:00 -05:00
geodb.h
[geodb] Additional filenames and struct members in geodb.h
2015-10-09 15:37:10 -04:00
geonames_fields.h
[geonames] Re-generating postal/geonames fields headers
2015-07-08 17:02:59 -04:00
geonames.c
[fix] Check the return of malloc() in geonames.c
2016-02-25 14:53:31 -06:00
geonames.h
[geodb] Adding separate bitset for geonames place types and using NFC normalization instead of NFD (requires retraining)
2016-07-21 17:04:57 -04:00
graph_builder.c
[geodb] Additional filenames and struct members in geodb.h
2015-10-09 15:37:10 -04:00
graph_builder.h
[geodb] Additional filenames and struct members in geodb.h
2015-10-09 15:37:10 -04:00
graph.c
[graph] graph_load and graph_save
2015-10-09 15:36:14 -04:00
graph.h
[graph] graph_load and graph_save
2015-10-09 15:36:14 -04:00
json_encode.c
[api] Simple JSON encoding for strings, UTF-8 rather than Unicode
2015-12-17 12:25:05 -05:00
json_encode.h
[api] Simple JSON encoding for strings, UTF-8 rather than Unicode
2015-12-17 12:25:05 -05:00
language_classifier_cli.c
[language_classification] Making directory optional on language_classifier client/test program
2016-01-27 03:18:53 -05:00
language_classifier_io.c
[language_classification] Automatic hyperparameter optimization using either the cross-validation set or two distinct subsets of the training set
2016-01-17 21:11:37 -05:00
language_classifier_io.h
[language_classification] Automatic hyperparameter optimization using either the cross-validation set or two distinct subsets of the training set
2016-01-17 21:11:37 -05:00
language_classifier_test.c
[language_classification] Making directory optional on language_classifier client/test program
2016-01-27 03:18:53 -05:00
language_classifier_train.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
language_classifier.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
language_classifier.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
language_features.c
[language_classification] Adding script feature for any non-Latin script. Even if the script doesn't directly identify the language, it can act as a modified intercept (all Han script addresses will share the Han feature, even if we haven't seen one of the > 80k Han characters)
2016-01-17 21:37:45 -05:00
language_features.h
[language_classifier] Features for address languages classification, quadgrams for most languages, unigrams for ideographic characters, script for single-script languages like Thai, Hebrew, etc.
2016-01-09 03:42:57 -05:00
libpostal_config.h
[config] language_classifier data dir
2016-01-10 03:05:36 -05:00
libpostal_data
[fix] Adding set -e to data download script so it fails if any subcommands fail
2016-05-04 23:08:06 -04:00
libpostal.c
[expansion] using easier-to-access data structure for address dictionaries
2016-11-27 00:56:48 -08:00
libpostal.h
[dictionaries] Making new component for near/nearby prepositions
2016-07-21 17:04:57 -04:00
logistic_regression_trainer.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
logistic_regression_trainer.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
logistic_regression.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
logistic_regression.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
logistic.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
logistic.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
main.c
[api] Default options accessible through getters, not static structs
2016-02-15 17:34:00 -05:00
Makefile.am
[build] configure/Makefile changes to use SIMD exp and BLAS when available
2016-08-06 00:43:24 -04:00
matrix.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
minibatch.c
[fix] NULL check
2016-01-09 01:43:25 -05:00
minibatch.h
[features] Functions for dealing with minibatches
2016-01-08 00:48:11 -05:00
msgpack_utils.c
[utils] cstring_array_terminate, moving msgpack_utils to separate file
2015-07-25 18:41:02 -04:00
msgpack_utils.h
[fix] Adding stdint.h include to most of the header files for portability
2015-08-07 02:43:44 -04:00
normalize.c
[utils] string_replace also creates a copy
2016-11-30 10:09:33 -08:00
normalize.h
[normalization] Adding NORMALIZE_STRING_SIMPLE_LATIN_ASCII option so parser can normalize punctuation and HTML entities, etc. without touching the alphanumeric parts of the original input
2016-08-21 19:45:32 -04:00
numex_data.c
[numex] regenerated numex data file
2016-11-30 15:58:55 -08:00
numex_rule.h
[numex] Separating rules from keys for Linux gcc compilation
2015-08-09 01:00:57 -04:00
numex_table_builder.c
[fix] numex_table builder program using new API (heap-allocated strings)
2016-03-02 16:28:28 -05:00
numex.c
[fix] Log errors in numex module if not loaded
2016-03-21 18:15:53 -04:00
numex.h
[numex] using module init method for building, otherwise passing NULL path uses the default path
2015-09-16 21:13:05 -04:00
postal_fields.h
[geonames] Re-generating postal/geonames fields headers
2015-07-08 17:02:59 -04:00
scanner.c
[tokenization] Re-generating scanner
2016-07-21 17:04:57 -04:00
scanner.h
[fix] stdint include
2015-09-20 20:10:47 -04:00
scanner.re
[tokenization] Hyphens, etc. between non-ASCII digits (e.g. Unicode full-width numbers) should be single tokens
2016-07-21 17:04:57 -04:00
shuffle.c
[parser] Forgot to add shuffle.h/.c
2015-12-01 11:25:28 -05:00
shuffle.h
[parser] Forgot to add shuffle.h/.c
2015-12-01 11:25:28 -05:00
sparse_matrix_utils.c
[fix] sparse_matrix_new_from_matrix uses new matrix types
2016-11-30 10:15:12 -08:00
sparse_matrix_utils.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
sparse_matrix.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
sparse_matrix.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
stochastic_gradient_descent.c
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
stochastic_gradient_descent.h
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
2016-08-06 00:40:01 -04:00
string_utils.c
[utils] string_replace also creates a copy
2016-11-30 10:09:33 -08:00
string_utils.h
[utils] string_replace also creates a copy
2016-11-30 10:09:33 -08:00
token_types.h
[tokenization] is_whitespace
2016-01-05 16:40:35 -05:00
tokens.c
[fix] calls and NULL checks
2016-07-21 17:04:57 -04:00
tokens.h
[tokenization] tokenized string with source
2015-10-04 18:27:04 -04:00
transliterate.c
[error handling] Checking that resize succeeded in transliterate.c
2016-08-21 19:43:09 -04:00
transliterate.h
[transliteration] Adding LATIN_ASCII_SIMPLE constant to transliterate.h
2016-08-21 19:42:10 -04:00
transliteration_data.c
[transliteration] Regenerating transliteration data files
2016-08-21 19:41:11 -04:00
transliteration_rule.h
[transliteration] Using revisit strings instead of keeping a backtrack count so we don't have to later map logical characters to the actual string, removing any duplicate keys in the table builder so that if any rules happen to overlap within a step, the first will take precedence
2015-05-29 16:54:05 -04:00
transliteration_scripts_data.c
[transliteration] Regenerating transliteration data files
2016-08-21 19:41:11 -04:00
transliteration_table_builder.c
[transliteration] strduping transliterator name while building table
2016-08-21 19:41:34 -04:00
trie_search.c
[fix] header defintion
2016-11-27 01:00:25 -08:00
trie_search.h
[fix] header defintion
2016-11-27 01:00:25 -08:00
trie_utils.c
[mv] Moving trie_new_from_hash to a module
2016-01-05 16:43:17 -05:00
trie_utils.h
[mv] Moving trie_new_from_hash to a module
2016-01-05 16:43:17 -05:00
trie.c
[fix] var declaration during trie creation
2016-03-02 16:05:32 -05:00
trie.h
[fix] All file_read_uint64 calls that use stack variables read into a uint64_t not a size_t so as not to smash the stack under a 32-bit arch (issue #18)
2016-02-29 22:36:00 -05:00
unicode_script_types.h
[unicode] Regenerating unicode_script_data.c
2016-01-17 20:53:44 -05:00
unicode_scripts_data.c
[unicode] Regenerating unicode_script_data.c
2016-01-17 20:53:44 -05:00
unicode_scripts.c
[unicode] Adding SCRIPT_INHERITED as a common script so diacritics like COMBING CEDILLA don't break the current script and produce false word breaks
2016-01-11 16:39:21 -05:00
unicode_scripts.h
[transliteration] Regenerating transliteration data files
2015-09-23 04:04:38 -04:00
vector_math.h
[math] Adding fast SIMD exponent using the Remez algorithm for vectorized exp
2016-08-06 00:31:16 -04:00
vector.h
[utils] Adding realloc checks to vector implementation
2016-08-05 23:02:52 -04:00