5496 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Al	8bc77372ef	[phrases] exposing trie_add_at_index and trie_get_from_index for more control in the transliteration tries	2015-04-26 22:24:02 -04:00
Al	6ebea11640	[transliteration] fixing transliteration rules, fixing escape characters, adding sizes to all the strings as they may have null characters	2015-04-26 19:47:54 -04:00
Al	ff9b6735f8	[transliteration] Adding header + generated C data file for simplified transliteration rules	2015-04-25 15:44:36 -04:00
Al	be29874f13	[transliteration] Parser for CLDR transforms to generate (simple) C transform rules	2015-04-25 15:42:21 -04:00
Al	1b33744956	[tokenization] Numeric tokens must end in number or letter	2015-04-22 14:55:18 -04:00
Al	9c0126a01c	[utils] two set types in collections.h	2015-04-19 09:32:53 -04:00
Al	908e3dc03c	[phrases] trie_search now only takes the original string and the token array. Fixed a bug where certain phrases were being found in string search but not in tokenized search	2015-04-19 09:32:20 -04:00
Al	606a669c01	[tokenization] breaking dashes or double hyphens break a word while other dashes don't	2015-04-17 19:14:42 -04:00
Al	6718182443	[tokenization] non-breaking dashes can be mid-word, em-dashes, etc. break words	2015-04-17 15:21:22 -04:00
Al	e21873635c	[utils] Using token offsets to calculate lengths for contiguous string arrays, inlining a few functions	2015-04-15 20:17:03 -04:00
Al	24e62b1c6c	[tokenization] Script to generate TR-29 ranges for re2c scanner	2015-04-14 15:50:50 -04:00
Al	5fa03587fb	[cldr] simple Python scanner for creating dynamic scanners for CLDR rule parsing	2015-04-14 15:49:24 -04:00
Al	efdcbc9eef	[project] adding a Python .gitignore for scripts, Python lib, etc.	2015-04-14 15:48:43 -04:00
Al	6e9295154a	[fix] local dirs for cldr data	2015-04-14 15:46:15 -04:00
Al	744231c148	[fix] cldr supplemental uses local copy	2015-04-13 19:03:44 -04:00
Al	a8b9981c9b	[fix] vars	2015-04-13 19:03:14 -04:00
Al	d1267145f7	[fix] args to wget	2015-04-13 19:02:50 -04:00
Al	d771da7c78	[i18n] unicode scripts file downloaded and cached locally	2015-04-13 19:02:29 -04:00
Al	cc4d2d08eb	[cldr] Adding script to download latest cldr release instead of pulling from the repo	2015-04-13 01:03:15 -04:00
Al	e241c1dfc8	[rm] Removing dependency on sds, char_array and cstring_array have similar benefits/functionality with fewer drawbacks	2015-04-12 18:07:33 -04:00
Al	83813bb980	[geodisambig] Models for geonames with msgpack serialization/deserialization	2015-04-12 16:47:01 -04:00
Al	acb575c84c	[fix] splitting out methods for unicode scripts	2015-04-12 15:21:23 -04:00
Al	1f9da05dd5	[geodisambig] C msgpack serialization dependency	2015-04-12 15:14:01 -04:00
Al	0234754c20	[fix] warnings in string_utils	2015-04-12 12:16:32 -04:00
Al	d50d7d182e	[fix] geonames import script for admin 1 codes	2015-04-12 12:16:08 -04:00
Al	888baa86f3	[fix] English dictionaries	2015-04-12 12:15:47 -04:00
Al	3a7f18581e	[utils] Adding min, max, argmin, argmax and log_sum_exp to generic vector math header	2015-04-12 12:11:04 -04:00
Al	fdd0c489f3	[fix] refactoring unicode script fetching into more reusable functions	2015-04-09 02:18:13 -04:00
Al	4729dfe178	[utils] string_[rl]strip => string_[rl]trim, removing warning about allocation	2015-04-06 02:19:19 -04:00
Al	53844067b1	[fix] better allocation sizes for tokenized strings	2015-04-05 22:02:31 -04:00
Al	198e51b8a3	[utils] more/better char_array methods	2015-04-05 22:01:46 -04:00
Al	79fd7a8ded	[tokenization/trie] simpler url regex reduces the scanner file size, accounting for a few more variations in word tokens, making trie suffix search use iteration instead of malloc'ing a new string	2015-04-05 16:33:14 -04:00
Al	5f3d74de18	[fix] contiguous string array	2015-04-03 11:22:50 -04:00
Al	fcaeebd656	[dictionaries] fixes to French dictionary	2015-04-01 19:02:38 -04:00
Al	c81aa72254	[utils] a few changes to contiguous string arrays	2015-04-01 19:02:11 -04:00
Al	fa59b63ab2	[fix] type name/import	2015-04-01 02:54:14 -04:00
Al	310acbed2c	[phrases] Adding prefix-only trie searches, primarily with Germanic languages in mind (spelled out numbers, concatenated prefixes). Making the prefix/suffix APIs for single tokens more consistent with trie searches over longer strings/token arrays	2015-04-01 02:52:57 -04:00
Al	1ac4438e39	[utils] More consistent naming in string_utils	2015-03-27 21:12:08 -04:00
Al	70831b5005	[dictionaries] French elisions	2015-03-27 21:03:55 -04:00
Al	127a61d492	[utils] adding pop method on the improved vectors	2015-03-27 21:00:03 -04:00
Al	3678d4a3ca	[gazetteers] string name doesn't need to be part of the gazetteer itself, adding two new dictionary types to the config for named people (e.g. JFK) and named organizations (e.g. UN)	2015-03-27 20:59:21 -04:00
Al	4ccd1b1fe2	[fix] update feature arrays to use the new APIs	2015-03-27 20:57:42 -04:00
Al	6768936953	[utils] Adding vector_math.h with some inline methods for vector operations (sum, dot product, arithmetic, etc.). Works with kvec dynamic vectors.	2015-03-27 20:57:03 -04:00
Al	70195fffd5	[utils] new methods on string_utils for better dynamic strings which retains the benefits of sds without having to worry about the pointer changing, renaming contiguous string array methods to something more succinct	2015-03-27 20:55:36 -04:00
Al	2d1c24a6e9	[tokenization] Adding url, email, US/international phone numbers, a separate type for ideographic numbers, more general quotes, paren types	2015-03-24 16:43:53 -04:00
Al	50187f28ce	[fix] .txt extension	2015-03-23 02:17:07 -04:00
Al	7ffe788913	[unicode] header	2015-03-18 17:25:53 -04:00
Al	d5a9041cd3	[unicode] Adding generated unicode script data	2015-03-18 17:01:03 -04:00
Al	e03c1f21a7	[unicode] generate C headers/data files from unicode.org scripts	2015-03-18 16:59:58 -04:00
Al	6c8e5b45a4	[fix] removing building alias (for OSm it means building category), fix to fetch script	2015-03-18 08:40:07 -04:00