Al
d575caba8a
[data] using UTC for libpostal data files on the Mac version of the download script as well
2016-12-09 19:43:05 -05:00
Al
c3f3896b48
[fix] update test for date function in data download script
2016-12-09 19:29:00 -05:00
Al
01afbf80ef
[data] Each curl process will retry the chunk up to 3 times
2016-08-25 23:18:39 -04:00
Tom Davis
18c8e90eb3
Use xargs to start workers as soon as possible
2016-07-27 17:46:44 -04:00
Tom Davis
11abf6cb22
Use posix sh for systems without bash
2016-07-26 20:17:18 -04:00
Tom Davis
2991ffd193
Don't call download_multipart for 1 chunk
...
Previously, where a file was larger than `$LARGE_FILE_SIZE` but smaller
than `$CHUNK_SIZE*2`, `download_multipart` would be called but would
only download one (1) chunk that was the whole file.
This fix keeps the same download performance as before but optimizes
processing chunks out.
2016-07-23 16:41:04 -04:00
Tom Davis
24e0314e71
Remove call to seq which may not exist
2016-07-23 01:03:15 -04:00
Al
ad9dfb46bd
[build] Using a process pool with 64MB chunks (similar to aws cli) for S3 downloads. Setting the max concurrent requeests to 10, also the default in aws cli.
2016-07-01 14:37:13 -04:00
Al
a9ba61585b
[fix] Adding set -e to data download script so it fails if any subcommands fail
2016-05-04 23:08:06 -04:00
Al
59e5fcd1b4
[fix] LC_ALL=C in data download script
2016-04-11 12:47:50 -04:00
Al
0d7f9f2032
[data] Using UTC dates for libpostal data file tracking for #38 . Also silencing curl when checking if file was updated
2016-03-10 16:44:02 -05:00
Al
c0b548833b
[fix] create data dir if it doesn't exist
2016-01-30 13:40:10 -05:00
Al
789db8f582
[build] Adding language classifier to data file download script. As the current file is rather large, added multipart downloads from S3 to speed things up
2016-01-27 03:31:45 -05:00
Al
2950358697
[build] address_parser client now links to libpostal, adding address_parser to download script with an "all" option
2015-12-12 12:49:50 -05:00
Al
6aaa08c220
[fix] Usage on libpostal_data script
2015-10-27 13:33:03 -04:00
Al
588cf1df86
[build] Changing options to libpostal_data script to allow downloading geodb, uploaded first version to S3
2015-10-11 22:25:37 -05:00
Al
91f4e477ad
[fix] typo
2015-10-06 12:04:07 -04:00
Al
abfa744d59
[build] Adding libpostal_data script for downloading data from S3, Makefile uses that now as part of the all-local target. Can be run periodically after install
2015-09-28 17:26:15 -04:00
