3 Commits

Author	SHA1	Message	Date
Al	ed05aaabb1	[utils] adding default chunk size to shuffle.h	2017-04-02 13:51:45 -04:00
Al	b76b7b8527	[parser] adding chunked shuffle as a C function (writes each line to one of n random files, runs shuf on each file and concatenates the result). Adding a version which allows specifying a specific chunk size, and using a 2GB limit for address parser training. Allowing gshuf again for Mac as it seems the only problem there was not having enough memory when testing on a Mac laptop. The new limited-memory version should be fast enough.	2017-03-05 02:15:11 -05:00
Al	b94264b745	[parser] Forgot to add shuffle.h/.c	2015-12-01 11:25:28 -05:00