tommy/libpostal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[tokenization] Adding a tokenizer method for appending to an existing tokens array (e.g. can stop/start tokenizing on a script change)

Browse Source
This commit is contained in:
Al
2015-06-25 10:03:34 -04:00
parent 581cf406a6
commit 2b69c185fa
3 changed files with 30 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
src/scanner.h
View File

@@ -14,8 +14,9 @@ typedef struct scanner {
uint16_t scan_token(scanner_t *s);
inline scanner_t scanner_from_string(const char *input);
scanner_t scanner_from_string(const char *input, size_t len);
void tokenize_add_tokens(token_array *tokens, const char *input, size_t len);
token_array *tokenize(const char *input);