[tokenization] non-breaking dashes can be mid-word, em-dashes, etc. break words

This commit is contained in:
Al
2015-04-17 15:20:31 -04:00
parent e21873635c
commit 6718182443
2 changed files with 187564 additions and 185374 deletions

372928
src/scanner.c

File diff suppressed because it is too large Load Diff