Module tokenizer

The tokenizer: turns a string into tokens, such as numbers, strings, and keywords

Structs

Tokenizer

The tokenizer. You may want to use the tokenize convenience function from this module instead.

Functions

tokenize

A convenience function for tokenizing the given input

Type Aliases

Token