Enumeration TokenKind

Enumeration Members

HardSeparator

Separate two tokens that are not in the same context (different phrases).

SoftSeparator

Separate two tokens that are in the same context (same phrase).

StopWord

the token is a stop word, meaning that it can be ignored to optimize size and performance or be indexed as a Word

Unknown
Word