Class Token

Index of the last byte of the original lemma.

The length in bytes of the normalized lemma.

Index of the first byte of the original lemma.

The count of characters of the normalized lemma.

Index of the last character of the original lemma.

Index of the first character of the original lemma.

Whether the token is a separator.

Whether the token is a stop word.

Whether the token is a word. A token is considered as a word if it's not a separator nor a stop word.

Kind of the Token assigned by the classifier.

Language of the Token.

The normalized lemma.

The length in bytes of the original lemma.

The count of characters of the original lemma.

Script of the Token.

The kind of the separator if the token is a separator.