Package io.github.protasm.jvmud.compiler.token

package io.github.protasm.jvmud.compiler.token
Token model classes representing lexical units recognized by the scanner.

Provides immutable Token records, a strongly typed TokenType enumeration that records expected literal classes, and traversal helpers such as TokenList for parser consumption.

Invariants include matching TokenType.clazz() to any attached literal and preserving scan order with an explicit EOF sentinel.

This package does not perform lexing itself; scanner is responsible for producing instances.

  • Class
    Description
    Token<T>
     
    TokenBuffer
    Immutable storage for a scanned token sequence, plus a stateful Cursor that provides parser-friendly operations (peek/consume/mark).
    TokenBuffer.ParseError
    Lightweight unchecked error a parser can catch at statement/decl boundaries.
    TokenClassifier
    Re-classifies identifier tokens into language keywords.
    TokenList
     
    TokenType