Class Lexer

java.lang.Object
com.iizix.prop.gunit.internal.Lexer
public final class Lexer extends Object
The left-to-right tokeniser for unit formulas.

It produces number, identifier, operator, parenthesis and comma tokens, each carrying its span in the original, untouched source string. Nothing is substituted, lower-cased or stripped beforehand.

Why this replaces the previous scanner. The old implementation replaced every unit name with a single control character across the whole expression before it scanned anything. That has two consequences this class is written to avoid:

  • Every source position after the first unit was shifted by a varying amount, since "dlg" collapsed from three characters to one, so an error could not be reported at the right caret position.
  • Pre-substitution and identifiers are structurally incompatible. No unit name is a substring of "min" or "max", so those two happen to be safe, but the next function would not be: exp() would become <UT_EX>p( because ex is a unit.

Whitespace. Whitespace separates tokens and is otherwise ignored, so "10 px" and "10px" are the same, matching the documented input format. Whitespace INSIDE a number is not accepted; the old scanner accepted "10.00 00" as 10.0000 because it stripped every space before scanning, which was an accident of the implementation rather than an intended format.

This also disposes of a bug in the shipping code: TAB, LF and CR were themselves valid unit markers, being the control characters assigned to %min, %max and vmin, while normalisation only stripped the space character. Any formula arriving with a stray tab or newline, from an XML or JSON round trip, a pasted value or a multi-line field, silently parsed as a different unit. Here they are simply whitespace.

Author:
Christopher Mindus

  • Constructor Details

    • Lexer

      public Lexer(String src)
      Creates a lexer over a source string.
      Parameters:
      src - The formula source, exactly as the author wrote it.

  • Method Details

    • getSource

      public String getSource()
      Gets the source string being scanned.
      Returns:
      The original, untouched source string.

    • tokenize

      public Token[] tokenize() throws FormulaException
      Scans the whole source into a token array terminated by a single TokenKind.EOF token.

      The parser wants an array rather than a stream because it needs two tokens of lookahead in one place, to tell a function call from a unit suffix, and a formula is short enough that scanning it whole costs nothing.

      Returns:
      The tokens, always at least one, the last always TokenKind.EOF.
      Throws:
      FormulaException - On an illegal character or a malformed number, with the offset and length of the offending span.