Tokenizer

simple tokenizer for DlangUI ML

class Tokenizer {

string[] _singleLineCommentPrefixes;

LineStream _lines;

dchar[] _lineText;

ushort _line;

dchar _prevChar;

string _filename;

Token _token;

anonymousenum EOF_CHAR;

this(string source, string filename, string[] singleLineCommentPrefixes);

dchar peekChar();

dchar peekNextChar();

dchar nextChar();

dchar skipChar();

void setTokenStart();

const(Token) parseEof();

const(Token) parseEol();

const(Token) parseWhiteSpace();

static bool isAlpha(dchar ch);

static bool isNum(dchar ch);

static bool isAlphaNum(dchar ch);

const(Token) parseString();

const(Token) parseIdent();

const(Token) parseFloating(int n);

const(Token) parseHex(int prefixLen);

const(Token) parseNumber();

const(Token) parseSingleLineComment();

const(Token) parseMultiLineComment();

const(Token) parseError();

const(Token) parseOp(TokenType op);

const(Token) nextToken();

string getContextSource();

string filename [@property getter];

int line [@property getter];

int pos [@property getter];

void emitError(string msg);

void emitError(string msg, Token token);

}

Constructors

this this(string source, string filename, string[] singleLineCommentPrefixes): Undocumented in source.

Destructor

~this ~this(): Undocumented in source.

Members

Enums

EOF_CHAR anonymousenum EOF_CHAR: Undocumented in source.

Functions

emitError void emitError(string msg, Token token): Undocumented in source. Be warned that the author may not have intended to support it.
emitError void emitError(string msg): Undocumented in source. Be warned that the author may not have intended to support it.
getContextSource string getContextSource(): Undocumented in source. Be warned that the author may not have intended to support it.
nextChar dchar nextChar(): Undocumented in source. Be warned that the author may not have intended to support it.
nextToken const(Token) nextToken(): get next token
parseEof const(Token) parseEof(): Undocumented in source. Be warned that the author may not have intended to support it.
parseEol const(Token) parseEol(): Undocumented in source. Be warned that the author may not have intended to support it.
parseError const(Token) parseError(): Undocumented in source. Be warned that the author may not have intended to support it.
parseFloating const(Token) parseFloating(int n): Undocumented in source. Be warned that the author may not have intended to support it.
parseHex const(Token) parseHex(int prefixLen): Undocumented in source. Be warned that the author may not have intended to support it.
parseIdent const(Token) parseIdent(): Undocumented in source. Be warned that the author may not have intended to support it.
parseMultiLineComment const(Token) parseMultiLineComment(): Undocumented in source. Be warned that the author may not have intended to support it.
parseNumber const(Token) parseNumber(): Undocumented in source. Be warned that the author may not have intended to support it.
parseOp const(Token) parseOp(TokenType op): Undocumented in source. Be warned that the author may not have intended to support it.
parseSingleLineComment const(Token) parseSingleLineComment(): Undocumented in source. Be warned that the author may not have intended to support it.
parseString const(Token) parseString(): Undocumented in source. Be warned that the author may not have intended to support it.
parseWhiteSpace const(Token) parseWhiteSpace(): Undocumented in source. Be warned that the author may not have intended to support it.
peekChar dchar peekChar(): Undocumented in source. Be warned that the author may not have intended to support it.
peekNextChar dchar peekNextChar(): Undocumented in source. Be warned that the author may not have intended to support it.
setTokenStart void setTokenStart(): Undocumented in source. Be warned that the author may not have intended to support it.
skipChar dchar skipChar(): Undocumented in source. Be warned that the author may not have intended to support it.

Properties

filename string filename [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.
line int line [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.
pos int pos [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.

Static functions

isAlpha bool isAlpha(dchar ch): Undocumented in source. Be warned that the author may not have intended to support it.
isAlphaNum bool isAlphaNum(dchar ch): Undocumented in source. Be warned that the author may not have intended to support it.
isNum bool isNum(dchar ch): Undocumented in source. Be warned that the author may not have intended to support it.

Variables

_filename string _filename;: Undocumented in source.
_len int _len;: Undocumented in source.
_line ushort _line;: Undocumented in source.
_lineText dchar[] _lineText;: Undocumented in source.
_lines LineStream _lines;: Undocumented in source.
_pos ushort _pos;: Undocumented in source.
_prevChar dchar _prevChar;: Undocumented in source.
_singleLineCommentPrefixes string[] _singleLineCommentPrefixes;: Undocumented in source.
_token Token _token;: Undocumented in source.

Meta

Source

See Implementation

dlangui dml tokenizer classes

enums

TokenType

functions

structs

Token