Token generator for a single source file. More...

#include <Compiler_Lexer.H>

Collaboration diagram for Aleph::Compiler_Lexer:

Public Member Functions
	Compiler_Lexer (const Source_Manager &sm, const Source_File_Id id, Diagnostic_Engine *dx=nullptr, const Compiler_Lexer_Options &opts={})
	Initializes a lexer for a registered source file.

Source_File_Id	source_file_id () const noexcept
	Returns the file ID currently being analyzed.

Source_Offset	current_offset () const noexcept
	Returns the current byte offset in the file.

void	reset (const Source_Offset offset=0)
	Repositions the reading cursor.

const Compiler_Token &	peek ()
	Peeks at the next token without advancing the cursor.

Compiler_Token	next ()
	Consumes and returns the next token from the stream.

bool	eof ()
	Indicates whether the end of the file has been reached.

Private Member Functions
bool	at_end () const noexcept
	Returns true if the cursor has reached the end of the input.

char	char_at (const Source_Offset off) const noexcept
	Safe character access at a given offset.

char	current_char () const noexcept
	Returns the character at the current cursor position.

char	next_char () const noexcept
	Returns the character following the current cursor position.

Source_Span	make_span (const Source_Offset begin, const Source_Offset end) const
	Creates a source span for the given range in the current file.

Compiler_Token	make_token (const Compiler_Token_Kind kind, const Source_Offset begin, const Source_Offset end) const
	Creates a token with the given kind and range.

Compiler_Token	make_eof_token () const
	Creates an End_Of_File token at the current position.

void	emit_error (const Source_Span &span, const std::string &message, const std::string &code, const std::string &note="", const std::string &help="") const
	Emits a diagnostic error through the engine.

void	skip_whitespace ()
	Consumes whitespace characters.

Compiler_Token	lex_line_comment ()
	Lexes a single-line comment.

Compiler_Token	lex_block_comment ()
	Lexes a multi-line block comment.

Compiler_Token	lex_identifier_or_keyword ()
	Lexes an identifier or classifies it as a keyword.

Compiler_Token	lex_number ()
	Lexes a numeric literal.

Compiler_Token	lex_string ()
	Lexes a string literal.

Compiler_Token	lex_char_literal ()
	Lexes a character literal.

Compiler_Token	lex_punctuation_or_operator ()
	Lexes punctuation or operator symbols.

Compiler_Token	lex_token ()
	Dispatches the next token from the input stream.

Static Private Member Functions
static bool	is_ident_start (const unsigned char ch) noexcept
	Checks if a character can start an identifier.

static bool	is_ident_continue (const unsigned char ch) noexcept
	Checks if a character can continue an identifier.

Private Attributes
const Source_Manager *	sources_ = nullptr

Source_File_Id	file_id_ = 0

const std::string *	input_ = nullptr

Diagnostic_Engine *	diagnostics_ = nullptr

Compiler_Lexer_Options	options_

Source_Offset	cursor_ = 0

bool	has_lookahead_ = false

Compiler_Token	lookahead_

Detailed Description

Token generator for a single source file.

Maintains the reading state (cursor) and provides a "consumption" interface for logical symbols.

Note: Thread-safety: Not thread-safe. Each file should have its own lexer instance if processed in parallel.

Definition at line 89 of file Compiler_Lexer.H.

Constructor & Destructor Documentation

◆ Compiler_Lexer()

Aleph::Compiler_Lexer::Compiler_Lexer	(	const Source_Manager &	sm,
		const Source_File_Id	id,
		Diagnostic_Engine *	dx = `nullptr`,
		const Compiler_Lexer_Options &	opts = `{}`
	)

inline

Initializes a lexer for a registered source file.

Parameters

sm	Source manager containing the file text.
id	Identifier of the file to tokenize.
dx	Diagnostic engine to report errors. If null, errors are only indicated via `Invalid` tokens.
opts	Trivia handling options (comments, etc.).

Exceptions

std::out_of_range If id is not valid in sm.

Note: Thread-safety: Not thread-safe.; Complexity: O(1).

Definition at line 534 of file Compiler_Lexer.H.

Member Function Documentation

◆ at_end()

bool Aleph::Compiler_Lexer::at_end ( ) const

inlineprivatenoexcept

Returns true if the cursor has reached the end of the input.

Definition at line 113 of file Compiler_Lexer.H.

References cursor_, and input_.

Referenced by lex_block_comment(), lex_char_literal(), lex_identifier_or_keyword(), lex_line_comment(), lex_number(), lex_string(), lex_token(), and skip_whitespace().

◆ char_at()

char Aleph::Compiler_Lexer::char_at ( const Source_Offset off ) const

inlineprivatenoexcept

Safe character access at a given offset.

Definition at line 119 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching(), and input_.

Referenced by current_char(), and next_char().

◆ current_char()

char Aleph::Compiler_Lexer::current_char ( ) const

inlineprivatenoexcept

Returns the character at the current cursor position.

Definition at line 125 of file Compiler_Lexer.H.

References char_at(), and cursor_.

Referenced by lex_block_comment(), lex_char_literal(), lex_identifier_or_keyword(), lex_line_comment(), lex_number(), lex_punctuation_or_operator(), lex_string(), lex_token(), and skip_whitespace().

◆ current_offset()

Source_Offset Aleph::Compiler_Lexer::current_offset ( ) const

inlinenoexcept

Returns the current byte offset in the file.

Useful for saving recovery states or for telemetry.

Returns: Cursor position in bytes.

Note: Thread-safety: Safe for concurrent readers.; Complexity: O(1).

Definition at line 560 of file Compiler_Lexer.H.

References cursor_.

Referenced by TEST().

◆ emit_error()

void Aleph::Compiler_Lexer::emit_error	(	const Source_Span &	span,
		const std::string &	message,
		const std::string &	code,
		const std::string &	note = `""`,
		const std::string &	help = `""`
	)		const

inlineprivate

Emits a diagnostic error through the engine.

Definition at line 159 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching(), Aleph::Diagnostic_Builder::code(), Aleph::code(), diagnostics_, Aleph::Diagnostic_Builder::emit(), Aleph::Diagnostic_Engine::error(), Aleph::Diagnostic_Builder::help(), Aleph::message(), and Aleph::Diagnostic_Builder::note().

Referenced by lex_block_comment(), lex_char_literal(), lex_punctuation_or_operator(), and lex_string().

◆ eof()

bool Aleph::Compiler_Lexer::eof ( )

inline

Indicates whether the end of the file has been reached.

Returns: true if the next call to next() would return End_Of_File.

Note: Thread-safety: Not thread-safe.; Complexity: O(N) (calls peek internally).

Definition at line 628 of file Compiler_Lexer.H.

References Aleph::Compiler_Token::is_eof(), and peek().

Referenced by TEST(), and TEST().

◆ is_ident_continue()

static bool Aleph::Compiler_Lexer::is_ident_continue ( const unsigned char ch )

inlinestaticprivatenoexcept

Checks if a character can continue an identifier.

Definition at line 107 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching().

Referenced by lex_identifier_or_keyword().

◆ is_ident_start()

static bool Aleph::Compiler_Lexer::is_ident_start ( const unsigned char ch )

inlinestaticprivatenoexcept

Checks if a character can start an identifier.

Definition at line 101 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching().

Referenced by lex_token().

◆ lex_block_comment()

Compiler_Token Aleph::Compiler_Lexer::lex_block_comment ( )

inlineprivate

Lexes a multi-line block comment.

Definition at line 194 of file Compiler_Lexer.H.

References Aleph::and, at_end(), Aleph::Block_Comment, Aleph::blossom_maximum_cardinality_matching(), current_char(), cursor_, emit_error(), Aleph::Invalid, make_span(), make_token(), next_char(), Aleph::Source_Manager::slice(), and sources_.

Referenced by lex_token().

◆ lex_char_literal()

Compiler_Token Aleph::Compiler_Lexer::lex_char_literal ( )

inlineprivate

Lexes a character literal.

Definition at line 293 of file Compiler_Lexer.H.

References Aleph::and, at_end(), Aleph::blossom_maximum_cardinality_matching(), Aleph::Char_Literal, current_char(), cursor_, emit_error(), Aleph::Invalid, make_span(), make_token(), Aleph::Source_Manager::slice(), and sources_.

Referenced by lex_token().

◆ lex_identifier_or_keyword()

Compiler_Token Aleph::Compiler_Lexer::lex_identifier_or_keyword ( )

inlineprivate

Lexes an identifier or classifies it as a keyword.

Definition at line 215 of file Compiler_Lexer.H.

References Aleph::and, at_end(), Aleph::blossom_maximum_cardinality_matching(), Aleph::classify_compiler_keyword(), current_char(), cursor_, is_ident_continue(), make_span(), Aleph::Source_Manager::slice(), and sources_.

Referenced by lex_token().

◆ lex_line_comment()

Compiler_Token Aleph::Compiler_Lexer::lex_line_comment ( )

inlineprivate

Lexes a single-line comment.

Definition at line 184 of file Compiler_Lexer.H.

References Aleph::and, at_end(), Aleph::blossom_maximum_cardinality_matching(), current_char(), cursor_, Aleph::Line_Comment, and make_token().

Referenced by lex_token().

◆ lex_number()

Compiler_Token Aleph::Compiler_Lexer::lex_number ( )

inlineprivate

Lexes a numeric literal.

Definition at line 228 of file Compiler_Lexer.H.

References Aleph::and, at_end(), Aleph::blossom_maximum_cardinality_matching(), current_char(), cursor_, Aleph::Integer_Literal, make_token(), and next_char().

Referenced by lex_token().

◆ lex_punctuation_or_operator()

Compiler_Token Aleph::Compiler_Lexer::lex_punctuation_or_operator ( )

inlineprivate

Lexes punctuation or operator symbols.

Definition at line 341 of file Compiler_Lexer.H.

References Aleph::Amp, Aleph::and, Aleph::AndAnd, Aleph::Arrow, Aleph::Assign, Aleph::Bang, Aleph::blossom_maximum_cardinality_matching(), Aleph::Caret, Aleph::Colon, Aleph::Comma, current_char(), cursor_, Aleph::Dot, Aleph::Double_Colon, emit_error(), Aleph::EqEq, Aleph::Greater, Aleph::GreaterEq, Aleph::Invalid, Aleph::LBrace, Aleph::LBracket, Aleph::Less, Aleph::LessEq, Aleph::LParen, make_span(), make_token(), Aleph::Minus, Aleph::MinusEq, Aleph::MinusMinus, next_char(), Aleph::NotEq, Aleph::OrOr, Aleph::Percent, Aleph::PercentEq, Aleph::Pipe, Aleph::Plus, Aleph::PlusEq, Aleph::PlusPlus, Aleph::Question, Aleph::RBrace, Aleph::RBracket, Aleph::RParen, Aleph::Semicolon, Aleph::Slash, Aleph::SlashEq, Aleph::Source_Manager::slice(), sources_, Aleph::Star, Aleph::StarEq, and Aleph::Tilde.

Referenced by lex_token().

◆ lex_string()

Compiler_Token Aleph::Compiler_Lexer::lex_string ( )

inlineprivate

Lexes a string literal.

Definition at line 254 of file Compiler_Lexer.H.

References at_end(), Aleph::blossom_maximum_cardinality_matching(), current_char(), cursor_, emit_error(), Aleph::Invalid, make_span(), make_token(), Aleph::Source_Manager::slice(), sources_, and Aleph::String_Literal.

Referenced by lex_token().

◆ lex_token()

Compiler_Token Aleph::Compiler_Lexer::lex_token ( )

inlineprivate

Dispatches the next token from the input stream.

Definition at line 485 of file Compiler_Lexer.H.

References Aleph::Compiler_Lexer_Options::allow_block_comments, Aleph::and, at_end(), Aleph::blossom_maximum_cardinality_matching(), current_char(), is_ident_start(), Aleph::Compiler_Lexer_Options::keep_comments, lex_block_comment(), lex_char_literal(), lex_identifier_or_keyword(), lex_line_comment(), lex_number(), lex_punctuation_or_operator(), lex_string(), make_eof_token(), next_char(), options_, and skip_whitespace().

Referenced by next(), and peek().

◆ make_eof_token()

Compiler_Token Aleph::Compiler_Lexer::make_eof_token ( ) const

inlineprivate

Creates an End_Of_File token at the current position.

Definition at line 152 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching(), cursor_, Aleph::End_Of_File, and make_span().

Referenced by lex_token().

◆ make_span()

Source_Span Aleph::Compiler_Lexer::make_span	(	const Source_Offset	begin,
		const Source_Offset	end
	)		const

inlineprivate

Creates a source span for the given range in the current file.

Definition at line 137 of file Compiler_Lexer.H.

References file_id_, sources_, and Aleph::Source_Manager::span().

Referenced by lex_block_comment(), lex_char_literal(), lex_identifier_or_keyword(), lex_punctuation_or_operator(), lex_string(), make_eof_token(), and make_token().

◆ make_token()

Compiler_Token Aleph::Compiler_Lexer::make_token	(	const Compiler_Token_Kind	kind,
		const Source_Offset	begin,
		const Source_Offset	end
	)		const

inlineprivate

Creates a token with the given kind and range.

Definition at line 143 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching(), make_span(), Aleph::Source_Manager::slice(), and sources_.

Referenced by lex_block_comment(), lex_char_literal(), lex_line_comment(), lex_number(), lex_punctuation_or_operator(), and lex_string().

◆ next()

Compiler_Token Aleph::Compiler_Lexer::next ( )

inline

Consumes and returns the next token from the stream.

Advances the internal cursor. If there is a lexical error, returns an Invalid token but continues advancing to attempt recovery.

Returns: The consumed token.

Note: Thread-safety: Not thread-safe.; Complexity: O(N) where N is lexeme length (amortized).

Definition at line 612 of file Compiler_Lexer.H.

References has_lookahead_, lex_token(), and lookahead_.

Referenced by main(), Aleph::Compiler_Parser_Stream::next(), TEST(), TEST(), TEST(), TEST(), TEST(), TEST(), TEST(), TEST(), TEST(), TEST(), and TEST().

◆ next_char()

char Aleph::Compiler_Lexer::next_char ( ) const

inlineprivatenoexcept

Returns the character following the current cursor position.

Definition at line 131 of file Compiler_Lexer.H.

References char_at(), and cursor_.

Referenced by lex_block_comment(), lex_number(), lex_punctuation_or_operator(), and lex_token().

◆ peek()

const Compiler_Token & Aleph::Compiler_Lexer::peek ( )

inline

Peeks at the next token without advancing the cursor.

Allows the parser to make syntactic decisions based on the next symbol without consuming it permanently. Successive calls return the same token.

Returns: The next available token in the stream.

Note: Thread-safety: Not thread-safe.; Complexity: O(N) where N is lexeme length (amortized).

Definition at line 593 of file Compiler_Lexer.H.

References Aleph::blossom_maximum_cardinality_matching(), has_lookahead_, lex_token(), and lookahead_.

Referenced by eof(), Aleph::Compiler_Parser_Stream::peek(), and TEST().

◆ reset()

void Aleph::Compiler_Lexer::reset ( const Source_Offset offset = 0 )

inline

Repositions the reading cursor.

Discards any pre-read tokens (lookahead) and restarts the analysis from the indicated position.

Parameters

offset New byte offset (default 0, the beginning).

Exceptions

std::out_of_range If the offset exceeds the file length.

Note: Thread-safety: Not thread-safe.; Complexity: O(1).

Definition at line 575 of file Compiler_Lexer.H.

References ah_out_of_range_error_unless, Aleph::blossom_maximum_cardinality_matching(), cursor_, has_lookahead_, lookahead_, offset, and Aleph::size().

Referenced by TEST().

◆ skip_whitespace()

void Aleph::Compiler_Lexer::skip_whitespace ( )

inlineprivate

Consumes whitespace characters.

Definition at line 177 of file Compiler_Lexer.H.

References Aleph::and, at_end(), Aleph::blossom_maximum_cardinality_matching(), current_char(), and cursor_.

Referenced by lex_token().

◆ source_file_id()

Source_File_Id Aleph::Compiler_Lexer::source_file_id ( ) const

inlinenoexcept

Returns the file ID currently being analyzed.

Returns: The stable ID associated with this lexer.

Note: Thread-safety: Safe for concurrent readers.; Complexity: O(1).

Definition at line 547 of file Compiler_Lexer.H.

References file_id_.

Referenced by TEST().

Member Data Documentation

◆ cursor_

Source_Offset Aleph::Compiler_Lexer::cursor_ = 0

private

Definition at line 96 of file Compiler_Lexer.H.

Referenced by at_end(), current_char(), current_offset(), lex_block_comment(), lex_char_literal(), lex_identifier_or_keyword(), lex_line_comment(), lex_number(), lex_punctuation_or_operator(), lex_string(), make_eof_token(), next_char(), reset(), and skip_whitespace().

◆ diagnostics_

Diagnostic_Engine* Aleph::Compiler_Lexer::diagnostics_ = nullptr

private

Definition at line 94 of file Compiler_Lexer.H.

Referenced by emit_error().

◆ file_id_

Source_File_Id Aleph::Compiler_Lexer::file_id_ = 0

private

Definition at line 92 of file Compiler_Lexer.H.

Referenced by make_span(), and source_file_id().

◆ has_lookahead_

bool Aleph::Compiler_Lexer::has_lookahead_ = false

private

Definition at line 97 of file Compiler_Lexer.H.

Referenced by next(), peek(), and reset().

◆ input_

const std::string* Aleph::Compiler_Lexer::input_ = nullptr

private

Definition at line 93 of file Compiler_Lexer.H.

Referenced by at_end(), and char_at().

◆ lookahead_

Compiler_Token Aleph::Compiler_Lexer::lookahead_

private

Definition at line 98 of file Compiler_Lexer.H.

Referenced by next(), peek(), and reset().

◆ options_

Compiler_Lexer_Options Aleph::Compiler_Lexer::options_

private

Definition at line 95 of file Compiler_Lexer.H.

Referenced by lex_token().

◆ sources_

const Source_Manager* Aleph::Compiler_Lexer::sources_ = nullptr

private

Definition at line 91 of file Compiler_Lexer.H.

Referenced by lex_block_comment(), lex_char_literal(), lex_identifier_or_keyword(), lex_punctuation_or_operator(), lex_string(), make_span(), and make_token().

The documentation for this class was generated from the following file:

Compiler_Lexer.H

Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Compiler_Lexer()

Member Function Documentation

◆ at_end()

◆ char_at()

◆ current_char()

◆ current_offset()

◆ emit_error()

◆ eof()

◆ is_ident_continue()

◆ is_ident_start()

◆ lex_block_comment()

◆ lex_char_literal()

◆ lex_identifier_or_keyword()

◆ lex_line_comment()

◆ lex_number()

◆ lex_punctuation_or_operator()

◆ lex_string()

◆ lex_token()

◆ make_eof_token()

◆ make_span()

◆ make_token()

◆ next()

◆ next_char()

◆ peek()

◆ reset()

◆ skip_whitespace()

◆ source_file_id()

Member Data Documentation

◆ cursor_

◆ diagnostics_

◆ file_id_

◆ has_lookahead_

◆ input_

◆ lookahead_

◆ options_

◆ sources_