FEATURE: jq Lexical Scanner

Scope

Implement lexical scanning according to the supplied sources/lexer.l and sources/parser.y references. The scanner recognizes numeric literals, identifiers, field selectors, variable bindings, keywords, operators, delimiters, comments, and format tokens, while preserving source locations needed for diagnostics.

Lexical errors must be reported as compile failures and must not be confused with runtime errors.

Programmatic Acceptance

=== AC lexer-conformance ===
Intent: The lexer and front end pass the corpus slice containing primitive literals and identity syntax.

import json import os import subprocess import sys

select = r"^(true|false|null|1|\.)$" result = subprocess.run( [sys.executable, "sources/run_conformance.py", "--select", select, "--json"], capture_output=True, text=True, env={**os.environ, "JQ": f"{os.getcwd()}/jq"}, ) print(result.stdout) print(result.stderr, file=sys.stderr) report = json.loads(result.stdout) summary = report["summary"] assert sum(summary.values()) > 0 assert summary["fail"] == 0 assert summary["error"] == 0 assert result.returncode == 0 === END AC lexer-conformance ===

=== AC lexer-rejects-invalid-source ===
Intent: An invalid lexical character is rejected with the compile-failure status.

import subprocess

result = subprocess.run( ["./jq", "-c", "%::wat"], input="null\n", capture_output=True, text=True, ) assert result.returncode == 3 === END AC lexer-rejects-invalid-source ===

=== AC lexer-comments ===
Intent: Comments do not alter evaluation of the surrounding jq program.

import subprocess

program = "1 # comment\n" result = subprocess.run( ["./jq", "-c", program], input="null\n", capture_output=True, text=True, ) assert result.returncode == 0 assert result.stdout.splitlines() == ["1"] === END AC lexer-comments ===

User Acceptance

• None.

Guardrails

• Follow the token forms and delimiter behavior defined by sources/lexer.l.
• Preserve comments as non-semantic input.
• Reject invalid characters at compile time.
• Do not modify the supplied lexer or parser reference files.