Tokenize the whole file before slicing

Tokenization splits source text into parts with syntax roles, such as strings and comments. A highlighter uses those roles to style code.

The code-snippets design brief14 requires whole-file tokenization. The highlighter receives the full source before the pipeline selects lines. Shiki exposes codeToTokens for token output.

An excerpt can start inside a multiline comment or string. Tokenizing only the excerpt removes the preceding source from the highlighter’s input.

The proposed pipeline:

  1. Tokenizes every source line.
  2. Selects the requested ranges.
  3. Keeps the original line numbers.
  4. Inserts a collapsed gap between separated ranges.

A collapsed gap labels omitted lines separately from source lines.

The planned property compares excerpt tokens with the same lines from whole-file tokenization. It includes multiline comments and template strings. See property-based testing18.

Both renderers consume one snippet model8. See the glossary for whole-file tokenization and collapsed gap.

Sources

  1. Shiki installation and usage. Primary source. codeToTokens API Accessed 2026-10-02.
  2. Code-snippets design brief. Primary source. Whole-file tokenization requirement Accessed 2026-10-02.

Last changed 2026-10-07 in f1a00f2. Source on GitHub. Change log.

Linked from

  1. Agent guide. An Effect stack so pure (aspirational) Kit Langton will blush.

    Tokenize the whole file before slicing — Select lines after tokenization so preceding code still guides recognition of multiline strings and comments.

  2. Change log. What changed in the files served here, newest first.

    Tokenize the whole file before slicing — edited: feat(learn): teach cards 63 to 82 with plain lines, diagrams and snippets

  3. Code snippets quote a pinned source. Shipped design brief: pinned Git objects, typed diagnostics, and one build-only snippet model for both views.

    Whole-file tokenization comes before range selection.

  4. Full agent guide. An Effect stack so pure (aspirational) Kit Langton will blush.
  5. One snippet model feeds both renderers. HTML and agent Markdown derive their code, repository, commit, path, and line numbers from the same snippet.

    Whole-file tokenization supplies the tokens.

  6. Rat Stack lore | rat-stack. Short, source-grounded notes on the ideas and decisions behind rat-stack.

    Tokenize the whole file before slicing — Select lines after tokenization so preceding code still guides recognition of multiline strings and comments.

  7. Source change history (log.md). The generated change log as Markdown.

    Tokenize the whole file before slicing — edited: feat(learn): teach cards 63 to 82 with plain lines, diagrams and snippets