█ Hackerman, Inc. 

Lexer Design in Koi Editor

Posted on July 20, 2026 by @asyncze | Go home

Koi Editor is built on Scintilla's core editing component.
Scintilla's companion library, Lexilla, provides a large collection of built-in lexers.

Koi Editor still uses some of Scintilla's built-in lexers, while others have been rewritten or pruned.
This creates a unified interface across all Koi lexers.
It also makes bugs and syntax-highlighting issues easier to track down because the implementations share the same structure.

Each lexer is customized to the language, yet normalised to fit the editor.
This means that the methods are the same, the base style ids are the same.
A keyword is styled as a keyword in every language that supports keywords.
This makes the codebase easier to maintain and reduces the effort required to add new languages.

Most lexers are pruned to only two methods, Lex and Fold. 
Lex is called by Scintilla to apply syntax styles to a range of text.
The requested range may be expanded when a language requires limited backtracking, such as for multi-line constructs.

Fold is the language-specific implementation of folding logic.
In some languages, this logic is very predictable, such as only folding code blocks { ... }.

In Python, folding is less straightforward.
Should subsequent indented lines be folded?
Yes.
Indented comments?
Yes.
Multi-line strings (docstrings)?
Probably.
What about non-indented comments or blank lines between indented blocks?
Those often need to be treated as part of the fold as well.

> Why not just use Lexilla as-is?

The goal isn't to replace Lexilla because its lexers are poor.
They're excellent.
But they are written for many editors, many configuration styles, and many historical expectations.

For Koi, we need something narrower:

- predictable style ids
- consistent lexer structure
- easier debugging
- fewer optional paths
- folding behavior that matches user interface

Performance considerations

Scintilla's built-in lexers are already very fast.
Pruning or reimplementing a lexer with fewer features and less branching can make it marginally faster.
The gains are usually small—often only a few milliseconds, but the simpler implementations are also easier to maintain over time.

Adding language support

Koi's lexers are part of the Hackerman-family HLex*, and organised per language outside of Scintilla.
Each lexer's files are placed in a single folder.
A build script then compiles, copies, and bundles everything into the Scintilla binaries.
Adding a new language is mostly a matter of creating a folder containing four files, three of which are largely boilerplate.

> Making all the lexers will be a lot of work, right?

Yes, but this build-stage re-organising and lexer pruning will make it less work.

Pending states

Some languages cannot know what something is until the next token.
Python provides a simple example:

> def hello(): ...

When the lexer sees def, it knows the next identifier should be styled as a function.
The identifier itself has not been seen yet, so the lexer sets a pending style state.

This is a very small state machine, but it avoids having to look backwards later or re-style already processed text.
Most languages have some variation of this.

Line states

Scintilla allows each lexer to store a single integer of state for every line.
Instead of rescanning the entire file every time a small edit happens, the lexer can restore whatever state the previous line ended in.

For the Python lexer this is mostly used for triple strings.
If a line ends inside a triple string, that information is stored as the line state.
The next lexing pass restores that state and continues lexing from there.

The stored state also records the kind of triple string:

- single or double quotes
- raw string
- format string
- raw format string

This means the lexer does not have to search backwards looking for where the string started.
It just restores the previous state and keeps going.

Line states are also useful for folding.
A fold implementation can quickly see whether a line belongs to a triple string without scanning the document again.

String substitution

JavaScript template strings are a good example.

> `Hello ${user.name}`

This starts as a string.
When the lexer reaches ${, the lexer temporarily stops parsing template text and switches back to normal JavaScript.

That expression can contain almost any valid JavaScript.

> `${foo(bar + baz())}`

Or even another template string.

> `${items.map(item => `${item.name}: ${item.value}`)}`

Now there are multiple template strings active at the same time.
To keep track of this, the lexer stores a small stack of template contexts.
Each context remembers whether it is currently parsing template text or a JavaScript expression, together with the current brace depth for that expression.
When ${ is reached, a new expression begins.
Every { increments the depth.
Every } decrements it.

When the brace depth returns to zero, the interpolation expression has ended, so the lexer switches back to parsing template text.
If another nested template string is encountered, another context is pushed onto the stack and the process repeats.

---

Posted on July 20, 2026 by @asyncze | Go home