Design constraints

Each lexer is intended to stay small, direct, and mostly self-contained. The current Python lexer is just over 1,000 lines.

A lexer should:

• classify tokens
• style identifiers, strings, comments, numbers, and operators
• recognize common structural names such as functions and classes
• store enough line state to resume correctly
• compute fold levels
• avoid unnecessary allocations
• avoid parser-level machinery

The practical model is a small C++ state machine built directly on Scintilla's lexer interfaces.

The Python lexer inherits from a thin HackermanLexerBase, which itself extends Scintilla's DefaultLexer. The base class mainly supplies default implementations for required lexer API methods that Koi does not currently use, such as substyles and secondary style handling. This keeps each language lexer focused on the language-specific logic.

The actual lexer then implements the standard Scintilla methods:

Sci_Position SCI_METHOD PropertySet(const char *key, const char *val) override;
Sci_Position SCI_METHOD WordListSet(int n, const char *wl) override;
void SCI_METHOD Lex(
    Sci_PositionU startPos,
    Sci_Position length,
    int initStyle,
    IDocument *pAccess
) override;
void SCI_METHOD Fold(
    Sci_PositionU startPos,
    Sci_Position length,
    int initStyle,
    IDocument *pAccess
) override;

That is the whole integration point. The lexer receives a changed range, styles the text, stores any relevant per-line state, and separately computes fold levels when folding is enabled.

Comparison with the upstream Lexilla Python lexer

The upstream Lexilla Python lexer is much more general, which is expected. Lexilla is not written only for one editor. It is a shared lexer library, so its Python lexer has to support more consumers, more configuration, more compatibility behavior, and more historical expectations.

The Lexilla Python lexer includes support for many details that Koi does not currently need to reproduce feature-for-feature:

• multiple lexer properties
• detailed indentation warning behavior
• substyles
• highlighted identifiers
• decorator and attribute options
• Unicode identifier options
• byte/string/f-string handling with more extensive state
• f-string expression stack handling
• quoted-string folding options
• more compatibility behavior for external editor consumers

That is a good design for Lexilla, but it is not automatically the right design for Koi.

Koi's Python lexer intentionally cuts the scope down. It keeps the parts that are important for the editor experience: common token styling, function and class names, triple strings, practical f-string expressions, indentation folding, comment/blank-line handling, and docstring folding.

The tradeoff is straightforward. Lexilla optimizes for broad correctness and reuse. Koi optimizes for small, readable, editor-specific lexers that can be changed quickly.

Token classification model

The lexer uses a direct token classification model. For Python, the lexer defines five word lists:

const char *const wordlists[] = {
    "Keywords",
    "Types",
    "Constants",
    "Built-ins",
    "Specials",
    0,
};

When an identifier is scanned, the lexer checks the configured word lists in order and changes the style if there is a match.

This avoids hard-coding every semantic category into lexer logic. The lexer only needs to know what an identifier looks like. The actual language vocabulary can be supplied through word lists.

The identifier rules are intentionally simple:

inline bool IsPyIdentifierStart(int ch) {
    return isalpha(ch) || ch == '_' || ch >= 0x80;
}
inline bool IsPyIdentifierChar(int ch) {
    return isalnum(ch) || ch == '_' || ch >= 0x80;
}

This is not a complete implementation of Python's Unicode identifier rules. It is a pragmatic editor rule. ASCII identifiers work as expected, underscores work as expected, and non-ASCII bytes are allowed rather than immediately breaking highlighting.

That is an example of the general design principle: prefer simple recovery over strict validation.

Pending name state

The lexer uses a small pending state to style function and class names. The relevant state is:

enum PendingKind {
    PendingNone = 0,
    PendingDef = 1,
    PendingClass = 2,
};

When the lexer sees the keyword def, it sets PendingDef, then the next identifier becomes a function/name style. When it sees class, it sets PendingClass, then the next identifier becomes a class style.

This handles the important visual cases:

def open_project(path):
    pass
class ProjectWindow(QWidget):
    pass

The lexer does not need to parse Python's full grammar to get this right. It only needs to recognize a common local pattern.

The implementation also marks identifiers followed by ( as names. That gives reasonable styling for function calls:

load_project(path)

Again, this is deliberately heuristic. It is not trying to determine whether a symbol is actually callable. It is only making the text easier to scan.

String detection

String handling is the most complex part of the lexer, because Python strings can be prefixed, raw, formatted, single-line, or triple-quoted. The lexer begins by detecting possible string prefixes:

inline bool IsStringPrefixChar(int ch) {
    switch (tolower(ch)) {
    case 'r':
    case 'b':
    case 'u':
    case 'f':
        return true;
    default:
        return false;
    }
}

It then scans up to a small prefix length, rejects duplicate prefix characters, and records whether the string is: raw, formatted, triple-quoted, single-quoted or double-quoted. This becomes a compact StringInfo value:

struct StringInfo {
    bool valid = false;
    bool isFormat = false;
    bool isRaw = false;
    bool isTriple = false;
    bool isDouble = false;
};

The lexer does not need a separate state for every spelling of a prefix. It only needs the properties that affect styling behavior.

For example:

"hello"
r"hello\n"
f"loaded {count} files"
"""long text"""

These all enter string styling, but with different internal flags. Raw strings suppress escape handling. Format strings enable expression handling. Triple strings persist across line boundaries.

enum TripleLineState {
    TripleNone = 0,
    TripleSQ = 1,
    TripleDQ = 2,
    TripleRawSQ = 3,
    TripleRawDQ = 4,
    TripleFormatSQ = 5,
    TripleFormatDQ = 6,
    TripleFormatRawSQ = 7,
    TripleFormatRawDQ = 8,
};

if (sc.atLineEnd) {
    styler.SetLineState(
        styler.GetLine(sc.currentPos),
        EncodeTripleLineState()
    );
}

DecodeTripleLineState(
    lineCurrent > 0 ? styler.GetLineState(lineCurrent - 1) : 0
);
if (tripleString) {
    initStyle = State_TripleString;
}

bool inFStringExpression = false;
int fStringBraceLevel = 0;
bool fStringTriple = false;
bool fStringDouble = false;
bool fStringRaw = false;

message = f"Loaded {len(files)} files"
path = f"{root}/{name}.txt"

text = f"{{literal}}"

if (sc.atLineEnd) {
    sc.ChangeState(SCE_HACK_ERROR);
    sc.SetState(SCE_HACK_DEFAULT);
}

Folding model

The folding pass is separate from the lexing pass.

For Python, most folding is indentation-based. Scintilla already provides indentation information through IndentAmount, including white-line flags. The lexer uses that as the base.

The high-level rule is:

• a non-blank line becomes a fold header if the next meaningful line is more indented
• blank lines and comment-only lines are skipped when deciding the next meaningful indentation level
• triple-quoted strings are folded as synthetic blocks
• fold levels are written back to Scintilla line by line

This gives normal Python block folding:

def run():
    setup()
    execute()

The def run(): line becomes a fold header because the next meaningful line is more indented.

def run():
    setup()

    # important step

    execute()

static bool IsCommentLine(Sci_Position line, Accessor &styler) {
    const Sci_Position pos = styler.LineStart(line);
    const Sci_Position eolPos = styler.LineStart(line + 1) - 1;
    for (Sci_Position i = pos; i < eolPos; i++) {
        const char ch = styler[i];
        if (ch == '#') {
            return true;
        } else if (ch != ' ' && ch != '\t') {
            return false;
        }
    }
    return false;
}

def explain():
    """
    This is a long explanation.
    It may contain indentation that is not code.
    """
    return True

const bool before = line > 0 && styler.GetLineState(line - 1) != 0;
const bool after = styler.GetLineState(line) != 0;
return before || after;

const bool before = line > 0 && styler.GetLineState(line - 1) != 0;
const bool after = styler.GetLineState(line) != 0;
return after && !before;

while (lineCurrent > 0) {
    --lineCurrent;
    const int indent = styler.IndentAmount(lineCurrent, &spaceFlags, NULL);
    if (!(indent & SC_FOLDLEVELWHITEFLAG) &&
        !IsCommentLine(lineCurrent, styler) &&
        !IsTripleStringLine(lineCurrent, styler)) {
        break;
    }
}

Performance characteristics

The lexer stays performant mostly because it avoids expensive work.

There is no AST.

There is no regex pipeline over the whole document.

There is no semantic index.

The lexer walks the requested range using Scintilla's StyleContext, changes style states, and stores line state where needed. The main performance properties are:

• single-pass scanning over the changed range
• limited backtracking
• line state used for multiline constructs
• word-list lookup for identifiers
• no heap-heavy token stream
• no parser recovery
• no external process or runtime dependency

This is important for Koi because the editor is built around reducing typing latency. Syntax highlighting should not feel like a separate subsystem that periodically catches up. It should feel like part of the text view.

The tradeoff is that the lexer accepts heuristic classification. For example, an identifier followed by ( is styled as a name. That is useful and cheap. It is not semantic proof that the identifier is a function.

Benefits of this approach

Why not use Tree-sitter for lexing?

Koi already uses Tree-sitter where it makes sense: symbol outlines, navigation, and other code-structure features. I do not think it is the right default path for lexing.

Lexing has a smaller job: decide how a range of text should be styled while the user is typing. For that, a small lexer can stay closer to the buffer. It receives a changed range, scans characters, assigns styles, stores line state, and writes fold levels directly into Scintilla's existing styling and folding pipeline.

Tree-sitter can also be used for highlighting, but it adds a larger system around that path: parsers, grammars, highlight queries, capture mapping, parser state, edit synchronization, and another layer to package and debug.

It is also harder to hand-tune for specific lexer behavior. With a small lexer, if Python docstring folding is wrong, or Odin procedure names after :: proc should be styled differently, I can usually change one local scanner or fold pass. With Tree-sitter highlighting, the behavior may involve the grammar, generated parser, query files, captures, and editor-side mapping.

That keeps syntax highlighting cheap and direct, while still leaving room for Tree-sitter where a real syntax tree actually is useful.

TLDR

The Koi lexer design is intentionally smaller than a general-purpose Lexilla lexer. It uses Scintilla's existing lexer interface, a compact state machine, word lists, line state, and indentation-based folding to provide the parts of syntax highlighting that matter most during editing.

Compared with the built-in Lexilla Python lexer, this approach gives up some edge-case completeness and configurability. In return, it gains control, readability, predictable performance, and a much smaller maintenance surface.

That is the right tradeoff for Koi at this stage. The editor does not need every lexer to be a full language implementation. It needs lexers that are fast, understandable, good enough for normal code, and easy to improve when real editing problems show up.

In other words: the lexer should stay close to the text. Not close to the compiler.