Implement streaming JSON tokenizer (Decoder/Encoder)

Summary

Implement low-level streaming JSON tokenizer for memory-efficient parsing and generation of large JSON documents.

Background

Go's jsontext.Decoder and jsontext.Encoder provide token-based streaming APIs that don't require loading the entire document into memory.

Proposed API

pub const TokenKind = enum {
    object_start,    // {
    object_end,      // }
    array_start,     // [
    array_end,       // ]
    string,
    number,
    true_literal,
    false_literal,
    null_literal,
    colon,
    comma,
};

pub const Token = struct {
    kind: TokenKind,
    bytes: []const u8,  // Raw token bytes
};

pub const Decoder = struct {
    pub fn init(reader: anytype) Decoder;
    pub fn readToken(self: *Decoder) !?Token;
    pub fn readValue(self: *Decoder) !Value;  // Read complete value
    pub fn skipValue(self: *Decoder) !void;   // Skip without allocating
    pub fn stackDepth(self: *Decoder) usize;
};

pub const Encoder = struct {
    pub fn init(writer: anytype) Encoder;
    pub fn writeToken(self: *Encoder, token: Token) !void;
    pub fn beginObject(self: *Encoder) !void;
    pub fn endObject(self: *Encoder) !void;
    pub fn beginArray(self: *Encoder) !void;
    pub fn endArray(self: *Encoder) !void;
    pub fn writeString(self: *Encoder, s: []const u8) !void;
    pub fn writeNumber(self: *Encoder, n: anytype) !void;
    pub fn writeNull(self: *Encoder) !void;
    pub fn writeBool(self: *Encoder, b: bool) !void;
};

Use Cases

• Processing multi-gigabyte JSON files
• JSON streaming over network connections
• Memory-constrained environments
• Building custom JSON processors

Reference

Go encoding/json/jsontext: Decoder, Encoder, Token, Kind

Priority

High - Enables memory-efficient processing