Skip to content

perf/decoder quadratic scan - #15

Merged
lcweden merged 3 commits into
mainfrom
perf/decoder-quadratic-scan-14
Jul 12, 2026
Merged

perf/decoder quadratic scan#15
lcweden merged 3 commits into
mainfrom
perf/decoder-quadratic-scan-14

Conversation

@lcweden

@lcweden lcweden commented Jul 12, 2026

Copy link
Copy Markdown
Owner

This pull request introduces resumable string parsing to the JSON decoder, allowing it to correctly handle string values that are split across multiple input chunks. The main changes include the addition of the consumeStringResumable utility, updates to the Decoder and Cursor classes to track partial string parsing state, and comprehensive tests to ensure correctness for chunked and escaped string scenarios.

Resumable string parsing support:

  • Added the consumeStringResumable function in src/utils/wire.ts to allow incremental/resumable parsing of string literals, supporting continuation across chunk boundaries and proper handling of escapes and UTF-8 validation. [1] [2] [3] [4]
  • Updated the Decoder class in src/modules/decoder.ts to use consumeStringResumable and maintain an internal #offset for tracking partial string consumption, enabling correct parsing when string tokens are split across chunks. [1] [2] [3] [4] [5] [6]

Internal buffer management improvements:

  • Enhanced the Cursor class in src/modules/cursor.ts to track buffer ownership and optimize how new byte chunks are appended, minimizing allocations and supporting efficient incremental parsing. [1] [2] [3]

Testing and validation:

  • Added integration tests in tests/integration/decoder.test.ts to verify correct parsing of large strings, strings split across many chunks, escaped quotes at chunk boundaries, and full object values after chunked strings.
  • Added unit tests for consumeStringResumable in tests/wire.test.ts, covering chunked string completion, escape handling, and deferred UTF-8 validation. [1] [2]

These changes collectively ensure that the JSON decoder can robustly parse arbitrarily chunked input streams, especially for large or escaped string values.

@lcweden lcweden self-assigned this Jul 12, 2026
@lcweden lcweden added the bug Something isn't working label Jul 12, 2026
@lcweden
lcweden merged commit 76973bd into main Jul 12, 2026
1 of 2 checks passed
@lcweden
lcweden deleted the perf/decoder-quadratic-scan-14 branch July 12, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: quadratic time and memory complexity when large string token spans multiple chunks

1 participant