A small embeddable data language with schemas, helpful diagnostics, units, comments, and clean JSON interop, ready to drop in
  • Zig 78.6%
  • TypeScript 17%
  • Rust 2.6%
  • JavaScript 0.5%
  • Shell 0.5%
  • Other 0.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Hugo Daniel eb63b3fc47 SJON 1.5.0
SJON is a schema-constrained data language for domain tools and the agents that operate them: deterministic S-expression data plus a pure, bounded expression layer, one front-end, a stable binary wire format, append-only diagnostic codes, and parity hosts in Zig, Node, Rust, and TypeScript.

## 1.5.0 — 2026-10-25

This release keeps binary wire v5 and the diagnostic-code enum unchanged.
The npm packaging fixes described under 1.4.2 first reach registry consumers
with 1.5.0: neither 1.4.2 nor 1.4.3 was published to npm (the previous registry
latest was 1.4.1). Packages ship compiled JavaScript and declarations, including
all documented subpaths; use pnpm to rewrite workspace dependencies when packing.

- **Decode once and query many:** `sjon.prepareBinary`, owning
  `PatternQuery.PreparedBinary`, and the four `queryPreparedBinary*` entries
  reuse a decoded tree, with fresh per-query environments and tree-path budgets.
  Preparation is outside those budgets. `sjon.BinaryDecoder` exposes the
  read-only decoder separately. See [the consumer guide](docs/BINARY_PATTERNS.md).
- **Evaluate a captured binary expression view:** `Expr.evalBinaryView` uses a
  borrowed cursor and view without whole-buffer root framing; consumers must
  honor cursor/input lifetimes. Existing whole-buffer evaluation is unchanged.
- **Carry author subtrees through Zig lowering:** `LoweringInput.subtree` and
  `EmittedValue.carry` preserve author syntax and spans while the final forest
  validates the destination context. This remains a Zig-only hook surface.
- **Rust distribution:** `sjon-host` is prepared for crates.io with explicit
  package contents and a corpus-independent consumer build. It requires
  Rust 1.92 and a matching external `sjon.wasm`.
- **Highlight runtime requirement:** the compiled grammar uses JSON import
  attributes, so its minimum Node version is corrected to 20.10.
- **Documentation and distribution:** corrected Node/Rust onboarding, eight-op
  editing and preserve-layout contracts, explicit host capability boundaries,
  current coverage counts, and stronger documentation drift checks.

- **The binary pattern query can now evaluate `(pure <form>)` expression
  leaves through new additive entries — the frozen legacy entries are
  unchanged.** `PatternQuery.queryBinaryWithEnv` and
  `queryBinaryWithBudgetAndEnv` compile a binary pattern with expression
  leaves captured as IR views, dry-run each at cycle 0, and evaluate them
  per cycle under the pattern-local `cycle` / `tick` / `seed` (which shadow
  same-named outer bindings), with the caller's `Expr.Env` chained beneath
  — where the legacy `queryBinary` / `queryBinaryWithBudget` silently
  degrade every expression leaf to `silence` and keep their byte-for-byte
  contract, including in the released read-only `sjon-binary.wasm` (no
  artifact, host-ABI, wire, or diagnostic-code change; the wasm closure
  allowlist is untouched). The entry, never a null env, selects semantics.
  When a leaf exists, the pattern bytes are copied once into the Result
  arena (one retained cursor, the compile-time pool index reused over the
  layout-identical copy), so a Result outlives the caller's bytes and env;
  the retention and pool-index steps preflight checked arithmetic against
  `budget.bytes` and refuse with `MemoryBudgetExceeded` before allocating.
  Malformed wire anywhere — including under a captured leaf, surfaced
  through the cycle-0 dry-run — is a hard, typed `BinaryError` from the
  existing error enum; static eval defects are the tree path's diagnostics
  at `[pure]`; later-cycle eval failures are counted drops. No new
  diagnostic codes, wire still v5, corpus still 393 cases.

A form value in a slot whose value-kind reference cannot resolve is reported
as the reference it is, and the composition that made it unresolvable is
named at preload: the aggregate phase resolves every kind reference every
manifest spells, so a bare name two loaded plugins both declare is a
schema-build diagnostic in every composing host, before any document. Nothing
moved on the stable surfaces: no wire-format change (still v5), no new
diagnostic codes (`ambiguous_element_kind` and `unknown_element_kind` both
exist and are corpus-covered), and the conformance corpus still at 393 cases.

- **The streaming binary cursor consumes the tree-trailing comment block on
  root exhaustion, and corrupt flag-set tails are now refused instead of
  silently accepted.** The wire appends a `[tree trailing comments]` block
  after the roots whenever flag bit 5 is set — the encoder writes the count
  varint even for an empty block — but `BinaryCursor.RootIter.next()`
  returned null at the last root without touching it, so an exact-end
  consumer landed one to eleven bytes short with no way to reach the block.
  `RootIter` now mirrors `ChildIter`/`VectorIter`: its null-producing call
  consumes the count and its entries through the same reader the mid-tree
  comment sites use and reports the count in a new `trailing_count` field.
  Error mapping uses existing codes only (`NodeCountExceeded` over-cap or
  over-long varint, `InvalidTag` bad kind byte, `Truncated` short read);
  entry text indices stay skipped, not pool-validated, matching the
  cursor's count-only stance at all four comment sites. This is a
  deliberate strictness increase on a host-facing surface: the three
  draining `Validator` walkers — and the released `sjon_validate_binary`
  wasm export — now surface those errors on a malformed tree-trailing tail
  where the walk previously succeeded silently, symmetric with the
  mid-tree trailing reads and `init`'s slack rejection. Non-draining
  callers (`evalBinary`, `queryPattern`) are unchanged; wire stays 0x05
  and no diagnostic codes moved.

### Fixed

- **A form child in a slot whose kind reference is ambiguous or unknown reports
  the reference, never a classified mismatch.** When a form lands in a typed
  slot whose `.named` reference does not resolve `.found` — two loaded plugins
  declare the bare name, or nobody declares it — both walkers used to fall into
  expression classification and emit `wrong_underlying` ("form `x` positional
  argument expects `y`, got form"), silently losing the head-set narrowing and
  burying the honest code. The form fast-path in `matchValueAgainstType`
  (`src/Validator.zig`) now falls through to the `.named` arm, and its binary
  twin `matchFormAgainstTypeBinary` gained the matching guard between its
  head-set step and its expression step, so both paths emit
  `ambiguous_element_kind` (naming the claimants and the qualified spelling)
  or `unknown_element_kind` at the child's span. A narrowing that cannot
  resolve is not a value that failed. A document loading one plugin is
  untouched — one claimant, the bare reference resolves, the narrowing runs as
  before — and a reference that resolves to a non-form kind keeps today's
  classified comparison exactly.

- **The aggregate phase resolves every manifest value-kind reference, so a
  bare-referenced kind name two plugins share is named at preload.**
  `Schema.validateKindRefs` joins the aggregate validators as the sixth,
  appended after cross-refs, unions, forms, lowering and defaults so the
  corpus-locked diagnostic-stream order of the other five does not move. It
  walks every plugin's forms (each key's `:type`, a form's `:positional` kind,
  recursing into slot-local `local_forms` on both carriers and into
  per-`:when` variant keys) and every value-kind's vector-shape `:element`,
  skips the primitive type shortcuts, and resolves each reference against the
  aggregate: `unknown_element_kind` for a miss, `ambiguous_element_kind` for a
  cross-plugin collision, span `{0,0}`, path `[<plugin>, <form>, <slot>]`,
  stamped `.aggregate`. Declaring the colliding name confers no priority, so a
  bare reference inside the declarant's own manifest is named too, which makes
  the ambiguity the composition's property, reported when the schema is built
  rather than one document's surprise. The two corpus fixtures of that class
  (`ambiguous-element-kind`, `unknown-element-kind`) were reviewed and moved to
  expect the preload diagnostic ahead of the document's own: their bytes pinned
  one phase of a two-phase class and were written before any preload walk
  existed, and the diff is the prepended diagnostic and nothing else, with the
  corpus still at 393 cases. The TypeScript parity host ports the walk in the
  same change (`hosts/typescript-parity/src/plugin.ts`,
  `validateKindRefs`), the corpus being the cross-host axis; `hosts/web` and
  `hosts/rust` drive the Zig artifact and inherit the walk.
2026-10-25 17:43:45 +00:00
conformance SJON 1.5.0 2026-10-25 16:54:38 +00:00
docs SJON 1.5.0 2026-10-25 17:43:45 +00:00
editors/vscode SJON 1.5.0 2026-10-25 16:54:38 +00:00
examples SJON 1.5.0 2026-10-25 17:43:45 +00:00
fixtures SJON 1.0 2026-06-09 22:58:55 +01:00
hosts SJON 1.5.0 2026-10-25 17:43:45 +00:00
landing-page SJON 1.5.0 2026-10-25 16:54:38 +00:00
manifests SJON 1.3.0 2026-08-27 14:57:32 +01:00
scripts SJON 1.5.0 2026-10-25 16:54:38 +00:00
src SJON 1.5.0 2026-10-25 16:54:38 +00:00
tools SJON 1.5.0 2026-10-25 16:54:38 +00:00
.editorconfig SJON 1.0 2026-06-09 22:58:55 +01:00
.gitignore SJON 1.4.3 2026-09-30 22:38:36 +01:00
biome.json SJON 1.3.0 2026-08-27 14:57:32 +01:00
build.zig SJON 1.4.3 2026-09-30 22:38:36 +01:00
build.zig.zon SJON 1.5.0 2026-10-25 16:54:38 +00:00
CHANGELOG.md SJON 1.5.0 2026-10-25 17:43:45 +00:00
coverage-baseline.txt SJON 1.1.0 2026-08-15 12:23:29 +01:00
LICENSE SJON 1.0 2026-06-09 22:58:55 +01:00
llms.txt SJON 1.4.0 2026-09-11 10:28:07 +01:00
package.json SJON 1.5.0 2026-10-25 16:54:38 +00:00
pnpm-lock.yaml SJON 1.4.1 2026-09-11 11:26:25 +01:00
pnpm-workspace.yaml SJON 1.1.0 2026-08-15 12:23:29 +01:00
README.md SJON 1.5.0 2026-10-25 17:43:45 +00:00
SJON.svg SJON 1.0 2026-06-09 22:58:55 +01:00

SJON

SJON is a schema-constrained data language for domain tools and the agents that operate them.

Domain tools grow config languages; agents write a growing share of the config. SJON gives both a shared substrate: an S-expression AST with the schema as data, validated identically in Zig, JS, and Rust.

Two layers, parsed by one front-end: deterministic S-expression data, plus a pure, bounded, closed-vocabulary expression layer evaluated by the consumer.

(scene :bpm 130
  (canvas :name "main"
    (camera :ortho :zoom (* 2 (b 1)))
    (stack :mode :mask
      (shape :sdf :radius 0.5
        :delay (delay :p+s (b 4))
        :lifespan (b 16))
      (shape :path :points [[0 0] [1 0] [1 1]]))))

Guarantees

Every claim below is machine-enforced, not asserted — each links the gate that checks it.

  • Totality — a tree always exists after parse; collection over abort. No parse can crash the caller.
  • Determinism — bit-identical eval native vs. WASM (vendored exp64/log64/pow64), checked by byte-diffed goldens.
  • Boundedness — no host-stack recursion anywhere in the Parser, Validator, or Expr evaluator; depth, step, and memory-budget ceilings, each with a test that trips it (oom_tests.zig).
  • Wire stability — Binary IR is versioned and diagnostic codes are append-only, both gated by the conformance corpus. Full layout in docs/DESIGN.md.
  • Diagnostics as product — every error carries (code, severity, span, path), a 130-entry explanations catalogue, and LSP quick-fixes. See it below.
Field Size Value
version 1 B 0x05

393 fixtures in the shared corpus; each host replays its supported subset (see the capability matrix below). 25 never-panic fuzz harnesses. 130 explained diagnostic codes. Zero dependencies in the core library. Every number here is drift-gated by zig build audit-docs; run zig build verify to check all of it yourself.

See it catch a mistake

$ printf '(scene :w 800)\n' | sjon validate -
<stdin>:1:2: error: unknown_form: unknown form `scene`
  at scene (phase: validation)
1 error

$ sjon explain unknown_form
unknown_form
  A form's head name is not declared by any loaded plugin.
...
https://hugodaniel.com/pages/sjon/errors/unknown_form

Every diagnostic is a real error page, not a string — same code in the CLI, the LSP's Problems panel, and --format=github CI annotations. See docs/TOOLING.md for the full CLI and LSP capability list.

Consuming

Every host shares the diagnostic vocabulary, the binary wire format, and the conformance corpus. Supported surfaces agree on the replayed fixtures; intentional boundaries and skip registries are documented in the capability matrix and quickstarts: docs/INTEGRATION.md.

Zig — the reference path, zero runtime dependencies:

// build.zig.zon
.dependencies = .{
    .sjon = .{ .path = "../sjon" },
},
// build.zig
const sjon_dep = b.dependency("sjon", .{ .target = target, .optimize = optimize });
exe.root_module.addImport("sjon", sjon_dep.module("sjon"));
const sjon = @import("sjon");

var tree = try sjon.parse(gpa, source);
defer tree.deinit();

const text = try sjon.print(gpa, tree, .{ .mode = .canonical });
defer text.deinit();

Node / Web (WASM) — install npm install @sjon-lang/web@1.5.0. Build matching artifacts from the 1.5.0 source with Zig 0.16 (zig build wasm-all); the npm package does not bundle WASM. From the source root:

import { SjonEncoder, SjonReader } from "@sjon-lang/web";

const encoder = await SjonEncoder.load("./zig-out/bin/sjon.wasm");
const reader = await SjonReader.load("./zig-out/bin/sjon-binary.wasm");
const binary = encoder.toBinary('(scene :bpm 130)');
const result = reader.validateBinary(binary);

See examples/quickstart-web.mjs and hosts/web/README.md.

Rust — cargo add sjon-host@1.5.0; wraps the matching external sjon.wasm via wasmtime (the crate does not bundle it):

use std::path::Path;
use sjon_host::{HostOptions, SjonHost};

let mut host = SjonHost::load(Path::new("zig-out/bin/sjon.wasm"), None)?;
let result = host.validate_document(source, &HostOptions::default())?;

See examples/quickstart-rust.rs and hosts/rust/README.md.

TypeScript — hosts/typescript-parity/ is a hand-ported reference for conformance verification, not for production; @sjon-lang/schema (hosts/schema/) is the typed schema builder for TS-first projects. See hosts/schema/README.md.

Docs

Building

zig build test      # all test suites (all in-source)
zig build verify    # every local gate: tests, fuzz, audits, all hosts, biome, clippy
zig build wasm-all  # both WASM artifacts
zig build lsp       # native sjon-lsp

Requires Zig 0.16.x.

Versioning & releases

The stable surfaces are the binary wire format and the diagnostic-code enum — versioned, and gated by the conformance corpus plus zig build verify. A change to either is deliberate and version-gated: the wire bumps its format integer, diagnostic codes only append, and the corpus lands updated in the same change.

Every package in this repo carries the one version declared in src/version.zig; zig build audit-format-versions names any manifest that falls out of step. The @sjon-lang/* packages carry full registry metadata and are published through scripts/publish-npm.sh. Use pnpm pack for local tarballs: pnpm rewrites the web package's workspace:* dependency into the matching registry version. Workspace and path dependencies remain supported. What each release changed is in CHANGELOG.md. The repeatable candidate and publication checks are in docs/RELEASING.md.

License

CC0 — public domain. See LICENSE for the legal text.