Sil-Q evaluation

A coding agent plays Sil-Q through structured public observations and its 80×27 terminal. The protected service owns the engine, private randomness, and scoring. A win requires taking a Silmaril from Morgoth's crown and escaping with it.

The September 2026 study completed ten Astra-medium sessions, each with eight practice games and two scored games: 0 wins in 20 scored games. Across all 100 games, 99 ended in death and one practice game was retired. Both Astra-max attempts were interrupted before scored play and excluded; the controls were canceled. There is no scored max result or measured effect of practice.

Website · Live trajectories · Local report and replays · Report source · Results · Methods · Scope amendment

Current harness

Revision 0.8.0 uses silq-session-v4 with silq-observation-v2 and a consistent silq-response-v1 envelope. It provides a readable default view, typed native prompts, displayed item choices, guarded input, named actions and a message-page acknowledgment helper. --json returns the same state/events structure for reads, single keys and batches. Native victory stays binary; depth, jewel acquisition and ascent are separate diagnostics. Durable receipts, verified restart replay and authenticated final scoring remain in place.

The new contract fixes total session time, per-game time, optional model cost, provider retries and container CPU/memory/process budgets before dispatch. These limits do not reset between games or after service recovery.

The primary metric is native victory within the declared budgets. Verified trial time/key exhaustion scores zero for that trial. Session expiry scores unfinished and unstarted games zero and retains earlier wins. Timeouts are reported separately from native deaths; infrastructure failures and operator stops remain unadmitted. The journal must prove session expiry before a partial schedule can be admitted.

See the interface upgrade for behavior and validation. A fresh public walkthrough passed native auditing; container recovery, Inspect integration, native replay and hidden-state checks also passed. A personally played game reached 100 feet and exposed clipped/stale terrain labels, now fixed. The original 375-frame recording remains preserved. The interface validation made no model requests. The following larger command-fuzz result belongs to the 0.6.2 baseline. The native command audit fixed smithing, projectile and knowledge-menu bugs. A 128-sequence fuzz run passed 139,923 sanitizer frames with exact native replay; the existing native and hidden-state checks also passed. Real containers passed recovery, task discovery, remote execution identity, all five scorers and resource telemetry checks. Direct public play and its native replay also passed. The first capped model pilots cost about $6.89 in recorded usage. The subsequent uncapped GPT-5.5 medium pilot completed naturally: death at 100 ft, roughly 24 minutes and $49.86 estimated cost. Its native replay and qualitative admission passed. The Astra-max release pilot also completed naturally and passed admission: a fatal fall from 150 ft to 250 ft after 82 minutes, at $180.35 estimated cost. Release 0.8.0 readiness is verified; research still needs a declared plan with readiness bound to its exact hash. See launch validation for pilot results, evidence and the next experiment commands, and the implementation notes for architecture. Historical results retain their frozen implementation and materials.

The grading and fairness audit fixes Rage information leaks, Listen RNG isolation, fatal-fall depth accounting and recovery deadlines. Native surface-exit tests cover wins, close misses, phase isolation and interrupted wins.

Development

Use Python 3.11 or later, uv, and Docker with Compose. From this directory:

uv sync --frozen
uv run ruff check .
uv run ruff format --check .
uv run ty check --error-on-warning
uv run python scripts/bootstrap.py
uv run pytest -q -rs
uv run python scripts/native_qa.py --output /tmp/silq-native-qa.json
uv run python scripts/qa_inspect.py --output .private/qa-new --report /tmp/silq-inspect-qa.json
uv run python scripts/qa_inspect.py --session-timeout --output .private/qa-timeout-new --report /tmp/silq-timeout-qa.json
uv run python scripts/qa_recovery.py --output .private/qa-recovery-new

These checks make no model requests. Bootstrap builds the development image tags. To preserve existing tags, build separate Docker service and client targets and pass --service-image and --client-image to the smoke tests. Output directories and run names are single-use. CI runs the engineering checks. Ruff and ty cover the runtime, scripts and tests. Their versions and the reporting dependencies are pinned in the development environment. Use uv run ruff check --fix . and uv run ruff format . to apply routine fixes.

Experiments

Copy a template from experiments/ and set a unique name, images, limits and bundle count. All templates use cost_limit: null: model cost is recorded but does not stop the run. Set a positive dollar amount to opt into a cap, or use --no-cost-limit with the low-level preparer.

Session time is shared across all games. The development settings match the completed pilot; the research templates are proposals, not a registered final study. Review the launch assessment before choosing a final schedule. Run-specific copies live with frozen private experiment manifests rather than accumulating dated templates in the source tree. Preparation freezes source, images, schedules, and commands without model calls:

cp experiments/development.template.json /tmp/silq-development.json
uv run python scripts/experiments.py prepare /tmp/silq-development.json --output .private/experiment-new
uv run python scripts/experiments.py status .private/experiment-new/experiment.json

To launch development model sessions, resume supervision, and review a closed run:

uv run python scripts/experiments.py run .private/experiment-new/experiment.json --workers 1
uv run python scripts/experiments.py resume .private/experiment-new/experiment.json --workers 1
uv run python scripts/audit_run.py .private/run-name

Workers survive supervisor exit. Locks and saved launch intents prevent duplicate Inspect launches. Failures stop queued launches; ambiguous exits require review. resume resumes supervision, not an interrupted Inspect conversation. Client notes, code, and request history persist in each run's workspace/. Only the service mounts state/. Keep both run directories and experiment manifests private.

Research dispatch requires fresh development, direct-play, resource, and production evidence, plus a readiness record whose experiment_sha256 binds the prepared plan. experiments.py gate derives the max gate from every registered, admitted medium session. A mixed medium/max plan pauses max in awaiting_medium_gate while medium runs finish and are reviewed. Add the gate's path and hash to a reviewed readiness record, then resume supervision. An exploratory gate permits a follow-up; it is not a population saturation claim. launch.py only prepares runs; all dispatch goes through the versioned supervisor.

The audit wrapper checks request identity, native replay, tool history, and public observations. It reads report findings as well as exit codes. Truncated views and boundary flags need review; they do not automatically invalidate a loss. Admission requires qualitative review and evidence hashes.

Requests and recovery

The game guide documents the CLI. GET /state reads the current public frame and GET /overall reads the schedule. Mutation bodies contain exactly these fields:

Each new mutation uses a canonical UUID. Retrying the same endpoint and body returns its stored status and response, including after restart. Reusing an ID with different content fails. The CLI saves requests before sending them; silq retry retrieves the last request's receipt.

Recovery replays the original random stream and compares every committed native frame, including private state. Changed code, data, configuration, or receipts block recovery. Only an unterminated final journal line can be removed; its bytes are retained. Historical v1/v2 sessions require their frozen service images.

An interrupted batch returns its executed prefix. A pending key is replayed on the same stream and its output accepted at recovery time; remaining keys are not sent. Read the returned state before continuing. A start interrupted before trial creation returns an error and can be issued deliberately with a new ID.

Deadlines remain absolute across downtime. An ending observed before the deadline retains its result after restart. Infrastructure failures remain invalid. One service can own a session at a time; HTTP connections and idle waits are bounded.

Results and historical records

For a new experiment, experiments.py report MANIFEST --output REPORT.json records every registered attempt, its admission status, available cost evidence and progress. Complete admitted conditions can be plotted with python -m scripts.reporting.plot_results --depth-only --input REPORT.json --output NEW_FIGURE_DIRECTORY. The schedule comes from the experiment instead of assuming ten games or two scored games. This produces depth by game and elapsed time, plus Silmaril acquisition and carrying-ascent figures. Unstarted games are marked explicitly. Verified time/key exhaustion remains in the capability denominator; infrastructure failures and unreviewed runs leave the condition incomplete.

The primary cost estimate is estimated_session_priced_usd. Each manifest freezes the standard rates and model-specific long-context rule: GPT-5.5 applies the uplift across the session; Astra applies it per request. The older request-wise field remains diagnostic. Missing usage makes the total unknown, including when a model request is canceled at the session deadline. These estimates exclude regional/tier adjustments and are not invoices.

The historical medium depth plot and its source data retain the original study's results. Old and new observation protocols are reported separately.

The current Python runtime uses strict Pydantic schemas for JSON boundaries and dataclasses for trial/replay state. The client, HTTP transport, native process, journal reader and recovery code have separate owners; see the architecture notes. Current audit and recovery accept only silq-session-v4. Use frozen source and images for historical runs.

analysis/ contains results, reviews, engineering records, screenshots, and the original guide versions. site/ contains the report, figures, and public replays. .private/ retains logs, journals, source snapshots, schedules, and image archives. Historical controller-specific validators remain with their frozen run sources and in Git history. Report and replay builders live in scripts/reporting/. See the script index.

Development history, selection rationale, study resource limits, and the September 9 audit describe earlier stages. Native errata explain errors in the materials used by the agents.

To rebuild the report and check its replay viewer:

uv run --frozen --with markdown python -m scripts.reporting.build_site
uv run --frozen --with playwright python scripts/reporting/qa_site.py

uv run --frozen --with matplotlib python -m scripts.reporting.plot_max_case renders the reviewed max case studies, including exclusions.

The website publisher serves the archive and follows new experiment runs automatically. It publishes public terminal frames only.