James Liu← All work

Local-first graphics AI system

RenderMasterBot

A schema-gated AI system that turns graphics requests into validated Unreal Engine scene plans, renders evidence, and proposes bounded corrections.

Unreal Engine material preview produced through RenderMasterBot
Focus

AI architecture, retrieval & Unreal automation

Tools & methods

Python · Unreal Engine 5 · Ollama · Pydantic

Project

Local-first graphics AI system

The model proposes. Deterministic code decides what can run.

RenderMasterBot does not allow a language model to execute arbitrary engine code. Planning crosses a versioned RenderSpec contract, strict schema validation, semantic preflight, asset allowlists, and host-owned evidence checks before Unreal receives an instruction.

Unsupported requests return an explicit capability or asset gap. The system is designed to refuse an invented engine path instead of disguising the problem with an unrelated scene change.

  • Versioned Pydantic contracts and exported JSON Schema
  • Observed Unreal capabilities constrain every plan
  • Hashes and run manifests preserve reproducible evidence
Execution gateunreal_executor.py · Python
preflight = run_preflight(spec)
if preflight.verdict == "fail" or (
    fail_on_warning and preflight.verdict == "needs_review"
):
    issue_ids = ", ".join(
        issue.issue_id for issue in preflight.issues
    ) or "unknown"
    raise UnrealSceneBuildError(
        f"RenderSpec preflight returned {preflight.verdict}: {issue_ids}"
    )

catalog: dict[str, AssetCard] = {}
for card in cards:
    if card.asset_id in catalog:
        raise UnrealSceneBuildError(
            f"duplicate AssetCard ID: {card.asset_id}"
        )
    catalog[card.asset_id] = card

The language model never talks directly to Unreal. A deterministic preflight and a validated asset catalog establish the execution boundary first.

Plans are grounded in assets the Unreal project actually contains.

A headless Asset Registry scan converts real project assets into validated AssetCards. Those records are indexed locally with explicit embeddings and retrieved by type, so the planner can assign only meshes and materials that survive catalog verification.

The execution layer can import a complete PBR material without overwriting existing content, build a transient scene, frame the camera from measured asset bounds, and render a one-frame Movie Render Queue preview.

Every correction must be small, inspectable, and measurable.

A local vision model evaluates the rendered preview against observed scene evidence. The correction planner then emits either a validated RenderSpecPatch or a precise statement of what the current system cannot repair.

The current milestone includes 122 automated tests across contracts, planning, retrieval, Unreal execution, preview integrity, evaluation, and patching. Broader scene editing and an Unreal Editor panel remain in development.

  • Tampered or mismatched preview evidence is rejected before inference
  • Patches are reapplied through the same validation boundary
  • Planning, vision, and embedding models remain independently replaceable
Next projectInput Recorder & Ghost Replay