Skip to content
Marcus Gonzalez

Agent UI SDK + Eval Framework

The evals changed the system, not just the score.

A fixed set of representative product tasks, the golden corpus, reveals whether a change removes a failure or only hides it. The loop cut the worst component misuse from ~95% to near zero and, in a small internal benchmark, produced work a designer rated higher at ~5× lower cost per task.

Fig. 00 · Method · The evaluation loop

Repeated evaluation changes the SDK context, a contract, or the component, so the same failure stops returning.

  1. 01Generate

    AI agents repeat the golden corpus of representative product tasks against the current tooling.

    same prompts, every run
  2. 02Check

    Validation contracts decide whether the result is ready for review.

    build · tests · stories · a11y
  3. 03Evaluate1

    A designer evaluates visual and interaction quality, pattern consistency, and brand.

    human
  4. 04Change the source

    Route each recurring failure to where it belongs, then run the same tasks again.

    • compile better SDK context
    • add a measurable validation contract
    • replace the component when its API is the cause
    ↺ next run

Hold the work constant · improve the system · measure again

1

Evaluate is the loop's one human stage. The contracts decide whether the result is ready for review; the designer decides whether it is good.

Formal density signal atlas · the loop's four stagesCorrectness, quality, and cost
The evaluation loop drawn as four ruled stages: generate, check, evaluate, and change the source. The designer's evaluation is the one stage marked in cobalt, and a returning rule holds the work constant so the system, not the score, is what moves.
Context01

Home Lending teams build with the firm's core design system, MDS, and a product-specific pattern library. Without one current source of truth, agents can choose the wrong pattern and still produce working code.

The Home Lending Employee Pattern Library extends MDS with new compositions and custom components, and sometimes replaces an option agents still see in the core system. The Agent UI SDK reads the packages installed in each project, resolves those overlaps, and builds one current catalog.

Fig. 01 · Signal atlas · Context

The SDK turns two systems into one usable catalog.

  1. Under test
    • MDS · core components and primitives
    • Employee Pattern Library · themes, compositions, custom components
  2. Agent UI SDK
    • build one current catalog from installed packages
    • resolve which patterns take precedence
    • check the result against measurable contracts
  3. Coding session
    • the right component from the right system
    • superseded options removed from view
    • targeted guidance and measurable checks
System map · two libraries into one catalogMDS + Home Lending · one current set of choices
A system trace: the two libraries under test, MDS and the Employee Pattern Library, run through the Agent UI SDK, which compiles one current catalog and resolves precedence, so a coding session sees the right component and measurable checks.

The SDK guides and checks the work without generating the interface itself. An agent can choose its own path, but it cannot finish the task without the required stories, tests, a successful build, and visual evidence.

Problem02

Poor APIs and defaults do more than create a bad screen. They make the same mistake a recurring development cost.

Agent mistakes surface in two ways. Type errors stop the build; other mistakes still render, with the wrong component, doubled padding, or off-system spacing. The deeper problem is that the agent can declare either result finished. A designer opens the work expecting to review the experience, but instead has to diagnose the failure, tell the agent what to fix, and start the review again. Every false finish wastes design time and adds another correction cycle.

Fig. 02 · Illustrative reconstruction · Problem

The same output, graded two ways.

Passes the build

✓ compiles · ✓ renders · ✓ looks done

Illustrative reconstructionSanitized capture pending
Fails the bar
  • wrong primitive
  • padding applied twice
  • off-system spacing
Illustrative reconstructionSanitized capture pending
1
1

Three failures, zero build errors. If these failures pass review, they create inconsistent design, harder-to-maintain code, and another correction cycle.

Illustrative reconstruction · production UI and values differA build error is not required to create debt
A generated card, drawn twice: on the left it passes the build, compiles, renders, and looks done; on the right the same card is graded against the quality bar and fails on the wrong primitive, doubled padding, and off-system spacing.

Design debt. Wrong primitives and inconsistent defaults spread patterns that future screens repeat.

Technical debt. Invalid composition and workarounds leave code that is harder to change safely.

Delivery cost. Repeat review slows development, while repeat instructions consume tokens on every task.

Method03

The corpus keeps the task constant. Automated checks establish readiness; a designer judges the quality of the result.

The corpus holds the work constant across tooling and model changes. Each run measures correctness, component misuse, design quality, and cost against the previous result.

Fig. 03 · Signal atlas · Method

Stable work runs through measurable contracts, then a designer judges what the checks cannot.

  1. Golden corpus

    Representative product tasks, prompts, and expected requirements stay stable across runs.

  2. Validation contracts1
    • no overflow · responsive behavior
    • internationalization · accessibility
    • test coverage · story coverage · build
  3. Designer evaluation

    Visual and interaction quality, pattern consistency, and alignment with brand.

1

The contracts decide whether the result is ready for review. The designer decides whether it is good.

Signal atlas · corpus, contracts, judgmentStable work · measurable correctness · human quality judgment
The evaluation stack as a trace: the golden corpus holds the tasks stable, validation contracts check correctness across overflow, responsiveness, internationalization, accessibility, coverage, and build, then a designer evaluates quality and brand.
Decisions04

Four failed architectures established the boundary: let the model generate, then use deterministic checks to decide whether the work is complete and correct.

Replace instructions with a completion contract

Sequenced instructions still allowed agents to skip tests, stories, and setup while claiming the work was finished. I replaced the sequence with a completion contract: the required evidence must exist, the build must pass, and rendered screenshots must expose what code checks cannot verify. Design review then focuses on hierarchy, interaction quality, consistency, and brand instead of preventable defects.

Fig. 04 · Signal atlas · Prior architectures

Four prior architectures taught one boundary: constrain the result, not the path.

  1. 01Deterministic generatorIts instructions grew too large to carry.
  2. 02Monolithic skillIts context stayed too broad to steer.
  3. 03Specialized agentsLatency and platform limits made it brittle.
  4. 04Prescribed sequenceAgents skipped work the sequence could not observe.

The model generates and repairs the work. Deterministic contracts decide when it is complete.

Four architectures preceded the current oneConstrain the result, not the path
Four prior architectures on a descent, each with the lesson it taught by failing: a deterministic generator, a monolithic skill, specialized agents, and a prescribed sequence. Beneath them, the doctrine they established: let the model generate and repair, and let deterministic contracts decide when it is complete.

Route each recurring failure to its owner

I use the evaluations to decide where each recurring failure belongs. A usage problem becomes prose guidance in the SDK-generated reference for that component. A measurable mistake becomes a validator rule. If neither can overcome the component's API or defaults, I replace the component.

Replace the API when the same correction keeps returning

Agents misused Layout in ~95%1 of runs. Validation caught each mistake, but the next run repeated it. Replacing Layout with familiar horizontal and vertical stacks moved misuse to near zero without more guidance.

Fig. 05 · Signal atlas · Observed misuse

Validation caught the misuse; only replacing the component removed it.

~95%

No validation

Layout misused in nearly every generation.

~95%

Validation added

Every mistake caught, and repeated on the next run.

~0%✓ held

Component replaced1

Stacks used correctly with no added guidance.

1

Validation cleaned up the result. Replacement changed the next generation. That distinction became the threshold for changing a component.

Normalized 100-cell proportion field · not a run countObserved misuse rate
One failure across three interventions, drawn as normalized proportion fields: with no validation the misuse is dense and solid; with validation added it is caught but repeated, shown as ringed squares; once the component is replaced the field is nearly empty and the correct use holds.

The best validation rule is the one the API makes unnecessary.

Compile each lesson so the next task uses fewer tokens

At setup, the SDK reads the MDS and Employee Pattern Library packages installed in the project and generates a current component catalog. During a task, the agent retrieves only the components and guidance it needs. The SDK adds validator guidance only when the agent breaks the corresponding rule.

Fig. 06 · Signal atlas · Context model

Compile both systems once; spend inference only on what the task needs.

  1. Compile once

    Installed MDS and Employee Pattern Library code becomes one current catalog with component-level guidance.

  2. Spend by task
    • resolve composition and supersession
    • retrieve only the relevant components and guidance
    • inject a validator correction only when its rule fires
  3. Next result

    Less repeated context, fewer recurring defects, faster completion.

Signal atlas · compile once, retrieve by taskCompile both systems · retrieve only what the task needs
The context model as a trace: installed code is compiled once into one current catalog; a task then resolves precedence, retrieves only what it needs, and injects a validator correction only when a rule fires; the next result carries less repeated context and finishes faster.

In a small benchmark using the golden corpus, a designer rated the Agent UI SDK's output higher than the stronger internal comparator. The SDK cost ~$4.50 per task versus ~$242 and finished in 8–11 minutes rather than ~20–25 because its references and validation live in the repository instead of behind a remote knowledge base.

Fig. 07 · Signal atlas · Benchmark

Repo-local context, not a bigger model, produced the cost and speed.

~5×lower AWS Bedrock cost per task, at higher designer-rated quality

AWS Bedrock cost per task~$4.50Agent UI SDK~$24 comparator
Completion time8–11 minAgent UI SDK~20–25 min comparator
Small internal benchmark on golden-corpus prompts · designer-reviewed qualityComparator teams anonymous outside Chase
A cost contrast: the Agent UI SDK against the stronger internal comparator, drawn to scale. Cost per task is about five times lower, and completion time is roughly half, with the sample limits and the repo-local mechanism stated in the figure's basis.
Status05

The corpus now evaluates every tooling change in the Agentic Originations product area, turning recurring failures into system improvements.

Scope
The component library and Agent UI SDK support the ~30-person Agentic Originations product area.
In motion
The team turns findings into updated references, validation contracts, or component changes and ships them through the normal build process.
Trajectory
The architecture supports multiple teams; one product area runs it today.
Measuring
Each run measures contract completion, component misuse, designer-rated quality, and token spend per task.
Impact
Each known failure becomes a reusable fix, reducing repeat review and preventing new design and technical debt.
Evidence and sources

1~95% misuse to near zero — The drop from ~95% misuse to near zero comes from evaluation audits before intervention and after Layout was replaced with stacks.

2~$4.50 vs ~$24 · 8–11 vs ~20–25 min — Cost and time come from a small benchmark using prompts from the golden corpus. A designer reviewed the resulting interface quality; comparator teams stay anonymous outside Chase.

Contact

How do you tell whether generated work is actually good, and not just green? If you're building evaluation into AI-generated work, let's talk.

marcus@marcusjg.comLinkedIn

Next case →

Designer Enablement

The threadThe evals define what production-ready output must contain. The next case shows how designers learn to produce work that can meet that bar.