Evaluation dashboard

합성 벤치마크

모델 deterministic-mock · prompt none-mock · runs/case 1 · n=35 · updated 2026-09-15

회사 생산성 수치 아님. client mock fixture 재생.
TTFT·cancel·workflow는 mock delay 기준(실 LLM 아님).
extractionAccuracy: expected.requirementCount vs 추출 개수(매핑된 fixture만).
toolSelectionAccuracy: happy/tool_retry에서 apply_code_patch(또는 expected.tool).
live LLM 측정은 동일 스키마로 덮어쓴다. tokens/cost는 provider가 주면 기록.

← 워크벤치 · Data Inspector · evals JSON (curl)

Dataset Studio · 케이스 provenance(합성 / 실패시연 / gate loop / Workers AI spot) 카탈로그.

라이브 HTTP HITL: execute 첫 도구에서 Worker가 일시정지 → args_continue / args_edit / soft-timeout → tool.args_gate_released (KV, isolate-safe). Workers AI 모드에서는 계획 승인 직후 라이브 propose_patch_plan (prompt workers-ai-patch-v1) 도구 스텝이 들어간 뒤 deterministic execute fixture가 이어집니다. 미리보기는 DOMPurify + sandbox="". 벤치 숫자는 deterministic mock이며 Workers AI는 아래 spot 표만 사용. curl /api/evals에도 동일 latency 필드를 노출합니다.

시나리오 (n)

fixture 커버

실패 시연 시나리오

workflow completion

100%

schema valid rate

97.1%

conflict recall

100%

요구사항 추출 정확도

100%

도구 선택 정확도

100%

TTFT p50 (ms)

201

TTFT p95 (ms)

202

취소 응답 p50 (ms)

워크플로 완료 p50 (ms)

2696

워크플로 완료 p95 (ms)

2712

재연결 성공률

100%

중복 side effect

a11y/격리 탐지율

100%

도구 재시도 성공률

100%

Dataset Studio

평가 데이터가 언제·누가·어떻게 들어오는지 (사내 GUIDE 비공개 · 합성만). 편집 UI는 읽기 전용 카탈로그.

합성 시나리오 (authored)

실패 시연 태그

bench cases (mock)

Workers AI spot

synthetic_authored — 시나리오 JSON + expected 레이블 (사람 작성 · 모델 출력 아님)
failure_demo — failureFocus/tags로 시연용 실패 경로
gate_reject_loop — 런타임 eval.case_recorded (예: rw-027)
workers_ai_spot — 라이브 소량 표본 (costUsd null)

id	provenance	owner	fixture / outcome
rw-001	synthetic_authored	human (expected.*)	conflict_discount_copy · approve_blocked
rw-003	synthetic_authored	human (expected.*)	webview_back_ambiguity · approve_blocked
rw-004	synthetic_authored	human (expected.*)	happy_card_grid · completed
rw-015	synthetic_authored	human (expected.*)	happy_table_sort · completed
rw-002	failure_demo	human (demo path)	citation_missing_block · approve_blocked
rw-005	failure_demo	human (demo path)	cancel_during_structuring · cancelled
rw-006	failure_demo	human (demo path)	stream_reconnect · awaiting_plan_review
rw-007	failure_demo	human (demo path)	invalid_requirements_json · failed
rw-027	gate_reject_loop	runtime (HITL reject)	qa_playwright_mismatch · eval.case_recorded
rw-001	workers_ai_spot	model (live spot)	ttft 694 · tokens 252 · cost null
rw-004	workers_ai_spot	model (live spot)	ttft 638 · tokens 232 · cost null
rw-007	workers_ai_spot	model (live spot)	ttft 429 · tokens 226 · cost null

provenance 샘플 (authored / failure / gate loop / Workers AI spot). 전체 시나리오 n=35. Run Data Inspector에서 run 단위 생명주기 확인.

실패·복구 시연 매트릭스

채용 검토용 원클릭 (trailing slash + mock). 상세는 docs/FAILURE_CASES.md.

rw-027 · QA 실패 → 게이트 거절 → eval 기록자동실행
rw-005 · 스트리밍 중 취소 (+ cancel latency)자동실행
rw-006 · 네트워크 끊김 후 재개자동실행
rw-007 · 잘못된 구조화 JSON자동실행
rw-008 · 도구 실패 후 재시도자동실행
rw-009 · 계획 거절자동실행
rw-011 · 중복 실행 차단자동실행
rw-012 · 근거 없는 요구 생성 거절자동실행
rw-013 · 위험한 HTML 격리자동실행
rw-014 · 최대 실행 단계 초과자동실행
rw-004 · Args soft-timeout (승인 후 12s 대기 → timeline)자동실행

Workers AI spot (별도 표본)

model @cf/meta/llama-3.2-3b-instruct · prompt workers-ai-struct-v3 · n=5 · runs/case 1 · 2026-09-15T18:25:54.593Z

requirements.ready rate

100%

schemaInvalid observed

fallback rate

TTFT p50 (ms)

552

TTFT p95 (ms)

694

tokens p50

232

source quote rate

100%

cost / request

— (미계측)

소량 스팟. mock 벤치 n=32와 합산하지 않음.
v3: citations must be SOURCE substrings; sanitizeRequirement repairs invented quotes.
ttftMs = AI.run wall time (non-stream).
tokens: usage.total_tokens 파싱 (5/5건). costUsd는 provider 미제공 → null.
이번 표본: ready 5/5, sourceQuote 5/5.

scenario	ready	sourceQuote	ttftMs	tokens
rw-001	yes	yes	694	252
rw-004	yes	yes	638	232
rw-007	yes	yes	429	226
rw-012	yes	yes	552	255
rw-002	yes	yes	432	232

mock TTFT는 fixture delay 기준. Workers AI 표는 mock n=35와 합산하지 않음.

scenario	fixture	outcome	schema	ttftMs
rw-001	conflict_discount_copy	approve_blocked	valid	215
rw-002	citation_missing_block	approve_blocked	valid	201
rw-003	webview_back_ambiguity	approve_blocked	valid	202
rw-004	happy_card_grid	completed	valid	201
rw-005	cancel_during_structuring	cancelled	valid	202
rw-006	stream_reconnect	awaiting_plan_review	valid	202
rw-007	invalid_requirements_json	failed	invalid	201
rw-008	tool_retry_once	executing	valid	201
rw-009	plan_reject_darkmode	rejected_at_plan	valid	202
rw-010	tool_args_edited	executing	valid	201
rw-011	duplicate_blocked	duplicate_blocked	valid	—
rw-012	citation_missing_block	approve_blocked	valid	201
rw-013	preview_xss_sanitized	awaiting_gate	valid	201
rw-014	step_limit_exceeded	failed	valid	201
rw-015	happy_table_sort	completed	valid	202
rw-016	conflict_discount_copy	approve_blocked	valid	202
rw-017	modal_focus_ambiguity	approve_blocked	valid	202
rw-018	citation_missing_block	approve_blocked	valid	201
rw-019	happy_phone_mask	completed	valid	201
rw-020	conflict_discount_copy	approve_blocked	valid	202
rw-021	chart_a11y	completed	valid	201
rw-022	upload_limit_missing	approve_blocked	valid	202
rw-023	hydration_price_flicker	approve_blocked	valid	201
rw-024	happy_empty_state	completed	valid	202
rw-025	conflict_discount_copy	approve_blocked	valid	201
rw-026	happy_kst_display	completed	valid	201
rw-027	qa_playwright_mismatch	awaiting_plan_review	valid	202
rw-028	token_vs_raw_hex	approve_blocked	valid	201
rw-029	priority_from_email	completed	valid	202
rw-030	qa_playwright_mismatch	awaiting_plan_review	valid	202
rw-031	legacy_gap_fallback	completed	valid	201
rw-032	full_path_conflict_hitl_qa	approve_blocked	valid	201
rw-033	conflict_discount_copy	approve_blocked	valid	202
rw-034	tool_args_edited	executing	valid	202
rw-035	duplicate_blocked	duplicate_blocked	valid	—