Evaluation prompts for pilots

Two-week pilots fail when success is argued from memory. Fix ten golden tasks up front and score each run with the same rubric.

Pick golden tasks

Choose tasks your team already does weekly: ticket summaries, FAQ drafts, SQL explanations, onboarding blurbs. Avoid exotic one-offs.

Scoring rubric

Rate 1–5 on accuracy, completeness, tone, and edit time required. A human editor times how long it takes to ship the output.

Blind comparison

When comparing models, shuffle outputs so reviewers do not know which model produced which draft until scores are recorded.

Exit criteria

Document minimum average scores and maximum edit time before expanding access beyond the pilot group.