you can score an AI-assisted work sample fairly without being an expert in the learner's job, as long as you score what you can see rather than whether the answer is right for that trade. The ed.cafe rubric scores four observable behaviors, 0 to 3 points each: a real job task, output checked against a source and fixed, zero private data, and a plain explanation. A sample passes at 8 of 12 with no zeros. Two scorers who calibrate on the same anchor samples for 30 minutes first will disagree far less than two scorers working from instinct.
The question you are actually asking
You run a group. Someone in it does medical billing. Someone else does bookkeeping. You do neither. At the end of the course each learner hands you one page of work they did with AI, and you are supposed to say whether it is good.
You are right to hesitate. If you judge the billing content, you are guessing. The fix is to stop judging the content and start checking the behaviors that make AI work trustworthy in any job.
Why behaviors, not domain answers
Think of a driving test examiner. The examiner does not need to know where you are going. They watch whether you check your mirrors, signal and stop at the line. Those are visible, and they predict safe driving anywhere.
AI work has the same kind of visible habits. Did the learner use AI on a real task? Did they check the output against something trustworthy and correct it? Did they keep private data out? Can they explain what they changed?
Employers have a reason to care. A WGU survey of 3,128 U.S. hiring professionals, cited by COABE's AI Hub, found that 60% say AI makes candidates' real skills harder to evaluate. COABE suggests "pairing AI practice with work samples and explanations that show learners' own reasoning." The Department of Labor's voluntary AI Literacy Framework (TEN 07-25).pdf) lists "Evaluate AI Outputs" and "Use AI Responsibly" among its five content areas, but sets no measurement standard and notes providers may be uncertain about "how to assess competency." This rubric is one practical answer to that gap.
The rubric, criterion by criterion
Each criterion scores 0 to 3. Read the level descriptions as "what you can point to on the page."
1. Real job task
- 0: no task, or a generic prompt with no job attached ("write something about customer service").
- 1: a task is named, but it is not from the learner's field or current or target job.
- 2: a real task from their field, but who it is for or what it will be used for is unclear.
- 3: a real task, with the audience and purpose stated, and the output in the format the job uses.
2. Checked and fixed
- 0: no check recorded.
- 1: "I checked it" with no source named.
- 2: a named source was compared with the output; no error was found, or only wording was changed.
- 3: a named primary source, at least one specific error identified, and the corrected version shown.
3. Zero private data
- 0: any real customer, patient or employee identifier, or any confidential figure, appears in the prompt or the output.
- 1: no names, but details that could identify a real person or account in context.
- 2: all data invented or public, but not labeled as such.
- 3: all data invented or public, labeled ("names and amounts invented"), and the learner notes what they replaced.
4. Plain explanation
- 0: missing.
- 1: present, but vague or full of jargon; it does not say what changed.
- 2: says what was kept and what was changed, but not why.
- 3: two clear sentences covering what was kept, what was changed, and why.
Anchor samples at 4, 8 and 12
Anchors are scored examples both scorers study before they score anything real. These are illustrative, written for this article, with invented data throughout.
Illustrative example, 4 points (fail). A customer support learner asks an AI tool to "write a good reply to an upset customer." Task: 1, because nothing ties it to a real situation. Checked: 1, the note says "looked fine." Data: 1, the prompt includes a ticket number in the employer's real format. Explanation: 1, "AI saved time." Total 4. No zeros, but far below 8.
Illustrative example, 8 points (pass). A medical billing learner drafts a letter explaining a denied claim to a patient, using an invented patient and invented amounts. Task: 2, real task, but the letter's reader and reading level are not stated. Checked: 2, the learner compared the AI's description of the denial code with the payer's published code list and found it matched. Data: 2, everything invented but not labeled. Explanation: 2, says what was kept and changed, not why. Total 8, no zeros. Pass.
Illustrative example, 12 points (pass). A bookkeeping learner asks AI to summarize when a small business must file a sales-tax return. Task: 3, the audience is a named type of client and the output is a one-paragraph client note. Checked: 3, the learner compared the summary with the state's official guidance page, found the AI gave the wrong filing frequency, and shows the corrected sentence. Data: 3, the business is invented and labeled. Explanation: 3, two sentences on what was kept, what was fixed, and why. Total 12.
The pass rule: 8 of 12, and why no zeros
Here is the catch with a points total on its own. A learner could score 3, 3, 0, 3 and reach 9. That sample shows a real task, a careful check and a clear explanation. It also shows real private data pasted into an AI tool.
Illustrative example, 9 points (fail). Same bookkeeping note as above, but the prompt contains a real customer's email address. Total 9, with a zero on criterion 3. Fail.
The no-zeros rule makes each criterion a floor. An employer reading "passed" should be able to trust that the learner did all four things at least partly, and never leaked data.
The 30-minute calibration routine for two scorers
Run this before scoring any real samples, and repeat it at the start of each cohort.
- Minutes 0 to 5. Read the four criteria and level descriptions aloud. Read the anchors with their scores.
- Minutes 5 to 15. Each scorer independently scores three practice samples with the scores hidden. Write scores per criterion, not just totals.
- Minutes 15 to 25. Compare criterion by criterion. Where you differ by 2 or more points, reread the descriptor and agree a rule. Write it in a one-page decision log ("A ticket number in the employer's real format counts as identifying").
- Minutes 25 to 30. Agree which real samples you will both score. Score the first ones independently, then compare.
How to know whether two scorers agree
Agreement must be measured, not assumed. ed.cafe has no agreement data yet, because no cohort has been scored. When you double-score, record three numbers:
- Exact agreement: the share of criterion scores that match exactly.
- Adjacent agreement: the share within one point.
- Pass/fail agreement: the share of samples where both scorers reach the same verdict.
Report each with its count, for example "38 of 40 criterion scores within one point." If pass/fail agreement is low, return to the decision log before scoring more.
Report the rubric separately from the certificate
The certificate and the work sample measure different things. Report them as two numbers: certificates earned, and work samples passing the rubric. Never add them together. A learner with a certificate and no passing sample has finished a course. A learner with a passing sample has shown a habit an employer can see. The Finish Ledger method explains how both are counted.
Where the simple version stops being true
Behavior scoring has limits. A scorer cannot tell whether the "primary source" a learner named is the right one for that trade. A learner checking a billing code against an outdated list would still score a 2 or 3. Two fixes help: require the source's name and date on the page, and invite an employer or experienced colleague to spot-check a few samples each cohort.
The rubric also does not show speed, volume or judgment over time. It is a one-page snapshot of four habits. Treat it as evidence, not as a license to practice. And skills-based hiring is still rare: a Harvard Business School and Burning Glass Institute report on skills-based hiring found that its promised opportunity "has borne out in not even 1 in 700 hires." A scored sample gives a hiring manager something to look at. It does not guarantee they will.
Decision checklist
- Can you point to the task, the source, the data label and the explanation on the page? If not, the sample is incomplete.
- Did two scorers calibrate on anchors first?
- Are any criteria at zero? Then it fails, whatever the total.
- Are certificate and rubric results reported as separate numbers?
To see a facilitator score a sample in about a minute, try the interactive rubric demo in the ed.cafe Staff Room.
