Build a small evaluation set | Introduction to Generative AI
Understand how generative AI works, write source-bounded prompts, validate outputs, evaluate grounded answers and design a responsible career-learning assistant.
Prerequisites: No prior AI experience required. Basic familiarity with reading short code examples is helpful; no paid API account is needed.
Build a small evaluation set
Evaluation turns an impression such as "the answer seems good" into a repeatable check. Begin with representative tasks and write down the expected properties of an acceptable answer. For a resume summary, useful checks include factual support, coverage of supplied experience, readable wording and absence of unnecessary personal information. Keep the test inputs small enough that a reviewer can understand the source evidence.
A rubric can combine automatic checks with human judgment. Automatic checks can verify that a response contains required keys, stays below a length limit or cites an allowed document ID. Human review is needed for subtler questions such as whether wording misrepresents experience. A model-based judge can help triage outputs, but it should not be treated as an infallible reference. Calibrate it against reviewed examples before using it for consequential quality gates.
Compare a candidate change against a baseline on the same evaluation set. Include ordinary examples, missing information, contradictory information and adversarial text that attempts to change the task. Record the model configuration and prompt version. If a new version improves fluency but introduces unsupported qualifications, it may be a worse release for a career product. The exercise below deliberately scores only an observable rule; it is not a complete quality metric.
Worked example
cases = [
{"expected_id": "certificate", "returned_ids": ["certificate"]},
{"expected_id": "enrollment", "returned_ids": ["certificate"]}
]
hits = sum(c["expected_id"] in c["returned_ids"] for c in cases)
print(hits / len(cases)) # 0.5 on this tiny teaching set
Try it yourself
Design a five-case evaluation set for a source-bounded career assistant. Define one automatic rule and one human-review criterion. Explain why five cases are useful for learning but insufficient to establish production reliability.
Check your understanding: explain why the example works, state one limitation, and record how you would test it.
["Check","What it establishes","What it does not establish"]
["JSON shape","Expected keys and types","Truth of the content"]
["Source support","Evidence for a claim","Complete coverage of every user need"]
["Human review","Contextual judgment on reviewed samples","A guarantee about every future response"]
1. Understand generative AI
2. Give clear instructions
3. Ground and evaluate answers
4. Build a responsible learning prototype