Open Weight Models · Lesson 4
Prompt clearly and evaluate answers
Ground answers in supplied notes and test accuracy, format, and uncertainty with repeatable prompts.
What you will learn
- Build a prompt with a specific task, supplied context, constraints, and output format.
- Ground a response in short synthetic notes and require unknowns to remain unknown.
- Evaluate factual support separately from fluency and presentation.
- Compare prompt revisions using the same cases and a recorded rubric.
Give the model a job it can be checked against
A vague request such as Help with this incident leaves the model to guess the audience, facts, and expected result. For this lesson, use four parts: task, context, constraints, and format. These are an authoring pattern for our exercise, not special keywords that guarantee obedience.
Keep the request short enough to inspect. Supplying relevant notes reduces the need to rely on the model's background knowledge, but it does not turn generated text into verified evidence. Treat each answer as a draft that must satisfy your checks.
| Part | Our handover task |
|---|---|
| Task | Write a factual incident handover |
| Context | Use only the synthetic NOTES below |
| Constraints | Do not invent a cause, owner, or recovery action |
| Format | Return four labelled lines with note references |
Run a prompt with a known evidence base
Start a fresh chat with ollama run gemma3:1b. Paste the entire block below into the model's chat prompt, including the opening and closing triple quotes. They let the CLI collect a multiline prompt. Do not paste this block directly into your operating-system shell.
All names and events here are invented. Note identifiers let you check support quickly: a reference is useful only when the cited note actually contains the claim. The model has not investigated the portal or verified a cause.
Paste inside the Ollama chat · text
"""
TASK: Write a factual incident handover using only NOTES.
CONSTRAINTS: Do not invent causes, owners, or recovery actions. Say "Unknown from notes" for missing information. Keep each line short.
FORMAT: Return exactly four labelled lines: Service, Status, Evidence, Unknowns. Add note IDs to supported claims.
NOTES
[N1] The affected service is the Harbour training portal.
[N2] Failed logins were reported at 09:10.
[N3] Access was restored at 09:20.
[N4] No root cause or incident owner has been confirmed.
"""Expected meaning, not guaranteed wording
A supported answer names the Harbour training portal, reports access restored at 09:20, includes the failed-login observation at 09:10, and leaves cause and owner unknown. It must not claim that a password reset, patch, or restart fixed the incident.
Check the answer before judging the prompt
Read the answer against the notes, not against how confident it sounds. Check each claim, time, and reference. Then check the requested format separately. A beautifully written answer with an invented cause fails the factual test; a factual answer in the wrong layout fails the format test.
Use the rubric below for every run. Mark each item Pass or Fail and copy a short piece of output explaining any failure. Do not ask the same model to be the only judge of its own work.
| Check | Pass condition |
|---|---|
| Service | Names Harbour training portal without adding affected services |
| Times and status | Preserves 09:10 failed logins and 09:20 restored access |
| Unknowns | Does not invent the root cause, owner, or recovery action |
| References | Every supplied note ID supports its associated claim |
| Format | Uses exactly the four requested labelled lines |
Change one thing and test again
Save the prompt as version A. Run it twice in separate fresh chats, using /bye and then starting the same model again. Record the model ID, Ollama version, complete input, output, and the five checks. Keeping the loaded model in memory is compatible with starting a fresh conversation.
For version B, add one short example of the required format using a different fictional service. Keep the Harbour notes and rubric unchanged, then repeat both runs. Compare all results rather than selecting only the best-looking answer. An improvement on this small exercise is evidence for this exercise, not proof of general reliability.
For this comparison, leave generation settings unchanged. A lower temperature can reduce sampling variation, but setting temperature to zero is not a guarantee of factual correctness or identical results across software, hardware, or model changes. Test the behaviour you need instead of treating a setting as a certificate.
Test the limits of the instruction
Create a missing-information case by removing N3. The model should no longer report a restoration time. This is a useful test because a fluent answer might carry over a familiar completion even when the supporting fact has gone.
Create a conflict case by adding a new note that says access was still unavailable at 09:20. The expected response should identify the conflict rather than silently choosing one status. Adjust the task to request conflicting observations explicitly and write down the expected outcome before running it.
Supplying notes is a grounding technique, not a security boundary. Text from an untrusted document might itself contain instructions. Later applications need their own input handling and output validation; a sentence saying use only these notes is not enough to make every use safe or correct.
A failed answer can complete the exercise
Success here means you run a controlled check, identify errors honestly, and explain a useful revision. You do not need to make a small model produce a perfect answer every time.
Create a small evaluation record
Run version A twice in fresh chats and record each complete output.
Apply all five rubric checks to both answers. Identify any unsupported detail by comparing it with the exact note text.
Create version B with one different fictional formatting example. Repeat two fresh-chat runs with the same Harbour notes and checks.
Remove N3 and test once more. Your expected result is that restoration status or time remains unknown rather than being reconstructed from the earlier case.
Write a short conclusion identifying a specific improvement, a remaining failure, or a result that did not change. Keep the evidence with your conclusion.
You have completed this task when…
- Your record contains both prompt versions, four baseline-case outputs, and one missing-information output.
- Each baseline output has five explicit Pass/Fail checks; any failure includes the relevant claim or format problem.
- The missing-information run is judged against the edited notes, and your conclusion does not claim that fluency, a single success, or a generation setting proves reliability.
Official documentation
Use these references for platform requirements, current options, and further detail.