A grounded assistant with an eval gate and a cost sheet

The situation

Build the fee-desk assistant properly: scripted API calls with the system prompt as the builder's lever, a cost sheet with real prices, structured extraction that returns null instead of inventing, a red-team pass with defences, a miniature RAG over ten chunks of academy knowledge, and a 20-case eval runner that scores every change before it ships. The final paragraph reads like a client deliverable because it is one.

What you'll be able to show

• Control behaviour through the system prompt and measure what it costs
• Extract structured data with validation and honest nulls
• Attack your own assistant and defend it, reporting what still gets through
• Ground answers in retrieved context and gate changes with an eval pass rate

What you need

• An LLM provider key on a free tier, Python, a .env file
• The provider's pricing page and embedding endpoint
• Ten chunks of invented-but-realistic academy knowledge (fees, timings, policies)

Tasks

1. 1The builder's lever
   Store the key in .env (verify .gitignore). Make three calls with a narrow system prompt and three user messages. Change only the system prompt — tone, language, refusals — and observe the change.
   A correct result: Six outputs showing behaviour moved by the system prompt alone.
2. 2The cost sheet
   With real current prices: count the system prompt's tokens (usage field or a tokenizer), an average question, a three-sentence answer. Compute cost per conversation and per month at 100/day and 2,000/day. Halve the prompt without losing correctness and recompute. Write the one-line conclusion.
   A correct result: A sheet with two monthly figures before and after, and the conclusion about prompt length × traffic.
3. 3Extraction that admits ignorance
   Add an /extract command that asks for a JSON lead record ({name, course, city, phone}) with the schema in the prompt, try/except parsing, one retry with the error, and validation. Test with a cooperative conversation and a messy one with fields missing.
   A correct result: Missing fields come back as null, never invented; the retry path is exercised once.
4. 4Red team, then defend
   Attempt a direct override, system-prompt extraction, a fake discount authorisation, and an indirect attack pasted inside an enquiry. Record what worked. Apply two defences and re-attack. Write the three-line report: what still gets through and its blast radius.
   A correct result: Four attacks before and after, and the honest three-line report.
5. 5Miniature RAG with an eval
   Embed ten knowledge chunks; similarity search in plain Python; a grounded prompt that answers only from context. Build a 10-question eval set with expected sources, log retrieved chunks per question, find one failure, diagnose retrieval vs generation from the log, fix the right layer, re-run.
   A correct result: An eval run before and after the fix, with the diagnosis written down.
6. 6The gate and the paragraph
   Extend the eval to 20 cases including 3 injections (expected: refusal) and 3 out-of-knowledge questions (expected: I do not know). Write the runner that prints a pass rate. Run it before and after one deliberate prompt change. Then write the three hiring-signal sentences for your assistant: eval pass rate, cost at 500 questions/day, attack behaviour.
   A correct result: Two pass rates around a change, and the three-sentence paragraph.

What to hand in

The repo (no key in it), the cost sheet, the extraction test outputs, the red-team report, the eval runs before and after, and the three-sentence paragraph.

How it is marked