Applied LLM Engineering

This is the hardest course in the catalogue and it is not a first AI course. It assumes Python at working level and the AI & Generative AI course, or equivalent: if you have not written Python that reads a file, calls an API and handles an error, start there. It is also the course whose subject changes fastest: model names, prices and APIs in these notes will be out of date within months, and you are being taught the shape of the problem rather than one vendor's current product. Model APIs cost money per call. The labs are sized to a few dollars, but you need a card and you are responsible for your own usage. Finally: this course does NOT teach you to train a model from scratch. Almost nobody in Pakistan is hired to do that. It teaches you to build things on top of models other people trained, which is what the jobs actually are.

Apply for this course

Enough of the mechanism to reason about failures, without pretending this is a maths course.

• Tokens, context windows and why the bill scales the way it does hands on
  Text becomes tokens; tokens cost money and fill a finite window. Counting them before you send them.
• Why a model states a wrong fact with total confidence
  Next-token prediction has no notion of true. This is the single most important thing to understand before shipping anything to a user.
• Temperature, sampling and reproducibility hands on
  Why the same prompt gives a different answer twice, and when you want that and when you very much do not.
• Calling an API properly hands on
  Keys kept out of the repository, timeouts, retries with backoff, and handling the model being down.
• Structured output hands on
  Getting JSON you can parse instead of prose you have to regex. Schemas, validation, and what to do when it comes back malformed anyway.

The existing AI course teaches prompting for your own use. This is prompting inside software other people depend on.

• System prompts, roles and instruction hierarchy hands on
  Where behaviour is set, and why user text must never be trusted as instruction.
• Few-shot examples and when they stop helping hands on
  Examples cost context. Measuring whether yours earn their tokens.
• Prompt injection hands on
  A user telling your model to ignore its instructions, and a document doing it silently. Defences, and the honest limits of them.
• Versioning prompts like code hands on
  A prompt is program logic. It belongs in git with a change history, not pasted into a text box.

The retrieval half of RAG, taught on its own first because most broken RAG systems are broken here.

• Embeddings: meaning as coordinates hands on
  Why 'car' and 'automobile' land near each other, and why that is not the same as understanding.
• Chunking, and why it decides everything downstream hands on
  Splitting documents. Chunks too big bury the answer; too small lose the context that made it an answer.
• A vector store end to end hands on
  pgvector and one hosted option. Indexing, similarity search, metadata filters.
• Measuring retrieval before blaming the model hands on
  If the right chunk was never retrieved, no prompt will save you. Building a small evaluation set and actually running it.
• Hybrid search hands on
  Keyword and semantic together, because exact terms, a part number, a name, are where pure vector search embarrasses itself.

Answering from your own documents, with the honesty to say when the documents do not contain the answer.

• The RAG loop hands on
  Retrieve, assemble context, generate, cite. Building it once by hand before touching a framework.
• Citations and grounding hands on
  Every claim traceable to a chunk. An answer a user cannot check is worse than no answer.
• Saying 'I do not know' hands on
  Making refusal the default when retrieval is weak. THE hardest and most valuable behaviour to get right.
• Evaluating a RAG system hands on
  A question set with known answers, run on every change. Faithfulness and relevance measured, not felt.
• Keeping the index fresh hands on
  Documents change. Re-indexing, deletions, and what a stale answer costs.
• A real corpus, with the mess left in hands on
  PDFs, scans, tables and Urdu-English mixed text. Clean corpora exist only in tutorials.

Letting a model call your code, and containing what happens when it calls it wrongly.

• Function calling hands on
  Describing a tool so a model can invoke it, and validating every argument it hands back.
• The loop, and stopping it hands on
  Plan, act, observe, repeat. Step limits, budgets and timeouts, because an unbounded loop is a bill.
• Blast radius hands on
  What a tool is allowed to touch. Read-only by default; anything destructive requires a human. Non-negotiable.
• When an agent is the wrong answer
  Most tasks are a script with one model call. Agents are slower, dearer and harder to debug, reach for one last.
• Building one small, useful agent hands on
  Two or three tools, a clear stopping condition, and a log of every step it took.

The part that separates a demo from something a business runs on.

• Cost and latency in production hands on
  Token accounting per request, caching, and choosing a smaller model where it is good enough.
• Logging every call hands on
  Prompt, response, tokens, cost, latency. Without this you cannot debug a complaint or forecast a bill.
• Evaluation as a gate hands on
  The question set runs in CI. A prompt change that regresses answers fails the build, exactly like a test.
• Safety, refusal and the limits of both hands on
  What guardrails do and do not achieve, and why the honest answer to a client is that this reduces risk rather than removing it.
• Data and privacy
  What leaves the country when you call a hosted model, what a client must be told, and when a self-hosted model is the only acceptable answer.

A RAG assistant over a real document set, with an evaluation set that gates it

Build an assistant that answers questions from a real body of documents: a company's policies, a set of manuals, a syllabus. Every answer cites the chunk it came from, and the system refuses when retrieval is weak instead of guessing. You supply a question set with known answers, and a change that lowers the score fails. Delivered with per-request cost and latency logged, and a written note on what it gets wrong. Marked mainly on the refusals and the evaluation set: anyone can produce a demo that answers the easy questions.

A final assessment marked out of 100; the pass mark is 60, with 120 minutes to complete it.

• Model behaviour, tokens and structured output (20 marks)
• Prompt engineering and injection defence (15 marks)
• Embeddings, chunking and vector search (25 marks)
• RAG, grounding and evaluation (25 marks)
• Agents, tool use and blast radius (15 marks)

Students who pass the final assessment and have project work verified by their trainer receive a completion letter. It states attendance, the assessment result and the projects verified, and carries a code any employer can check at the verification page.

No verified NAVTTC alignment. No published lesson plan has been checked against this syllabus module by module, as the CEH mapping was. Map it before quoting it in a bid. This course is taught to BvLogic's own syllabus, published in full above.

Interested in this programme?

Apply now, or try the subject free first: there is no reason to pay before you know you like it.