Scenario design, scoring and after-action review

Start from the behaviour, not the feature

A training objective written as 'understands the shutdown procedure' cannot be observed and therefore cannot be scored. The same objective written as 'isolates the supply before opening the housing, and does not proceed while the indicator is lit' names an action, an order and a condition — all three of which a simulator can detect.

So the first piece of work on any trainer is rewriting the objectives as observable events. It is unglamorous, it is usually done with the subject-matter expert rather than the engineer, and everything downstream — the scenario, the scoring, the report — is derived from it.

Scripted, randomised, and the useful space between

A fully scripted scenario is repeatable and fair, which is what assessment needs, and completely predictable once word gets round a cohort. A fully randomised one cannot be memorised and cannot be compared between candidates.

What usually works is a fixed skeleton with varying detail: the same objective, the same decision points, the same difficulty, but different starting conditions, different fault, different weather. The examiner gets a defensible comparison, the trainee gets a situation they have not seen, and both come from one scenario definition rather than twenty hand-built ones.

The instructor is part of the system

• A console showing what the trainee is seeing and doing, in real time, without being in the headset.
• The ability to inject a fault, a message or a condition mid-run, because the most valuable training moments are the ones an instructor creates by reading the room.
• Pause, resume and reset, so a run can be stopped at the instructive moment rather than played to the end.
• A way to annotate as it happens — a marked timestamp during the run is worth more at debrief than a note written afterwards.

Score what matters, not what is easy to count

Time to completion is trivial to measure and often the least informative number in the report. Correct sequencing, hesitation at a decision point, recovery after an error and whether a check was actually performed are harder to capture and are what the assessment is for.

The failure to watch for is a metric that can be satisfied without the underlying competence. Any score that rewards speed will produce trainees who are fast, so a scoring scheme needs the penalty structure designed alongside it, and it needs reviewing after the first cohort against what the instructors actually observed.

Log events, derive scores later

The durable design is to record a timestamped stream of what happened — every interaction, state change and scenario event — and to compute scores from that stream afterwards, rather than incrementing a counter as the run goes.

That separation pays off repeatedly. The scoring scheme can be revised and the whole history re-scored without re-running anything. A disputed result can be examined rather than argued. And the same stream is what makes replay and after-action review possible at all, so it is worth getting in from the first milestone even if nothing reads it yet.

After-action review is where the learning lands

Most of the value of a simulated run is realised in the debrief, not during the run. What that needs is the ability to replay the session with the timeline visible, jump to the decision points, and show the trainee the moment rather than describe it.

A good review is specific and short: here is where the sequence diverged, here is what the system was showing you at that moment, here is what it would have shown had you done it in the other order. The ability to answer that last question is the strongest argument for simulation over practice on the real equipment.

Reports have to leave the system

Training programmes keep records, and they keep them somewhere else. So the output needs to be an exportable artefact — a per-run report for the trainee's file and structured data for whatever tracks competence across a cohort.

I build the export as a documented format from the start rather than a feature added at the end, because the organisation's record-keeping requirements usually turn out to be the fixed constraint that everything else has to meet.