Education & EdTech
Build assessment AI that survives a disputed score
Automated scoring, adaptive practice and tutoring only earn trust when every mark can be reconstructed and defended. Digiaeon builds education systems with rater agreement, fairness testing and a replayable audit trail designed in from the first sprint.
Education & EdTech
Scoring, tutoring and credentialing that a candidate can dispute — and you can defend
- Now
- 3 applications
- Next
- 3 applications
- Frontier
- 1 application
- Constraints we design around
- 5
Context
Three forces reshaping this sector.
What changed recently enough that the old architecture is now the wrong one.
Language models became grading infrastructure, quietly
Through 2024 and 2025, LLM raters moved from demos into the path between a candidate and a released score. The interesting question stopped being whether a model can grade an essay. It is now whether the score holds up when a candidate appeals, a rater disagrees, and the model version that produced it was silently deprecated three months ago.
Credentials are turning machine-verifiable
Open Badges 3.0 and W3C Verifiable Credentials 2.0 put a cryptographically signed claim in the learner’s hands instead of a PDF in an inbox. In India, the Academic Bank of Credits, DigiLocker and the National Academic Depository push the same direction under NEP 2020. Employers are starting to verify rather than trust — institutions that cannot issue in these formats become the slow path.
Integrity shifted from the exam hall to the model
Items leak to Telegram and Discord within hours of a live window. Submissions are model-written. AI-detection classifiers remain unreliable and their false positives fall hardest on non-native writers — several universities have retired them after exactly that. Integrity work is moving from a detector verdict towards accumulated process evidence.
Applications
What we would build here, in order.
Grouped by how close each one is to production today — not by how impressive it sounds in a board pack.
Build it this quarter
Current models, current data. The engineering is understood and the risk is schedule risk, not research risk.
Rubric-aligned scoring for written responses
Decompose the rubric into observable traits and score each trait separately against anchor exemplars, rather than asking one model for a holistic band. Calibrate on a double-marked set that spans the full score range, report quadratic weighted kappa per trait against human raters, and watch for length bias — a model that rewards word count will look excellent on aggregate and fail on short, strong answers.
Speaking and pronunciation scoring
Transcription with Whisper-class ASR, forced alignment with the Montreal Forced Aligner, then goodness-of-pronunciation features at phone level plus fluency features drawn from pause distribution and articulation rate. The failure mode is structural: word error rate is higher for some first-language groups, so ASR quality becomes a fairness problem that propagates straight into the score.
Tutoring assistants that teach rather than answer
A tutor that solves the problem is a homework machine. The build is a hint ladder held by an explicit state machine — elicit, hint, worked analogue, then full solution only on an earned path — grounded in the institution’s own syllabus edition so a physics answer does not arrive from a different board’s curriculum. Answer-leakage evaluations run in CI alongside correctness.
One thing has to change first
Buildable, but gated on something outside the model — labels, a consented holdout, an integration, or a process the business has to agree to change.
Item generation with psychometric screening
Generation is the cheap half. The screen is the product: near-duplicate detection against the live bank by embedding similarity, key-plausibility and distractor-function checks, readability bounds, then a field test on a small sample. Items with a point-biserial below roughly 0.2, or a flat discrimination parameter under a 2PL fit, get dropped before they ever reach a scored session.
Adaptive practice and item selection
An IRT-calibrated bank with maximum Fisher information selection shortens a test dramatically — and burns the bank, because the most informative items get served constantly. Sympson-Hetter exposure control and a shadow-test formulation keep content balance and exposure inside their constraints. For practice rather than high-stakes testing, a knowledge-tracing model over a scheduler such as FSRS is usually the better fit.
Learner analytics and early warning
A Caliper Analytics or xAPI event spine, a warehouse that models a learner’s week honestly, and a risk model judged on precision in the top decile rather than on AUC. The unglamorous part decides whether it works: a flag with no counsellor, no capacity and no intervention script is a dashboard, not an outcome.
Scope it as research, with a gate
Credible and worth a prototype. We would run it behind an evaluation gate and say plainly if it does not clear the bar.
Portable credentials an employer can verify without calling you
Issue a Comprehensive Learner Record as signed Verifiable Credentials, with revocation that actually functions and a verification endpoint that does not leak the holder’s history to the verifier. Key custody is the hard part — a compromised issuing key means revoking and re-issuing everything signed with it, and a lost one means no rotation and no revocation at all, so hardware-backed keys and a rotation plan come before the first badge is minted.
Constraints
What makes this sector different.
The part of the build that does not transfer from anywhere else. Every one of these has to be answered in the architecture, not in the project plan.
A disputed score is the real specification
Every released score must be reconstructable months later: the model version, prompt, rubric revision, retrieved exemplars and the raw response, all pinned and stored as an immutable scoring artefact. Calling a hosted endpoint whose weights move underneath you, and keeping only the number, is disqualifying. There must also be a documented path to a human re-mark that can overturn the machine.
Fairness is measured, not asserted
Differential item functioning by Mantel-Haenszel, standardised mean differences in scores across groups, kappa computed per subgroup rather than pooled, ASR word error rate broken out by first language. The operational bind is that the demographic data needed to measure this is data you may not be permitted to retain — which usually means a consented, sampled fairness panel held separately from operational records.
Item security and exposure control
The calibrated bank is the most valuable asset in the business and it degrades every time an item is seen. Exposure caps, staged release windows, and leak detection from response-pattern anomalies — person-fit indices catch a candidate who answers hard items correctly and easy ones wrong. Live items must not cross into a third-party inference endpoint without a data processing agreement and zero retention in writing.
Minors, consent and data residency
India’s DPDP Act 2023 requires verifiable parental consent for users under eighteen and bars behavioural tracking and targeted advertising at children; COPPA, FERPA and GDPR Article 8 draw comparable lines elsewhere. Practically: no training on learner content by default, regional inference, short retention on transcripts and recordings, and a deletion path that genuinely reaches the vector store.
Accessibility is a legal requirement, not a backlog item
WCAG 2.2 AA is the floor, and assessment makes it harder than a marketing site — items must be screen-reader navigable without disclosing the answer, mathematics needs MathML rather than an image, and extended-time accommodations interact with every timer and auto-save in the stack. The European Accessibility Act and Section 508 procurement make this a gating item in a tender, not a polish pass.
Every one of these is discoverable in week one. Most programmes discover them in month three, after the architecture is already load-bearing.
Outcomes
What we design toward.
Targets and typical ranges for this class of system. Each one carries its basis, because a number without one is decoration.
- Quadratic weighted kappa against human raters, per trait
- 0.70–0.80Quadratic weighted kappa against human raters, per traitTypical range for trait-level automated essay scoring on public corpora such as the ASAP set; we hold a build to it as a design target, not a delivered client result.
- Responses deferred to a human rater
- 10–25%Responses deferred to a human raterDesign target for a conformal deferral threshold — the band is set by the agreement the exam owner requires, not by inference cost.
- Submission to released score, batch pipeline
- Under 24 hoursSubmission to released score, batch pipelineDesign target at practice-test volume; high-stakes windows with mandatory double marking are slower by construction.
- Any released score reproducible from pinned versions
- Fully replayableAny released score reproducible from pinned versionsArchitectural property of the design, enforced by a replay test in CI — not a measured outcome from a client engagement.
These are design targets and published benchmark ranges, not delivered client results. We do not publish results attributed to clients.
Solutions
What we bring to this sector.
The practices that do most of the work here. Each one is the same discipline applied against the constraints above.
Blueprints
Reference architectures behind this work.
How the systems are actually assembled, with the engineering decisions and their costs stated. Reference designs, not case studies.
Grounded Answer Engine
A question-answering system over a company’s own documents that cites every claim to a source span, respects the permissions of the person asking, and refuses when the evidence is not there.
Read the blueprintDocument Intelligence Pipeline
A high-volume extraction pipeline that turns invoices, claims, KYC packs and trade documents into validated structured records, attaches a calibrated confidence to every field, and sends only the fields that genuinely need a person into an exception queue sized to the real error rate.
Read the blueprintQuestions
The awkward questions.
The ones a technology committee actually asks, answered the way we would answer them in the room.
Next step
Bring us the constraint you keep working around.
A 45-minute working session. We'll tell you what we'd build, what we'd not build, and roughly what it costs. No deck.