Building a Multimodal Agentic Pipeline for Educational Content Ingestion

Educational publishers have decades of valuable content locked inside worksheets, textbooks, and scanned assessment papers. Converting it into structured, interactive learning material is harder than it looks: a single document can mix questions in multiple languages, inconsistent mathematical notation, diagrams that are essential to the question, multi-column layouts, and imperfect OCR. Doing this manually is slow and error-prone, and plain OCR only solves the easy part — it extracts text without understanding question boundaries, formulas, answer choices, or educational intent.

The motivation is concrete. A structured question set ingested into Wayground works two ways: it can be printed as a proper classroom worksheet, or played live as a formative assessment. Publishers wanted both from the same source material, with every question tagged by Depth of Knowledge (DoK) so a teacher can control the rigor of what they hand out. That dual use is why structure is non-negotiable — a printed worksheet forgives loose formatting, but an interactive assessment needs clean question boundaries, machine-readable answers, and explicit metadata.

We built a publisher content ingestion system that handles two related jobs:

1. Extract existing questions from worksheets and assessment documents.
2. Generate new questions from textbooks and other reference material.

Both share the same foundation but need very different reasoning and quality control. This post walks through how each pipeline works and what we learned building them.

Why OCR Isn't Enough

A PDF is a visual representation of a document, not a semantic one. Consider a worksheet containing the expression:

(x^2 - 4) / (x - 2)

A basic OCR system might produce:

x2 - 4
------
x - 2

or:

x^2 - 4 / x - 2

These outputs are textually similar but mathematically different. The same ambiguity appears everywhere: chemistry superscripts and reaction arrows carry meaning, physics diagrams define the geometry needed to answer a question, and biology labels can be part of the question itself. Language adds another dimension — a Gujarati or Spanish worksheet may mix English scientific terminology, mathematical notation, and locally formatted instructions on one page.

A reliable ingestion system therefore needs more than OCR. It needs a sequence of specialized reasoning steps.

System Overview

The platform is a FastAPI application with LangGraph orchestrating the two workflows. LangGraph carries a typed state object through each pipeline — raw text, detected subjects, extracted questions, quality scores — which makes runs easy to inspect, pause, and retry, and keeps agents from relying on information hidden inside prompts.

The supporting services:

• Mistral Document AI for OCR and document parsing
• GPT-4o for structured extraction, formula conversion, and image understanding
• Portkey for model routing across providers
• SymPy for deterministic mathematical verification
• PostgreSQL for document, question, image, and processing metadata
• A publishing layer that converts the internal representation into the platform schema

Part 1: Extracting Existing Questions

Extraction is not transcription. For each question, the system must reconstruct the author's intended structure: the stem, instructions, answer choices, correct answers, associated images, formulas, subject, language, and question type. We implemented this as an eight-stage sequential workflow.

1. OCR extraction

Pulls raw text, page structure, images, and language signals — and deliberately preserves as much source information as possible. Cleaning too early destroys useful signals like column structure and the relationship between an image and nearby text.

2. Structure analysis

Determines what kind of document this is — question numbering, section headings, multi-column layouts, repeating answer-choice formats — and picks a chunking strategy. Arbitrary token-based splitting can separate a question from its answer choices or detach a diagram from the text that references it. The useful question is not how many tokens fit in a context window, but what the smallest self-contained educational unit is.

3. Subject classification

Detects the subjects present, which drives later processing: a chemistry document needs different normalization rules than a biology worksheet. It also flags whether the document contains formulas, so text-only documents skip formula processing entirely.

4. Formula cleanup

Converts inconsistent OCR output into standardized LaTeX. This stage must be conservative: its job is to repair representation, never to change mathematical meaning. Where confidence is low, it preserves the original expression and flags it for review instead of inventing a correction.

5. Text normalization

Repairs broken whitespace, incorrect line breaks, split answer choices, repeated headers and footers, and encoding artifacts — while retaining references to the original source.

6. Structured question extraction

Converts normalized content into a validated schema:

{
  "question_text": "What is the value of x?",
  "question_type": "multiple_choice",
  "options": [
    { "id": "a", "text": "2" },
    { "id": "b", "text": "4" },
    { "id": "c", "text": "6" },
    { "id": "d", "text": "8" }
  ],
  "correct_answer": "b",
  "subject": "mathematics",
  "language": "en",
  "source_page": 4,
  "confidence": 0.91
}

Structured output here is critical. Free-form model responses are difficult to validate and dangerous to publish directly.

7. Language specialization

Corrects spelling and language-specific OCR errors without translating or changing meaning — important for documents that mix a regional language with English terminology or non-Latin scripts. The system currently supports Gujarati and Spanish alongside English.

8. Final validation

Checks that required fields are present, options are structurally valid, referenced images exist, LaTeX parses, questions are not empty or duplicated, and confidence meets the publishing threshold. The output is a set of structured questions with a quality report.

Why Multiple Agents Instead of One Big Prompt?

Sending the whole document to one large prompt is simpler — and nearly impossible to debug. When the output is wrong, you cannot tell whether the failure came from OCR, layout understanding, question boundaries, formula parsing, or schema generation.

Specialized stages give us three things: observability (every stage produces inspectable intermediate output), targeted retries (a failed formula conversion does not repeat OCR for the whole document), and independent improvement (the formula agent or validator can be upgraded without redesigning the pipeline).

That said, not every stage needs an LLM. Deterministic code is preferable whenever the task can be solved reliably with rules, parsers, or mathematical libraries.

Part 2: Generating New Questions

Generation has a different objective: use textbook content to create new questions that test understanding at specific cognitive levels. The workflow contains nine agents and four subject-specific generation modules, with a quality-gated retry loop — questions must score at least 75 against a rubric, with up to three generation attempts.

Content analysis

Beyond identifying topics, the analyzer extracts core concepts, relationships between them, important values, and — most valuably — common student misconceptions. A high-quality distractor should represent a believable reasoning mistake, not a random incorrect value. When testing order of operations, a plausible wrong answer comes from evaluating addition before multiplication, because it reveals what the learner misunderstood.

Cognitive-level planning

Before generating anything, the system plans a distribution of questions across cognitive levels. At publishing time, each question maps to the Depth of Knowledge tag Wayground uses, so a worksheet can be assembled against an explicit rigor target instead of an implicit difficulty:

This planning step exists because without it, language models produce many questions that sound different but test the same shallow skill.

Subject-specific generation

A generic prompt is rarely sufficient across subjects, because correctness means something different in each:

Deterministic Answer Verification

Language models are useful generators, but they should not be treated as calculators. For mathematical questions, the generated answer is independently solved with SymPy:

The generator proposes; the verifier solves the problem independently. If they disagree, the question does not proceed to publishing. The same principle extends to unit validation, chemical equation balancing, and option uniqueness checks.

Distractors from Known Mistakes

This is my favorite part of the system. Instead of asking a language model to invent incorrect options, the system generates distractors by solving modified versions of the original problem with SymPy — where each modification encodes a specific mistake a student could plausibly make.

Consider:

f(x) = (3x^2 + 1)^4

The correct derivative requires the chain rule:

f'(x) = 4(3x^2 + 1)^3 * 6x = 24x(3x^2 + 1)^3

The distractor generator applies realistic mistake transformations — forgetting the chain rule, differentiating the inner function incorrectly, dropping the outer coefficient — and SymPy evaluates each altered procedure:

A. 24x(3x^2 + 1)^3    correct chain rule
B. 4(3x^2 + 1)^3      forgot the inner derivative
C. 12x(3x^2 + 1)^3    inner derivative taken as 3x instead of 6x
D. 6x(3x^2 + 1)^3     dropped the outer coefficient

The validator confirms each distractor is mathematically distinct from the correct answer and from the others, and does not simplify into the correct expression. The result is distractors that are realistic, deterministic, and traceable: the system can explain not only that an option is wrong, but which mistake produces it.

Quality-Gated Generation

Every generated question is scored against a rubric covering correctness, grounding in the source material, clarity, cognitive-level alignment, distractor quality, and language quality. Questions below the threshold are retried — but instead of a bare "generate again," the validator returns structured reasons:

{
  "score": 68,
  "passed": false,
  "issues": [
    {
      "category": "cognitive_alignment",
      "message": "The question tests direct recall instead of the requested level 3 reasoning."
    },
    {
      "category": "distractor_quality",
      "message": "Two answer choices are obviously unrelated to the concept."
    }
  ]
}

This feedback becomes input to the next generation attempt.

Extraction and Generation Are Different Products

Although they share infrastructure, the two workflows should not be treated as one generic operation:

This distinction shapes prompts, validation, metrics, and failure handling throughout the system.

What We Learned

1. OCR is only the first step. Layout analysis, formula reconstruction, image association, and semantic extraction are separate problems.
2. Preserve provenance. Without page numbers and source spans, reviewing an incorrect question becomes a search problem.
3. Use models for interpretation, deterministic code for verification. Schema validation, math solving, deduplication, and retry limits do not need an LLM.
4. Validate intermediate representations. Each stage needs its own contract — waiting until final output makes failures impossible to isolate.
5. Quality requires a feedback loop. Plan, generate, verify, score, and retry beats one-shot generation.
6. Correctness is subject-specific. Symbolic proof in mathematics, unit checks in physics, source grounding in biology.

What's Next

The current architecture is a solid foundation. The improvements I care most about:

• Page-level provenance with bounding boxes, so a reviewer sees the exact source region beside each structured question.
• Hard gates instead of one aggregate score — a question should never pass because strong writing compensates for a wrong answer.
• Confidence-based review routing: auto-approve the easy cases, send formula-heavy or low-confidence output to subject experts.
• Semantic deduplication across editions, teacher copies, and answer keys, where questions differ only in formatting or values.
• Cached document understanding and resumable execution, so a retry resumes from the failed stage instead of re-running OCR.
• An evaluation dataset with human-reviewed ground truth — the metric that matters is cost per accepted question, not cost per token.
• Treating uploaded documents as untrusted input, with strict structured outputs so embedded text can never act as instructions.

Conclusion

Turning educational PDFs into reliable questions is not a single-model problem — it is a document-understanding, reasoning, verification, and orchestration problem. The most important architectural decision was treating extraction and generation as pipelines of specialized stages rather than one large prompt: models interpret, deterministic tools verify, and quality gates decide what reaches publishing. The payoff is publisher content that lands in Wayground ready for both lives it needs to lead — printed as a worksheet, or played as a formative assessment — with a DoK tag on every question. And with provenance, evaluation, and reviewer feedback in place, the same system keeps improving with every document it processes.