Extracting roof measurements from PDFs with LLMs, and why the model is the smallest part

A roofing distributor cannot price a tile roof without numbers: total area, the area at each pitch, and the lengths of every ridge, hip, rake, eave and valley. Those numbers already exist. They sit inside PDF reports from EagleView, RoofScope, Hover, Roofr and a few others, each with its own layout, units and quirks. Before we built DocScan for the Brava Roof Tile Pro Portal, someone re-typed them into a quote form by hand.

It sounds like a weekend job: send the PDF to a model, ask for JSON, fill in the form. It is not. Below is why that version breaks on real uploads and what we shipped instead. The prompt turned out to be the smallest part.

The naive version and how it breaks

The first prototype was exactly that: one PDF, one prompt, one JSON object. It worked on the demo report and fell over inside a week of real uploads. Each failure was specific, and each one became a test.

Units. Reports label area in "SQ" (a roofing square is 100 sq ft) or in square feet. A model that reads 32.4 SQ as 32.4 sq ft gives you a roof one hundredth of its real size. A dropped decimal does the opposite. Nothing about the JSON looks wrong.

Double counting. RoofScope reports have a "Project Totals" table grouped by slope category (Flat, Low, Standard, Steep) and a separate "Totals (SQ)" table by actual pitch. Read both and you count the roof twice. Hover repeats the same geometry on several pages.

Pitch notation. Some tables write "6" in the header and mean 6/12. Some write 16:12. A model will happily normalise one of them and drop the other.

Empty answers. Reasoning models can spend their whole output budget thinking and return nothing at all. That is not an exception you can catch; it is a successful response with no text in it.

One confidence number. Ask a model how confident it is and it returns 0.92 for the whole document, which tells a reviewer nothing about which field to check.

What shipped

The advanced pipeline is a sequence of small stages. Each one writes its name and a progress number to the job row, so the UI can poll and support can see exactly where a job stopped.

1. Validate the upload before anything else: it must end in .pdf, be non-empty, be under 15 MiB, and start with the %PDF- header.
2. Check the user's monthly quota for the advanced pipeline (20 by default). An exhausted quota drops to the cheaper single-call pipeline and records why.
3. Fingerprint the vendor locally.
4. Extract with a parser where one exists, and with a model where it does not.
5. Run vendor-specific corrections.
6. Render only the pages that matter and run the vision stages on those.
7. Normalise, score every field, validate, geocode the address.

Read the PDF yourself before you pay for tokens

Most vendor reports have a real text layer. PyMuPDF can read it in milliseconds, and the vendor is obvious from a handful of markers in that text. When the local fingerprint matches we know the vendor and the page numbers of the length diagram, the pitch diagram and the measurement table, with confidence 0.99 and no model call. The AI discovery step only runs when that fails.

EagleView goes further. Its reports carry vector geometry, so we parse the measurements locally from text and strokes. Those values are deterministic and are marked as read directly from the report. The model only fills the gaps.

That cuts cost and raises accuracy at the same time. The best LLM call is the one you never make.

Send the model four pages, not forty

Counting intersections (where an eave meets a hip, a hip meets a ridge) is a vision task. We render at most four pages: the length diagram, the pitch diagram and the measurement table, at 2x zoom. The prompt names the legend explicitly and tells the model to ignore the colour swatches, count each junction once, and not to count the same geometry from two different pages. Any count we could compute locally overwrites the model's.

Check the answer against the document itself

A roof report contains redundancy. The per-pitch areas must add up to the total area. That gives you a free validator that needs no second model:

AREA_BY_PITCHES_TOLERANCE_SQFT = 10.0

def repair_area_unit_mismatch(total_area, area_unit, rows, tolerance_sqft=AREA_BY_PITCHES_TOLERANCE_SQFT):
    total = float(total_area)
    pitched_total = sum(float(row["area"]) for row in rows)
    if pitched_total <= 0:
        return total_area, rows
    if area_unit == "squares" and abs(total / 100 - pitched_total) <= tolerance_sqft / 100:
        return total / 100, rows  # total was in sq ft, label said squares
    if area_unit == "sq_ft" and abs(total - pitched_total * 100) <= tolerance_sqft:
        return total_area, [{**row, "area": float(row["area"]) * 100} for row in rows]
    return total_area, rows

If the numbers only agree after dividing by 100, the unit label was wrong. If they do not agree at all, the job is flagged for review.

RoofScope gets a second, narrow model pass that reads only the "Totals (SQ)" table, and that correction is accepted only when its rows sum back to the total within 10 sq ft. Otherwise we keep the earlier data and flag it.

Confidence per field, recomputed on the server

The JSON schema asks for a confidence score and a short reason for every field, and the prompt says so plainly: score each field on its own, use 0 when a value is missing, do not copy the document score everywhere. Then the server ignores half of what the model said.

DIRECT_CONFIDENCE_THRESHOLD = 0.9

def _field_confidence(context, supplied, field, fallback) -> float:
    if field in supplied and not _clamp(supplied[field]):
        return 0.0                      # the model said 0, keep it
    if context.parser_source == ParserSource.LOCAL_PARSER:
        return 1.0                      # deterministic parse
    if field in supplied:
        return _clamp(supplied[field])  # model's per-field score, clamped
    return fallback                     # provider-level confidence

The score is a product of where the value came from and what the model said about it.

The reason text the user sees is generated from the final score, not taken from the model, so the label can never disagree with the number. A value read by the local parser says "Read directly from the EagleView report." A value at or under 0.9 says "Estimated from related measurements in the roof report."

Fallback across vendors, not retries on one

The OpenAI SDK's built-in retries are switched off (max_retries=0). When a stage fails, it is retried once on the other vendor: OpenAI to Gemini, or the reverse, using the same JSON schema converted to Gemini's format. The job records which provider and which model actually answered.

def _call_with_fallback(self, *, primary, fallback):
    try:
        result = primary()
        self.used_ai_provider, self.used_model_name = self.ai_provider, self.model_name
        return result
    except Exception:
        if not self.fallback_model_name or (
            self._fallback_client is None and not self._credentials_available(self.fallback_provider)
        ):
            raise
        result = fallback()
        self.used_ai_provider, self.used_model_name = self.fallback_provider, self.fallback_model_name
        return result

Two lessons from running this on AWS Fargate with Gemini through Vertex AI. First, Vertex rejects a Content object with no explicit role="user", which the public Gemini API accepts, so code that worked locally failed in production. Second, check whether the fallback can authenticate, not whether an API key variable exists. We moved Gemini to workload identity federation, and a check for GEMINI_API_KEY would have silently disabled the fallback the day that key was retired.

Never leave the user with nothing

For a sales tool, a failed job is worse than a wrong one. A wrong value in an editable form costs a reviewer ten seconds. A failed job means the report is re-typed from scratch.

So the pipeline has three levels. The full run. If any stage throws, an editable fallback that keeps whatever evidence exists. If that throws too, a forced result with deliberately ugly defaults (area 0, pitch 0/12) so a reviewer cannot miss them.

try:
    self._discover(context)
    self._extract_provider(context)
    self._render_selected_pages(context)
    self._extract_intersections(context)
    self._classify_hips_ridges(context)
    return self._normalize_validate_complete(context)
except Exception:
    try:
        return self._complete_editable_fallback(context)
    except Exception:
        return self._force_editable_complete(context, reason="forcedEditableFallback")

Every path ends on the same screen: the original report on the left, editable fields on the right, a confidence badge and reason on each field, and a button that sends the file to Brava's team in a support chat if the user wants a human to check it.

What I would tell someone starting this

• Fingerprint and parse locally first. Use the model to fill gaps, and record per value whether a parser or a model produced it.
• Use a strict JSON schema with a per-field confidence and a bounded reason string. Then recompute the confidence on your side.
• Find the redundancy in your documents and use it as a validator. Totals, sums, counts that must match.
• Send the model the fewest pages that answer the question.
• Retry across vendors, not on the same one, and log which model answered.
• Log empty responses with the stop reason and token counts. They will happen.
• Design the failure path first. The user should always land on something they can edit.

Nothing here is clever. It is the checking and the fallbacks that make it safe to hand a model's numbers to someone who is about to quote a real roof.