Overview

Complete Confidence Calculation Flow

High-Level Flow

Confidence Score Components

1. LLM Score (0.0-1.0)

• Source: Confidence value directly from the LLM JSON response
• Extraction: Parsed during JSON parsing from the confidence property
• Meaning: Represents the LLM's own confidence in its extraction

2. Parsing Score (0.0-1.0)

• Calculation: Determined during JSON parsing based on format validation success
• Algorithm:
• Dates: 1.0 if matches strict format (yyyy-MM-dd), 0.5 if parsed leniently
• DateTimes: 1.0 if matches ISO 8601 format, 0.5 if parsed leniently
• Numbers: 1.0 if parses successfully, 0.0 otherwise
• Strings/Booleans: 1.0 if parses successfully, 0.0 otherwise

3. OCR Agreement (0.0-1.0)

• Purpose: Measures how well extracted values match the OCR text
• Calculation: Uses type-specific matching algorithms (detailed below)

4. OCR Confidence (0.0-1.0)

• Purpose: Measures OCR quality for the matched text
• Source: Uses symbol-level confidence from Google Cloud Vision
• Calculation: Type-specific calculation methods (detailed below)

5. Final Score (0.0-1.0)

• Purpose: Weighted combination of all components
• Formula: Varies based on OCR agreement strength (detailed below)

OCR Agreement Calculation by Data Type

String Type

1. Normalize both extracted value and OCR text (lowercase, trim, normalize whitespace)
2. Exact match check: If normalized OCR contains normalized extracted value → Score = 1.0
3. Fuzzy match: Use Fuzz.PartialRatio to find best substring match
4. Threshold check: If fuzzy score ≥ 0.75 → Return fuzzy score, else 0.0

• Extracted: "Invoice Number"
• OCR: "Document: Invoice Number 12345"
• Result: Exact match found → Score = 1.0

Number Type (Long/Double)

1. Parse extracted value (remove currency symbols, commas)
2. Extract all numbers from OCR using regex: [$€£]?\s*[-+]?\d{1,3}(?:,?\d{3})*(?:\.\d+)?
3. Exact match: If any OCR number matches within tolerance (0.01) → Score = 1.0
4. Partial match: Find closest number and calculate relative error:
• relativeError = |extracted - closest| / max(|extracted|, |closest|)
5. Graduated scoring:
• relativeError < 0.01 (1%) → Score = 0.9
• relativeError < 0.05 (5%) → Score = 0.8
• relativeError < 0.1 (10%) → Score = 0.5
• relativeError ≥ 0.1 → Score = 0.0
6. Threshold check: Only return score if ≥ 0.5

• Extracted: 1234.56
• OCR: "1234.65" (decimal digits swapped)
• Relative error: 0.09 / 1234.65 = 0.000073 (0.0073%)
• Result: Score = 0.9 (excellent match)

Date Type

1. Normalize extracted date to yyyy-MM-dd format
2. Extract all dates from OCR using multiple patterns:
• MM/DD/YYYY or DD/MM/YYYY
• YYYY/MM/DD
• Month name formats (e.g., "Jan 15, 2024")
3. Normalize all OCR dates to yyyy-MM-dd
4. Exact match: If normalized OCR date equals normalized extracted date → Score = 1.0
5. Component similarity: Compare year, month, day components:
• Calculate similarity: (matchingComponents / 3)
• If similarity ≥ 0.67 (2 out of 3 components) → Return similarity score
6. No match → Score = 0.0

• Extracted: "2024-01-15"
• OCR: "01/15/2024"
• Normalized OCR: "2024-01-15"
• Result: Exact match → Score = 1.0

DateTime Type

1. Normalize both values (lowercase, trim)
2. Exact match only: If normalized values match → Score = 1.0
3. No partial matching → Score = 0.0 if not exact

String Type

1. Threshold check: If agreement score < 0.75 (and not exact match) → Return 0.0
2. Find OCR word matching the extracted value:
• Strategy 1: Exact word match → Use word confidence directly
• Strategy 2: Word-by-word matching (for multi-word strings)
3. For word-by-word matching:
• Tokenize extracted value into words
• For each word:
• Step 1: Try exact match in OCR words
• Step 2: If no exact match, try fuzzy matching using Fuzz.Ratio (threshold ≥ 0.75)
• Step 3: If multiple fuzzy matches, use context disambiguation (previous/next words)
• Step 4: For fuzzy matches, use symbol-level matching
4. Symbol-level matching (for fuzzy matches):
• Compare characters position-by-position
• Only matching characters contribute to confidence
• Average confidence of matching character symbols
5. Return average confidence across all words

• Extracted: "test"
• OCR words: ["test"] with confidence 0.95
• Result: Exact match → Confidence = 0.95

Number Type (Long/Double)

1. Threshold check: If agreement score < 0.5 → Return 0.0
2. Find OCR word matching the matched text:
• Try exact match of normalized number
• Fallback: Extract numeric components (e.g., "1234.65" → ["1234", "65"])
• Find all matching components in OCR words
3. For exact matches: Use word confidence directly
4. For partial matches: Use symbol-level matching (digits only):
• Filter to digits only (ignore commas, periods, currency symbols)
• Compare digits position-by-position
• Only matching digits contribute to confidence
• Average confidence of matching digit symbols
5. If multiple components found: Combine all components for symbol-level matching

• Extracted: "1234.56"
• OCR: "1,234.65" (split into ["1", "234", "65"])
• Matched: "1234.65" (closest number)
• Components: ["1234", "65"] → Both found in OCR
• Symbol-level: Compare "123456" vs "123465" → Positions 0-3 match → Average confidence of matching digits

Date Type

1. Only calculate for exact matches (agreement score = 1.0)
2. Find OCR word matching the matched date text:
• Strategy 1: Try exact match as single word
• Strategy 2: Word-by-word matching (tokenize by spaces)
3. For word-by-word matching:
• Match each component (year, month, day) individually
• Use context disambiguation if multiple matches exist
• Average confidence across all matched components
4. Return average confidence

• Extracted: "2024-01-15"
• OCR: "2024 01 15" (spaces between components)
• Components: ["2024", "01", "15"]
• Match each component → Average their confidences

DateTime Type

Boolean Type

1. Only calculate for exact matches (agreement score = 1.0)
2. Find exact OCR word match
3. Return word confidence directly

Final Score Calculation

Formula Selection Logic

Rationale

• No OCR: Relies primarily on LLM confidence (90% weight)
• Strong OCR agreement (≥0.8): Balanced weighting across all components
• Weak OCR agreement (<0.8): Heavily weighted toward LLM (65%) since OCR is unreliable

Overall Extraction Confidence

1. For each field in the extraction schema:
• If field has extracted values → Use its FinalScore from confidence breakdown
• If field has no extracted values → Use 0.0
2. Calculate average: overallConfidence = average(FinalScore for all schema fields)

Symbol-Level Matching Algorithm

1. Filter characters: Extract only relevant characters (digits for numbers, all characters for strings)
2. Position-by-position comparison:
• Compare targetChars[i] with ocrChars[i]
• If characters match → Add corresponding symbol confidence to collection
• If characters don't match → Skip (don't add to confidence)
3. Calculate average: confidence = average(matching symbol confidences)

Component Handling for Numbers

1. Extract numeric components: Split on non-digits (e.g., "1234.65" → ["1234", "65"])
2. Find matching components: Search for each component in OCR words
3. Combine components: If multiple components found:
• Concatenate digits in order
• Concatenate symbol confidences in order
• Create virtual OCR word with combined data
4. Symbol-level matching: Use combined components for position-by-position comparison

Thresholds and Configuration

Agreement Thresholds

• String: 0.75 (75% fuzzy match required)
• Number: 0.5 (50% relative error acceptable)
• Date: 0.67 (2 out of 3 components must match)
• DateTime: 0.75 (higher precision required)

Number Relative Error Thresholds

• Excellent: < 0.01 (1% error) → Score = 0.9
• Good: < 0.05 (5% error) → Score = 0.8
• Acceptable: < 0.1 (10% error) → Score = 0.5
• Unacceptable: ≥ 0.1 → Score = 0.0