CoKeeper: teaching a classifier to say "I'm not sure"

Automating General Ledger classification for accountants using specialized ML pipelines.

Impact: Reduces manual categorization time by ~16×, turning 8 hours of manual entry into 30 minutes of high-level review. 92.5% accuracy in production.

The problem

Accountants spend significant billable hours manually sorting hundreds of bank transactions. The patterns are repetitive, but the data is “dirty”: merchant strings are inconsistent, containing transaction IDs, locations, and cryptic abbreviations that standard rule-based systems fail to catch.

The solution: confidence-tiered automation

CoKeeper doesn’t just categorize; it assesses its own certainty. By outputting predictions in three confidence tiers, it creates a clear, trustworthy workflow for the user.

Success metric: ~59% of all transactions land in the green tier, ready for instant import.

Pipeline

1. Bank CSV ingest: raw exports, any institution’s format.
2. Merchant string cleaner: strips transaction IDs, locations and cryptic abbreviations.
3. 5-level vendor cascade: exact, alias, fuzzy, pattern, then ML fallback.
4. Dual-model classifier: CatBoost and scikit-learn.
5. Confidence tiers → QuickBooks export: green auto-imports, yellow gets a spot-check, red goes to review.

System architecture

The system is built as a modular, serverless pipeline deployed on GCP Cloud Run.

1. Vendor intelligence (5-level cascade)

A custom feature engineering system that resolves messy bank descriptions into clean vendor identities:

• Level 0: Normalization (stripping prefixes, locations, and phone numbers).
• Level 1-2: Exact and fuzzy matching (SequenceMatcher ≥ 0.75).
• Level 3: Universal merchant database and keyword inference.

2. Dual-model routing

Instead of a single classifier, CoKeeper routes data through two specialized CatBoost models:

• Matched vendor model: used when vendor intelligence recognizes the merchant (92.1% accuracy).
• Unmatched vendor model: relies on raw text features for unknown merchants (73.7% accuracy).

Logic highlight: the confidence engine

This snippet shows how the system converts raw model probabilities into actionable business tiers.

def get_confidence_tier(probabilities: list[float]) -> dict:
    """
    Categorizes the prediction based on the model's top-1 certainty.
    Ensures 'Green' tier maintains near-perfect precision for auto-posting.
    """
    top_prob = max(probabilities)

    if top_prob >= 0.80:
        return {"tier": "GREEN", "action": "AUTO_POST", "color": "#22c55e"}
    elif top_prob >= 0.50:
        return {"tier": "YELLOW", "action": "SPOT_CHECK", "color": "#eab308"}
    else:
        return {"tier": "RED", "action": "MANUAL_REVIEW", "color": "#ef4444"}