TRACER: Learn-to-Defer for LLM Classification with Formal Teacher-Agreement Guarantees

I'm releasing TRACER (Trace-Based Adaptive Cost-Efficient Routing), a library for learning cost-efficient routing policies from LLM traces.

The setup: you have an LLM handling classification tasks. You want to replace a fraction of calls with a cheap local surrogate, with a formal guarantee that the surrogate agrees with the LLM at least X% of the time on handled traffic.

Technical core:

• Three pipeline families: Global (accept-all), L2D (surrogate + conformal acceptor gate), RSB (Residual Surrogate Boosting: two-stage cascade)
• Acceptor gate predicts surrogate-teacher agreement; calibrated on held-out split
• Calibration guarantee: coverage maximized subject to TA >= target on calibration set
• Model zoo: logreg, MLP (1h/2h), DT, RF, ExtraTrees, GBT, XGBoost (optional)
• Qualitative audit: slice summaries, contrastive boundary pairs, temporal deltas

Results on Banking77 (77-class intent, BGE-M3 embeddings):

• 91.4% coverage at 92% teacher agreement target
• 96.4% end-to-end macro-F1
• L2D selected; method automatically determined by Pareto frontier

Paper in progress. Feedback welcome.