Context Payload Optimization for ICL-Based Tabular Foundation Models

In-context learning (ICL) for tabular foundation models shifts optimization from training time to inference time, introducing accuracy-latency trade-offs around context payload size. The post covers the 'iron triangle' of response quality, cost, and latency in ICL workflows, then outlines payload optimization strategies across two dimensions: method (task-agnostic vs. task-aware techniques like KNN, clustering, RAG) and moment (offline precomputation vs. on-the-fly, client-side vs. service-side). A hands-on Python demo using the Solar Flare dataset and SAP-RPT-1 model demonstrates KNN-based context prefiltering, comparing inference time and accuracy with and without the optimization.