🧠 Food for thought

Implications, context, and why it matters.

Why top AI models struggle with routine business work

• Company data looks different from the public web text used to train large models. It can have vague column labels, many blank fields, and codes stored as plain text 1.
• That gap can wreck results. In one academic study, an AI model’s F1 score, which balances precision with recall, fell from 0.94 on public data to 0.07 on enterprise data for a data engineering task 1.
• Broad models can be the wrong size for narrow operational jobs. Their extra capacity and wide training can reduce accuracy 2.
• They also lean toward familiar patterns from training. Some defaulted to Structured Query Language (SQL) even after receiving instructions and docs for a company’s proprietary query language 1.

The AI business is moving from giants to specialists

• Firms are building smaller AI agents for specific workflows. Databricks’ KARL uses reinforcement learning for multi-step grounded reasoning with company documents 3.
• Competition now centers on running the full AI lifecycle, including feedback systems for tracking errors and improving models over time 4.
• That supports a hybrid setup. Small efficient models handle routine volume, then send only unclear or hard cases to larger costlier systems 2.
• That makes evaluation and tuning tools more valuable after launch. Databricks added to that stack by buying Quotient AI, which helps large companies run AI agents more reliably 4.