How to avoid vendor dependency in an unpredictable AI market

This article summarizes an episode of Sourcery with Molly O’Shea’s video series featuring Scott Wu, CEO of Cognition.

Photo credit: Cognition AI

The next major challenge for AI tools is knowing when a job is done.

Scott Wu, CEO and co-founder of Cognition, argues that AI progress relies on tasks having a clear test for success.

This measurable approach helps companies assign tasks effectively and adapt their workflows around AI.

Measurable tasks highlight where AI excels

Software development is currently leading AI adoption because code gives instant feedback: a program either runs or it doesn’t.

That certainty gives AI systems the exact right or wrong answers they need to improve, offering a level of certainty that standard office work cannot provide.

To get useful results from AI, companies should follow a targeted approach:

• Start small: Select tasks where success can be tested quickly.
• Define parameters: Convert vague assignments into tasks with clear pass or fail criteria.
• Enable self-correction: Provide testing environments so AI tools can find and fix their own mistakes.
• Focus on outcomes: Measure success based solely on the final completed work.
• Reinvest resources: Use saved time to build better software capabilities.

Wu notes that standard tests, or benchmarks, give AI systems a clear path to follow by defining exactly what success looks like. However, teams must avoid “teaching to the test” because high benchmark scores do not always guarantee real-world success.

Staying flexible reduces vendor lock-in

Measurable results alone cannot protect a company from the rapidly shifting AI market.

With the top-ranked models changing constantly, locking into a single tech provider is a financial risk, as retraining engineers on new systems is prohibitively expensive.

To remain independent from any single AI provider, companies should:

• Use flexible tools that can connect to the best available AI model at any given time.
• Select platforms that maintain a consistent interface regardless of the underlying algorithm.
• Build vendor partnerships focused on daily work outcomes rather than specific tech brands.
• Track exactly which AI model completed each specific task for accountability.

Workflow integration drives real business value

Workflow continuity drove the Windsurf merger