We have talked to over 100+ customers on AI agents, current implementations and expectations.

Our findings show that test-time compute and "reasoning" is less valuable than current set of models used in production:

1. Enterprise customers still are concerned about data privacy.

Even though many AI labs point to their policies that customer data is not used for training, this promise is not taken at face value.

Hence first solution that big companies come up with is using open source models to keep their data inhouse or on prem.

OpenAI O1 like models (except Deepseek) do not have an open source version and even if they did, the compute necessary to be able to run these models inhouse greatly increase the cost.

2. End users are impatient.

When a model runs for hours to show a piece of work, however complex it is, the user is gone.

Even though models can do human-level work, they are still viewed as "computers". Hence patience level towards a human doing work vs AI doing work is very different.

3. Users are pleased with the current set of fast models.

The feedback towards the current results, particularly from Claude 3.5 Sonnet, please the end users.

For complex business cases, the expectation is not that reasoning towards a problem should be better.

Instead it is about:

- connectivity with internal data

- higher context windows

- ease of use

I believe this makes a case against test time compute models in favour of current set of models. There is still so much UX and customisation to figure out, we do not need to wait for the next gen models as apparently they will only be better in reasoning but also much slower and more expensive.