Free AI benchmark tool launched

Shrewsbury-based technology consultancy Shoothill has launched Shoothill AI Signal - a free tool that regularly tests the major AI models and publishes how their performance shifts between updates. 

Live at signal.shoothill.ai, it addresses the issue that the models behind today’s AI business tools aren’t static. Providers tweak them between major releases and as a result accuracy, reasoning and consistency can move with no warning.  

Shoothill AI Signal runs the same fixed set of tests against each major model on a regular cycle, scoring them on the things that matter in commercial use. These are truthfulness, reasoning, instruction-following, stability and overall business readiness. The methodology is then published in full. 

"We've spent years building AI into real projects for UK businesses, and the same issue keeps surfacing," said Simon Jeavons, group managing director at Shoothill.

"The models change. Reasoning shifts, output quality moves, and there's no clean way for a business to see it happening. We built Signal because we needed it for our own work. The day-to-day variation is genuinely fascinating, but if it's affecting your business, you need to know about it. That’s why we are sharing it." 

Shoothill expects it to be used mainly by IT and operations leaders weighing up which models to deploy, compliance teams getting ahead of issues, and senior managers who want a straight answer on whether the AI inside their tool is still doing the job it was picked for. Users get access to advanced notifications that flag up changes in model behaviour so action can be taken. 

Shoothill AI Signal is free, and Shoothill has committed to keeping it that way, ensuring its tested and new models are added as they become relevant to the UK market.