Inaccuracies of Generative AI Based Search Tools for Extracting Data

Forward

Given their current trajectory, chatbots are on course to become the Goto tool to summarise information on the web. LLMs are trained to answer natural language questions with well summarised and confidently expressed text. This makes trusting their output as fact very compelling. However, due to how current models are built and even with the supporting information systems around them at runtime, the accuracy of their output still should not be blindly trusted. The authors demonstrate through a simple experiment how most current LLMs fail to accurately answer a UK statistics focused question. All answers provided were within expected bounds, and all were presented in an authoritative voice, but few were right and none were repeatedly right. The trouble is, as a result of their convincing tone these models can easily be assumed to be reporting the truth. As such, the risk of a propagating and using false statistical data will only grow.

Some of the companies in this sector are becoming aware of these issues and solutions are being proposed. However, these tools are currently being rolled out on supranational data and have yet to be tested on national ‘internal’ datasets.

View the full PDF