Why LLMs Hallucinate Numbers in Analytics (and How to Catch It)
LLMs predict plausible text, not computed results. See why they invent numbers in analytics work, the patterns to watch for, and how to catch it before it reaches a report.
Short answer: An “agent” for analyst work is a loop that gives an LLM tools — read a file, run SQL, run a script — and lets it plan across several steps toward a goal, instead of answering in a single reply. You can build a genuinely useful one with Claude Code and a couple of scripts; you don’t need a framework to get started.
“AI agent” gets used loosely enough that it’s worth being precise about what actually makes something one, before building it.
A chatbot answers a question in one shot: you ask, it replies, done. An agent does something closer to what a junior analyst does with a task: it breaks the goal into steps, uses tools along the way (read a file, run a query, check a result), and decides what to do next based on what it finds, without you specifying every step in advance.
The mechanical difference is tool use plus a loop: the model is given a set of tools it can call, it calls one, sees the result, and decides whether to call another tool or give a final answer. Claude Code is built exactly around this loop already — it’s a general-purpose agent with file, search and shell tools, which is why it’s a reasonable starting point rather than something you need to build from scratch.
Take a recurring task: a weekly sales summary. Manually, this means opening last week’s export, comparing it to the prior week, noting anything unusual, and writing a short summary for a Monday message.
As an agent task, you’d give it:
The agent reads the file, computes the comparison itself (ideally by running actual code rather than eyeballing numbers — see why LLMs hallucinate numbers for why that distinction matters), and produces a draft. You review it before it goes anywhere.
Least privilege, same as you’d apply to a new hire’s system access on day one:
Resist the urge to give it broad access “just in case it needs it later.” Add access when a specific task genuinely requires it, not preemptively.
This is really the same split as the broader question of what to delegate — see Which analyst tasks to hand to AI, which to keep for the fuller framework this fits into.
If the task happens once, or rarely, building an agent around it costs more setup time than it saves. Agents pay off for recurring, well-defined tasks where the format of the input is stable enough that “read this week’s file” reliably means the same thing each week. A one-off analysis is usually faster done directly, with the LLM as a assistant in the conversation rather than as a standing agent with its own tools and loop.
My take: start with the smallest possible version — one tool, one clearly bounded task, human review on every run — before adding more tools or more autonomy. The temptation is to build something impressive; the useful version is usually something boring that reliably saves twenty minutes every Monday.
If you haven’t set up an environment for this kind of work yet, Claude Code for data analysis: a beginner setup covers the starting point.
Part of: AI for Analysts