Short answer: use language models to write and explain the analysis, not to be the calculator. Let the model generate SQL or Python, run it on the real data, and have the model interpret the result. Numbers that only exist in the model’s answer are not data.
Where LLMs help in analysis
| Task | Good fit? | How |
|---|---|---|
| Writing SQL or pandas code from a question | Yes | Give the schema and sample rows; run the code yourself |
| Explaining a chart or a variance | Yes | Paste the actual numbers; ask for drivers and caveats |
| Cleaning messy text fields (categories, names) | Yes | Run in batch and spot-check a sample |
| Doing arithmetic over many rows in the prompt | No | Models make silent mistakes; use code |
| Inventing benchmarks or market numbers | No | Ask for sources and verify them |
A prompt pattern that keeps you honest
prompt.txt
You are a data analyst. Answer the question by writing SQL only.
Schema:
orders(order_id, customer_id, created_at, total_usd, region)
customers(customer_id, segment, signup_date)
Question: Which segment had the largest drop in average order value
between Q2 and Q3 2026, by region?
Rules:
- Return one SQL query for PostgreSQL.
- Do not compute or guess any numbers.
- List the assumptions you made as SQL comments.Run the query, paste the result table back, and only then ask: “Explain the three biggest changes and what could cause them. Say which explanations need more data.”
Checks before you share an AI-assisted analysis
- Every number in the summary appears in a query result you ran.
- The model’s assumptions (date ranges, filters, currency) match the business definition.
- Totals reconcile with a trusted report.
- Causal claims (“sales dropped because…”) are labelled as hypotheses.
Models are fluent with numbers they did not compute. A confident percentage in a summary is the most common way AI-assisted analysis goes wrong.
Where the data should run
For company data, prefer tools where the code runs inside your environment and the model sees only schemas, samples and results, for example a notebook connected to your warehouse, or Athena queries generated by a model on Amazon Bedrock. Check what your AI usage policy allows before pasting real data into any chat tool.