Short answer: measure generative AI like any other process change: a baseline before, the same metric after, and the full cost including people’s review time. Ignore “number of prompts” and “time saved” surveys on their own; they measure activity, not results.
What to track
| Metric | Example | Why it matters |
|---|---|---|
| Cycle time | Hours from ticket opened to resolved | Captures real time saved, including review |
| Throughput | Reports, tickets or PRs completed per week | Shows whether saved time turns into output |
| Quality | Rework rate, defects, customer corrections | Speed with more rework is not a gain |
| Adoption that sticks | Weekly active users after 8 weeks | Novelty fades; habits show real value |
| Total cost | Licences + usage + setup + review time | The denominator most business cases forget |
A simple ROI formula
roi.txt
monthly_benefit = (baseline_minutes - new_minutes) / 60
* tasks_per_month * loaded_hourly_cost
- extra_review_minutes / 60 * tasks_per_month * loaded_hourly_cost
monthly_cost = licences + model_usage + maintenance_hours * hourly_cost
roi = (monthly_benefit - monthly_cost) / monthly_costFor model usage on Amazon Bedrock, estimate it with the token cost calculator.
How to get a fair baseline
- Pick one process and measure it for two to four weeks before the change.
- Use a pilot group and a comparison group doing the same work, when possible.
- Measure the same metric, the same way, after the change.
- Report the range, not just the best week.
Self-reported time savings are usually higher than measured ones. Use them as a signal for where to measure, not as the result.