AI Needs a Net Work Test, Not an Adoption Dashboard
PMs should stop treating AI activity as proof of progress. An agent, automation, or internal tool is only valuable when it removes more work than it creates across the whole workflow: setup, review, exceptions, maintenance, and coordination included. If the team cannot show that difference, it has built a demo—not an operating advantage.
The Operating Rule
Measure AI by net work removed, not artifacts produced or prompts sent.
Before scaling an AI workflow, name the job it changes, establish its current cost, count every new human step, and set a decision date. Keep it only when the whole system is better for the people and customers who have to live with it.
The Problem Is Not AI Slop. It Is Unpriced Work.
A polished internal tool can be a terrible product decision. It can look impressive in a leadership demo, generate a satisfying trail of agent runs, and still make everyone else's day worse. Someone has to learn a new workflow. Someone has to review its output. Someone has to handle the cases it cannot handle. Someone has to fix it when the model, data, policy, or adjacent process changes.
That work is easy to hide because it arrives in small pieces. A support lead corrects an AI-generated categorization. An analyst checks an exception. A PM rewrites an automated update before it reaches leadership. An engineer gets pulled into another integration edge case. Each task looks minor. Together they can exceed the work the automation was meant to remove.
The most dangerous metric in this environment is adoption. Adoption tells you that people touched a tool. It does not tell you whether the workflow improved. Microsoft makes the same distinction in its guidance on measuring AI business impact: active users, prompts, and feature use can show usage without showing that work or business results improved. Its newer guidance is even blunter: measure agent value and risk, not the number of agents shipped.
Product managers should make that distinction their job. The AI era does not need another dashboard that proves people are busy. It needs an economic discipline for deciding whether a new workflow deserves to exist.
The Real Tradeoff: Visible Output Versus Actual Leverage
The temptation is understandable. AI has made it possible for one product leader to spin up prototypes, internal tools, summaries, and automations at a pace that used to require a queue. That creates a new kind of theater: visible output becomes the proxy for strategic relevance.
A current r/ProductManagement discussion puts the cost in plain language. The original poster described a Head of Product building internal AI tools and process automations that require more work than using a model directly. The highest-rated comments did not reject experimentation. They objected to leaders treating output as the job: one commenter argued that a product head should lead teams toward a direction, not push output; others described half-built internal workflows that people must learn, debug, and give feedback on despite never asking for them.
That is the argument PMs should take seriously. The problem is not that an executive has Claude open. The problem is that an experiment becomes organizational work before it has earned that burden. Curiosity is cheap. Workflow change is not.
The 2025 DORA report reaches a compatible conclusion from a much broader research base: AI acts as an amplifier, and the largest returns come from improving the underlying organizational system rather than adding tools in isolation. In a weak system, AI does not create leverage. It speeds up the creation of more things for the system to absorb.
Run the Net Work Test Before You Scale
Do not ask, “Can we automate this?” Ask, “Does this leave less total work behind?” The difference matters because a task can be technically automated while the workflow around it gets heavier.
| Test | What PMs Need to Define | Failure Signal |
|---|---|---|
| Baseline job | The user, trigger, handoffs, cycle time, error rate, and current manual effort. | The team cannot say what the old workflow actually costs. |
| Human work added | Review, exception handling, prompt maintenance, training, escalation, and integration upkeep. | The workflow only looks efficient because cleanup lives in someone else's queue. |
| Customer or team outcome | The behavior, quality, speed, cost, or risk result that should change. | Success is described as “people are using it” or “we shipped an agent.” |
| Decision date | A 30- or 60-day review with an owner who can expand, redesign, or stop the workflow. | The automation becomes permanent by default. |
| Net result | Whether the outcome gained exceeds all added work and operating cost. | The case depends on a demo, a usage count, or a promise instead of before-and-after evidence. |
This is not a demand for perfect ROI modeling. It is a demand for an honest counterfactual. Microsoft’s agent measurement guidance recommends the same basics: establish a pre-rollout baseline, use both leading and lagging indicators, and, where practical, keep a comparison group. PMs do not need a finance degree to apply that rule. They need to stop calling an initiative successful before they know what it displaced.
Three Ways PMs Mistake Activity for Value
1. The AI status update
An agent turns project notes into a weekly update. The output arrives faster, but the PM still corrects context, rewrites the recommendation, and manages stakeholder follow-ups created by the summary's omissions. Keep the agent only if it meaningfully reduces preparation time or improves decision quality. Otherwise, it has automated formatting and preserved the real work.
2. The internal support copilot
A tool drafts answers, but every response needs a specialist review because the source data is stale or the customer situation is nuanced. The right next move may be to narrow it to one bounded request type, refresh the knowledge source, and measure resolution time and reopens. A broad rollout just turns experts into a hidden quality-control department.
3. The AI research pipeline
A team asks a model to summarize interviews, then treats the summary as discovery. The process feels fast, but researchers and PMs spend more time untangling generic themes and defending decisions without direct evidence. Use AI to prepare, organize, and compare research; retain the source trail and validate consequential interpretations with real customer context. PM Prompt’s customer interview analyzer is designed for that evidence-first sequence.
What PMs Should Do Next Week
Pick one workflow that is already annoying people
Do not start with a model capability. Start with a recurring job where the team can name the user, pain, handoffs, and current cost. The best AI pilots are not impressive in isolation; they remove a real constraint.
Write the before state in the brief
Capture the current cycle time, manual steps, error or rework rate, and the people who absorb exceptions. If you cannot write the before state, you are not ready to claim an after state. Use an AI product management workflow to make the handoffs and decision points explicit first.
Give the pilot a kill rule
Decide what would make the team stop, shrink, or redesign the workflow: no time returned, poor review-pass rate, more escalations, higher support load, or no measurable outcome. The kill rule protects experimentation from becoming permanent process debt.
Review the whole workflow, not the tool
Bring the people who execute, review, maintain, and receive the work into the review. A tool can look excellent to its creator and still be a tax on everyone downstream.
What PMs Should Stop Doing
Stop celebrating an AI workflow because it generated a polished artifact.
Stop using active users, agent counts, or prompt volume as a business case.
Stop letting review, exceptions, and maintenance disappear into other teams' capacity.
Stop turning a leader's experiment into everyone else's process before the result is defensible.
Takeaways
- AI adoption proves contact with a tool, not improvement in a workflow.
- The hidden costs are review, exceptions, training, maintenance, and coordination.
- Every AI pilot needs a baseline, an outcome, a decision owner, and a stop rule.
- The right question is not whether AI can perform a task. It is whether it removes net work from the system.