You are reading this because someone senior has asked what AI returned, and the honest answer is a list of initiatives with a status next to each one. Platforms now exist that promise to turn that list into a number Finance will sign. This guide is for the executive who has to choose one: a Chief Data, Digital or AI Officer, a CFO or a CIO, and the team they ask to run the evaluation.
It describes practices and tests, not products. Every criterion can be applied to Propelon, to a competitor, to a tool you build in-house, or to doing nothing, and we score ourselves with the same scale. Where Propelon does not meet a criterion today we say so in our own response, with a date.
The guide runs in nine parts. The companion scorecard carries the 26 criteria, the default weights and the scoring scale; the RFP kit carries the template, the requirements matrix, the 90-day test plan and the reference questions.
Get the guide, the scorecard and the RFP kit Scope a 90-day launch
1. Why this is not a tool evaluation
A reporting tool that is wrong produces a stale chart, and someone notices. A value realization platform that is wrong produces a confident number nobody can defend to the audit committee, funds the next three initiatives on the strength of it, and stops nothing. The failure mode is quiet, and it shows up a year later as a line in a board deck that Finance will not stand behind.
Three principles follow. A dashboard is not a verification: a status someone typed is evidence of a status someone typed, and only telemetry from the systems of work and a signature from Finance turn it into a fact. A demo on the vendor’s data proves nothing about your portfolio; insist on your backlog, your baselines and your head count. And the governance of value, who may attest, who may decline, and whether declining is recorded, deserves the same rigor you would give to the arithmetic.
2. Start with your own baseline
Collect these eight measures before the first vendor meeting. They take about a week, they are rough on the first pass, and they are the only way to know afterwards whether anything changed. They also tell you which criteria deserve the most weight.
| Measure | How to collect it in a week | Why it matters |
|---|---|---|
| AI initiatives in flight, and where the list lives | Pull every backlog: Jira, Azure DevOps, ServiceNow, Aha!, SharePoint lists, the CoE’s spreadsheet. Merge duplicates. | The portfolio is the unit of value. Most organizations find 20-40% duplication across business units on the first pass. |
| Share of initiatives with a baseline attested from a system of record | For each initiative, ask for the starting number, its source system, the period and who signed it. | Without a baseline nothing can be verified later; the share is usually under 20%. |
| Share with any Finance-signed value in the last 12 months | Ask FP&A which AI initiatives it has signed a number for. | This is your organization’s version of the 37%. |
| Idea-to-production time, P50 and P95 | Dates from the backlog tools: intake, first decision, production. | The compression lever is measured against this. |
| Spend per initiative through pilot | People, data access, model and API usage, cloud, licenses, by phase, for the last ten pilots. | Sets the price of every initiative that should have been stopped earlier. |
| Initiatives stopped after pilot in the last 12 months, and spend at stop | Finance and the CoE together; include the quiet ones that stalled. | Avoided spend is the first lever and the one Finance will believe. |
| Adoption against the target population for the last three launches | Weekly active users from the systems of work over head count in scope from HR, not logins to the vendor’s tool and not a survey. | Value is realized in proportion to adoption; this is the denominator argument. |
| Monthly AI operating cost, and who approves it | Model and API invoices, the LLM gateway, cloud AI line items, agent run counts. | One in five organizations now says AI operating cost limits its use of AI (McKinsey, 2026). |
3. Ten questions every vendor should answer
Ask all ten in the first meeting and write the answers down in the vendor’s words. The first five are about what the platform does to an initiative; the last five are about what it does for you in a time-boxed evaluation. A good answer is specific and demonstrable; a red flag is a sentence that sounds fine until you try to test it.
| Question | A good answer | A red flag |
|---|---|---|
| 1. How do you take in the initiatives we already have? | Imports from the systems where they live, merges duplicates, and keeps the team’s tool as the place work happens. | A new form for every initiative, or “export a CSV and we will load it.” |
| 2. What has to be true before an initiative may consume AI capacity or people? | An attested baseline, a value hypothesis with unit economics, a named owner and sponsor, and a completed Responsible AI checklist, all on the record. | A stage field that someone sets by hand. |
| 3. How is a baseline captured, and who attests it? | From a system of record, for a stated period, attested by a named person with a date; without it the first gate refuses. | “The team enters the starting number.” |
| 4. How do you keep Modeled, Committed, Realized and Verified apart? | Five value states, reported separately and never summed; a number becomes Verified only when Finance signs it after the lock month. | One blended ROI number on a dashboard. |
| 5. How is adoption measured? | From telemetry in the systems of work, against a target population taken from HR or identity data. | Surveys, logins to the vendor’s own tool, or a promised adoption percentage. |
| 6. How is AI operating cost budgeted and attributed? | A budget per initiative in a unit Finance can plan with, attribution on every call, an alert at 80% and a stop at 100% unless re-approved. | A monthly invoice with no way to split it by initiative. |
| 7. Who can decline, and what happens when they do? | Finance, Model Risk and Security can decline at a gate; declining is a recorded outcome with a reason, and the initiative stops consuming capacity. | Approvals only, with no refusal path. |
| 8. What does a Finance sign-off look like in the system? | A named verifier, a lock month, a signature with a date, and lineage from the number to its source, method and owner. | An email thread outside the system. |
| 9. How does the first 90 days run, and how does it end? | Readiness before Day 1, a weekly gate cadence, and a Day-90 executive review that ends in buy, extend or stop with Finance present. | An open-ended pilot with no decision date. |
| 10. Where does our data live, who can see it, how long is it kept, and is it used to train models? | Written answers in the order form and the data processing agreement, including a clause that your data is never used for training. | “We will get back to you,” or “anonymized data may be used to improve the service.” |
4. The 26 criteria and the first-90-day tests
Score every criterion from 0 to 5: 0 is not available; 1 is described but not shown; 2 is shown on vendor data; 3 is shown on your data; 4 is measured in the first 90 days; 5 is measured and meets the target you set before Day 1. Nine criteria are must-haves. A must-have scored below 3 removes a vendor from the shortlist unless the vendor commits in writing to a delivery date; the item is then a roadmap item and is re-scored at the Day-90 review. Two or more roadmap must-haves remove the vendor.
The default weights in the scorecard favor the baseline, the value states, lineage and adoption telemetry, because those are the criteria that decide whether a number can ever be verified. Move weight toward your own biggest problem, and fix the weights before the vendors respond.
Group A. Intake, gates and portfolio
| Criterion | What to look for | How to test it in the first 90 days |
|---|---|---|
| 1. Backlog import and de-duplication (must-have) | Imports from Jira, Azure DevOps, ServiceNow, Aha! and SharePoint lists; merges duplicates and near-duplicates across business units; nothing is re-keyed. | Import the real backlog in week 0. Count the duplicates found and confirm no team re-entered anything. |
| 2. Gate criteria in plain English, with the reason for refusal (must-have) | Criteria an executive sponsor can read; a refusal names what is missing and who can supply it. | Pass five initiatives and refuse three on the record in weeks 1-2. Read a refusal aloud to the sponsor and ask whether it is clear. |
| 3. Kill or go inside 14 days | A clock starts at intake; initiatives that stall are surfaced to the sponsor; stopping is a recorded outcome, not a failure. | Measure median days from intake to the first gate decision over the cycle; the target is 14 or fewer. |
| 4. Responsible AI checklist at intake | Ten points (copyright, privacy, bias, explainability, security, oversight, accuracy thresholds, monitoring, provenance, record of decision) completed before code; Model Risk can decline. | Run one initiative through the checklist with Compliance present and confirm the record shows who signed each point. |
| 5. Headless operation inside the tools teams already use (must-have) | Status moves because work moved in Jira, Azure DevOps or ServiceNow; nobody logs in to update a field; no new system of record. | For two weeks, confirm no team member opened the platform to update status while gate status still moved. |
Group B. Baselines, value model and attestation
| Criterion | What to look for | How to test it in the first 90 days |
|---|---|---|
| 6. Attested baseline from a system of record (must-have) | Metric, source system, period, attestor and date captured on the record; no baseline, no gate. | Attest baselines for ten initiatives. Try to pass one without a baseline and confirm the gate refuses. |
| 7. Value hypothesis with unit economics | Baseline, target, volume and dollars per unit of change, with the arithmetic visible and a Finance factor applied by category. | Finance reproduces three hypotheses from the record without asking anyone. |
| 8. Five value states kept apart (must-have) | Opportunity, Modeled, Committed, Realized and Verified reported separately and never summed. | Ask for “total value.” The right answer is five numbers. |
| 9. Finance verifier role and sign-off inside the system | A named verifier, a lock month, a signature with a date; declining to verify is a recorded outcome. | Run one initiative to a Finance decision inside the 90 days, including one declined verification. |
| 10. Realization window and lock month | A configurable realization window (12 months is typical), a fiscal calendar and a snapshot at close. | Confirm a value moves from Realized to Verified only after the lock month, and that the snapshot cannot be edited afterwards. |
| 11. Decision lineage (must-have) | Every number traces to source, method, owner and date; every gate decision to its evidence and attestor. | Pick three numbers from the executive view and trace each to its source in under five minutes. |
Group C. Adoption and engagement telemetry
| Criterion | What to look for | How to test it in the first 90 days |
|---|---|---|
| 12. Adoption measured against a target population from telemetry (must-have) | Numerators from the systems of work, denominators from HR or identity data; no surveys and no vendor-tool logins. | Compare the platform’s adoption figure with your own telemetry pull for one initiative; agree within five points. |
| 13. Named integrations | Jira, Rally, Azure DevOps, Aha!, ServiceNow, Teams and Slack for work and engagement; the LLM gateway or Copilot administration for usage. | Connect two integrations in week 0 with configuration only, no custom code. |
| 14. Engagement mechanics tied to real adoption | Recognition rides on telemetry, not on self-reported activity; leaderboards are opt-in; the design aims at behavior, not points. | Observe gate-review attendance and recognition events through the cycle; ask three users whether it changed what they did. |
| 15. Adoption decay alerts | When adoption falls below target after launch, the adoption owner is told, with the population affected. | Exclude one team from an initiative for a week and confirm the alert reaches the adoption owner. |
| 16. Privacy-preserving aggregation (must-have) | Reporting by population; individual-level detail only where policy allows; HR and Privacy have reviewed the data model. | Privacy reviews the data model and confirms no individual productivity ranking is exposed by default. |
Group D. AI cost governance
| Criterion | What to look for | How to test it in the first 90 days |
|---|---|---|
| 17. AI capacity budgets per initiative | Capacity counted in a unit the business can budget (Propelon uses Agent Actions) and set at the first gate. | Set budgets for ten initiatives and confirm every call is attributed to exactly one. |
| 18. Alerts and stop rules | An alert at 80% of budget, a stop at 100% unless re-approved, and the re-approval on the record. | Drive one initiative past 80% and confirm both the alert and the stop. |
| 19. Cost attributed to value | Cost per initiative beside its value state; cost per verified dollar at the portfolio level. | At Day 90, read cost and value for every initiative on one page. |
Group E. Deployment, security and data
| Criterion | What to look for | How to test it in the first 90 days |
|---|---|---|
| 20. Identity, access and isolation | SSO (SAML or OIDC), role-based access by title and project role, tenant isolation, an audit log. | Security reviews the configuration and attempts an unauthorized read with a test account. |
| 21. Data residency, retention and training | Data location named, retention set, and a written clause that your data is never used to train models. | Read the data processing agreement. Score “not available” if the clause is verbal. |
| 22. Security program evidence | SOC 2 status (Type I, Type II, or a dated plan), a penetration test summary, a vulnerability disclosure policy and a security.txt. | Request the documents in week 0 and score what arrives, not what is promised. |
Group F. Proof, economics and partnership
| Criterion | What to look for | How to test it in the first 90 days |
|---|---|---|
| 23. Measured results from the first 90 days (must-have) | The vendor proposes a Day-90 executive review that ends in buy, extend or stop, with results presented in value states. | The review happens on the planned date with Finance present. |
| 24. Pricing that does not penalize adoption | Never per seat; scope by business unit or portfolio; capacity priced as a budget; engagement and adoption included in every tier. | Model the year-two cost at double the adoption. It should not double. |
| 25. References and roadmap transparency | A reference you can call with Finance on the line; roadmap items labeled as such, with dates in the order form. | One reference call, and one roadmap item retested at the Day-90 review. |
| 26. Executive sponsorship and an operating cadence | The vendor names executive sponsorship as a precondition and runs the cadence with you: weekly gate reviews, a monthly value review, the Day-90 executive review. | The cadence holds for 13 weeks with the sponsor present at the reviews. |
5. Compare against real alternatives
Every evaluation has three competitors that never submit a proposal: doing nothing, building it yourselves, and the AI features inside tools you already own. Score each of them with the same 26 criteria. Two of them will do surprisingly well on cost and surprisingly badly on everything that leads to a verified number.
| Alternative | What it does well | Where it fails the criteria |
|---|---|---|
| Doing nothing: Jira, SharePoint and a spreadsheet | Costs nothing new; everyone already knows it. | No gate, no attested baseline, no value states, adoption by anecdote. The 37% stays where it is. |
| Building in-house: Power BI over Jira and the LLM gateway | Fits your data exactly; your team owns it. | Reporting is not verification. The build usually stops at a dashboard; attestation, lock months and Finance sign-off never get built, and the team that built it moves on. |
| Idea-management tools | Good at collecting and voting on ideas. | Their last step is our first gate: nothing happens after selection, and nothing is measured. |
| Project and portfolio management tools | Schedules, resources and status. | Status is self-reported; value is a field, not a verified number; AI cost is invisible. |
| AI governance control planes | Decide whether an agent may act, and log what it did. | They answer whether agents may act. Whether the work paid is a different question, and the two sell together. |
6. Designing the evaluation
Entry criteria
Do not start until all of these are true. Each one that is missing becomes the reason the evaluation ends without a decision.
- An executive sponsor who owns the outcome and will attend the Day-90 review.
- A Finance verifier named before Day 1, with the haircuts and the realization window agreed.
- The baseline measures above collected, however rough.
- Written success criteria: the kill rate, the cycle time, the adoption target population and the first value to be verified.
- A decision date for buy, extend or stop, on the calendar before the evaluation starts.
- Target populations defined from HR data for every initiative in scope.
- Access to the systems of work and the LLM gateway, through your identity provider, read-only where possible.
Scope
One business unit, or three if the sponsor owns all three, with ten to twenty-five initiatives in flight including the existing backlog, three strategic focus areas, and read access to the systems of work through your identity provider. Resist the urge to evaluate on a toy portfolio: the import, the duplicates and the refusals are the test.
Timeline
The commercial unit is a 90-day launch. The clock starts at launch; configuration and readiness happen before Day 1, and the first cycle ends in an executive review that decides.
| When | What happens | Exit |
|---|---|---|
| Weeks 0-2 (before Day 1) | Readiness: portfolio imported, baselines attested, budgets set, integrations connected, sponsor and verifier named. | Everything configured; the clock has not started. |
| Day 1 | Launch. The first cycle opens with the gate calendar published. | Sponsor present; gates scheduled. |
| Weeks 1-4 | First gate reviews: every active initiative gets a decision on the record; at least three are stopped. | Median intake-to-decision at or under 14 days. |
| Weeks 5-8 | Adoption telemetry live for every production initiative; first values Modeled with source, method and owner. | Finance can reproduce three hypotheses from the record. |
| Weeks 9-12 | Cost beside value on one page; budget alerts tested; Finance review prepared; one declined verification recorded. | The executive view answers “what did AI return?” in five numbers. |
| Week 13 (Day 90) | Executive review with Finance present: adoption against target population, value by state, cost by initiative. | Buy, extend or stop, recorded. |
Commercial
Pay for the 90-day launch and have it credited against the first-year subscription. A free pilot with no sponsor and no decision date is the most expensive thing a vendor can offer you, because nobody on either side has a reason to stop. If the vendor cannot name executive sponsorship as a precondition, the vendor has not run one of these before.
7. Reading the economics honestly
The business case for a value realization platform rests on three levers, and they should be presented separately. The first is avoided spend: initiatives that would never have produced verified value are stopped at the first gate, inside 14 days, instead of after a pilot. The second is timing: initiatives that do succeed reach production sooner, so realization starts earlier. The third is adoption: value is realized in proportion to the share of the target population that uses the result, and that share is measured, never promised.
Propose on the first lever and let the other two stay Modeled. At typical costs, a prototype and its AI operating cost run to $100,000-$150,000 before anyone decides anything, against $30,000-$40,000 to stop at Discovery; three initiatives stopped at the first gate cover a year of a mid-tier subscription. That is the break-even framing a CFO will defend, because every input in it is the organization’s own.
Six habits keep the model honest. Separate hard-dollar value (operating cost, working capital) from business impact (risk avoided, experience), and count only the first in ROI. Agree the realization factors with Finance before Day 1, by category, and write them down. Label every vendor figure a target until it is measured in your environment. Distinguish capacity from head count: hours redeployed are real, and they are not a layoff. Check that the price does not rise because more people adopted. And never, in any document, add the five value states together.
8. Red flags
Any one of these is a reason to slow down. Two together are a reason to stop.
- Demos on the vendor’s data rather than your backlog and your baselines.
- No concept of an attested baseline, or a baseline the team types in.
- A single blended ROI number anywhere on the dashboard.
- Adoption measured by survey, by logins to the vendor’s tool, or promised as a percentage.
- “AI ROI” claims about reference customers that no Finance organization signed.
- AI cost that cannot be attributed to an initiative.
- Per-seat pricing, or any price that rises because more people adopted.
- A pilot with no decision date, or one that starts before the sponsor and the Finance verifier are named.
- Vague answers on data location, retention or model training.
- A vendor who spends the meeting on competitors instead of your criteria.
9. Sources
- McKinsey & Company, The state of AI in 2026: On the road to ROI, August 25, 2026 (80% report individual productivity gains; 37% attribute EBIT impact; one in five constrained by AI operating cost).
- Boston Consulting Group, The Widening AI Value Gap, September 2025 (about 5% of companies achieving value at scale).
- Propelon, Delivering Innovation in an AI World: The Enterprise AI Value Realization Platform (whitepaper), September 2026 (the five value states, the five gates and the 90-day launch as used in this guide).
Propelon has one Fortune 5 healthcare reference with telemetry-verified adoption at scale and design partners in their first cycles. It does not yet have a public Finance-signed value reference, and this guide does not claim one.
Get the guide, the scorecard and the RFP kit
One package by email: the guide, the weighted scorecard, the RFP template, the 104-requirement matrix, the 90-day test plan and the reference questions.