Twelve Questions to Ask Your PMIS Vendor About AI

Every major platform serving capital delivery is shipping AI capability this year. If you have sat through more than two vendor presentations you will have noticed that the announcements are difficult to tell apart. The same adjectives. The same chat panel. Frequently the same demonstration, which is a natural-language question producing a well-formatted answer.

The differences that matter are not in the announcements. They are in the answers to questions the announcements do not address, and most of those questions have to be asked deliberately because no vendor volunteers them.

What follows is a framework, not an argument for a platform. It applies to whichever product you are running or evaluating.

Every major platform is shipping AI capability this year and the announcements are difficult to tell apart. The differences that matter are in questions no vendor volunteers.

Group one: what is it reading?

Everything downstream depends on this, and it is the group most often skipped because it sounds technical.

• What data does the AI actually read, and is it structured records or inferred from documents? An answer drawn from validated records with defined relationships is a different class of answer from one assembled by reading files. Both can be correct. Only one can be traced.
• What accuracy is claimed, and how was it measured? The honest answer from most vendors today is that no figure exists. That is acceptable. What is not acceptable is a percentage with no methodology behind it, and you should ask what was tested, on what sample, and against what ground truth. this region the project record is in English and this is not a capture question at all. It is a presentation question, and it applies to the board and ministry reporting layer rather than to the working records beneath it.

Group two: what can it do without you?

• Can it act without a human? Ask for a demonstration that ends in the system preparing something, then watch what happens. Prepared changes should remain visibly pending. If the action completes, you are being offered an unattended actor inside your control environment.
• Is every AI action in the audit trail, and does the trail name the person who approved rather than the assistant that prepared? This is the question that will matter in a dispute two years from now, and it is almost never asked during evaluation.
• Can capabilities be enabled individually? All-or-nothing forces a decision most organisations are not ready to make. Ask what the unit of control is — per capability, per user group, per project, per environment — because that determines how much of your rollout you actually govern.

Group three: what happens to your data?

• Does the AI honour existing permissions, including where a result connects activity across records the user would reach separately? Synthesis is where permission models fail. A summary that combines several sources can reveal something none of them would have revealed individually.
• Is your data used to train or improve models, and how long is AI-related data retained? Require this in writing. It is the question your audit committee will ask, and a verbal assurance given during a demonstration is not something an audit committee can accept.
• Is there a published governance framework, with a date, that constrains the product roadmap? The difference between a vendor with a framework and one without is not the document. It is whether the answers above are design commitments or current behaviour, and current behaviour changes without notice.

Group four: what are you actually getting?

• Is what you are being shown shipped, or planned? This deserves a direct answer and it is reasonable to ask for it in writing. Roadmaps in this category are frequently marked as proof-of-concept, and a version number on a proof-of-concept list is a target rather than a date. A vendor who distinguishes clearly between the two is telling you something good about how they operate.
• Is AI usage metered, and what happens when you exceed the allowance? Several platforms are introducing token, query or per-seat limits. This matters more than it appears to, because a capability people are told to ration does not get adopted — teams revert to the workarounds the capability was meant to replace. Ask whether the limit is per seat or per tenant, because only one of those lets you turn a project team loose on it.
• Can you run your own analysis on your own data?

The twelfth question

The last one is the pivot, and it is worth more attention than the other eleven combined.

Ask a vendor whether you can point your own tools at your own project history — whether there is a database with a documented schema, a reporting replica, direct query access. Then listen carefully, because the answer tells you who the vendor believes owns your project record. decision, taken for sound architectural reasons in a multi-tenant environment. But the consequence for you is that every question your organisation ever wants answered has to travel through the vendor's product roadmap. Your ability to analyse fifteen years of your own delivery history is bounded by somebody else's release cycle.

Ask who can analyse your project history. The answer tells you who the vendor thinks owns it.

This also determines what happens if you ever decide to leave, which is not a hypothetical over a ten-year horizon.

The practical difference between a documented relational database and an API-only export is the difference between a migration that is engineering and a migration that is archaeology.

Applying it honestly

It would be convenient to run this framework against PMWEB and report a clean sweep. That is not what an evaluation framework is for, and any framework whose author scores perfectly on it should be treated with suspicion.

PMWEB's positions are strong on the control and permission questions. AI access follows existing platform permissions. The assistant will not finalise an action without explicit approval, and prepared updates stay pending.

Capabilities can be enabled individually. Relationships are surfaced for analysis rather than created automatically. Customer data remains contained within the platform. On question 11, PMWEB's stated position is that usage carries no limit.

On question 2 there is no accuracy figure, because none has been published. On question 3 the presentation-layer position is a fair question to put and one we are putting. And on question 10 the distinction between what is shipping and what sits further out is one this series has been careful to observe, which is why several articles in it describe what the discipline needs rather than what any product does today.

Those are the honest answers. A vendor or partner giving you a complete set of positive answers to twelve questions of this kind is not describing a product. They are describing a brochure.

How to use it

Send the twelve before the demonstration rather than after it, and ask for written responses. This changes the meeting. A vendor who has answered in writing arrives prepared to discuss substance, and the demonstration becomes a verification of the answers rather than a performance.

Score the answers rather than the features. Two platforms with near-identical capability lists can produce very different scores here, and the difference will predict your experience over the following three years far better than the feature comparison will.

And record the questions a vendor cannot yet answer, without treating that as a failure. This field is moving quickly and honest gaps are normal. What matters is whether the gap is acknowledged or papered over, because the second tells you what the rest of the relationship will be like.

Score the answers rather than the features. Two platforms with near-identical capability lists can produce very different scores here, and the difference will predict your next three years

better than the feature comparison will.

That concludes Part III. Part IV turns to the use cases organisations build for themselves, and to the question of how you start without moving fifteen years of history.