Five Questions That Kill a Bad AI Tool Demo

Every AI tool demo I have sat through was designed to be watched, not tested.

That is craft, not dishonesty. A demo is a piece of marketing, and the people building it are good at their jobs. The problem is that the buyer usually shows up to watch, which means the two parties agree on the format and nobody finds out anything.

I have bought tools that worked and tools that did not, and the difference was almost never visible in the demo. It became visible in week three, when the person who had to use it every day quietly went back to the old way. So I stopped watching demos and started interrupting them.

1. The demo data is perfect. Yours never will be.

Watch what they load. It is always a clean spreadsheet, consistent naming, no duplicates, no records from 2019 where somebody typed the company name into the contact field. Your data does not look like that. Nobody’s does.

The question: “Can we stop and run this on a file I bring?” Bring a genuinely ugly export. If the answer is that they will do a data-preparation exercise first, that is not a no, but it is a cost, and it belongs in the price you compare against alternatives. Most of the time the preparation is the actual work and the tool is the easy part.

2. They show you output, never the edit you will do after

Every AI demo ends at the moment the output appears. Applause. Next slide. The output is not the deliverable. The deliverable is what you send, and between those two things sits an edit. That edit is where the tool’s real cost lives, and it is the one number nobody in the category wants to quote.

The question: “Show me what you would change in that output before you sent it to a client.” Watch how long it takes them. If a vendor cannot find anything to change in their own tool’s output, they either do not use it or they are not being straight with you.

3. “It learns your brand voice”

This is the most-repeated sentence in the category and the least examined.

The question: “Show me it getting your brand voice wrong.” A vendor who has genuinely used their own product knows exactly where it breaks. They will tell you it flattens humour, or defaults to American spelling, or writes every third sentence as a negation. That answer is a good sign. It means someone has actually shipped work through it. A vendor who cannot produce a failure case has never left the demo.

4. No pricing on the website

Absence of pricing is not a pricing strategy. It is a negotiation strategy, and it means the price is a function of how much they think you can pay.

The question: “What does a company our size pay, and what is the range you have sold this at?” You will not always get an honest answer, but you will learn something either way. And you now know that any number you are quoted is an opening position, which changes how you respond to it.

5. A case study with a percentage but no base number

“40 percent lift” is not information. Forty percent lift on what? From what? A forty percent improvement on five leads a month is two leads. That may still be worth buying. But it is a very different purchase from the one the slide implies, and the slide is built so that you do not ask.

The question: “What were the absolute numbers before and after?” If those numbers do not exist, the case study is a testimonial with decoration.

What I do not treat as a red flag

A young company with a thin case-study list is not a red flag. Everyone was young once, and early customers get the best attention they will ever receive. Judge the product and the people, not the logo wall.

A tool that only does one thing is usually a good sign rather than a warning. The suites that promise to replace your whole stack are the ones that end up half-adopted and fully paid for.

The rule underneath all five

Every one of these questions does the same job. It moves the conversation from what the tool can do to what you will actually experience on an ordinary working day.

That is the only useful frame. Nobody is buying a capability. Everyone is buying a Tuesday.

Who this applies to

Founders and marketing leaders evaluating AI tooling without a procurement function to hide behind, which is most teams under twenty people. My challenge to you: take the ugliest file you own into the next demo and ask them to run it in front of you. Whatever happens in that ten minutes is your answer.

Frequently asked questions

How do I evaluate an AI marketing tool before buying?

Test five things live in the demo: your data instead of theirs, the post-output edit, a failure case for brand voice, the pricing range, and the base numbers behind any percentage.

Why does hidden pricing matter?

Because it means the price is set by what the vendor thinks you can pay. Knowing that changes how you negotiate and what you compare it against.

Is a percentage in a case study always misleading?

Not always, but it is incomplete without the base. A forty percent lift on a small number is a small number, and that may still be worth it.

What if the vendor refuses to use my data in the demo?

That is not automatically a no. It does mean a data-preparation cost exists, and it belongs in the total price you compare against alternatives.