A Shopify AI chatbot will answer your customers' questions on day one. Whether it sells anything depends on five things you can check before the free trial ends: whether it can ask a question before listing products, whether it can say "we don't have that", whether it can capture a contact, what it measures, and whose knowledge it is working from. We checked none of these before installing one in our own Shopify store. Four months and 238 conversations later we had the answers, and no attributed orders.
Key facts
- The store: Shopify, bathroom and plumbing equipment, about 1,000 products, Serbia
- The app: a paid AI chatbot from the Shopify App Store, around 30 USD a month, well rated
- The result: 238 conversations, 108 product clicks, 2 carts, 0 purchases attributed, July to October 2026
- The replacement: the same catalogue behind a chat with three written sales rules
- Who this is for: Shopify merchants comparing chatbot apps, or wondering why the one they have is not selling
Why a Shopify chatbot looks like it works
The App Store listing shows conversations handled, products recommended, clicks generated. Our dashboard showed all of that. It was accurate. The chat replied in seconds, knew the product codes, gave correct store hours and delivery terms, and wrote tidy paragraphs.
What the dashboard did not show was that the chat asked the customer a question first in 7 of 379 inputs, and returned a priced product list in 248. Customers clicked a product and left. The conversation count went up. The order count did not move. The full export is in the case study.
The five checks below are what we would run on any Shopify AI chatbot now, during the trial, with the export in hand.
Check 1: can it ask before it lists?
Type a two-word product query into the chat: "shower drain", "toilet seat", whatever your customers type most. Count how many products come back and whether a question came first.
A chat that returns nine items with prices is a search engine. A chat that asks "for a shower tray, washbasin or kitchen sink?" and then shows two is a salesperson. Then look in the app's settings for a way to make the question happen per product group. If there is no such setting, no amount of prompt tuning will produce it reliably.
Check 2: can it say "we don't have that"?
Ask for something you do not stock: a size, a brand, a code. The chatbot we used offered a 36,780-dinar backflow valve to a customer who wanted a 160 mm transition piece, because the valve's name contained "DN160". Three times.
The default behaviour of any AI chatbot is to return the nearest thing. In a store, the nearest thing is a wrong part sold with confidence, and a customer who knows what they asked for stops trusting every other answer. Check whether the app lets you forbid substitution and require "we don't have that, what is it for?" instead.
Check 3: can it take a number?
Ask the chat something it cannot answer, late in the evening. Does it say "call us during working hours", or does it ask for your number and promise a call?
In our export, 123 of 384 replies ended with "call us" or "email us". None offered to take a contact. A quarter of the conversations happened between 10 pm and 7 am. Every one of those customers was shown the door politely. A Shopify chatbot that cannot capture a lead and pass it to you is losing exactly the customers it was installed to catch.
Check 4: what does it measure?
Open the analytics. If the headline numbers are conversations, messages and clicks, ask where the orders are. Shopify knows which sessions converted. A chatbot app that integrates properly can attribute orders to conversations. One that reports clicks only is reporting activity, not sales.
Set your own two numbers before the trial starts: orders where the chat was the last touch before checkout, and contacts captured when the chat could not close. Read those at the end of the trial, not the click count. The general version of this mistake, with the other five, is in why chatbots fail to sell.
Check 5: whose knowledge is in it?
Every Shopify chatbot syncs your product catalogue. That is the easy part. The question is whether there is a place to put what your salesperson knows: that "I need a bath tap, just something that works" should get the two cheapest models immediately, that toilet seats come back because of hinge spacing, that one fan model is bought for the wrong room every week.
If the app has only a catalogue sync and a generic "tone" setting, it will multiply your catalogue. That is what ours did, 238 times. If it has a knowledge base you can write to, per product group, with the salesperson's corrections, it can multiply your sales method instead. The work of writing that down is yours either way. The seven best practices for an ecommerce chatbot are the list of what to write.
What we did instead
We exported every conversation, read them, and replaced the app with our own chat on the same Shopify store, same catalogue, with three rules a shop salesperson gives every new colleague: one question before the list, no closest match, take the contact. Alongside the rules, the salesperson's knowledge went in group by group.
Results will be published when there are enough conversations to mean something. The honest statement for now is that zero attributed orders in four months was measurable, and that any Shopify AI chatbot can be checked against the five points above before it costs you the same four months.
How a salesperson's knowledge moves into a system, and stays there when the salesperson leaves, is on the AI integration page.