CaretAI customer support reply drafts that don't send the wrong thing
Our marketing bot once wrote an answer for our FAQ that said there was "nothing to pay" while we were invite-only. It read well. It was also wrong: billing was already live and invited teams were seeing a monthly price. The bot wasn't guessing wildly. It had an older fact, it was confident, and nothing in the process made it check.
That was an FAQ, and we caught it before it mattered. Put the same thing in a support reply and a customer has it in writing, in their inbox, from you. That's the real risk with AI customer support reply drafts. The writing is usually good. The facts are where it goes wrong, and confident wrong answers are the hardest to spot.
Not every reply carries the same risk
A password reset link and a refund promise are different jobs. Sort incoming questions into tiers and treat each one differently.
| Tier | Example questions | What can go wrong | Draft must include | Who approves |
|---|---|---|---|---|
| 1. How-to | "How do I invite a teammate?" | Wrong steps, old screenshots | A link to the current help page | Anyone on support |
| 2. Account and bugs | "I can't log in", "the export is empty" | Blaming the user for a real bug | What was checked, and the bug note if handed to engineering | Anyone on support |
| 3. Money | Refunds, plan changes, failed charges | A refund or price you can't honor | The exact amount and the policy it comes from | The person who owns billing |
| 4. Promises | "When will X ship?", "Do you support Y?" | A date or feature that slips | Who confirmed it, and when | The person who owns the roadmap |
| 5. Upset or legal | Threats to leave, data requests, complaints | Tone, liability | Nothing sent from a draft without a rewrite | A founder |
Most of the volume sits in tiers 1 and 2. Most of the damage sits in 3 to 5. So put your review time where the damage is: skim the how-to drafts, and read the money and promise drafts line by line.
Ask for drafts that show their working
A draft that's only the reply text makes you redo the bot's research to check it. Ask for this shape every time:
- The question, restated in one line. If the restatement is wrong, the reply is wrong, and you see it in two seconds.
- The reply, ready to send.
- Sources. The help page, the past reply or the policy it used, as links.
- Not confirmed. Anything in the reply it couldn't check. "I couldn't find whether annual plans get prorated refunds" is the most useful line a draft can have.
- Promises in this reply. Any date, amount, refund or feature, pulled out on its own line.
- Tier, from the table above.
Point 4 is the one that would have saved us. Our "nothing to pay" line came from a fact the bot believed and never flagged. A draft that has to list what it couldn't confirm has a much harder time sounding sure about something stale.
The review checklist
Before you approve, check:
- The customer's name, company and plan are right.
- Every fact matches a source in the draft, and the source is current.
- No date is promised unless the owner confirmed it.
- Any amount matches the policy and the billing record.
- Links open and point to the right page.
- Nothing mentions another customer.
- It answers the question they asked, in fewer words than you'd expect.
Then write down every change you make. Anything you fix twice goes into the bot's instructions.
An example: a refund question, start to finish
The customer and the policy in this example are made up.
The customer wrote: "We were charged for a year but meant to pick monthly. Can we switch and get the difference back?"
The bot drafted this:
Question: Charged annual by mistake, wants monthly and a refund of the difference. Reply: "Hi Priya, sorry about that. I've switched your plan to monthly, and the difference will be refunded to your card within 24 hours..." Sources: refund policy page, billing record (annual charge on 14 Oct). Not confirmed: whether the 24-hour refund time applies to card payments. Promises: plan switch, refund of the difference, 24 hours. Tier: 3, money.
Then the billing owner read it. The policy allows a switch within 14 days, and the charge was 6 days old, so the refund was fine. The timing wasn't: card refunds take several business days. They cut "within 24 hours", wrote "in the next few business days, depending on your bank", and removed "I've switched your plan" because nothing had been switched yet.
Two approvals went out, as two cards: "Change plan to monthly" and then "Send 1 reply". Both waited for the billing owner's click.
Without the "Not confirmed" line, the 24-hour promise would have read like a fact.
Question: charged annual by mistake, wants monthly and the difference back. Reply: "Hi Priya, sorry about that. I've switched your plan to monthly, and the difference will be refunded to your card within 24 hours..." Sources: refund policy, billing record. Not confirmed: whether 24 hours applies to card payments. Promises: plan switch, refund, 24 hours. Tier: 3, money.
Cut "within 24 hours", say "in the next few business days, depending on your bank". Drop "I've switched your plan", nothing's switched yet.
Change plan to monthly
ApproveDenyApproved.
Send 1 reply to Priya
ApproveDenyApproved.
An example thread, shortened.
How to set it up in Caret
We built Caret around exactly this step: bots do the work in the thread and ask for your yes before anything goes out. A support setup looks like this.
- Create a support bot and write its job: "triage new support messages, draft replies in the six-part format, and never send without approval".
- Put your help docs, refund policy and five or six good past replies in its files. Memory lasts between conversations, so it also remembers what each customer asked last time.
- Paste the tier table into its job, with who approves each tier, and connect only the apps it needs.
- Add it to your support thread. It's bridged to Slack both ways, so you can approve from Slack. Before it sends, it asks, and an admin approves or denies with one click.
- Have it hand bugs to your engineering bot by its @handle in the thread, with an ask like "this looks like a bug, here are the steps the customer sent, can you check it".
- Give it a schedule for the weekly roundup: "every friday 16:00 post the top issues this week and which ones are still open".
The audit trail records what each bot sent and when, which helps the day a customer says nobody answered.
FAQ
Is it safe to let AI answer customer support emails? It's safe to let AI draft them. Sending is where it gets risky, because a wrong refund amount or a promised date is in the customer's inbox for good. Keep a person on the send, at least for anything involving money or dates.
How do I stop AI support replies from making things up? Give it your real docs and policies to work from, require sources in every draft, and make it list what it couldn't confirm. Then check pricing and dates with the person who owns them.
Can AI draft support replies in Slack and wait for approval? Yes. In Caret, the bot drafts in a thread that's bridged to Slack, and the approval card shows up there. One click sends it, one click stops it.
If support is your job, or the job nobody has this week, here's what a support bot can take on. Put a bot on it: start your 7-day trial or download the Mac app.