What to ask an AI automation agency before you sign.
This industry has almost no barrier to entry, which means the range between the best and worst people offering to automate your business is enormous. These are the questions that tell them apart, and I have included the ones that are awkward for me to answer.
Ownership and lock-in
1. Whose accounts will this run in?
Good answer: yours. Your OpenAI or Anthropic key, your Twilio account, your automation platform, billed to your card.
Warning sign: it runs in theirs and you pay them a monthly fee that covers it. That is not a service arrangement, it is a hostage arrangement, and you will discover the terms on the day you try to leave.
2. If I stop working with you tomorrow, what happens?
Good answer: the system keeps running, you revoke their access, and you have documentation for the parts you might need to change.
Warning sign: vagueness, or an answer about how that never happens. Ask it directly and listen for whether they have thought about it before.
3. Can I see the actual build?
The workflows, the prompts, the logic. Good answer: yes, here it is. Warning sign: proprietary. There is very little genuinely proprietary in this work, and "our secret framework" usually means a Make scenario they would rather you did not see the size of.
Scope and honesty
4. What would you tell me not to automate?
The most useful question on this list. Anyone who has done this work has a list: emotionally loaded calls, anything requiring judgement you would not delegate to a new employee, processes too broken to encode.
Warning sign: everything can be automated. It cannot, and someone who says otherwise has either not done much of it or is not going to tell you when they hit a wall.
5. Would an off-the-shelf product do this instead?
Ask it plainly. For a lot of straightforward requirements the honest answer is yes, and a fifty dollar a month tool will do it.
Warning sign: immediate dismissal of every product on the market. The cheap tools are good now. An agency pretending otherwise is hoping you have not tried one.
6. What does this cost to run once it is live?
Separate from the build. API usage, telephony, platform subscriptions. Good answer: an itemised estimate with the assumptions stated.
Warning sign: "minimal" or "just a few dollars". Voice minutes and model usage are real costs that scale with your volume, and finding out after launch is a bad way to find out.
Competence
7. What happens when it breaks?
Not if. Good answer: monitoring and alerting so you hear from the system rather than from a customer, plus a defined route to get it fixed.
Warning sign: no answer, or an assurance that it will not. Everything breaks. The question is whether you find out from a dashboard or from an angry client.
8. How did you test it before it went live?
Good answer: specifics. Real scenarios, edge cases, deliberately awkward inputs, a staging period.
Warning sign: "we tested it" with no detail. Ask what broke during testing. Everyone who has genuinely tested something has a story.
9. Show me something that went wrong on a previous build
Not a case study. A failure. Everyone doing real work has one, and the ones who tell you about it are usually the ones worth hiring. It also tells you how they behave when things go badly, which you will find out eventually one way or the other.
Commercials
10. What exactly am I paying for?
Hours, outcomes, a retainer, a licence. All are legitimate; being unclear is not. If it is a retainer, ask what happens in a month where nothing needs doing.
11. Who actually does the work?
Relevant for agencies with a sales layer. The person selling is often not the person building. Warning sign: you cannot speak to whoever will build it before you commit.
12. What do you need from me, and how much of my time?
Good answer: a realistic figure, usually several hours of your attention early on, because nobody can automate a process they have not had explained properly.
Warning sign: "nothing, we handle everything". Automation built without the owner's input automates a guess.
The one that is awkward for me
Since this list would be dishonest without it: ask whether you are hiring a company or a person.
I am one person. That has real advantages, which is that the person you speak to is the person who builds it and there is no account manager between you and the work. It also has a real risk, which is that I am a single point of failure. If I am unavailable, there is no bench.
How I handle it is that everything runs in your accounts with documentation, so another competent developer could pick it up. That is not the same as having a team, and if it matters to you, you should hire an agency with one. Any solo operator who does not acknowledge that trade-off is not being straight with you.
How to use this
You do not need all twelve. Questions 1, 4 and 6 will tell you most of what you need to know within about ten minutes, because they are the three that are hard to answer well without having actually done the work.
And if you ask me any of them, you will get the answers above. That is roughly the point of publishing them.
Frequently asked questions
What should I ask an AI automation agency first?
Whose accounts the system will run in. If the answer is theirs rather than yours, everything else is negotiating the terms of a dependency. Yours means you keep the system if the relationship ends.
How do I know if an automation agency actually knows what they are doing?
Ask what they would tell you not to automate, and ask about a build that went wrong. Both are hard to answer convincingly without real experience, and both are questions someone selling you a fantasy will dodge.
Should I hire a freelancer, an agency, or use a product?
For a straightforward requirement, a product. For something genuinely specific to your business, a person or an agency. The honest test is whether you can describe what you need in one sentence without naming three systems. If you can, buy software.
What are normal running costs after an automation is built?
It depends on volume, but it is never zero. API usage, telephony minutes and platform subscriptions all scale with how much you use them. Any agency describing running costs as negligible either has not measured them or is hoping you will not.
Is it risky to hire a one-person agency?
It is a real trade-off. You get the builder rather than an account manager, but there is no bench if they are unavailable. The mitigation is that everything runs in your accounts with documentation so someone else could take it over. Ask directly, and be sceptical of anyone who says it is not a risk.