When the CEO wants AI features in the product by next quarter, agree to the date, but make it the date for a gated pilot, not for a launch. Turn the request into three written commitments: one narrow workflow, a monthly run cost, and a pass mark on an eval set. The launch follows only if the pilot passes that mark.
More on the people side of these calls is in our engineering management guides for CTOs.
Is the CEO's request about an AI feature or a story for investors?
Probably some of both, and the same scoping answers either one. Before you agree to anything, get three answers from the CEO.
- What should change for a customer? Customers closing tickets faster is a feature. Wanting to be seen doing AI is a story, and you should know which one you're building.
- What did the competitor actually ship? Open their product. A press release and a working feature are different things, and the working one is often smaller.
- What metric would tell us it worked? Pick one, with a number.
A Hacker News thread from a lead engineer whose CEO was moving teams into an AI group with no roadmap drew practical advice: test whether AI strengthens your core value, build the roadmap with the CEO rather than resisting, bring objective data instead of conclusions, and ask how likely a competitor using AI well is to beat you. One commenter's rule was that if that likelihood is above even odds, shift some focus to AI.
All of it points the same way: arrive with questions and a plan. If the CEO's answers are vague, the questions still force the conversation, and you haven't said no.
What do you scope so a date is possible?
One narrow workflow, not a general assistant. Nielsen Norman Group's guidance on scoping generative AI features argues that narrow scope makes an "AI for AI's sake" feature less likely, because a narrow feature has to justify itself against a specific task. A broad assistant can't be tested, so it can't be finished.
Write the not-in-scope list down and get the CEO to read it. Summarising a support conversation for the agent is a scope. Also drafting replies, tagging tickets and answering customers directly is three more projects. Each item you leave off the page now is an argument you won't have in week six.
For the pilot frame, we'd borrow SVA's CEO guide: one high-impact, low-complexity use case, with proof of return inside 90 days. It fits inside a quarter, and it makes the date about a pilot.
Then put what gets bumped on the roadmap, in writing. An AI pilot isn't free capacity, and the CEO should see what moves out for the quarter. For a method to size that trade, size the roadmap to real capacity covers the mechanics.
What will the feature cost to run each month?
Anthropic's pricing page lists Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens. That's enough to put a number in front of the CEO before anyone writes code.
Take a support-reply summariser with these example inputs: 50,000 requests a month, 3,000 input tokens and 500 output tokens per request.
- Input: 50,000 x 3,000 is 150 million tokens, which at $2 per million is $300.
- Output: 50,000 x 500 is 25 million tokens, which at $10 per million is $250.
- Total: $550 a month at list price.
If usage doubles, so does the bill: $1,100. That's the figure to give the CEO, with the volume attached, so they see that the cost follows adoption. Anthropic's Batch API takes 50% off input and output, but it suits work nobody is waiting on. An agent waiting for a summary is waiting, so don't plan on the discount.
This is model spend only. Engineering time and the eval work below are on top.
How do you know it is good enough to ship?
Build an eval set before you agree a launch date. No published figure tells you what accuracy a model will reach on your own inputs, so measure it.
Anthropic's documentation on defining success says criteria should be specific, measurable, achievable and relevant. Its testing guidance adds three principles for the eval set itself: make it task-specific, with edge cases included, automate the grading where you can, and prefer a large volume of cases over a few polished ones.
In practice:
- Collect real cases from your own data, including the ugly ones. For the summariser, that means long threads, threads in two languages, and threads where the customer is angry.
- Write what a pass looks like. "Summary includes the customer's request and the current status, no invented details" can be graded. "Good summary" can't.
- Run two or three candidate models on the same set. The cheaper one may be enough, and you won't know until you try.
- Agree the pass mark with the CEO in writing before the results arrive. Agreeing afterwards turns the number into a negotiation.
A commenter on a second Hacker News thread, on a CEO who wanted AI to do everything, cited a support bot that made things up and an airline chatbot error. That failure is the one a pass mark exists to catch.
Who builds it, and what happens to the roadmap?
There are three options, and each costs something different.
Your existing engineers build it. Use them when the scope is one workflow and the roadmap items that move out are ones the CEO has agreed to lose. It's the cheapest in cash, and the roadmap pays.
One contracted engineer joins for the pilot. Use a contractor when the roadmap can't give up the capacity but the scope is already tight, since a contractor on a vague scope just burns money faster. Vetting an AI engineer covers what to check. HighCircl's no-minimum-hours contracts suit a pilot that might end at the gate.
Or defer. That's the right call if the CEO's answers to the three questions don't hold up. Say so with the competitor analysis attached. A permanent AI hire only makes sense after the pilot passes and the feature has a roadmap of its own.
What this means for your AI feature decision
Your decision is whether to commit the team to a date, and the answer we'd give is yes to a date for the pilot, and a launch date only after the pilot passes. Put four things on one page for the CEO: the workflow and the not-in-scope list, the monthly run cost at today's volume and at double, the pass mark, and what comes off the roadmap.
If the CEO accepts the page, you have a mandate that survives a bad week. If they won't accept a pass mark, the decision was made before the evidence, and you're planning a launch rather than testing a feature. Know which you're in before you commit the team.
The bill after launch is the next surprise. When the pilot passes and usage grows, when the LLM bill runs above what the feature earns covers the levers. And if the same CEO is also asking for a smaller team because of AI, when the CEO wants a smaller team because of AI is the companion conversation.
FAQ
How do I respond when the CEO wants AI because a competitor has it?
Ask what the competitor shipped and what it changes for their customers. Then offer a pilot on one workflow, with a date, a monthly run cost and a pass mark. You're agreeing to the urgency without agreeing to an unscoped launch.
How long does an AI feature take to build?
Estimate it from your own eval. Build the eval set, run two or three candidate models on it, and see how far the best one is from your pass mark. A model that clears it on the first run needs integration work; one that misses needs more iteration. Give the CEO a pilot date after that run, not before.
Should we hire an AI engineer for this?
Not before the scope is written. If the pilot fits your current team and the roadmap trade is acceptable, you may not need to. If it doesn't, a contracted engineer for the pilot is a smaller commitment than a permanent hire.
Should we copy the competitor's feature?
Copy the outcome, not the feature. Find what customers get from it and whether your product can deliver it with one narrow workflow. A pilot on your own data will tell you more than a screenshot of theirs.
What if the CEO just wants an AI story for investors?
The same page works. A scoped pilot with a metric gives the CEO something true to say, and it gives you protection if the story outruns the product. Don't build anything you can't put a pass mark on.
Sources
- Hacker News: Ask HN, My CEO wants to go hard on AI
- Hacker News: Dear Lewis, my CEO wants AI to do it all
- Nielsen Norman Group: Scope in generative AI features
- SVA Consulting: A guide for CEOs for prioritizing high-impact AI projects
- Anthropic: Claude pricing
- Anthropic: Create strong empirical evaluations