AI tools were used to help draft this post, with a human reviewing, fact-checking, and editing throughout.
When we're helping customers scope an AI solution, one question comes up almost every time: "what's this actually going to cost once it's live?" Estimating a normal serverless workload is hard enough already. AI makes it harder still, because you're no longer billed for compute time, you're billed for tokens, and how many tokens a request uses depends on what's in that prompt and that response.
There's a second complication: these systems aren't deterministic. Run the same use case for the same user twice, with the same prompt template, and you can still end up with two different bills, because the model decides on its own how much to generate, and on reasoning models, how long to "think," each time it runs. This isn't just theory. AWS's own model card for Claude Opus 4.7 says outright that even a temperature of 0 "did not guarantee identical responses across invocations," and Anthropic's docs on extended thinking note that a thinking budget "is a target rather than a strict cap." Put usage-based billing on top of that kind of variability and a single, confident cost figure stops being realistic.
Where the cost comes from
Here's what makes up an AI workload's bill, and what we'd do about each piece.
| Component | AWS example | Billed by | Tip / best practice |
|---|---|---|---|
| Model inference | Amazon Bedrock | Input + output tokens per request | Count real prompts with a token-counting endpoint rather than guessing from word counts, and cache repeated context. |
| Embedding model | Bedrock Titan / Cohere embeddings | Tokens embedded per document or query | Batch embedding jobs and only re-embed content that has actually changed. |
| Vector store | Amazon OpenSearch Serverless | OCU-hours, not query volume | Check the collection type. "Classic" collections hold a 2-OCU minimum 24/7 regardless of traffic; newer collections can scale to zero when idle. |
| Compute / orchestration | Lambda, API Gateway, Fargate | Requests, duration, memory provisioned | Right-size memory and timeout, and avoid always-on tasks for bursty traffic. |
| Observability | CloudWatch Logs, X-Ray | GB ingested/stored, traces recorded | Log selectively instead of full prompts and context by default, and set a retention period. |
| Networking / data transfer | NAT Gateway, Transit Gateway, PrivateLink | GB processed, hourly attachment or endpoint charge | Use VPC interface endpoints instead of a NAT Gateway for AWS-to-AWS calls where the pattern allows it. |
| Environments | All of the above, per account | Multiplied by every environment running continuously | Split each cost into "scales with usage" or "scales with time provisioned," then multiply the second kind by every environment, not just production. |
A worked example: tokens versus the vector store floor
Say you're running 50,000 queries a month, at roughly 800 input tokens and 200 output tokens each. On Amazon Bedrock, Claude 3.5 Sonnet ($6 and $30 per million input/output tokens) puts that at about $540 a month in production. Now compare it to a single OpenSearch Serverless Classic collection sitting on its 2-OCU minimum: 2 × $0.24 × 24 hours × 30 days, around $345 a month, per environment, whether anyone queries it once or a million times. Multiply that across dev, staging, and production and the idle floor alone, about $1,035 a month, can end up bigger than the token bill it's supposedly supporting.
A simple self-check before you send an estimate
Before you quote a monthly cost for an AI workload, ask yourself:
- Have you counted actual tokens for representative prompts against the specific model you'll run, rather than guessing from word counts?
- For every stateful component (vector store, provisioned throughput, cache), do you know whether it scales to zero when idle, or holds a minimum floor regardless of traffic?
- Have you multiplied every "scales with time" cost by the number of environments that actually run continuously, not just production?
- Is logging and tracing volume in the estimate, not treated as a rounding error?
- Does the estimate account for NAT Gateway, Transit Gateway, or PrivateLink charges on the actual network path between the application and the model or vector store?
If any answer is "no," that's usually where the first surprised Slack message about the AWS bill comes from.
Estimation is the starting line, not the finish
Everything above only gets you the cloud consumption number, the recurring line on the AWS bill. That's real, and it matters, but it's not the whole cost of the solution. Total cost of ownership also includes the build effort, evaluating the model, and the ongoing time your team spends running it. Get that fuller number right and the business has something solid to weigh against the value the solution is meant to deliver, which is the real question when deciding whether to build it at all.
And getting it right once doesn't mean it stays right. Usage patterns change, model pricing and tokenizers shift between versions, and the NAT Gateway someone wired up for a proof of concept has a habit of quietly becoming permanent infrastructure. The FinOps Foundation describes FinOps as an operational and cultural practice that matures through continuous iteration, not a one-off cost-cutting exercise, and that's true here too. Token usage, OCU or provisioned throughput commitments, and logging retention all need a second look as a solution moves from pilot to production, not a single estimate at the start. Get the initial number right and you've earned the investment decision. Skip the ongoing FinOps work afterward, and that number quietly stops being true.
