AI时代云成本剧变：决策成本反超实现成本

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

We spent a week going through the cloud bill for our internal AI platform. The results, up front:

Cloud Run instance-based billing down 84% (44% on the single service we tested first)

Container image storage down 65% after revisiting how many versions we keep

The thing that mattered most wasn't any single cut. It was the gate at merge time and the daily visibility

But the savings aren't really what I want to write about. When implementation gets cheap, the cost of deciding whether to build something stays exactly where it was, and suddenly it's the expensive part.

Slow implementation used to be a decision gate

This isn't a post about AI, exactly. But AI is what makes this particular thing grow.

Building anything used to cost real time. You'd estimate it, review the design, carve out the engineering time. That process is tedious, and it also had a side effect nobody designed for: it forced you to ask whether the thing was worth building at all. The slowness itself was the gate.

Make implementation fast and that gate comes off. You can ship an idea the same day you have it. That's genuinely good, and it makes "just build it and find out" the right call far more often than it used to be.

What also comes off, though, is any reason to hold back. Each individual thing is small enough that nobody objects. And what's left behind isn't engineering hours, it's the running cost of whatever you shipped. That part doesn't show up on the day you build it. It accumulates quietly afterward.

The starting point: we couldn't attribute spend to an app

What kicked this off was a handful of occasions where usage-based spend grew more than we expected after a particular change. BigQuery scan volume, and Vertex AI / Gemini calls.

The frustrating part was that we couldn't attribute the spend back to a specific app after the fact. BigQuery does let you label query jobs, and job history is there. But several of our apps run under the same service account, so knowing which principal ran a query didn't tell us which app was responsible. We could push labels through every call path, but until that's fully rolled out you're in the same position.

If you can't attribute after the fact, you stop it before it grows. That's where the idea of putting a gate at the entrance came from.

Visibility comes before any of the cuts

Here's the conclusion first: the highest-leverage part of this whole week was the gate at merge time and the daily reporting, not any individual optimization. Cuts are one-time. Left alone, the same things pile up again.

The gate: changes above a threshold go to a human

We added cost as a review dimension for our AI reviewer. If a PR pushes usage-based spend past a threshold, it gets handed to a human. Three thresholds:

The design choice I care about here: the AI doesn't do the arithmetic. It reads the change and measures quantities like scan volume, call counts, and token counts. Converting to currency and comparing against the threshold happens in a deterministic script. Hand the money math to a model and you get occasional arithmetic slips, the wrong unit price, and answers that drift between runs. A gate that gives different answers on different days isn't a gate.

Unit prices come from our actual billing export (spend ÷ usage), not from the published rate card. Hardcoding list prices means carrying three separate sources of drift: USD conversion, exchange rate, and committed-use discounts.

Storage gets its own threshold because cumulative cost behaves differently from execution cost. Stop a job and its daily cost goes to zero. Data you've written keeps billing until someone deletes it. So for storage we ask a different question: is there an expiration policy, and what does this add per month going forward? That's a forward projection, computed from call volume times payload size, not a measurement of what already happened.

Daily reporting: post to Slack twice a day

We post per-app cost to Slack twice a day. A few decisions in there worth mentioning.

Post even on days when nothing crossed a threshold. "Nothing to see today" is information, and more importantly, the reader needs to be able to tell the difference between a quiet day and a broken reporter. A monitor that only speaks up when something's wrong can't tell you it's dead.

Compare against the same weekday last week, not yesterday. Batch volume varies by day of week, so a day-over-day comparison flags every Monday as a spike, and people stop reading it within a week.

Treat "no data last week" as an increase. This is the dangerous case. A batch that just started running is exactly the thing that surprises you. Percent-change metrics tend to drop these on the floor because the math is undefined when the baseline is zero, so we handle that case explicitly.

The backstop: put a ceiling on usage-based services

Gates and reporting still leak. There are paths that don't go through a PR at all, like manual runs, external triggers, and usage patterns nobody anticipated.

So for anything that bills by usage, set a ceiling that matches how you actually operate. BigQuery, Vertex AI, and Gemini all let you cap this with quotas.

The important part is to derive the ceiling from measured usage, not from what the platform allows.

Take BigQuery. On-demand pricing has a default cap of 200 TiB per day, per project (that's been the default since September 2025; before that it was unlimited). Per-TiB pricing varies by region, but at a few dollars per TiB, that ceiling works out to somewhere north of a thousand dollars a day. Run that for a full month and you're at tens of thousands of dollars.

Which means the default setting is: spend that much, every month, without anyone being told. That isn't a ceiling.

We cap BigQuery scan volume at roughly 1.5x our normal usage. "Ten times normal, to be safe" has the same problem as the default. A ceiling only means something if it sits where a runaway actually stops.

One thing to be clear about: we can do this because this is internal infrastructure. Capping at 1.5x means accepting that things stop when they hit the cap. For an internal platform, you rerun it tomorrow. Do the same thing on a customer-facing path and you've built yourself an outage. Only put this kind of ceiling on things that are allowed to stop. For anything touching end users, set the ceiling much higher and handle it with alerting and autoscaling instead.

Given that constraint, the side effects are fine. A legitimate job occasionally hits the cap, and when it does, that's a prompt to go find out why something needed 1.5x normal volume. Which is its own kind of detection. Alerts tell you afterward. A quota stops it while it's happening. With usage-based billing that difference matters.

If you try to run any of this as a habit rather than a system, it evaporates the first busy week. It only works once it's structural. Gate at the entrance, daily reporting for the continuous view, quota as the hard stop. Three layers, and only together do they make the invisible visible.

Case 1: when the default is the expensive one, speed builds debt

Now the things we actually found. Cloud Run first.

Cloud Run has two billing models: instance-based, where CPU is always allocated, and request-based, where it's allocated only while handling a request. 97% of our Cloud Run spend was instance-based.

The cause was structural. Looking at where we define Cloud Run services in our repo, most of them didn't specify cpuIdle at all. There's no shared factory, so every new service silently landed on the expensive side unless someone thought to set it.

Worth knowing: deploy from the console or gcloud and the default is request-based, the cheap one. Define the same service declaratively in IaC with explicit resource limits and cpuIdle ends up false, which puts you on instance-based. So click it together and you get the cheap default; write it as code and you get the expensive one. The more disciplined your infrastructure practice, the easier this is to miss.

Which is the whole point from earlier. When the default sits on the expensive side, going faster builds debt automatically. The per-service difference is small enough that nobody notices.

The name reads backwards

There's a trap in the naming. The setting is called cpuIdle.

Read true as "keeps running while idle" and you have it exactly backwards. And the failure mode is nasty: no exception, no error, just the work that happens after the response quietly not happening. Flip a service that does fire-and-forget background work to true and the work disappears without a trace in the logs.

把 cpuIdle 理解为"空闲时继续运行"，你就完全搞反了。而且失败模式很恶心：没有异常，没有错误，只是响应之后的那些工作悄悄地没有发生。把一个做 fire-and-forget 后台工作的服务切换到 true，工作就会在日志中毫无痕迹地消失。

So we reduced the decision to one question: does this service keep working after it returns a response?

所以我们把决策简化成一个问题：这个服务在返回响应之后是否继续工作？

The exclusion list is narrower than it looks

排除列表比看起来要窄

When we picked which services to switch, we started with a generous exclusion list of anything that "probably needs CPU all the time." Most of it turned out not to. Two things about Cloud Run's behavior are not obvious.

当我们挑选要切换哪些服务时，我们先用一条宽松的排除列表起手——任何"可能需要持续占用 CPU"的东西。但事实证明大部分都不需要。关于 Cloud Run 的行为，有两点并不显而易见。

First, CPU is allocated during container startup regardless of this setting. A service that loads a large dataset from BigQuery at boot, synchronously, before the web server starts, looks like an obvious "needs constant CPU" case. But that work happens during startup, so switching it is fine.

首先，CPU 在容器启动期间就已经分配了，与这个设置无关。一个在启动时（Web 服务器启动之前、同步地）从 BigQuery 加载大型数据集的服务，看起来显然是"需要持续占用 CPU"的情况。但那部分工作发生在启动阶段，所以切换它是没问题的。

Second, long timeouts are not a reason to exclude something. A batch that takes 3,600 seconds of synchronous work is handling a request that entire time, so it has CPU. "It's a heavy job, so it needs constant CPU" doesn't hold.

其次，长超时不是排除的理由。一个需要 3,600 秒同步工作的批处理任务，在这整个时间段内都在处理请求，所以它是有 CPU 占用的。"这是一个重活，所以需要持续占用 CPU"这个逻辑并不成立。

What actually needs excluding is services that keep working after the response goes out, the fire-and-forget ones. That's the whole list. Getting there took several rewrites of the exclusion set.

真正需要排除的是那些在响应发出之后继续工作的服务——即 fire-and-forget 类型。这就是整个列表。得到这个结论花了我们对排除集合的多次重写。

Settings like this always over-exclude when you decide by intuition. Exclude based on a condition you can state, not on a service feeling like it needs it. Whether you can compress the rule into one line is a decent proxy for whether you've understood it.

这类设置如果凭直觉做决定，总是会过度排除。基于你能陈述的条件来排除，而不是基于某个服务"感觉上需要它"。能不能把规则压缩成一行，是判断你是否真正理解了这个规则的好标准。

Results, and stopping the regression

结果，以及阻止回归

The first service we switched came down 44% on its own. Rolling it out brought instance-based billing down 84% overall. Everything is switched except the handful of services doing fire-and-forget work.

我们切换的第一个服务独自下降了 44%。推广之后，基于实例的计费总体下降了 84%。除了少数做 fire-and-forget 工作的服务外，其他全部已切换。

Then we added a CI guard that fails on service definitions with no cpuIdle. The part I like: specifying cpuIdle: false explicitly is also a violation. If a service genuinely needs constant CPU, it goes in an allowlist with a written reason. The point is to make someone type out why.

然后我们添加了一条 CI 护栏：对没有 cpuIdle 的服务定义报错。我喜欢这个设计的地方是：显式指定 cpuIdle: false 同样是一种违规。如果某个服务确实需要持续占用 CPU，它就进入一个白名单，并附带书面理由。重点是让某人必须把原因打出来。

And with guards like this, we now write a violation on purpose and confirm the build actually fails. A guard that's been written isn't necessarily a guard that runs, and there's nothing to tell you when it isn't. All you're left with is the belief that you're covered, which is worse than knowing you aren't.

而且有了这类护栏，我们现在会故意写一条违规记录，并确认构建确实会失败。一条写好的护栏不一定是正在运行的护栏，当它没有运行时也没有任何提示告诉你。你剩下的只有"我是安全的"这种信念，这比知道自己没有 coverage 还要糟糕。

Case 2: the data can't tell you how recovery actually works

案例 2：数据无法告诉你恢复实际上是如何运作的

Container image storage was the other large line item. The fix was trivial: we went from keeping 10 image versions to keeping 2. Storage dropped 65%.

容器镜像存储是另一大笔支出。修复很简单：我们从保留 10 个镜像版本改为保留 2 个。存储费用下降了 65%。

The interesting part is how you get to that number. From the data alone, all you can say is "maybe we need 10." How far back you might roll to isn't something the data records.

有趣的是如何得出这个数字。仅从数据来看，你只能说"也许我们需要 10 个"。你能回滚到多远的版本，这不是数据会记录的事情。

Here's the actual reasoning. If the image is gone, you restore the code from git and deploy it again. So keeping images isn't an archive that lets you return to any point in history. It's a way to get back to the previous version quickly in an emergency. Framed that way, two is enough.

以下是真正的推理。如果镜像没了，你从 git 恢复代码并重新部署。所以保留镜像不是一种让你能回到历史上任意时间点的归档。它是一种在紧急情况下快速回滚到前一个版本的方式。这样理解的话，两个就够了。

What's doing the work there isn't data, it's knowing how recovery actually happens in your organization. Whether restoring from git is a safe assumption. How long a deploy takes. Whether anyone has ever needed to go further back than one version in an emergency. None of that is in your logs or your metrics.

在这里起作用的不是数据，而是了解恢复在你的组织中实际是如何运作的。从 git 恢复是否是安全的假设。部署需要多长时间。是否有人在紧急情况下曾经需要回滚到超过一个版本之前的版本。这些都不在你的日志或指标里。

Ask an AI to look into it and you'll usually get "keep more versions, to be safe," because that's what the data supports on its own. What a human brings is knowing what the retention is for.

让 AI 来调查，你通常会得到"保留更多版本以防万一"，因为这是数据本身能支持的结论。人类带来的是知道保留的目的是什么。

Case 3: we were running full CI for every review round

案例 3：我们在每个评审轮次都跑完整 CI

The CI change that mattered most was about when things run, not what.

最重要的 CI 改动是关于什么时候跑，而不是跑什么。

A reviewer flags something, you push a fix, they look again. Every one of those pushes ran the full set of jobs: build, lint, test, and knip.

评审者标一个问题，你推送一个修复，他们再看一眼。每一个这样的推送都会运行完整的 job 集合：build、lint、test 和 knip。

The structure is the same with human reviewers. It's just that AI review makes the rounds more frequent and much faster, so waste that was always there becomes obvious. Measured, we average 3.4 rounds per PR (counting a round as one flag-and-fix cycle), with 6.8 minutes of heavy jobs per round. Which means running the same test suite three or four times before review even converged. We were paying for a full test run against code that still had open comments on it.

人工评审的结构是一样的。只是 AI 评审让轮次变得更频繁、也快得多，所以一直存在的浪费变得明显了。实际测量下来，每个 PR 平均 3.4 轮（以一次标问题-修复为一个轮次），每轮 heavy job 耗时 6.8 分钟。这意味着评审收敛之前，同一个测试套件要跑三四次。我们在为仍有问题待处理的代码支付完整的测试运行费用。

So we moved the heavy jobs behind review approval. Every push runs a lightweight check set (1.2 minutes), and when the reviewer approves, a bot triggers the heavy CI.

所以我们把 heavy job 移到了评审通过之后。每次推送运行轻量级检查集合（1.2 分钟），当评审者批准时，一个 bot 触发 heavy CI。

Light checks per round, heavy jobs once at the end. 3.4 × 6.8 min becomes 3.4 × 1.2 min + 6.8 min, roughly 53% less.

每轮轻量检查，最后一次性 heavy job。3.4 × 6.8 分钟变成了 3.4 × 1.2 分钟 + 6.8 分钟，大约节省了 53%。

Skipping and not-running are not the same thing

跳过和不跑不是一回事

There's an implementation trap here. Our first attempt used if: conditions to skip jobs inside the same workflow, and that was the wrong design.

这里有一个实现陷阱。我们的第一次尝试使用 if: 条件来跳过同一 workflow 内的 job，这是错误的设计。

Skip a required check and you've effectively disabled branch protection. The green check appears without the tests having run, and the PR is mergeable. That fails open.

跳过一个必需的检查，你实际上就禁用了分支保护。绿色勾选出现了，但测试并没有跑过，PR 仍然可合并。这是 fail open。

Never start the workflow and no check is created, so the unreported required check blocks the merge. That fails closed.

永远不启动 workflow，就不会创建检查，所以未报告的必需检查会阻止合并。这是 fail closed。

That asymmetry is the whole design. The heavy jobs now live in a separate workflow triggered only by workflow_dispatch. If the dispatch fails, the checks stay unreported and the merge stays blocked.

这种不对称性就是整个设计的核心。heavy job 现在位于一个独立 workflow 中，仅通过 workflow_dispatch 触发。如果 dispatch 失败了，检查保持未报告状态，合并也被阻止。

The cost optimization and the safety net came out of the same decision. "Spend less" and "stop when something's wrong" tend to share a shape.

成本优化和安全网来自同一个决策。"少花钱"和"出问题就停"往往有共同的结构。

Also: deciding not to do something, with arithmetic

同样：用算术来决定不做某事

We also skip the heavy jobs on PRs that only touch documentation. The design question there was whether to split the docs-only detection into its own job.

我们也跳过仅修改文档的 PR 的 heavy job。设计上的问题是：是否要把 docs-only 检测拆成独立 job。

A dedicated job is structurally cleaner, but every PR then pays for one more runner start, about 25 seconds. Docs-only PRs are around 4% of recent merges, and they save roughly 300 seconds of heavy jobs each.

独立 job 在结构上更干净，但每个 PR 都要多支付一次 runner 启动费用，大约 25 秒。docs-only PR 约占最近合并的 4%，每次可节省大约 300 秒的 heavy job。

Run the expected value: 96% of PRs paying 25 seconds outweighs 4% of PRs saving 300. So the detection happens inside an existing job and we skipped the dedicated one.

算一下期望值：96% 的 PR 支付 25 秒，超过 4% 的 PR 节省 300 秒。所以检测在现有 job 内部进行，我们跳过了独立 job。

"Make the structure cleaner" always sounds right. In a cost context, you can decide against it with arithmetic. Go by which version feels more elegant and you'll miss reversals like this one.

"让结构更干净"听起来总是对的。在成本语境下，你可以用算术来决定反对它。按照哪个版本感觉更优雅来选择，你就会错过像这样的反转。

Case 4: it looks free at first, then the data grows

案例 4：一开始看起来免费，然后数据增长了

One more: BigQuery MERGE.

再一个：BigQuery MERGE。

When you want to insert data without creating duplicates, MERGE is the obvious choice. Match on a key, update if it's there, insert if it isn't. One statement, and it's idempotent.

当你想插入数据但不产生重复时，MERGE 是显而易见的选择。用 key 匹配，如果存在就更新，不存在就插入。一条语句，而且是幂等的。

The catch is that MERGE reads the target every time, no matter how many rows you're inserting. Even for a single row, it has to scan the target side to confirm that row isn't already there.

问题是 MERGE 每次都会读取目标表，不管你插入了多少行。即使只有一行，它也必须扫描目标侧来确认那行不在那里。

That's awkward from a cost perspective. While the table is small, it looks like nothing. Nothing is wrong on the day you write it. But as the data accumulates, the scan per run grows with it. Your execution frequency hasn't changed, and the cost climbs anyway.

从成本角度看这很尴尬。当表很小时，看起来不算什么。写代码的那天什么都没问题。但随着数据积累，每次运行的扫描量也随之增长。你的执行频率没有变，但成本却在增加。

And because nobody edited any code, this never shows up in PR review. A gate at merge time asks "how much does this change add," so anything that grows without being changed is out of scope by construction. This is the purest version of what I described at the top: invisible on the day you build it, accumulating quietly afterward.

而且因为没有人编辑过任何代码，这永远不会出现在 PR 评审中。合并时的门禁问的是"这个变更增加了多少成本"，所以任何在未被修改的情况下增长的东西，按构造来说都是超出范围的。这是我开头描述的最纯粹版本：构建那天不可见，之后悄无声息地积累。

Where the duplicate check should live

重复检查应该放在哪里

So we moved the duplicate check to Firestore, and BigQuery only gets the insert.

所以我们把重复检查移到了 Firestore，BigQuery 只做插入。

Look the key up in Firestore, and only insert the rows that come back as new. BigQuery becomes append-only, and the scan for reconciliation goes away entirely.

在 Firestore 中查找 key，只插入返回为新的行。BigQuery 变成仅追加， reconcile 扫描完全消失。

Same shape as Case 2. Not "how do we optimize the MERGE" but does BigQuery need to be the thing doing the duplicate check at all. BigQuery is excellent at scanning large volumes and aggregating. Asking it to confirm whether one key exists is using it against the grain. Firestore is the opposite: point lookups are its job, aggregation isn't. Push each to the side it's good at and both end up doing something natural.

与案例 2 相同的思路。不是"我们如何优化 MERGE"，而是 BigQuery 是否需要成为做重复检查的那一方。BigQuery 擅长扫描大量数据和聚合。让它来确认某个 key 是否存在是在逆势使用它。Firestore 相反：点查询是它的工作，聚合不是。让每个去到它擅长的方向，两边都能做得很自然。

Why you need something that notices

为什么你需要某种能注意到变化的东西

The point here isn't "don't use MERGE." It has its place, and we haven't replaced all of ours.

这里的重点不是"不要用 MERGE"。它有它的用武之地，我们也没有替换掉所有的 MERGE。

The point is that cost grows in two different ways: expensive from day one, and expensive after it grows into it. The first kind, review catches. The second kind, review never sees.

重点是成本以两种不同方式增长：从第一天就贵，以及长成之后才变贵。第一种，评审能catch到。第二种，评审永远看不到。

Which is what the daily reporting is for. Comparing against the same weekday last week, "this app's cost keeps climbing and nobody has touched it" eventually shows up on its own. That's the reason a gate alone isn't enough.

这就是每日报告的用途。与上周同一天相比，"这个应用的成本一直在涨，但没人动过它"最终会自己冒出来。这就是为什么仅靠门禁是不够的。

What the freed-up time is actually for

释放出来的时间实际上是用来做什么的

When implementation gets cheap, the gate that slow implementation used to provide comes off with it

当实现变得廉价，慢实现曾经提供的门禁也随之消失

What's left is the running cost of everything you shipped, which is invisible on the day you ship it

剩下的就是你交付的所有东西的运行成本，这在交付当天是不可见的

And it grows in two ways: expensive from day one, and expensive once the data grows into it. Review only ever catches the first kind

而且它以两种方式增长：从第一天就贵，以及数据长成之后才贵。评审只能 catch 到第一种

所以你节省下来的时间，要花在思考这件事是否应该存在

习惯会崩坏，所以要把它做成结构性的：在合并时设置关卡、每日报告、按量付费服务的配额、CI 中的护栏

同时要接受有些决策无法自动化：恢复机制如何运作、回滚到哪个版本、这个东西到底是否需要存在

能把系统化的东西推入系统，把人类的注意力留给无法系统化的部分。这就是重新分配时间的意思

更快地交付无疑是好的。但如果把所有收益都用来交付更多，那么你所交付的东西的重量会在之后追上你。我们花了一周做这次清理。下一步不是安排下一次清理，而是构建更多防止问题堆积的机制。

For further actions, you may consider blocking this person and/or reporting abuse