← All posts

AI Is Everywhere at Work. Use It Smarter and Cheaper

A couple of years ago, "we use AI at work" was a conference talk. Now it's just Tuesday.

Recent developer surveys put weekly use of AI coding tools at around 90% of professional developers, and a big chunk of them use it daily. Some vendors claim that over 40% of committed code is now AI-generated. Take the exact numbers with a grain of salt, since every survey defines "use" differently. The direction is not in doubt, though. AI in the workplace isn't a pilot program anymore, it's the default.

The more interesting number is trust. In the same surveys, the vast majority of developers say they don't fully trust what the AI hands them. So we've got a weird situation: everyone uses it, almost nobody believes it. That gap is where all the good (and bad) engineering happens.

The Part Companies Forget: Policy and Privacy

When a tool spreads this fast, it usually spreads faster than the rules around it. In plenty of companies, half the team is on an approved assistant and the other half is quietly pasting code into whatever free chatbot they like best. People aren't being sneaky for fun. The official tool is slow to arrive, so they grab what works.

That's how you end up with API keys, customer data or half of a proprietary service sitting in a chat history nobody controls. A few boring habits go a long way:

  • Decide what's allowed in a prompt. Source code, yes. Secrets, customer records and anything under an NDA, no. Write it down in one page, not forty.
  • Give people a sanctioned tool that's actually good. If the approved option is worse than the free one, people will route around it.
  • Check the data terms. Does the provider train on your inputs? How long do they keep them? Business and API plans usually differ from consumer ones here, so read the fine print.
  • Keep secrets out of the repo to begin with. An assistant that can read your project can read your .env file too, so use ignore rules and a proper secrets manager.

None of this is exciting, but it's the difference between "we use AI" and "we use AI and our lawyers are fine with it."

Making AI Code Actually Better

Studies comparing AI-assisted and human-only pull requests keep finding the same thing: the AI code tends to have more issues, especially logic mistakes and security problems. The size of the gap changes a lot depending on who's measuring and what they count, so I wouldn't quote any single "1.7x" figure at a party. But one finding stuck with me. A model passing functional tests doesn't tell you much about whether its code is clean or secure.

So "it runs" is the lowest bar there is. Here's what moves the needle for me:

  • Give it context, not just a task. Tell it your stack, your conventions, and what "done" looks like. A short project instructions file (like a CLAUDE.md) pays for itself in the first hour.
  • Ask for small changes. A 40-line diff is reviewable. A 1,500-line one is a trust fall.
  • Let tests be the judge. Have the AI write tests first or alongside, then make it run them. I wrote about why unit tests matter even more with AI and I stand by it.
  • Automate the boring checks. Linters, type checkers and static analysis catch the unused variables, leftover debug code and sketchy patterns that AI loves to leave behind. Run them in CI so nobody has to remember.
  • Still read the diff. Review AI code like you'd review a confident intern's code: probably fine, occasionally very wrong, and always worth a look.

One more habit that's saved me a lot of grief: tell the AI what not to do. "Don't add new dependencies." "Don't touch the public API." "Don't rewrite files I didn't mention." Models love to be helpful, and "helpful" sometimes means rearranging half your project. Guardrails in the prompt are cheaper than cleaning up after the fact.

Also, be honest about when to stop. If you've re-prompted the same problem four times, the model isn't going to magically get it on try five. Clear the context, rewrite the prompt with what you learned, or just write it yourself.

Match the Model to the Workload

Here's the mistake I see most: using the biggest, shiniest model for everything. It's like driving a truck to grab a coffee.

Most providers sell a tier of models now. A flagship for hard reasoning, a mid-tier workhorse, and a small fast one that costs a fraction as much. The price gap between the top and bottom can be well over 10x, sometimes closer to 100x, per million tokens. That's not a rounding error, that's a budget line.

A rough way to split things up:

  • Small and fast: classification, tagging, extracting fields, summarizing short text, simple rewrites, autocomplete-style stuff.
  • Mid-tier: day-to-day coding, refactors, writing tests, code review, most chat-style help.
  • Flagship: gnarly debugging, architecture decisions, big multi-file changes, anything where a wrong answer is expensive.

A nice trick is to start cheap and escalate. Try the small model, check the result (a test, a schema validation, a quick sanity check), and only hand it to the big one if it fails. Plenty of tasks never need the escalation.

Here's what that can look like in code. It's a sketch, not a framework, and callModel stands in for whatever client you use:

type Tier = 'small' | 'mid' | 'large';

async function answer(task: string, check: (out: string) => boolean): Promise<string> {
	for (const tier of ['small', 'mid', 'large'] as Tier[]) {
		const out = await callModel(tier, task);
		if (check(out)) return out;
	}
	throw new Error('No model produced a valid answer');
}

The check function is doing the real work. It could validate JSON against a schema, run a test, or confirm the output contains the fields you asked for. If you can't write a check, you can't safely route cheaply, because you won't know when the small model got it wrong.

Rate Limits and API Costs

If you're building on an API instead of just using a chat app, two things will eventually bite you: the bill and the 429.

Costs come from tokens in and tokens out, and output tokens usually cost several times more than input. A few things help a lot:

  • Prompt caching. If you keep sending the same long system prompt or document, cached input is commonly priced at around a tenth of the normal rate. Put the stable stuff first and the changing stuff last.
  • Batch APIs. If nobody's waiting on the answer (nightly summaries, bulk tagging), many providers offer discounted asynchronous batch processing. Check your provider's current terms, since this varies.
  • Trim your prompts. Don't paste an entire repo when three files will do. Cap max_tokens so a rambling answer can't run up the bill.
  • Set a budget alert. Please. A retry loop gone wrong can burn through money while you sleep.

Rate limits are trickier than they look. You usually have several at once: requests per minute, tokens per minute, and sometimes daily caps. It's easy to stay far below the request limit and still get throttled because each request is huge. Fifty requests of 50,000 tokens each is 2.5 million tokens, which blows past a lot of per-minute limits even though 50 requests sounds tiny.

When you hit a 429, it usually means the provider is healthy and you're just sending too much. The fix is to shape your own traffic:

async function callWithBackoff<T>(fn: () => Promise<T>, maxRetries = 5): Promise<T> {
	for (let attempt = 0; ; attempt++) {
		try {
			return await fn();
		} catch (err: any) {
			if (err.status !== 429 || attempt >= maxRetries) throw err;
			// exponential backoff with jitter, capped at 30s
			const delay = Math.min(30_000, 500 * 2 ** attempt) * (0.5 + Math.random() / 2);
			await new Promise((r) => setTimeout(r, delay));
		}
	}
}

Respect the retry-after header when the provider sends one, and keep a retry cap so a stuck request doesn't spin forever. Naive retries without limits are a great way to turn a rate-limit problem into a cost problem. If you truly are down (not just throttled), that's when a fallback to a second model or provider earns its keep.

What It Actually Costs (A Made-Up But Realistic Example)

Say you've got a feature that makes 10,000 requests a day, each with about 2,000 tokens in and 500 tokens out. These prices are illustrative, loosely in the range you'll see across providers, so plug in your own:

Setup Input price Output price Rough daily cost
Flagship model, no caching $5 / M $25 / M ~$225
Mid-tier model $3 / M $15 / M ~$135
Small model $1 / M $5 / M ~$45
Flagship, 1,500 of 2,000 input tokens cached $0.50 / M cached $25 / M ~$157

Same feature, same traffic, and the bill swings by 5x depending on choices. That's 20 million input tokens and 5 million output tokens a day, and the output side alone is more than half the cost on the flagship. Caching rescued a big chunk of the flagship bill without changing the model at all, which is why I'd reach for it first.

Multiply by 30 and a "we'll just use the best model everywhere" decision turns into a real monthly line item. It's also why I like running the small model first and keeping the expensive one for the cases that need it.

Keeping an Eye on It

You can't manage what you can't see, and AI spend is weirdly invisible until the invoice lands. A few cheap things make it visible:

  • Log tokens per request. Input, output and cached counts are in the API response. Store them next to the feature name, so you can say "the summarizer costs $40 a day" instead of "AI is expensive."
  • Tag by feature or team. Separate API keys or metadata per project make it obvious who's spending what, and make it easy to cut off a runaway job without taking everything down.
  • Watch the 429 rate too. A rising number of throttled requests is an early warning that you're about to need a higher tier, a queue, or a smaller prompt.
  • Review it monthly. Models get cheaper and better on a regular basis. The setup that was right six months ago may be overpaying now, and a newer small model may handle work you used to give the big one.

Pricing and limits change all the time, so check the official docs before you budget anything. But the habits don't change: give the AI good context, verify what it writes, pick the smallest model that does the job, and treat tokens like the metered resource they are. Do that, and AI is a genuinely great coworker. Skip it, and you'll have a very fast way to produce expensive bugs.