erabot.ai
ProductPricingCase studiesEvalBlogDocs
Book a demo

We ran erabot on OpenAI's own Swarm — it extracts names with a model that costs ~65× too much

First in a series where we scan well-known AI repos for wasted LLM spend and show the fix. Every number here is reproducible — run the scan yourself in 10 seconds (how, at the bottom).


The one-liner

OpenAI's Swarm — their experimental multi-agent framework — has an eval step that extracts names from a sentence using gpt-4-turbo-preview. That's a legacy model whose input tokens cost ~65× more than gpt-4o-mini, which handles name extraction just as well. It's a one-word fix.

What we ran

Scan this public repo for wasted LLM spend: https://github.com/openai/swarm

That's it — erabot's free repo scan. It found 11 LLM call sites (test/mock files excluded), and flagged this one as the top line item (high confidence on the model, because it's a plain string we could resolve).

The finding

examples/customer_service_streaming/src/evals/eval_function.py:

def name(self):
    extract_name_prompt = "You will be provided with a sentence. Your goal is to extract the full names you see in the sentence. Return the names as an array of strings."
    response = self.plan['step'][-1]
    completion_result = self.client.chat.completions.create(
       model="gpt-4-turbo-preview",   # <-- legacy, expensive, for a trivial task
       max_tokens=100,
       temperature=0,
       messages=[
        {"role": "system", "content": extract_name_prompt},
        {"role": "user", "content": f"SENTENCE:\n{response}"}]
    )

Pulling full names out of a sentence is about the simplest NLP task there is. gpt-4-turbo-preview is a 2024-era model that OpenAI itself superseded with gpt-4o — which is cheaper and better. Using it here is the classic "wrong model for the job" leak: a model string that got written once and never revisited.

The numbers — honestly

Per 1M tokens:

ModelInputOutput
gpt-4-turbo-preview (in the code)$10.00$30.00
gpt-4o (modern, like-for-like)$2.50$10.00
gpt-4o-mini (plenty for extraction)$0.15$0.60

So the fix is worth, per call:

  • → gpt-4o: ~70% cheaper, and a better model.
  • → gpt-4o-mini: ~98% cheaper (input is 65× less; output 50× less), with no quality loss on a task this simple.

The honest part: the "$/month" depends entirely on how often this runs, which static code can't know — so erabot marked it volume-assumed and estimated ~$9.60/mo for this call on a default volume. On Swarm (example code) the real number is small. The point isn't the $10 — it's the pattern. This exact line, copy-pasted into a production agent that extracts names on every request, is where it quietly becomes thousands a month. The percentage above is exact; the volume is yours.

The fix

One word:

model="gpt-4o-mini"   # name extraction; ~98% cheaper, same result

Candidate swap — as always, verify it holds quality on your real inputs before you ship it. (On name extraction, it will. On something subtler, that's exactly what shadow-testing is for.)

Why this is the most common leak we find

After "re-sending the same context uncached in a loop," a frontier/legacy model doing a simple job is the #1 thing we see. Models get cheaper and better every few months; the model string in your code doesn't update itself. Classification, routing, extraction, yes/no, short labels — these almost never need a frontier model, and the price gap is often 20–100×.

If OpenAI left one in Swarm, it's worth 10 seconds to check yours.

Scan your own repo (free, 10 seconds)

  • In Claude Code: claude plugin marketplace add erabot-ai/erabot-plugin then /erabot:scan — it runs on your own agent and reads your real loop and config, where the big money actually hides.
  • In ChatGPT / anywhere: paste a snippet or a public repo URL and ask erabot to scan it.

Free gives you the diagnosis. The verified audit proves each swap holds quality on your real traffic (shadow-tested) and ships apply-ready diffs → erabot.ai.


How erabot stays honest: every assumed number is marked assumed, and a cheaper-model swap is always a candidate until it's shadow-tested against your real traffic — never claimed as "proven" from static code. We'd rather be trusted than impressive.

EB
erabot.ai

Scan any codebase. See exactly what you spend on AI. Get one-click fixes.

No spam. Only launch updates and cost optimization tips.

Product

  • Product
  • CLI
  • Pricing
  • Changelog
  • Docs

Resources

  • Example audit
  • Blog
  • Case studies
  • Eval

Company

  • Security
  • Privacy
  • Terms
  • Support

© 2026 erabot.ai. All rights reserved.