We ran erabot on OpenAI's own Swarm — it extracts names with a model that costs ~65× too much
First in a series where we scan well-known AI repos for wasted LLM spend and show the fix. Every number here is reproducible — run the scan yourself in 10 seconds (how, at the bottom).
The one-liner
OpenAI's Swarm — their experimental multi-agent framework — has an eval step that extracts names from a sentence using gpt-4-turbo-preview. That's a legacy model whose input tokens cost ~65× more than gpt-4o-mini, which handles name extraction just as well. It's a one-word fix.
What we ran
Scan this public repo for wasted LLM spend: https://github.com/openai/swarm
That's it — erabot's free repo scan. It found 11 LLM call sites (test/mock files excluded), and flagged this one as the top line item (high confidence on the model, because it's a plain string we could resolve).
The finding
examples/customer_service_streaming/src/evals/eval_function.py:
def name(self):
extract_name_prompt = "You will be provided with a sentence. Your goal is to extract the full names you see in the sentence. Return the names as an array of strings."
response = self.plan['step'][-1]
completion_result = self.client.chat.completions.create(
model="gpt-4-turbo-preview", # <-- legacy, expensive, for a trivial task
max_tokens=100,
temperature=0,
messages=[
{"role": "system", "content": extract_name_prompt},
{"role": "user", "content": f"SENTENCE:\n{response}"}]
)
Pulling full names out of a sentence is about the simplest NLP task there is. gpt-4-turbo-preview is a 2024-era model that OpenAI itself superseded with gpt-4o — which is cheaper and better. Using it here is the classic "wrong model for the job" leak: a model string that got written once and never revisited.
The numbers — honestly
Per 1M tokens:
| Model | Input | Output |
|---|---|---|
gpt-4-turbo-preview (in the code) | $10.00 | $30.00 |
gpt-4o (modern, like-for-like) | $2.50 | $10.00 |
gpt-4o-mini (plenty for extraction) | $0.15 | $0.60 |
So the fix is worth, per call:
- →
gpt-4o: ~70% cheaper, and a better model. - →
gpt-4o-mini: ~98% cheaper (input is 65× less; output 50× less), with no quality loss on a task this simple.
The honest part: the "$/month" depends entirely on how often this runs, which static code can't know — so erabot marked it volume-assumed and estimated ~$9.60/mo for this call on a default volume. On Swarm (example code) the real number is small. The point isn't the $10 — it's the pattern. This exact line, copy-pasted into a production agent that extracts names on every request, is where it quietly becomes thousands a month. The percentage above is exact; the volume is yours.
The fix
One word:
model="gpt-4o-mini" # name extraction; ~98% cheaper, same result
Candidate swap — as always, verify it holds quality on your real inputs before you ship it. (On name extraction, it will. On something subtler, that's exactly what shadow-testing is for.)
Why this is the most common leak we find
After "re-sending the same context uncached in a loop," a frontier/legacy model doing a simple job is the #1 thing we see. Models get cheaper and better every few months; the model string in your code doesn't update itself. Classification, routing, extraction, yes/no, short labels — these almost never need a frontier model, and the price gap is often 20–100×.
If OpenAI left one in Swarm, it's worth 10 seconds to check yours.
Scan your own repo (free, 10 seconds)
- In Claude Code:
claude plugin marketplace add erabot-ai/erabot-pluginthen/erabot:scan— it runs on your own agent and reads your real loop and config, where the big money actually hides. - In ChatGPT / anywhere: paste a snippet or a public repo URL and ask erabot to scan it.
Free gives you the diagnosis. The verified audit proves each swap holds quality on your real traffic (shadow-tested) and ships apply-ready diffs → erabot.ai.
How erabot stays honest: every assumed number is marked assumed, and a cheaper-model swap is always a candidate until it's shadow-tested against your real traffic — never claimed as "proven" from static code. We'd rather be trusted than impressive.