Jev’s (almost) free lunch: Jev vs. GPT-5.4 vs. Claude Opus 4.7 on real phishing emails

TypeSafe’s new model architecture shows similar detection rate as Claude Opus 4.7, at 1/190 the cost. We gave four models the same 232 handpicked, real emails and the same question. Here is what Jev gets wrong, mostly financial emails and how to fix it for under a third of price of running Opus on everything.

Jev is TypeSafe’s first System One model: instead of generating text, it answers typed questions (yes or no, pick one, rate on a scale) with probabilities that a system can act on directly. TypeSafe claims Jev is “193.6x Faster, 444.6x Cheaper” than LLMs “based on workflows for System One tasks”. We wanted to know what that looks like on a real job, and whether the accuracy holds up.

Note: Jev is a new architecture that got released just last week. We will update our approach as we refine its usage in our detection engine.

TL;DR

• We gave the same JSON description of each email as input and the same question: should this be flagged as phishing, a scam, malicious, or deceptive junk? The models were Jev, GPT-4o, GPT-5.4 and Claude Opus 4.7. We added GPT-4o to the mix as we wanted baseline numbers from an older model (2024).
• The test set is 232 real emails: 64 that should be flagged (45 malicious, 19 spam) and 166 that should be delivered, plus 2 we excluded.
• Catching threats, Jev is close to the best model. Of 45 malicious emails: Opus 4.7 caught 44, Jev 43, GPT-5.4 41, GPT-4o 38.
• False alarms are where Jev pays. On 166 legitimate emails, Jev at a 0.5 cut-off wrongly flagged 21 (12.7%). GPT-4o flagged 11. Opus flagged 1. GPT-5.4 flagged none. Jev’s false alarms were almost all bank alerts, tax notices and confirmation codes: real email that asks you to sign in or act now.
• Where Jev beats Opus: speed, cost and control. 134 ms against 1,350 ms. $72 per million emails against $13,500. And a probability you can set a threshold on, where Opus gives one bit.
• Where Opus beats Jev: judgement. It tells a real TurboTax “ACTION REQUIRED” from a fake one, flags empty emails with a suspicious attachment from the metadata alone, labels threats correctly, and can explain itself.
• Use both, and the false alarms go away. Every one of Jev’s 21 false alarms scored under 0.85. Send Opus everything Jev scores between 0.3 and 0.85, plus anything with an attachment and almost no text. That is 29% of mail. It catches 61 of 64 with 1 false alarm, matching Opus on malicious email (44 of 45), at 30% of the cost. We set those rules on the first 82 emails and then added 150 more; they held.
• Ask the direct question. We first built Jev a battery of narrow questions with hand-set weights. It did worse than asking outright. Narrow questions are for explaining a verdict, not reaching one.

The Test Setup

We created a dataset with a mix of malicious, legitimate, and marketing emails with the following breakdown:

What each model reads

All four models receive byte-for-byte the same JSON and are asked the same flag question. None of them gets header analysis: no authentication results and no routing headers. They see who the sender claims to be, the sender’s domain, whether that domain is free webmail, and the Reply-To address. Code unwraps redirectors first, so a link written as google.com/url?q=https://evil.example is presented by where it lands.

This is the actual JSON for a synthetic test phish in our set:

Results

Two stories are in this table.

On threats, Jev is nearly as good as the best model. At 0.5 it caught 43 of 45 malicious emails, one fewer than Opus and more than GPT-5.4 or GPT-4o. Its only two misses are near-empty messages from throwaway Gmail accounts with a fake invoice attached as a PDF or an image. There was nothing to read.

On legitimate mail, Jev raises too many false alarms to run alone. At 0.5 it flagged 21 of 166 real emails. GPT-5.4 flagged none and Opus flagged one. Jev’s false alarms are not random. Seventeen came from the 148 newly added legitimate emails, and nearly all of them are the same kind of message:

These are real. They also do exactly what phishing does: name a bank or a tax service, warn that something needs attention, and ask the reader to sign in. GPT-4o flagged six of the same emails. GPT-5.4 and Opus passed all of them.

Jev against Opus: what to take away

Opus 4.7 was the most accurate model we tested and Jev the fastest and cheapest, so this is the comparison that matters for a design decision.

Where Jev outshines Opus

1. It is fast and cheap enough to sit in front of every email. Jev answered in 134 ms at the median and 249 ms at the 95th percentile. Opus took 1,350 ms and 2,271 ms. Jev costs $72 per million emails; Opus costs $13,500, about 190 times more. For that, Jev caught 43 of the 45 malicious emails to Opus’s 44. In practice: a check that runs before delivery, on all mail, has to be Jev-shaped. At Opus’s latency and price you start choosing which emails deserve a look.

2. It gives you a dial, and the dial holds still. Opus returns true or false, although it can be prompted to return probabilities. Jev returns a probability that ranked our flag-worthy mail above our deliverable mail with an AUC of 0.95. 59 of 64 caught with 21 false alarms at 0.5, 53 with 8 at 0.6, 48 with 1 at 0.8, 43 with none at 0.85. In practice: you can set one threshold for a warning banner and a stricter one for quarantine, and you can see which emails are uncertain. To move an LLM’s operating point you rewrite its prompt and re-test.

3. Its mistakes announce themselves. All 21 of Jev’s false alarms scored between 0.5 and 0.80. Nothing legitimate scored 0.85 or higher. On the one malicious email Opus got wrong, the RFP phish below, Opus answered “clean correspondence” with no hedge, and Jev answered 0.51. In practice: “I don’t know” is an instruction to escalate. A confident wrong answer is not, and Jev gave very few of those.

Where Opus outshines Jev

1. Judgement on the hard calls. Opus raised 1 false alarm on 166 legitimate emails. Jev raised 21. The gap is entirely in mail that looks like phishing and is not: a TurboTax notice headed “ACTION REQUIRED”, a bank’s “check your balance now”, a registrar’s confirmation code, a class-action settlement notice. Opus passed every one. It also caught all 19 spam emails; Jev caught 16 and scored two cold-outreach pitches at 0.11. In practice: Jev reads surface features, and a real bank alert has the same surface as a fake one. Telling them apart takes knowing that the sender, the links and the request all fit together, and that is where a second opinion pays.

2. It reasons about what is missing. Four malicious emails in our first batch have an empty or near-empty body and the scam inside an attached image or PDF. No model could see the attachment. Opus flagged all four anyway, from a throwaway sender, no text, and a file. Jev flagged two (0.85 and 0.58) and missed two (0.46 and 0.22). In practice: Jev judges the text it is given. If your threats arrive as pictures, put OCR in front of it or route those emails elsewhere.

3. It names the threat and can say why. Asked for a category, Opus put 60 of the first batch’s 62 flag-worthy emails in a threat category. Jev managed 39, because “what is the purpose of this message?” is a question about form, and phishing is built to have a benign form: Jev called a fake Comcast voicemail a notification at 0.78 and a Costco reward scam marketing at 0.65, while flagging both. And Opus can write the sentence a user reads. Jev returns numbers; to explain a verdict you have to ask it a separate set of narrow questions and assemble the reason yourself. In practice: if the output is shown to a person, an LLM has to be somewhere in the loop.

Running both: how the two-model design works

The idea is to spend Opus only where it changes the answer. An important note here is to use discretion on what emails to send to Jev. In our tests, we did not use any sensitive or personal emails. Every other email takes the following path:

1. Parse in code. Extract the sender, links and attachment names, unwrap redirect links, and pull text and links out of attachments where you can. This runs first because neither model can flag what it is not shown.
2. Ask Jev the flag question. About 150 ms and $0.00007.
3. Route on Jev’s probability.
   • 0.85 or higher: flag it. Jev is sure. In our set nothing legitimate scored this high.
   • Below 0.3: deliver it. Jev is sure the other way.
   • Between 0.3 and 0.85: ask Opus, and take Opus’s answer.
4. One override rule: an attachment with almost no text goes to Opus whatever Jev said. We used “under 40 words of body text.” Jev judges the text it is given, and on an empty email with a PDF attached it has nothing to judge. Opus reasons from what is missing.
5. Explain only what gets flagged. Users read the reason, and Jev cannot write one. For a flagged email, either take the reason from Opus or ask Jev its narrow questions and assemble one (“claims a brand its sender domain does not match, 0.98”). Delivered mail needs no explanation, so most email never pays for one.

We fixed these rules on the first 82 emails, then added the 150 legitimate ones without changing anything. On the new emails the design raised no false alarms and caught both scams that had been hiding in the folder. All 17 of Jev’s false alarms there fell inside the band, and Opus cleared every one.

And the result, against each model alone:

What it still gets wrong. It misses three emails and raises one false alarm. The false alarm is Opus’s own: a cold sales email we labelled graymail and Opus called spam. The misses are the RFP phish, which nothing text-only can catch, and two cold-outreach spam emails (“I found errors on your website”) that Jev scored a confident 0.11, so they never reached Opus, which would have flagged them. A confidence band cannot rescue a confident mistake. That is why the attachment case is handled by a rule and not by a threshold, and it is the argument for adding more such rules as you learn where your fast model is confidently wrong.

Setting this up on your own mail. Our edges are 0.3 and 0.85 and our rule is 40 words. Yours will differ. Run the fast model over a few hundred labelled emails, sort by probability, and look at where its mistakes sit. Put the band edges just outside the mistakes that are near the middle. For every mistake that is far from the middle, ask what the email has in common with others like it, and write a rule. Then check what share of mail the band captures, because that share is your LLM bill. Include plenty of bank, tax and account-security email in the legitimate half: that is where a fast model’s false alarms come from, and a test set without it will flatter you, as our first one did.

Conclusion

Jev is not a free lunch, but it is a cheap one. On our 232 emails it caught 43 of 45 malicious messages, one fewer than Claude Opus 4.7, in a tenth of the time and for about half a percent of the cost. What it lacks is judgement about legitimate email that looks like phishing: bank alerts, tax notices, confirmation codes. Running alone, it raised 21 false alarms where Opus raised one.

The fix is simple, because Jev is rarely wrong when it is sure. Let it decide the verdicts it is confident about, and send the rest to a model that can reason. That matched Opus on malicious email and on false alarms, for 30% of the price.

Use the cheap check everywhere and the expensive one where it counts.

—