One Benchmark Row and a Median at the Ceiling: Reading OrcaCyber Zero 1.0

The OrcaCyber Zero 1.0 card carries exactly one benchmark row: CyberGym Level 1 pass@1 at 98.07, tagged `orcarouter` as its source [OURS]. Everything else on the page is four FAQ entries and a price row — there are no prose sections at all. That makes this the thinnest card in the batch, and the orcacyber zero 1.0 query that lands on it is answered by a page with almost nothing to read. The card itself is served by OrcaRouter.

What one row can and cannot say

A single benchmark row is not a summary. It is one measurement, and on this card it is a measurement of one thing: whether the model can find and exploit a vulnerability in a controlled environment, at the easiest difficulty tier that CyberGym defines.

98.07 is a high number. It is also a number that says nothing about the other things a security model is asked to do — analysing a patch, writing a detection rule, triaging a report, or refusing a request it should refuse. The card has no row for any of them.

Two readings are available, and only one of them is useful.

The charitable reading: this is a specialist card, and specialists publish the one number their buyer cares about. A 98.07 on CyberGym Level 1 is the pass rate a red-team buyer wants to see before anything else.

The sceptical reading: a card with one self-published row and no sections has not been evaluated, only asserted. The row’s source tag is `orcarouter`, which means the number comes from us rather than an independent harness.

Both are true at once, and the practical consequence is the same either way: the row is a starting claim, not an assessment. Anyone routing security work to this card should reproduce a task of their own against it before trusting the 98.07 to generalise.

A cache read priced at a quarter

The rate row is more informative than the benchmark row, because it is comparable to fifty other cards at once.

Line Rate per million Against input
Input $3.00 100%
Output $7.50 250%
Cached read $0.75 25%

 

A 25 percent cache read is unusual. Most cards in this catalogue price cached input at 10 percent of the input rate; a smaller group sits at 20 percent; this one sits at a quarter [OURS]. No cache-write rate is published at all, so a strategy that budgets a write premium has nothing to budget against.

The output-to-input ratio is 2.5×, which is low for a frontier-priced card. Reading is expensive relative to writing here — the opposite of a document-processing model, and consistent with a workload where the prompt is short (a target description, a scope) and the answer is long (an analysis, a proof of concept).

At 100 million input tokens a month, the bill is $300; at the same output volume it is $750 [OURS arithmetic on the published rates]. Cached input at the same volume would be $75.

Every day at the ceiling

Seven-day telemetry reports a median time to first token of 10,000 ms and a p95 of 10,000 ms [OURS]. That is not a measurement; 10,000 ms is the recording ceiling, and a median that equals the p95 at the ceiling means more than half of the sampled days were capped.

The error rate is 2.51 percent — roughly one request in forty — and output speed is 156.8 tokens per second, which is fast for this catalogue. Traffic over the window is 5,770,953,891 tokens.

So the card is doing heavy volume at high throughput, and its starting latency is unmeasured rather than bad. Those are different facts, and the card does not distinguish them: a reader who sees 10,000 ms will assume the model is slow to start, when the honest statement is that the instrument stopped recording.

For a batch workload — a scan, a triage sweep, a nightly run — a second of first-token latency barely registers against a 156 tok/s generation rate. For an interactive console it is the number the operator stares at.

Traffic without a benchmark table

Five point seven seven billion tokens in a week is more than most cards in the catalogue see, and it sits on a page with one row of evaluation and no prose [OURS].

That combination is worth stating plainly because it inverts the usual assumption that cards are thin because nobody uses the model. Here the model is heavily used and the card is thin because the vendor chose to publish one number.

Which means the card is not a guide to the model. It is a rate sheet with a claim attached.

The takeaway

OrcaCyber Zero 1.0 carries one benchmark row — CyberGym Level 1 pass@1 at 98.07, self-reported — and four FAQs, against $3.00 input, $7.50 output and $0.75 cached per million tokens [OURS].

The rate row is the reliable part: a 25 percent cache read, a low 2.5× output ratio, and no cache-write rate at all. The latency is unmeasured — the median sits on the recording ceiling — and the benchmark is one claim about one task, which is the right thing to check first and the wrong thing to generalise from.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top