Speed that changes what you can build · Cerebras demo

Setup

One key and a compatible endpoint.

The notebook uses the same openai library as the workshops. Only the service address and the key change. On connecting, it lists the available models and picks gpt-oss-120b.

The setup cell lists the gpt-oss-120b and qwen-3.8-27b models and picks gpt-oss-120b — Two models available on the account. We used gpt-oss-120b in every test.

Test 1

A full document in 2.26 seconds.

We asked the model for a 90-day plan for a customer service team to adopt generative AI: diagnosis, use cases, risks, metrics and a calendar. The notebook counts the tokens as they arrive.

The first token arrived at 0.35 seconds. From there it wrote 3,006 tokens in 1.92 seconds: 1,568 tokens per second. The full document, counting that initial wait, took 2.26 seconds.

Test 1 card: 1,568 tokens per second, 3,006 tokens in 1.92 seconds, first token at 0.35 seconds — The card measures from the first token. The counter is not an animation: it is the speed at which the tokens arrived.

Test 2

24 customer messages, classified in 5.5 seconds.

A contact center never stops receiving messages. The notebook classifies 24 at once by intent and urgency, eight in parallel, and suggests the next action for each one.

The whole batch took 5.5 seconds: 0.23 seconds per message on average.

Burst: 24 messages in 5.5 seconds, each with intent, urgency and a suggested action — Each message comes back with intent, urgency and a suggested action.

What we learned

The same cell took 363.6 seconds on October 27. In that run each agent draft came back in 1 or 2 seconds, but some steps took almost a minute. That points to queueing or the account's rate limit, not to the model. The speed of one call does not guarantee the speed of a batch: how much the provider lets you ask for counts too.

Test 3

An agent that corrects itself three times, in 6.4 seconds.

The case: a customer is upset because her car loan payment was charged twice. The agent writes a reply, critiques it, fixes it, and repeats the cycle three times. That is seven chained calls.

Each step took between 0.54 and 1.39 seconds. All seven together, 6.4 seconds.

Four drafts and three critiques, each with its time, and the total: 7 chained calls in 6.4 seconds — We trimmed the text of each step so the times show. The notebook shows every draft in full.

What we learned

With slow inference, an agent that reviews itself is expensive and people lose patience. When each call takes a second, reviewing three times stops being a luxury. Speed changes which agent designs are reasonable.

Test 4

The audience question.

At the event, the audience writes the question. In this run we used the one the notebook ships with: why inference speed changes the cost of an AI agent, in five points for a chief executive.

Audience question: 1,401 tokens per second, 697 tokens in 0.50 seconds — 697 tokens in 0.50 seconds from the first token, which arrived at 0.22 seconds. 0.72 seconds in total.

Summary

What the notebook measured.

The last cell gathers the four tests. Total time includes the wait for the first token and the network latency of the place where it ran.

Long document: 3,006 tokens in 2.26 seconds.
Burst of 24 classifications: 4,890 tokens in 5.45 seconds.
7-step agent: 6,446 tokens in 6.38 seconds.
Audience question: 697 tokens in 0.72 seconds.

Summary table with tokens per second, tokens and total time for the four tests — Results depend on the network, the model and the provider's load at that moment.

In short

Four ideas to take with you.

A fast call changes the experience: a 3,000-token document arrives in a couple of seconds.
A fast batch also depends on how much the provider lets you ask for. Measure both.
With fast inference, an agent can review itself several times without anyone noticing.
Measure on your network and with your load. A number from an event is not your production number.

Run the notebook with your key →

Speed that changes what you can build.