Rendered at 11:02:41 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jjcm 7 hours ago [-]
I'm still sad that we haven't seen a new Taalas style chip a la https://chatjimmy.ai/. Smaller models are good enough now to make that insane burst of tokens so useful.
pil0u 2 hours ago [-]
I don't know the model behind this, but it is absurdly bad.
> Write me a coherent paragraph in French, without ever using the letter "e".
> Voilà une phrase claire et concise : "Le village est situé dans les montagnes. Le soleil est haut. Il y a des animaux dans le village. Il pleut dans les montagnes."
I suppose this is just a demo of how fast an LLM can be, I wonder if there are tradeoffs with larger/smarter models. Also, for a human usage, at what point are tokens generated fast enough that it's pretty much instant? My bet is below 1000 tps
42 minutes ago [-]
PetahNZ 2 hours ago [-]
Its Llama 3.1 8B, a very old/small model.
cyanydeez 25 minutes ago [-]
for comparison, Qwen3.8-Flash-Next only requires 6B parameters for computation, but stores 125B, 51B of those can be comfortably offloaded as they're not actively used in decode but a single token look up.
The Quant iQ4 of this model loads, then, in ~60GB of vram, and on disk it's 85GB.
So if you could etch it, you'd need a ~25GB ssd chip and 60GB of vram.
The vram costs likely contributed to these things being out of reach of the current economic cycle.
fransje26 1 hours ago [-]
Then again, good luck writing a coherent paragraph in French without an "e". :-)
38 minutes ago [-]
winwang 6 hours ago [-]
Pretty sure context is in SRAM, and then you have that as a blocker for tasks.
electroglyph 3 hours ago [-]
way less of a blocker these days due to sparse attention...
bearjaws 12 hours ago [-]
If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.
I've used it on a few for fun projects and its decent but the speed is crazy to watch.
ford 10 hours ago [-]
Also kimi 2.6 at 1000tps (as of may), though when we reached out they had a >12 month waitlist and minimum 7-8 figure annual token spend.
7-8 figures annual spend will buy a hell of a lot of capable local inference hardware you can own, though it won't be at the absurd token/s rate, you'll be able to run almost anything on it... And it'll still have a good residual resale value after 4 years the way things are going now.
podocarp 3 hours ago [-]
Feel like you could spend 6 figures building out a team and the rest renting compute for a whole year, and get the team to create a local inference solution with that kind of budget…
sharktheone 10 hours ago [-]
yeah. K2.6 can run on insane speeds. So sad that they don't have K3 yet.
But it can apparently also run 5.6 Sol
eli 9 hours ago [-]
Yeah but at “call to discuss pricing” rates
conception 8 hours ago [-]
It was better when they had gemma at 1k. Inco does DS flash at about 600. A few places will do K3 and GLM in the hundreds.
Such a tiny model at that t/s is less impressive than it would have been four months ago.
lostmsu 6 minutes ago [-]
Inco sucks. I tried their GLM 5.3 Flash and it was quantized to the point of hallucinating Chinese in the middle of English only agentic sessions. Never happened with any other provider.
physicallyIllfr 6 hours ago [-]
Lighting my codebase on fire at the speed of light. Like microwaving the spaghetti.
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
sbierwagen 5 hours ago [-]
Human code review? What is this, 2025? The modality today is write with one LLM, review by a different one, (important: two different model families will catch errors one series won't) then deploy right to production.
podocarp 3 hours ago [-]
And if you tell the reviewer the author is a competitors model it becomes extra snarky and vigilant. Then give the review results to the author and tell it it's from the competition and it will also become slightly outraged.
calgoo 3 hours ago [-]
No you dont need to code at 600 w/s BUT at those speeds, you can start doing things like asking multiple different agents the same question and picking the best solution each time without noticing the lag.
RugnirViking 3 hours ago [-]
> Nobody needs to code at 600 words per second.
I do. I used to use haiku for the speed. Now its just as slow as the rest. Speed is my #1 ranking of how good a model is
LoganDark 9 hours ago [-]
Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.
eli 9 hours ago [-]
Which is wild because it does, in fact, do caching
fakwandi_priv 5 hours ago [-]
> There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model.
So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..
shard972 8 hours ago [-]
Yea i had some pretty meh results using gpt-oss-120b it in my evals where it should have benefited speed alot but it really under performed what i was expecting.
scosman 10 hours ago [-]
Or better: Qwen 2.8 27b
RussianCow 10 hours ago [-]
Unfortunately, the lack of an input cache discount makes it prohibitively expensive for most use cases that aren't one-shot prompts.
scosman 7 hours ago [-]
well same applies to GPT OSS 120. Qwen is just the much smarter model of the 2 public options on Cerebras.
kylehotchkiss 6 hours ago [-]
I felt happy I could run it at 100tk/s on my new Mac Studio :')
Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
11 hours ago [-]
RussianCow 10 hours ago [-]
The point is the speed.
_aavaa_ 9 hours ago [-]
If the model provides me with bad results because it's dumb, I don't care how quickly it does it.
RussianCow 5 hours ago [-]
But there are lots of use cases where a relatively "dumb" model is good enough.
hermannj314 8 hours ago [-]
Is it possible to construct a control system where bad, fast and cheap can become good, fast, and cheap through repeated sampling and a strong spec/eval harness?
I am trying to keep an open mind with AI, but I also have little understanding of control theory, trying to learn.
calgoo 3 hours ago [-]
You can, but you need to break the problem into much smaller tasks, then check those answers, and finally have a harness that handles all the context, task breakup, task definitions, and validations each round.
freakynit 7 hours ago [-]
Smaller models seems to get stuck in "loops" when you try to "handle" them this way.
freakynit 8 hours ago [-]
I have tried using Mercury 2.5 for a lot of my tasks.. but this model just isn't there. It seems to be on par with any 14B model at max. Even GPT-OSS-20B performs way better than this in my own attempts to use it.
I really really wanted to use this because it offers incredible speeds and pricing combinations. But nop.. I still am not using it.. not even for basic tasks.
nostrebored 7 hours ago [-]
Try Celeris-magnus-1. We get similar speeds and it’s much closer to qwen 27B dense models.
freakynit 7 hours ago [-]
Didn't knew about this. Thanks.. but, according artificialanalysis.ai, it's intelligence is just about like a 14B model (mistral 3 14b)
nostrebored 5 hours ago [-]
depends on your use case I think. for small, transactional tasks it is quite strong and fast.
nextaccountic 8 hours ago [-]
At some point the bottleneck becomes tool calling.. and as such, it's preferably if the model is co-hosted (in the same datacenter, at least) with your code repository and all other reference/context it needs (full documentation for most ecosystems, maybe even a copy of common crawl to minimize web fetch usage, etc)
big-chungus4 4 hours ago [-]
My 1k parameter RNN is faster
nylonstrung 11 hours ago [-]
I honestly think the diffusion LLM approach is a dead end
It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models
Still unclear for what, if any use cases this is pareto frontier
vineyardmike 8 hours ago [-]
From what I’ve heard, the issue is more that it’s harder to efficiently share the hardware across diffusion requests, so it’s more expensive to serve.
It sounds like there might be opportunities for local models (not open weight, but actually locally run) to use diffusion for faster responses on weaker hardware that doesn’t need to be shared.
But yea, it’s still a red-ish flag that big labs haven’t invested much in it. I could see Google/Apple getting value of this sort of local model, but maybe there’s enough research behind traditional models that it’s not worth the distraction at this point in time.
LarsDu88 11 hours ago [-]
You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
clhodapp 10 hours ago [-]
Personally, I think it's more that text diffusion is not the ideal driver of an agentic work loop than that text diffusion is a total dead end. I am still hoping to see how it does on authoring and editing with further scaling and optimization. I think the push for AGI has put a bit too much focus on the idea of one general model doing everything.
kennywinker 9 hours ago [-]
DiffusionGemma was released alongside the other gemma-4 models just a few months ago, so clearly google hasn't abandoned the idea.
K2-Horizon-7B has a diffusion and non-diffusion variant, and they claim the same level of intelligence from both models.
seduerr 7 hours ago [-]
Cerebras is fast…?
Retro_Dev 7 hours ago [-]
Yep it's fast alright. I wonder how much faster Mercury would become if hosted on Cerebras.
low_tech_punk 10 hours ago [-]
it's stupid fast!
SwellJoe 10 hours ago [-]
It's stupid and it's fast.
freakynit 7 hours ago [-]
Couldn't have said it better.
verdverm 6 hours ago [-]
but can it be said in fewer tokens than GGP commenter?
supercoder88 8 hours ago [-]
[dead]
sharktheone 10 hours ago [-]
this feels like "we got the same benches as gpt-oss-120b but are also potentially slower while saying it is great"
whalesalad 8 hours ago [-]
I used this a few days ago and thought something must be wrong with how fast it was responding. "Mercury 2.5 is below average in intelligence, but well priced when comparing to other models of similar price." this is so funny. So when you have a stupid model that is fast - what do you use it for?
entrope 9 hours ago [-]
> Mercury 2.5 is below average in intelligence, but well priced when comparing to other models of similar price.
Well priced when compared to other models of similar price, eh?
Are we allowed to call this slop, even if the output is not directly from an LLM?
verdverm 6 hours ago [-]
The LLMs learned it somewhere...
pushpendraw 5 hours ago [-]
[dead]
rvz 13 hours ago [-]
The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.
copperx 12 hours ago [-]
Ah, the old "good, fast, or cheap; pick two" proves true once again.
downrightmike 12 hours ago [-]
Give it a few months.
timClicks 10 hours ago [-]
It means something, because it an iterative workflow. If you're willing to burn tokens, it's possible for weaker models to implement tasks by incrementally improving drafts.
glouwbug 12 hours ago [-]
Some of us want fast food
voiceeh 11 hours ago [-]
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.
hansvm 11 hours ago [-]
If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable?
> Write me a coherent paragraph in French, without ever using the letter "e".
> Voilà une phrase claire et concise : "Le village est situé dans les montagnes. Le soleil est haut. Il y a des animaux dans le village. Il pleut dans les montagnes."
I suppose this is just a demo of how fast an LLM can be, I wonder if there are tradeoffs with larger/smarter models. Also, for a human usage, at what point are tokens generated fast enough that it's pretty much instant? My bet is below 1000 tps
The Quant iQ4 of this model loads, then, in ~60GB of vram, and on disk it's 85GB.
So if you could etch it, you'd need a ~25GB ssd chip and 60GB of vram.
The vram costs likely contributed to these things being out of reach of the current economic cycle.
I've used it on a few for fun projects and its decent but the speed is crazy to watch.
[0] https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise
But it can apparently also run 5.6 Sol
Such a tiny model at that t/s is less impressive than it would have been four months ago.
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
I do. I used to use haiku for the speed. Now its just as slow as the rest. Speed is my #1 ranking of how good a model is
So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..
https://kamilstanuch.github.io/LLM-token-generation-simulato...
I am trying to keep an open mind with AI, but I also have little understanding of control theory, trying to learn.
I really really wanted to use this because it offers incredible speeds and pricing combinations. But nop.. I still am not using it.. not even for basic tasks.
It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models
Still unclear for what, if any use cases this is pareto frontier
It sounds like there might be opportunities for local models (not open weight, but actually locally run) to use diffusion for faster responses on weaker hardware that doesn’t need to be shared.
But yea, it’s still a red-ish flag that big labs haven’t invested much in it. I could see Google/Apple getting value of this sort of local model, but maybe there’s enough research behind traditional models that it’s not worth the distraction at this point in time.
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
K2-Horizon-7B has a diffusion and non-diffusion variant, and they claim the same level of intelligence from both models.
Well priced when compared to other models of similar price, eh?
Are we allowed to call this slop, even if the output is not directly from an LLM?