Mixed
the speed from the cards we have now can blaze at 130-200 tokens per second of generation with q5 and a full context in fp16
alexellisuk on Hacker News
130-200 tokens per second
In the thread “Local Qwen isn't a worse Opus, it's a different tool”, replying to this comment
Helped: Dense models provide faster generation speeds than MoEs.
Got in the way: MoE quality is not as good as dense models.
The important thing about MoEs which I mention in the conclusion is that they carry fewer (way fewer) active tokens during inference/generation. 35B-A3B is what we started out with in the days of only having the 3090, but the quality is not as good, and the speed from the cards we have now can blaze at 130-200 tokens per second of generation with q5 and a full context in fp16. Not to say that MoEs don't have their place. …