Yes
I was surprised that even the quantized 7b version easily chomped my 3090's vram - the context takes a lot
eurekin on Hacker News
60 T/s
In the thread “Show HN: I built an open source AI video search engine to learn more about AI”, replying to this comment
Helped: Low hallucination impact on meeting notes use case.
Got in the way: High VRAM consumption for 7b model.
I'm using the tiny mistral 7b for those 1hr long transcripts internal in my company. I was surprised that even the quantized 7b version easily chomped my 3090's vram - the context takes a lot. I think it goes up to 32k tokens (I go up to 20k). …