I've always wondered how it's possible. No way they've got some crazy software optimisations that nobody else can replicate right? They've gotta just be throwing a ridiculous amount of compute power at every request?

[–] webghost0101@lemmy.fmhy.ml 4 points 1 year ago* (last edited 1 year ago)

Well there are 2 things.

First there is speed for which they do indeed rely on multiple thousands of super high end industrial Nvidia gpus. And since the 10Billion investment from microsoft they likely expanded that capacity. I’ve read somewhere that chatgpt costs about 700,000 a day to keep running.

There are a few others tricks and caveats here though. Like decreasing the quality of the output when there is high load.

For that quality of output they do deserve a lot of credit cause they train the models really well and continuously manage to improve their systems to create even higher qualitive and creative outputs.

I dont think gpt4 is the biggest model that is out there but it does appear to be the best that is available.

I can run a small llm at home that is much much faster then chatgpt.. that is if i want to generate some unintelligent nonsense.

Likewise there might be a way to redesign gpt-4 to run on consumer graphics card with high quality output… if you don’t mind waiting a week for a single character to be generated.

I actually think some of the open sourced local runnable llms like llama, vicuna and orca are much more impressive if you judge them on quality vs power requirement.