This is dumb. Moore’s law may be mostly dead, but chips are still progressing at an absurd pace. In 6 years you’ll be able to run the o1 model on a raspberry Pi with no internet access.
Maybe, but i never mentioned years into the future. Of course technology will improve. The hardware will get better and more effcient, and the algorithms and techniques will improve.
But as it stands now, i still think what i said is true. We obviously don’t have exact numbers, so i can only speculate.
Having lots of memory is a big part of inference, so I was going to reply to you that prices of memory stopped going down at a similar historical rate, but i found this, which is interesting
The cost when down by about 0.1x from 2000 to 2010. 2010-2020 it was only about 0.23x. 2020-2023 shows roughly another halving of the price, which is still a pretty good rate.
The available memory is still only one part. The speed of the memory and the compute connected to it also plays a big part in how these current systems work.
There’s absolutely no doubt that lower-end models are going to keep improving and that inference will keep getting cheaper. It won’t be on a Raspberry but my money’s with you. In 6 years you’ll be able to buy some cheap-ish specialized hardware to run open models on and they’re gonna be at least as capable as today’s frontier models while burning a fraction of the energy.
In fact i wouldn’t be surprised if frontier models were somehow overtaken by vastly cheaper models in the long run. The whole “trillion parameter count” paradigm feels very hacky and ripe for radical simplification. And wouldn’t it be hilarious ? All those suckers spending billions building a moat only to see it swept under their feet.
This is dumb. Moore’s law may be mostly dead, but chips are still progressing at an absurd pace. In 6 years you’ll be able to run the o1 model on a raspberry Pi with no internet access.
lmaoooo
Maybe, but i never mentioned years into the future. Of course technology will improve. The hardware will get better and more effcient, and the algorithms and techniques will improve.
But as it stands now, i still think what i said is true. We obviously don’t have exact numbers, so i can only speculate.
Having lots of memory is a big part of inference, so I was going to reply to you that prices of memory stopped going down at a similar historical rate, but i found this, which is interesting
https://ourworldindata.org/grapher/historical-cost-of-computer-memory-and-storage?time=2020..latest
The cost when down by about 0.1x from 2000 to 2010. 2010-2020 it was only about 0.23x. 2020-2023 shows roughly another halving of the price, which is still a pretty good rate.
The available memory is still only one part. The speed of the memory and the compute connected to it also plays a big part in how these current systems work.
There’s absolutely no doubt that lower-end models are going to keep improving and that inference will keep getting cheaper. It won’t be on a Raspberry but my money’s with you. In 6 years you’ll be able to buy some cheap-ish specialized hardware to run open models on and they’re gonna be at least as capable as today’s frontier models while burning a fraction of the energy.
In fact i wouldn’t be surprised if frontier models were somehow overtaken by vastly cheaper models in the long run. The whole “trillion parameter count” paradigm feels very hacky and ripe for radical simplification. And wouldn’t it be hilarious ? All those suckers spending billions building a moat only to see it swept under their feet.