The world got rattled a few days back with the DeepSeek release of its latest model and how they have managed high performing models with significantly low compute requirements.
The stock markets noticed, tech pundits have been having a field day on Linkedin and X.
But what if this is not a one-off and we are getting carried away with the noise around the China angle !
What if, this was start of a Moore’s Law for GenAI :
The GenAI models will need less than half the compute every 18 months !
Image Source:https://semianalysis.com/2023/02/04/a-century-of-moores-law/
But then there is Huang’s Law – the same dollar buys twice the computing power every 18 months !
While Huang and Nvidia may want us to believe the Huang’s Law to be true, but in the world which is getting taken over by the GenAI euphoria, the compute hunger going down for foundational models has more over-arching impact !
I just had this thought flash through and couldnt sleep. Am typing this early morning and also realize that I am surely out of my technical depth when it comes to AI and foundational models.
I also understand that the compute reduction for DeepSeek can probably be attributed to their approach and design of the model(as the popular Whatsapp forward explains). And may be this is a one-off fork in the road rather than a regular outcome in the coming years.
Yet, humor me for a bit and lets run with this thought-experiment:
If we are indeed entering a GenAI Moores Law era, what should we expect the future to hold?
- Significant drop in cost of using/deploying genAI across enterprise and consumer systems. Cost may not be the reason for most application-layer builders to choose between LLMs. Apart from performance bench-marks, how else would one filter/evaluate models?
- Self-hosted use-cases would emerge. Full models may run on a smartphone. Combine that with need for privacy and we may be unlocking a lot of new-use-cases.
- On the other hand, if models do end up running on closed-systems, would the models be starved of the data needed for them to keep improving?
- Stocks of infrastructure providers will drop and correct to incorporate the long term revenue projections. Combined with Huang’s Law, it may be a double whammy !
- What would this mean for SLMs and fine-tuned models?
Maybe this would never happen ! May be it would !
Nonetheless, exciting times ahead !
Leave a Reply