Discussion about this post

User's avatar
Caleb's avatar

I think it's still something of an open question whether or not there's a material advantage to running LLMs locally, but my initial thought here would be no for a few reasons.

First, it's a computationally- and data-intensive computation with utterly trivial network requirements. The query/response is simple plain-text, with the data requirements being the model itself and whatever supporting data may be desirable to flesh out the results or enable further computation. Why shell out top-dollar to compete with tech company wallets for constrained hardware supply when you can pay a subscription fee? Other than the privacy of your question/response, you're not gaining much here and paying a lot of money for it.

Second, running LLM computations on the edge means moving the models themselves to the edge. We're still in the early days, so the models/weights are quite valuable and expensive to produce. If there's very little practical incentive to run the models locally to begin with (see above), why risk something taking the model and running when you can instead just host the model in the cloud?

Third, we're still largely in the "hope" phase of AI/LLMs. There's a lot of hope that this will generate real economic return commensurate with the cost, but I'm not sure to what extent this return has been realized. In my experience it's a net-productivity improvement, but for newer or more junior people I can almost see that reversing since they're not in as good of a position to judge when they're getting a bad answer.

TLDR, I think we're still in the "hope" phase of the technology itself, with a lot of hype and still a mixed bag on real-world return. Added to that, the incentives to push this compute to the edge seems weak at best and likely there's more incentive to keep it off the edge for a while from a data-value perspective. There might be a time for this trade, but I think you're early.

Expand full comment
David C Reutter's avatar

AI is like the oh boy moment, I can crunch a spreadsheet on my PC. Did nothing for productivity. And costs are a lot lower, considering that first computer might have cost you a grand in 1990's dollars. AI is going to start a data stampede. Ask Bing a question and he goes to the same sources you already knew. Its' faster (and less critical). Ultimately are you better at picking stocks (or race horses - James Quinn wrote a book on that). So if you thought information overload was a problem, get ready. Bing will help you sort through the data? (see lack of critical parsing of sources), same old garbage in garbage out admonitions. And Bill Gates will get rich all over again.

Expand full comment
25 more comments...

No posts