I am still pretty skeptical of large language models. I went pretty deep with them a couple of years ago, and most of what I found was either disappointing (in regard to LLM performance and capabilities) or annoying (in regard to all the marketing blather). In the ensuing months, I tended to discount the whole effort as just hype being inflated so startup founders could pump up their stock valuations and dump their holdings to get rich quick.
I think some of that remains true. But I also try to take a look at what’s going on with LLM development, because even with all the marketing nonsense and overpromising, there really is something going on with the technology.
Just taking a look at the LLM field is getting pretty difficult. It’s not that hard to spin up a neural net (you can try it yourself with open-source code; I did). What gets more difficult is training. But it’s not that difficult, and if you have a look at the Chatbot Arena, they have a leaderboard listing nearly 200 LLMs — and those are just the good ones.
Anyway, AI technology hasn’t been standing still, and training expense, resource overuse, and results (in many areas) have all improved. I’ve found you have to be pretty discriminating about which LLM you use for a given task. I’ve finally found a couple of productive ways to use an LLM for programming assistance, but only with Cody (from SourceGraph) and Claude (from Anthropic).
In my experience, LLMs can write code, but they’re a lot better at reading code and suggesting improvements. It reminds me of coding with a partner, where you might write a function and ask your colleague to have a look and find problems you missed. It’s a lot like writing and editing; it’s far easier to edit someone else’s work than your own. The other big win, at least for me, has to do with Python. I get the most out of Python when I use the (kajillions of) libraries with incredibly well-done functions already finished. My problem is that I just don’t know about all of them and what’s in them. But the LLMs do (or at least they know a lot more of them than I do). This has saved me many hours of writing code that somebody else has already done and tested, and others have used.
That, by the way, is what I’ve found to be the best thing about Python: the whole experience of using the language feels more like a community effort than a lone programmer figuring out one little step at a time. Many, many years ago I did some programming and “lone programmer” thing was what propelled me into different pursuits. I knew there were plenty of people producing, for example, C code that would apply to what I was working on, but at the time there wasn’t an easy way to find or access their work — even though, generally speaking, programmers are very generous with the code they write.
I wonder if that’s a hidden benefit of LLMs — that by being better, more engaging search engines, they can help us create better communities of practice. Guilds, but without the exclusionary parts.
Leave a Reply