Something to See Here
Last month, I wrote up some of my thoughts on ChatGPT. That post was a reaction to other opinions I’d seen, namely, the “it’s so over” and the “machines have become sentient” takes, which I strongly disagree with. However, I also don’t want to be lumped in with the “nothing to see here” crowd, who are also obviously wrong.
Clearly, there is something here. The question is what that something is, and I continue to disagree with the idea that this is the birth of some, new machine form of life. In contrast, I think what we are seeing is that GPT[x] has unlocked human intelligence that was previously unavailable to us.
By that, I do not simply mean that, since GPT-4 was created by humans, we deserve the credit. Neither am I referring to the widespread tendency of humans to hide the helping hand they give to AI when it solves problems — leading it through a maze like a kitten following laser pointer and then telling us how smart the kitten was to have solved the maze “all on its own”. I mean, I agree with both points, but that’s not what I meant about unlocking human intelligence.
To explain what I mean, we need a little background.
What We Know LLMs Can do
Rather than speculating about what LLMs might be capable of, let’s focus on what we know can be achieved by deep learning. For that, we can cite someone who wrote the book on Deep Learning and authored one of the most widely used tools for doing it:
Deep Learning models (like LLMs) can successfully interpolate between examples they have seen in order to make good guesses for values on examples that lie in between those.
In all likelihood, GPT-4 was trained on a data set hundreds of times larger than Wikipedia. That training set contains documents in which human beings explained how they solved all manner of problems, a vast treasure trove of human knowledge.
Previously, the only way that we could access that information was via search. Despite how much better Google is than what came before it, Google is still giving us only a glimpse, I think, of what is out there because it requires us to guess what words will appear in a document with the answer. If we guess the wrong words, Google won’t show it to us, and if we guess the right words, it still may not show it to us because there are many more “popular” web sites that also contain those words. As a result, the answer we are looking for is locked beyond the first page of results, where few of us ever go.
LLMs can gain access to this because they have super-human abilities in at least two aspects: (1) memory and (2) pattern matching. Clearly, they can “remember” far more than any human could. The size of GPT-4 is likely measured in trillions of parameters, which can encode an absurd amount of knowledge (in principle, a non-trivial fraction of the entire internet).
As François Chollet said above, pattern matching is how deep learning words. For pattern matching on text data, while I don’t know for certain, I strongly suspect that GPT-4s ability already strongly exceeds what humans can do. I say that because, usually, once AIs can do something at a level near to human performance, it’s only a blink of an eye before they move beyond it. See, for example, AlphaZero and AlphaGo.
Humans spent the last 25 years recording a significant fraction of all human knowledge on the internet, and GPT-4, I think, has cracked the code on how to interpolate between the data points encoded in those documents. The result is that a significant fraction of all human knowledge that was accessible only far into the depths of the Google search results, is now accessible to anyone in the world via an easy-to-use interface. That is an incredible achievement.
What We (Wrongly) Assume LLMs Can Do
None of us can understand what problems would be easy if we had read a significant fraction of the Internet and had a super-human ability to pattern match on its contents. So instead, when we see GPT-4 answer a question, we assume that it must have done so in the way that humans would, by developing an “understanding” of how things work.
Human understanding does not seem like a good model of how GPT-4 works. None of us would be able to easily produce mathematical proofs but fail at simple arithmetic. None of us would be able to ace hundreds of coding interview questions but then fail on every example created in the last three months. The understanding that you or I would need to develop in order to solve the first one would allow us to solve the second one as well, but it does not with GPT-4. Its abilities are just different from ours.
Even if, some day, GPT-4 does learn how to do arithmetic, its inefficiency in learning that specific task compared to other tasks highlights a difference compared to human learning and reasoning. None of us needed to see a billion examples in order to learn arithmetic.
My goal here is not to make predictions about what LLMs can’t do. I want to highlight the incredible benefit they have given us: making a significant fraction of human knowledge (more than any single person could ever learn) easily available to the average human. That is an incredible gift, even if that is “only” accomplished by pattern-matching.
Where We Go Next
If all problems that are “well solved” (having thousands of examples) can now be answered easily by anyone, that fact will have significant impact on society. Long-term, it means that the only answers worth paying for are answers to questions that are new and novel. In effect, you must be a Ph.D., working on problems on the frontier of knowledge or building new things substantially different from what already exists, to add value.
That would suggest that more people will spend longer in college. Ironically, the largest problems near-term, though, are in college, where it’s never been easier to cheat. Similar issues have arisen before, e.g., when calculators were created. I think education will adapt just fine, even if by the “obvious” method of taking all tests on paper while closely monitored. If the goal of an education (being able to solve new and novel problems) is still valuable, then we will find a way to deliver it.
Similar issues arise for training on the job:
As long as senior knowledge workers are valuable, I think we’ll figure this out. For example, Ben says that “of course, you need to double-check GPTs work” (paraphrase) because it does make mistakes. Hmm… that seems like something a junior knowledge worker could be doing! In fact, that sounds like a good way for junior workers to hone their skills until they are ready to take on the new and novel problems.
As I noted above, human reasoning is different from how machines solve problems. Human reasoning works by “understanding”, by building mental models, and it works especially well on new and novel problems, as François Chollet said in the rest of the quote I cited above:
Ultimately, this is what our education system teaches. It’s built to teach human beings, so it works to build “understanding”. While the problems of cheating are now more difficult, the goals and value remain the same.
It will take time to adapt. None of us previously had access to this store of answers to all well-solved problems, so we didn’t realize how many problems we were solving day-to-day were already well-solved. They felt novel when they weren’t. Solving those problems for the thousandth time was dead weight loss, in some sense, and it can henceforth be avoided.
Now that we all have easy access to the database of well-solved problems, we will, I think, quickly gain an understanding of what is new and novel and focus our efforts on the latter category. The end result, I think, will be more humans pushing out the edges of knowledge and creativity than ever before, giving a productivity gain to humans as a whole.
