One thing that makes large language models (LLMs) so powerful is the variety of tasks to which they may be applied. The identical machine-learning model that may help a graduate student draft an email could also aid a clinician in diagnosing cancer.
Nevertheless, the wide applicability of those models also makes them difficult to judge in a scientific way. It will be not possible to create a benchmark dataset to check a model on every variety of query it may possibly be asked.
In a recent paper, MIT researchers took a special approach. They argue that, because humans resolve when to deploy large language models, evaluating a model requires an understanding of how people form beliefs about its capabilities.
For instance, the graduate student must resolve whether the model could possibly be helpful in drafting a specific email, and the clinician must determine which cases could be best to seek the advice of the model on.
Constructing off this concept, the researchers created a framework to judge an LLM based on its alignment with a human’s beliefs about how it would perform on a certain task.
They introduce a human generalization function — a model of how people update their beliefs about an LLM’s capabilities after interacting with it. Then, they evaluate how aligned LLMs are with this human generalization function.
Their results indicate that when models are misaligned with the human generalization function, a user could possibly be overconfident or underconfident about where to deploy it, which could cause the model to fail unexpectedly. Moreover, attributable to this misalignment, more capable models are likely to perform worse than smaller models in high-stakes situations.
“These tools are exciting because they’re general-purpose, but because they’re general-purpose, they will probably be collaborating with people, so we have now to take the human within the loop under consideration,” says study co-author Ashesh Rambachan, assistant professor of economics and a principal investigator within the Laboratory for Information and Decision Systems (LIDS).
Rambachan is joined on the paper by lead creator Keyon Vafa, a postdoc at Harvard University; and Sendhil Mullainathan, an MIT professor within the departments of Electrical Engineering and Computer Science and of Economics, and a member of LIDS. The research will probably be presented on the International Conference on Machine Learning.
Human generalization
As we interact with other people, we form beliefs about what we predict they do and have no idea. As an illustration, in case your friend is finicky about correcting people’s grammar, you may generalize and think they’d also excel at sentence construction, although you’ve never asked them questions on sentence construction.
“Language models often seem so human. We wanted as an example that this force of human generalization can also be present in how people form beliefs about language models,” Rambachan says.
As a place to begin, the researchers formally defined the human generalization function, which involves asking questions, observing how an individual or LLM responds, after which making inferences about how that person or model would reply to related questions.
If someone sees that an LLM can accurately answer questions on matrix inversion, they may also assume it may possibly ace questions on easy arithmetic. A model that’s misaligned with this function — one which doesn’t perform well on questions a human expects it to reply accurately — could fail when deployed.
With that formal definition in hand, the researchers designed a survey to measure how people generalize once they interact with LLMs and other people.
They showed survey participants questions that an individual or LLM got right or flawed after which asked in the event that they thought that person or LLM would answer a related query accurately. Through the survey, they generated a dataset of nearly 19,000 examples of how humans generalize about LLM performance across 79 diverse tasks.
Measuring misalignment
They found that participants did quite well when asked whether a human who got one query right would answer a related query right, but they were much worse at generalizing concerning the performance of LLMs.
“Human generalization gets applied to language models, but that breaks down because these language models don’t actually show patterns of experience like people would,” Rambachan says.
People were also more prone to update their beliefs about an LLM when it answered questions incorrectly than when it got questions right. Additionally they tended to consider that LLM performance on easy questions would have little bearing on its performance on more complex questions.
In situations where people put more weight on incorrect responses, simpler models outperformed very large models like GPT-4.
“Language models that get well can almost trick people into considering they’ll perform well on related questions when, really, they don’t,” he says.
One possible explanation for why humans are worse at generalizing for LLMs could come from their novelty — people have far less experience interacting with LLMs than with other people.
“Moving forward, it is feasible that we may get well just by virtue of interacting with language models more,” he says.
To this end, the researchers wish to conduct additional studies of how people’s beliefs about LLMs evolve over time as they interact with a model. Additionally they wish to explore how human generalization could possibly be incorporated into the event of LLMs.
“Once we are training these algorithms in the primary place, or attempting to update them with human feedback, we’d like to account for the human generalization function in how we take into consideration measuring performance,” he says.
In the intervening time, the researchers hope their dataset could possibly be used a benchmark to match how LLMs perform related to the human generalization function, which could help improve the performance of models deployed in real-world situations.
“To me, the contribution of the paper is twofold. The primary is practical: The paper uncovers a critical issue with deploying LLMs for general consumer use. If people don’t have the proper understanding of when LLMs will probably be accurate and when they’ll fail, then they will probably be more prone to see mistakes and maybe be discouraged from further use. This highlights the difficulty of aligning the models with people’s understanding of generalization,” says Alex Imas, professor of behavioral science and economics on the University of Chicago’s Booth School of Business, who was not involved with this work. “The second contribution is more fundamental: The dearth of generalization to expected problems and domains helps in getting a greater picture of what the models are doing once they get an issue ‘correct.’ It provides a test of whether LLMs ‘understand’ the issue they’re solving.”
This research was funded, partly, by the Harvard Data Science Initiative and the Center for Applied AI on the University of Chicago Booth School of Business.
