Is ChatGPT BS-ing you?

For all their amazing abilities to do things like writing Paddington Bear short stories on the fly and offering recipes that look like they were written by William Shakespeare, generative artificial-intelligence systems like ChatGPT have a big shortcoming.

They can’t always be trusted.

Such systems sometimes offer patently false or misleading information and present it as fact. In some cases, they respond to prompts with bizarre suggestions that seem to indicate little conception of the real, physical world .

AI researchers have taken to calling such dubious responses “hallucinations.” But a team of philosophers at the University of Glasgow in Scotland thinks there’s a better term to describe what’s going on.

Bulls---, they call it — and they argue it applies to all of ChatGPT’s output.

In a paper published online last month in the journal Ethics and Information Technology, University of Glasgow lecturers Michael Townsen Hicks, James Humphries and Joe Slater take Harry Frankfurt’s book “On Bulls---” as their springboard. In that book, Frankfurt defines B.S. as something that’s said with “a lack of concern for the truth.”

That’s essentially what’s happening with large language models, such as the one underlying ChatGPT , Hicks and his colleagues argue in their paper. Such systems generate text by predicting what word should come next, based on their training data.

But they generate that text without any conception of what’s actually true, the Glasgow team wrote. Worse, they do so using natural language, in a way that’s meant by their designers to be indistinguishable from what a human might say.

“ChatGPT functions not to convey truth or falsehood but rather to convince the reader of — to use [Stephen] Colbert’s apt coinage — the truthiness of its statement,” Hicks and his colleagues said in their paper, entitled “ChatGPT is Bulls---.”

Representatives of ChatGPT developer OpenAI did not respond to a request for comment from The Examiner.

Hicks, Humphries and Slater said they became interested in what ChatGPT was doing after seeing their students turn in increasing numbers of papers that were clearly written by or with the help of such systems. Such papers had certain telltale signs, they told The Examiner.

They never had any spelling mistakes but seemed to be written by someone who had no idea how to organize paragraphs, Humphries said. They often included lots of bulleted points and, in longer papers, would repeat the same points numerous times, Hicks recalled. They also used certain distinct phrases over and over and would get citations wrong — making up the names of particular authors, he said.

“There’s this uncanny valley effect,” said Humphries. “It’s almost, but not quite, a thing a human would write.”

The concern the researchers had was with the lack of thought or concern for the truth that was going into such papers. When a student plagiarizes another’s work or gets someone to write a paper for them, at least someone is thinking about what’s going into the paper and is thinking about the arguments they’re making, Humphries said.

“Students cheating on their essays is a bummer and it’s bad for them, but it’s not the end of the world,” he said. “But when people don’t actually care about what the facts [are], don’t really care about making a cogent argument, then it gets really worrying, really fast.”

In effect, that’s what ChatGPT and like systems are doing, the Glasgow team argues. They’re “bulls--- machines” — they generate responses without regard to the responses’ underlying truth, Hicks and his colleagues say.

Because of that disregard for the truth, the authors write that models’ responses are still B.S., even if they offer true answers more often than not. After all, the researchers point out, one of the things that makes bulls--- convincing is that it sounds true and often has an element of truth to it.

Although they make their case in an academic paper, the debate over how to conceptualize the output of ChatGPT and similar systems is anything but academic, the Glasgow team argues. The metaphors we apply to technologies affect our expectations of them and how we use and regulate them.

The problem with describing faulty outputs of such models as hallucinations is that the term implies that there’s something wrong in how those systems are perceiving the world, rather than in how they are designed, the researchers say. What’s more, the term implies that something unusual or unexpected has happened. But, Hicks and his colleagues argue, such systems are doing the same thing when they give false responses and true ones.

“There are all these ways of talking about ChatGPT that kind of obscured what it was programmed to do and what its function was,” Hicks said.

“The kind of labels we give to technology does affect how people actually go and use the technology,” Slater said.

AI researchers interviewed by The Examiner agreed that it’s important to think about the labels used to describe what these systems are doing.

Hallucination is a particularly problematic term because it implies that systems like ChatGPT have some sort of intelligence or intent, said Chirag Shah, a professor in the Information School at the University of Washington. But that’s not what’s going on, he said.

They don’t have intent, Shah said. No matter what they may say in the text they generate, the models don’t understand what we’re saying or feeling. They’re just designed to pretend that they do, he said.

In effect, they bulls---, he said.

“They’re just pretending, and pretending really well,” Shah said.

For Alex Hanna, the director of research at Distributed AI Research, “bulls---” gets us a little closer to understanding what the AI models are doing, but it’s imperfect. The term accurately conveys the idea that systems like ChatGPT are generating responses without regard for their truth, she said.

But she added it also can anthropomorphize those systems, implying that there’s some kind of intelligence at work with them.

Hanna said she thinks a more apt metaphor is one University of Washington professor Emily Bender came up with: the Magic 8 Ball. That conveys the randomness and the lack of any kind of intelligence in their responses, she said.

“You ... have to factor in the idea that these things have no internal states, and you don’t want to personify them in such a way,” Hanna said.

A problem with the Glasgow researchers’ paper is that they overstate the case that systems like ChatGPT just generate bulls---, said James Landay, co-director of Stanford’s Institute for Human-centered Artificial Intelligence.

Typically, the developers behind such systems refine them before making them available to the public, Landay said. They use reinforcement learning methods to encourage them to offer correct answers, rather than wrong ones, he said.

Still, whether you use the term “bulls---” or “hallucinations” or something else, Landay agreed that it’s important to convey to users of AI models such as ChatGPT that they can’t fully expect the systems to be truthful. As he and Shah both noted, there’s a particular danger for people turning to such systems looking for medical or mental health advice.

“We want the general public to understand what some of the problems and limitations of this technology [are] so they don’t over-rely on it for things that may be dangerous for them,” he said.

If you have a tip about tech, startups or the venture industry, contact Troy Wolverton at twolverton@sfexaminer.com or via text or Signal at 415.515.5594.