Tom Coles (3:00)

This is Alex Tway, the CTO and co founder of GPT0, the company that found these 100 plus AI hallucinations in research papers.

Alex Tway (3:09)

They're like, hey, like this. LLM knows so much about my research, I think I wrote all these papers. I didn't. In some ways it's a weird point of pride, I think, to be hallucinated by an AI. That's definitely one sign that you've made it in the industry.

Tom Coles (3:23)

If you're new to this subject, it might sound strange to talk about a computer program hallucinating. They're not out in the desert taking mind altering substances after all. The reality is a little more prosaic.

Alex Tway (3:35)

What happens is a researcher might say, oh, can you write this section of the paper for me and make sure to add a lot of citations? And the AI will do that, but it's essentially doing it without any references. And so it has to start making things up to make them look like real high quality citations, but they're actually not corresponding to anything real.

Tom Coles (3:59)

Alex certainly has skin in this game. The company he runs offers a service to organisations publishing these papers to help root out AI slob. But no one is denying that there are AI hallucinations in these papers. So what exactly did they find? First, the context. The papers in question were published as part of a big AI conference known as Neurips.

Alex Tway (4:21)

It's essentially the premier event for machine learning.

Tom Coles (4:25)

This get together attracts the brightest minds in AI, or machine learning as Alex calls it. From academic researchers at top universities to industry researchers at the big tech companies like Meta and Google. Alex says that in this booming industry, getting your paper published really matters.

Alex Tway (4:43)

Having a couple of papers in these conferences can get you an OpenAI job. If you're a startup company, having a couple of these papers in these conferences can mean raising $100 million from investors.

Tom Coles (4:57)

This means there's a massive incentive for researchers to pump out a lot of papers for consideration at conferences like this one.

Alex Tway (5:04)

Once a year they'll receive about, let's say around 20,000 submissions and then they'll accept about 5,000 of them.

Tom Coles (5:13)

This is where Alex's company comes in. They took those 5,000 odd papers that were selected for publication and oh wait a minute, asked AI to take a look.

Alex Tway (5:24)

Yeah, so that's the funny thing. The most accurate methods to do this are using AI itself, but extremely specialized for this purpose. And so to make our specialized AI be able to find hallucinations, we have to train it on countless examples where we've actually labeled like hey, this is a hallucination, this isn't. And it drastically improves in that task compared to an off the shelf chatgpt or something like that. So essentially we ran our hallucination detector on these papers for this experiment.

Tom Coles (6:05)

They didn't look at the text of the papers themselves, but just the citations, the references to other papers which lodged the research in the wider web of academic publishing. These are easier to verify, as wrong as you can easily check against the real thing.

Alex Tway (6:19)

And we would go through each citation and try to verify whether or not it exists, searching through like massive buck scale search engines, academic databases and so on. And so we might get a bunch of potential matches.

Tom Coles (6:36)

Most of the time the details of the paper cited, author title and date published did match a real scientific paper. Sometimes they really didn't.

Alex Tway (6:45)

We were able to pretty easily find like at least 100 hallucinations over 50 papers.

Tom Coles (6:52)

This isn't an exhaustive list by the way, they just stopped counting when they found a suitable round number. And these hallucinations took a variety of forms.

Alex Tway (7:01)

About 39 were just completely non existent publications then. The other 61 had a combination of fabricated authors, people who don't exist or exist but never wrote a paper like that, fake titles, fake links or URLs and so on.

Tom Coles (7:21)

Some of the dodgy citations contained real authors, but who didn't write those papers. Some had wonky names or odd titles, but others were just completely made up. This one is my absolute favourite.

Alex Tway (7:32)

The authors were first name, last name and others, which I imagine is quite a coincidence that all of those three were real people.

Tom Coles (7:41)

We asked professor first name and doctor last name for comment, but didn't hear back. Look, it seems clear from this research that incredibly busy under pressure, researchers are using AI to write the boring bits of their papers. It's quite funny really that AI researchers aren't immune to its charming overconfidence. But beyond the obvious irony, why does this matter?

Alex Tway (8:04)

In the culture of computer science, it often can be challenging to reproduce some of these experiments. And the reviewers, they don't have time to do so in their own capacity. And so there's so much trust that goes into like, did you write your code correctly? Is your data correct? They're often at scales that are impossible to verify without a lot of extra work. And so if we can't trust that your paper is even a human reviewed, so the AI is making mistakes in your paper and you're not catching it, then how can you trust that everything else created by the researcher was also reviewed by human and not hallucinated by AI?

Tom Coles (8:47)

For their part, the organizers of the Neurips conference told us that while they accept researchers are using AI to write their papers and hallucinations can get through the review process, they do not believe the research in these papers would necessarily be invalidated by the discovery of AI hallucinations. At the same time, they're continuously refining their guidance for both the authors and reviewers as the use of AI rapidly evolves. There is an implication for society in all of this too. Alex says that because of the biases in the stuff the AI has learned from citations from non anglophone researchers seem to go wrong more than for others. Although this wasn't from a Neurips paper, there are some pretty odd things going on.

Alex Tway (9:29)

We found that it would just start chaining together highly likely names of researchers such as you'd start chaining Chinese initials like hyx, xz, N, blah blah blah blah blah blah. Just a string of 10 three letter acronyms and you could just tell that the LLM thinks oh, if I had to make up a citation, all I have to do is just write Chinese names.

Tom Coles (9:58)

Thanks to Alex Tway that's it for this week. If you've seen a number in the news you think we should take a look at email more or lessbc.co.uk. we'll be back next week. Until then, goodbye.

MyFico Advertiser (10:20)

If you're an H Vac technician and a call comes in, Grainger knows that you need a partner that helps you find the right product fast and hassle free. And you know that when the first problem of the day is a clanking blower motor, there's no need to break a sweat. With Grainger's easy to use website and product details, you're confident you'll soon have everything humming right along. Call 1-800-GRAINGER clickgrainger.com or just stop by Grainger for the ones who get it done.

MyFico Advertiser (10:49)

Ready to buy a car, a home, or just want to take control of your money? Your FICO score matters and 90% of top lenders use it to make decisions. Checking your FICO score for free today without hurting your credit score? Visit myfico.com free or download the MyFico app today. MyFico gives you the score lenders use most, plus credit reports and real time alerts to help keep you on top of your credit, visit myfico.com free and take the mystery out of your FICO score.