Why AI detectors think the US Constitution was written by AI

If you feed America’s most important legal document – the US Constitution – into a tool designed to detect text written by AI models like ChatGPT, it will tell you that the document was almost certainly written by AI. But unless James Madison is a time traveler, this cannot be the case. Why do AI type detectors give false positives? We spoke to several experts — and creator of the AI ​​type detector GPTZero — to find out.

Among the news stories of overzealous professors failing an entire class due to suspected AI writing tool use and falsely accusing children of using ChatGPT, generative AI has education in a frenzy. Some think it represents an existential crisis. Teachers who draw on didactic methods developed over the past century have been scrambling to find ways to maintain the status quo—the tradition of relying on the essay as a tool for measuring student mastery of a subject matter.

As tempting as it may be to rely on AI tools to detect AI-generated writing, evidence so far has shown them to be unreliable. Due to false positives, AI type detectors such as GPTZero and ZeroGPT and OpenAI’s Text Classifier cannot be trusted to detect text composed of Large Language Models (LLMs) such as ChatGPT.

If you feed GPTZero a piece of the US Constitution, it says that the text “will likely be written entirely by artificial intelligence.” Several times over the past six months, screenshots of other AI-powered detectors have surfaced showing similar results The virus moved on social media, inspiring confusion and plenty of jokes about the founding fathers being robots. It turns out that the same thing is happening with the Bible anthology, which has also been shown to be generated by artificial intelligence.

To explain why these tools make such obvious mistakes (and often lead to false positives), we first need to understand how they work.

Understand the concepts behind AI discovery

Different AI handwriting detectors use slightly different methods of detection but with a similar premise: an AI model trained on a large set of text (consisting of millions of writing examples) and a predicted set of rules that determine whether handwriting is likely to be man made. or artificial intelligence.

For example, at the heart of GPTZero is a neural network trained on “a large variety of human-written text generated by artificial intelligence, with an emphasis on English prose,” according to the service’s FAQ. The system then uses properties such as “perplexity” and “explosion” to evaluate the text and determine its classification.

In machine learning, perplexity is a measure of how much a piece of text deviates from what the AI ​​model learned while training it. As Dr. Margaret Mitchell of AI firm Hugging Face told Ars, “Puzzling is a function of ‘How amazing is this language based on what you’ve seen?'” “”

So the thinking behind measuring puzzlement is that when they write a script, AI models like ChatGPT will naturally access what they know best, which comes from their training data. The closer the manager is to the training data, the lower the confusion rating. Humans are offensive writers—or at least that’s the theory—but humans can also write with low bewilderment, especially when imitating a formal style used in law or certain types of academic writing. Also, many of the phrases we use are surprisingly common.

Let’s say we guess the next word in the phrase “I’d like a cup of _____”. Most people fill in the blanks with the word “water”, “coffee” or “tea”. A language model that is trained on lots of English text will do the same because these phrases appear frequently in English writing. Confusion from any of these three outcomes will be very low because the prediction is quite certain.