Man or machine? GPTZero uncovers AI texts

Writing an essay, composing a poem or conjuring up an essay out of thin air in a matter of seconds? Since the end of November, this has been possible with the artificial intelligence (AI)-powered chatbot "ChatGPT"

While pupils, students and office workers are delighted with the useful tool, a computer science student at Princeton University in the US, Edward Tian, sees potential dangers in it. Over the New Year, he therefore created a programme that recognises whether a text comes from ChatGPT or from a human. It is called GPTZero.

"There is so much hype around ChatGPT. Is this and that being written by AI? We humans deserve to know!" writes the 22-year-old on Twitter. This is particularly important in the context of school. "Do secondary school teachers want pupils to use ChatGPT to write their history essays? Probably not," Tian continues.

Interested parties have been able to use the beta version of the app since the beginning of January. Either directly via Tian's own website or via the open source platform Streamlit.

How GPTZero works

For the analysis, users enter text into the GPTZero text field. After a few seconds, the app responds with an assessment based on a score. If the number is comparatively high, it is likely that the text was written by a human.

GPTZero uses two criteria to assess whether a text was written by an AI or not: "perplexity" and "burstiness". Perplexity measures the randomness in a sentence. GPTZero analyses how the sentence is constructed and whether its structure is unusual or surprising. If this is the case, it indicates that the sentence was written by a human.

Burstiness analyses the randomness within the entire text. AI tools tend to write texts with constantly low complexity. Humans, on the other hand, tend to write a more complex sentence and then a simpler one.

Tian "completely overwhelmed" by the huge interest

Since its release, the rush for GPTZero has been huge. "I am completely overwhelmed by the support this app has received," says Tian on Substack. This comes after he had apologised for the website crash in a previous message. It crashed in the meantime due to high demand.

Despite the huge rush, there are also more critical voices such as this one towards GPTZero. On Twitter, they point out to Tian that his app will not be useful for long. That's because they think ChatGPT will outsmart such programmes in the future.

Could GPTZero be outwitted quickly?

Sandro Luck, Machine Learning Engineer at Digitec Galaxus, thinks so too. Nevertheless, he is not immediately writing off the new AI app. "I suspect that GPTZero will probably uncover long texts from ChatGPT," he says. "That's because OpenAI - the company behind ChatGPT - simply has no interest in hiding the fact that ChatGPT is an AI." However, as soon as developers set themselves the goal of tricking tools like GPTZero, this happens "very quickly".

Tian definitely wants to concentrate fully on expanding GPTZero in the coming months, the student writes on Substack. He wants to improve the modelling capabilities and concentrate on fully scaling the app. He also hopes to publish his findings with the programme in a scientific paper soon.