It’s hard not to notice the growing number of “AI detector” tools flooding the market, with many individuals believing in their efficacy.
In reality, many users are even willing to pay for these services, but the question remains: Do these tools genuinely deliver on their promises?
To evaluate the effectiveness of AI detectors, I’ll put three popular tools to the test: Originality AI, Contentatscale.ai, and OpenAI’s own AI Text Classifier. I will examine their performance on three distinct 500-word topics: a personal story, a brief product review, and an informative article.
Each topic will be generated in three different ways:
- Pure AI output based on a single, straightforward command.
- AI-assisted writing, where unique and specific prompts are provided for each paragraph.
- Content written solely by me, a non-native English speaker.
Disclaimer: As an affiliate marketer, I typically earn income through referral links. Many similar articles promote Originality AI as the best tool and provide affiliate links for readers to use. However, I’m a bad affiliate marketer, so I won’t be including any referral links in this article. If, after reading this piece, you decide not to use any AI detection tools and feel that I’ve saved you some money, I’d be grateful if you’d consider buying me a coffee.
Let our test begin.
1. Testing AI Detector Tools on a Personal Story
Detecting AI-written content with a single prompt
First, let’s examine how well AI detector tools can distinguish between an AI-generated and a human-written personal story. I’ll begin with a single prompt: “Please write a 500-word story about my travel to Kalon Surf Resort in Costa Rica.” I really did, good resort by the way.
This is the story I got from ChatGPT (there are more words, it wrote me the story which is over 700 words, you can check the full text here):
Wow, that sounds like a book. Okay, time to test it with AI detection tools:
As the results show, none of the AI detectors accurately identified the content as purely AI-generated. However, the Content at Scale tool came the closest to making a correct assessment but still not even close.
Detecting AI-written content with multiple prompts
In order to write the story about my travel to Kalon Surf Resort in Costa Rica I used the following prompts:
- I have recently traveled to Costa Rica and stayed and Kalon Surf resort. Can you write me a short intro about this, 2-5 sentences.
- Next, could you write a few words about Kalon Surf resort. Say it’s a great resort with a beautiful location and offers nice features.
- Next, describe the surfing part and say that Kalon provides great surfing classes and instructors.
- Next, talk about the food at the resort, say it was delicious and their chefs were the best.
- Next, I’d like to give them kudos for massage therapy, please describe it as well
- Talk about non-surfing times and activities to do, e.g. you can go hiking near city of Dominical and mention a few spots.
- Now, talk about great people at Kalon and say I would recommend it to everyone who is into surfing
- Finally, talk about price. Say it’s expensive but worth every single penny
That’s what I got from ChatGPT:
This is how AI-detector tools evaluated such text:
As you could see, none of the tools think it was written by AI.
Detecting AI-written content with my own text
Finally, I wrote my own story about traveling to Costa Rica. English is my second language and I don’t have good writing skills so please do not judge:
Let’s see what AI detectors will say about this brilliant piece of content:
Ok, good work AI detectors, even though the OpenAI’s own AI text classifier did not identify me as a human, I guess the text quality was so bad.
2. Testing AI Detector Tools on a Product Review
Okay, it’s understandable that AI detectors may struggle to identify AI-generated content in a personal story. However, product reviews should be easier to detect, making this test particularly relevant for SEO people. I recently purchased a Garmin Fenix 7 smartwatch (oops, sorry just a habit of placing affiliate links), which provides an excellent opportunity for me to write my own review. Additionally, we can easily generate an AI-written review to compare and test the AI detectors’ performance.
Let’s begin the second test.
Detecting AI-written content with a single prompt
To generate an AI-written review, let’s provide ChatGPT with a straightforward, single prompt: “I recently purchased a Garmin Fenix 7 smartwatch. Write a 500-word review of the Garmin Fenix 7 watch. In particular, please describe how it tracks surfing activity, its solar charging features and also compare reviews of other users”. I want to avoid making the prompt overly specific or excessively broad.
This is the prompt I got (full text here):
The generated review appears to be quite legitimate, especially since it includes a few unique features I wanted to highlight. Let’s evaluate how well the AI detectors can identify it as machine-generated content.
It appears that Originality AI performed well, while the other tools had partial success. This raises the question: Can Originality AI consistently identify AI-generated content in review-type articles? To further investigate this, let’s use the single ChatGPT prompt “Share your thoughts on the Garmin Fenix 7 watch, covering its features, design, and user experience in a 500-word review” and test the generated content with Originality AI to see if it can reliably detect AI-generated content. It should have been extremely easy to identify with such a general prompt (full text here).
Okay, now what? I tried to find the logic of how it identifies “AI content” but I can’t find any…
These are the rest of the results for your review:
Detecting AI-written content with multiple prompts
As a Garmin Fenix 7 user, I have the advantage of tailoring my prompts to create a more authentic review using ChatGPT. These are the prompts I use:
- I have recently bought Garmin Fenix 7 and I am in love with this smartwatch. I want to publish a review on my website, can you help me to write a short intro? 3-5 sentences
- First, talk about the design. I like how it looks and it’s very user-friendly
- Next, I extremely love how it tracks all kinds of sports with unique metrics. E.g. with surfing it shows how many waves you got, what was the max speed, height, length, and more. With strength, it shows the group of muscles you are worked + it calculates how many sets you did. It has unique metrics for rowing, running, yoga, and more. Please describe this section thoroughly and more detailed
- Next is battery life and a solar-powered charge feature. Please explain how come this model can work over 20 days without charging.
That’s enough for 500-words and ChatGPT assisted me really well to describe my personal opinion about this product (full text here):
I liked the quality of the text and how it covered the features I wanted. Time for a test:
Oops, busted! Interestingly, all the AI detectors identified a similar pattern in the text, which they flagged as AI-generated content. What do you think could be the giveaway? Share your thoughts in the comments section below.
Detecting AI-written content with my own text
Okay, finally I wrote my own review. I never realized how hard it is to review a technical product on your own, so my review turned out pretty general:
I know, it is quite a terrible piece of content and I hope you did not waste your time reading it. Okay, what do AI detectors say:
It seems that AI detectors have no trouble identifying my writing as human-generated (even though Originality AI identified 3% of the content as AI), which is not surprising since I’m not a professional writer and English is my second language. My text is quite simplistic and features basic English grammar. In truth, the review I wrote is far from ideal, and I would never publish it in its current form. To improve the review, I’ll ask ChatGPT for assistance and then see how the AI detectors respond to the enhanced version.
Interestingly, after improving the review with ChatGPT’s help, Originality AI detected even less AI-generated content than my original text, while Content at Scale showed better results. Ironically, OpenAI’s own AI Text Classifier appears to be the least accurate of the bunch.
Evaluating these AI detector tools is proving to be quite challenging, as the results seem to be somewhat inconsistent and unpredictable.
3. Testing AI Detector Tools on an Informational Article
Lastly, we’ll create an informative article. As a Canadian, I’d like to generate content related to the country I’m currently living in.
Detecting AI-written content with a single prompt
I’ll start with a simple prompt: “Describe the best and the worst things about Vancouver (Canadian city) within 500 words” (you can find the full version here):
Pretty decent list of pros and cons.
Time for a test:
It appears that Originality AI has once again demonstrated the strongest performance among the AI detectors we’ve tested.
Detecting AI-written content with multiple prompts
I entered the following prompts:
- I am writing an article about Vancouver, BC and I am going to explain the city’s pros and cons. Please write an intro paragraph
- Let’s start with the best things first The first on my list is its natural beauty. Please sounds like a Vancouver resident
- Next, I’d like to give kudos to climate
- Let’s proceed to bad things. Please talk about affordability and crazy real estate prices.
- Finally, rainy weather sucks.
Results:
I am getting tired of finding logic in these results and I can’t find a clear pattern in the AI detection methods.
Detecting AI-written content with my own text
By now, I’m fairly confident that AI detector tools will classify my content as human-written. Nevertheless, let’s give them another opportunity to demonstrate their ability to accurately assess my writing. You can check my text here.
Good job, Originality AI.
Indeed, it seems that my less-than-perfect English writing skills make it easier for AI detectors to identify my text as human-generated. I’m aware that many professional writers craft high-quality content on their own, but they may be wrongfully accused of using AI due to the specific algorithms of these detection tools.
Okay, now what? Should I Use AI Detectors For Checking Content?
My tests have shown that the results from AI detector tools can be quite inconsistent. In some cases, the outcomes were not accurate, while in others, they were reasonably close.
These AI-detector tools often use complex algorithms and machine learning techniques that analyze various aspects of the text, such as vocabulary, grammar, sentence structure, and patterns that may be typical of AI-generated content. However, this doesn’t necessarily imply that the text was created using an AI tool. What if a writer naturally used similar vocabulary, grammar, or sentence structure?
I wouldn’t recommend relying solely on these tools, as their results are inconsistent.
If you’re still concerned about AI-generated content on your website, consider why that is. Are you worried about Google penalizing you? Google has officially stated that they reward high-quality content, regardless of how it’s produced.
Is your concern due to suspicions that a highly-paid writer is using AI tools to create content? In that case, ask yourself whether you’re satisfied with the provided text. If you are, embrace it. If not, consider leveraging AI to optimize your work process and scale your workload more efficiently.
Personally, I don’t see a promising future for AI-detection tools, especially after Google’s announcement that they accept AI-generated content. However, that’s not the primary reason for my skepticism. AI technology is evolving rapidly, and it’s improving every day.
AI detection tools struggle to keep up with the rapid advancements in AI technology. As AI-generated content becomes increasingly sophisticated and indistinguishable from human-written text, it becomes more challenging for detection tools to accurately identify such content, which may limit their effectiveness in the future.
I view these “detection” tools as just another way to extract extra money from users.
I hope you found this content helpful. If I managed to save you some money by dissuading you from investing in AI-detection tools, feel free to show your appreciation by buying me a coffee 🙂
