Battle of the emails: Does human-written or AI-generated content perform better?

By Sean Tinney February 22, 2023

Artificial intelligence (also referred to as AI) is taking over. It is everywhere from social media, digital assistants, chat bots, to the transportation industry. It’s even in food services. Have you seen these AI-robot fast food restaurants?

As the use of artificial intelligence continues to expand, it’s no surprise that it has made its way into email marketing. One area in particular where AI has been gaining attention is in the creation of email content. 

While human-written emails are what we all know and have written for years, many marketers are turning to AI-generated content. But is this better? Surely people reading those emails can tell the difference between a human and robot, right?

Imagine if you could prompt an AI-content tool to write great quality emails. No more sleepless nights wondering if your email is going to perform. Plus, think of all the hours you would get back in your week because you didn’t need to write emails.

I didn’t want to imagine the possibility anymore, so I decided to test a human-written email vs AI-generated content. 

Setting up the human vs AI email content test

Here are the steps I took to set up my email split test:

1 – Determine the email topic

First I needed a topic, so why not write the content about using artificial intelligence in email marketing.

2 – Find an AI-writing tool 

I chose ChatGPT since I had already used them and was familiar with the tool. But there are plenty of other AI writing solutions that I could have used for this test.

3 – Write my email copy

When I wrote my copy I wanted to highlight the value of email marketing but also communicate my pain point with emails – the time it takes me to write them. Then I wanted to introduce AI-powered writing tools as an option for people that may also be looking for a quick solution

Here’s my copy of the email:

4 – Create the AI generated content

When using ChatGPT, the main page has a section where you type your instructions. I’ve found the more detail you provide here, the closer this AI content writing tool provides a draft inline with your vision. 

I structured my initial instructions as if I were typing something to a copy-writer. I wanted the tone to be witty because this is how my draft felt and I wanted similar writing styles. And I wanted the AI generated content to follow a similar structure as my version, which was to establish me as a credible source, reference my pain point, then provide the solution.

Here were the instructions I provided for my first email attempt:

The first AI generated content was good. It sounded like a human wrote it however I felt the copy was too strong at promoting ChatGPT as a solution. Here’s what was generated based on my initial instructions:

Now if I worked at ChatGPT, this copy would be perfect. But, the purpose of my email was not to sell ChatGPT, I simply wanted to suggest them as an option.

So at this point I needed to update my instructions to make it sound less like a sales email. Here’s where artificial intelligence in content marketing is powerful, I simply typed instructions as if I was providing feedback to a professional content writer.

These instructions gave me my final AI email marketing content draft:

You may notice three of the final four paragraphs are the same as the human (that’s me) written content. That was strategic on my part because I wanted to let readers know they were part of this test since the email topic was on AI content writing.

Test Results

• The email written by me had 6.2% more clicks.
• The link asking readers to try ChatGPT had 19.9% more clicks in the AI generated content. The AI email did a better job promoting the benefits of AI content writing.

What’s better – human or artificial intelligence?

Could you tell the difference between the two emails? If I didn’t write and set up this test, I honestly would not be able to distinguish between the two.

While the human written email had more clicks, I can’t overlook the fact that the AI-written email had nearly 20% more clicks on the “try ChatGPT” link.

My conclusions

So, is an email written by artificial intelligence better than a human?

Depending on the type of content you plan on writing, that AI-generated content could perform better than a human-written email.

If you’re writing email content for a product or service you plan selling, than I think AI-generated content could be really powerful. As evident by the number of people who clicked the “try ChatGPT” button on the AI generated email.

However, I think one area that human written content will always be more powerful is when writing a personal story or something based on your experience. 

If you look at the email written by me, I wrote about MY personal experience. I did this to try to get the readers to relate to me. I attempted to provide ChatGPT instructions to create a similar intro, but I don’t think it’s capable.

What’s next

Overall, I believe AI content writing is pretty close to human written content.

The results of my test showed a 6.2% increase in clicks for the human written email. This is a good increase but not insurmountable. Next time we test this, I won’t be shocked if the AI email saw more clicks. Why? Because AI is learning and getting better each day.

So will I use an AI email writing assistant in the future? Yes. We all get writer’s block and I think these AI tools are great to complement your writing. 

I recommend experimenting with an AI content writing tool. You may find that it writes better copy and gets better results. At the very least you could use AI to generate ideas for your, subject lines (see our list of the top subject line generator tools), write a portion of your newsletter, or just to pull bits and pieces of content you feel is stronger than what you would write.

I’d love to hear your success stories with artificial intelligence in content marketing. Leave your comments below.