Table of Contents
Since the launch of ChatGPT late last year, there’s been a wave of new tools that leverage its technology. The promise of many of these tools is that they’ll help marketers create content faster and more efficiently.
In our experience, AI has generally been more useful for expediting parts of the content creation and editing process than actually creating long-form content on its own. This seems to align with what many others in the industry and in our own network have said: it’s a great tool but it won’t replace a human writer.
However, we recognized the speed at which new tools were launching and their capabilities improving. We've also come a long way in terms of prompt sophistication, which has a significant impact on the quality of the output.
We wanted to cut through the noise and figure out whether or not it really is possible to create decent-quality content that could pass our clients’ review process.
So, acknowledging our priors, we set out to prove ourselves wrong by doing some experimentation.
Experiment #1: Using ReWord to Create SEO Content
ReWord is a tool that combines AI-powered content creation with a sleek UI to help marketers research, create, and edit content content all in one place. There were a few things that made ReWord appealing:
- The fact that it was also an editor (think ChatGPT meets Google Docs) meant that we wouldn’t have to continually feed it prompts and then copy and paste the output into a doc. This made it feel like we’d be getting the best of both worlds: AI-powered content combined with a UI that made it easy to edit, structure, and reformat the text.
- ReWord is “trained” on your existing blog content in order to try to mimic brand voice. The way you train it is by connecting Google Search Console and turning it loose on your published blog articles.
- ReWord dynamically inserts internal links into article drafts, potentially saving users the time of having to do so manually.
Our hypothesis was simple: using ReWord, we’d be able to double content production for our client by creating articles that would be accurate, on-brand, and up to standards. And, importantly, we’d be able to do so quickly and easily enough that the incremental cost to do so would be minimal.
The Experiment
Our experiment with ReWord wasn’t a strict experiment that would live up to any scientific standards. We just wanted to see if we could use it to produce content quickly and without the input of an actual writer. So, we simply signed up for an account and got started.
After connecting ReWord to our client’s GSC account, we dove in and got started with our first few articles. Immediately, some cool features jumped out at us.
In the “articles” section of ReWord, you can see your entire blog portfolio in one place. ReWord pulls in the title, publishing status, comments, and word count. While it didn’t help us create our articles, we thought it was cool to have that information collated within the platform.
When we started actually creating our first articles, we noticed the “research” functionality that ReWord uses to help inform article creation. You can ask it to suggest subtopics, FAQs, relevant statistics, and more. Without a doubt, this feature alone could save content creators a lot of time by pulling in the information they’d otherwise need to find in disparate sources on the internet (of course, you should still fact-check before publishing).
You can also see important information about your article at a glance, which was nice to have on hand.
Aside from the extra features, the editor itself was very user-friendly. The tool allows users to create a single section of content at a time using either their own writing or the AI-powered content generator. Since we wanted to see if we could produce content using only AI, we created each section of each article using AI rather than writing any of it ourselves.
While the tool had some great features, we weren’t in search of great features. We wanted to know if we could use it to produce quality content quickly and efficiently.
The Results
After creating four articles using ReWord, we felt we had reached a solid conclusion: while the tool itself was cool on many levels, we couldn’t use it to produce articles in a way that would be significantly faster or cheaper than simply hiring a copywriter.
To get a sense of what that means from a more quantitative standpoint, we spent nearly an hour and a half creating each article. You might think, “Hey, that’s probably way faster than if they wrote it themselves.” And you’d be right. But we weren’t comparing the experience to writing the articles ourselves - we were comparing it to hiring a professional copywriter.
Our assessment was that a copywriter could produce a better article at only a slightly higher cost than the time investment needed to do it ourselves using ReWord.
Here’s why:
- While AI-based tools have been improving, much of the underlying technology has stayed the same. Even with great prompts, AI content is still AI content.
- The amount of editing needed to get the content to an acceptable level was a pretty significant time suck. Because the AI was trained on our client’s existing content, we assumed there would be less editing involved, but this was far from true.
- The dynamic link inserter, while a cool feature in theory, didn’t really work in practice. We still needed to go in and manually add links.
Takeaways
To be clear, none of this is meant to detract from ReWord as a tool. It’s a very impressive tool and they’re upfront about the fact that it’s meant to be used in tandem with writers to support collaborative writing, not as a replacement for human writers. We have no doubt it can be extremely useful for that application.
However, for our purposes - using AI to produce long-form content quickly - it ultimately didn’t prove to be a viable solution.
Experiment #2: Using ChatGPT to Create SEO Content
ChatGPT is a generalist’s tool that uses generative AI to answer prompts with it’s vast learning database. It has been co-opted as a marketing tool to develop marketing content like long and short-form content. ChatGPT is an obvious contender for AI-driven content, here are a few reasons we tested it with this experiment:
- It is largely free to use. A big consideration in content creation is the cost. Content is one of the biggest line items for marketing teams, and the majority of the time, content needs to be spun up quickly. With ChatGPT, that cost factor becomes minimal for unlimited, ready-to-use content.
- ChatGPT is able to integrate tools like WebPilot - a live data integrator, which gives it an advantage over a less evolved tool like ReWord. Tools expand the capability of ChatGPT and make it more flexible and easy to use with real-life data from today.
- The more you use ChatGPT, the more it learns your writing and editing styles. This can save you time in the long run.
Our hypothesis was that ChatGPT could create content for an e-commerce website that would be accurate and able to rank as well as human-written content. Our secondary hypothesis was that we would be able to cut the production time and cost of this content in half.
The Experiment
This experiment can be broken down into three sections: Content Identification and Comparison, Content Generation, and Content Measurement. Knowing how ChatGPT isn’t great at having an opinion, we wanted to ensure that the content we were creating had the best chance of success.
Content Identification and Comparison
We decided that our e-commerce client would be a great fit for AI content based on our upcoming content calendar. We were looking for a client that had little to no brand voice with a content plan that focused on splicing their product catalog into niche segments. This client’s content plan had multiple “[number] best [products] for [season]” posts coming up (eg. 10 best sweaters for fall). This content is low human effort and high impact, with other similar content created by writers already performing well for the site.
We had a variety of this type of content in the pipeline, which means we were able to do a loose A/B test. We split the topics in half, with similar topics being split between human writers and AI. For example, “Top Polos for Corporate Workers” and “Top Polos for the worksite” would go head to head in writer v machine.
An exciting premise, no?
Content Generation
The execution of this content would not require any special training on our content specialist’s part. The process would remain relatively simple. We would:
- Create detailed content outlines, with recommended keywords, lengths, and concepts to include.
Our outlines are already designed to be detailed, with any information that a writer could want. We include things like suggested talking points per paragraph, recommended keywords, and recommended structure per paragraph.
- Create detailed prompts to execute our outlines
Our prompts had to be modified a few times to get the right flavor of what we wanted. In some cases, we re-inputted a segment of the article with an additional prompt to really get what we wanted from the AI tool.
- Edit AI-driven content
This editing and creation of content was a symbiotic relationship. As ChatGPT output content, we were consistently tweaking and editing the content. The content process went from being segmented into multiple days work, into a process that was start -to-finish completed in one sitting.
- Wait for the content from the writer and edit that content
- Publish at the same time
Content Measurement
Measurement - everyone's favorite word. We set up GA4 tracking for our new posts. As of writing, these pieces have no traffic in these dashboards. Which is why it was important to us to treat this content as we would any other content, by tracking leading indicators like impressions and rank. We used Google Search Console to begin tracking the upward momentum of our content using a direct comparison between AI and Human written content.
Another metric we tracked that is less performance marketing and more organizationally important is time spent on content. We - like any marketing department - have limited resources. Content requires intensive effort and coordination with multiple teams, as well as being a consistent line item on marketing budgets. Not only was ChatGPT cost-effective - essentially nil except our content specialist’s salary - it was extremely labor-effective. A piece of content that would otherwise take a week or two to turn around from a human writer took 1-3 hours to complete internally.
The headline - If cost and resources are your biggest barrier to content entry, ChatGPT is a great place to start. But if you’re going for performance and long-term SEO performance, human-written content leads our performance indicators in the early stages.
The content that we got back from ChatGPT was consistent - and we mean that in a bad way. It was contrite, in some places non-sensical, and in others, just plain boring. There’s only so much you can say about clothing, and turns out ChatGPT found every way to say it. There was little variation from piece to piece initially, despite different keyword targets and concepts being written.
In this respect, while ChatGPT content is quicker, it requires more consistent attention and really does not lead in performance metrics to that degree where it would make sense to use consistently over a human writer.
Despite leaving a lot to be desired, if you’re creating content that just needs to get out there and don’t have the resources to establish clear content generation processes internally, using ChatGPT is a great way to get started. It’s quick and inexpensive, and if you have one person working on it, it can be an awesome tool to use in adjunct with other content initiatives.
Comparing ReWord vs. ChatGPT
Methodology
Tools
To successfully conduct our experiments, we wanted to be sure to get as well-rounded results as possible. To do this, we opted to use two popular AI tools and compare the results.
ChatGPT
Perhaps one of the most “buzzing” AI tools available, ChatGPT is a chatbot that allows users to ask for and receive information in a conversational way. The model is trained to respond to users based heavily on the prompt provided, by analyzing the intent and audience of the prompt entered.
While ChatGPT offers a conversational interaction, there are no editing capabilities built into the system itself, requiring users to copy, paste, and edit in outside locations. What is beautiful about ChatGPT is the simplicity and comfort of the conversational interaction. You simply enter your prompt into the chat box, hit send, and wait for your response!
ReWord
ReWord is a collaborative AI editor that can be trained to generate suggestions and content suited for your specific blog(s). In set up, you have the option to link or manually upload your published blogs, allowing the AI to learn your brand’s voice, tone, and quirks. The more information it has, the more it learns about you.
Once in the main editor, there are several interactive tools available to you:
- Research: This feature allows you to ask questions or prompt the AI. The feature also has built in questions that will assist in creating your copy such as:
- What subheadings matter most to my readers?
- What facts should my readers know
- What other titles could work for this article?
- Rephrase: This feature allows you to have AI rephrase highlighted sentences that may be confusing.
- Simplify: This feature allows you to consolidate multiple sentences.
- Enrich: This feature allows you to reword sentences that are simple, boring, or just need a good fluff.
- Command: This feature allows you to prompt or command the AI to write or edit sections or the entirety of your article.
How We Did It
We’ve talked about the tools we used, now let’s dive in to our approach.
ChatGPT
Rather than allowing ChatGPT to generate both the content outlines and the content, we prompted ChatGPT with human-generated outlines. To do so, we used our prompt:
For our listicle pieces, product descriptions were a necessary component. We leaned on the product descriptions prompt from our ChatGPT Prompt Library to help us generate these.
We compiled these prompts into our outlines so ChatGPT received them as one outline and would align them with each other.
While the content itself was sufficient, we noticed that the product descriptions were not. The product descriptions provided needed to be edited for tone and voice to fit in the article. The amount of editing it took us to reach a product description acceptable for a listing highlighted to us that we should save these sections to be written by humans.
ReWord
When generating a new article, ReWord asks for the title, a brief description of the intent of the piece and who you think will be reading it, in order to learn more about your brand and audience. As it learns about you and your brand, it will provide you with a list of recommendations of topics that your audience may want to read about.
When generating these prompts, we tried to be as specific as possible. We focused on a one sentence description of the objective of the article, followed by a one sentence description of the target audience. This description was not heavily demographic based, rather experience.
For example, if we were to write a prompt about copywriters, we would say “Our intended audience is novice copywriters looking to begin their career” rather than “college-aged women who are just getting their start”.
When writing the pieces, we took two approaches: (1) Allow ReWord to generate the copy completely and (2) Generate the content in stages, using the tools for support.
In Approach 1, we used prompts similar to our ChatGPT prompts:
“Write an article targeting the SEO keyword “insert keyword here”. Your intended audience is ______. Write as if you are an expert in _____. Write no less than 1,000 words.”
This prompt changed as we learned the nuances of the tool. If there were specific things we wanted it to avoid saying, we added it into the prompt at the end.
In Approach 2, we asked the tool to assist section by section.
“Write an introduction for an article targeting the SEO keyword “insert keyword here”. Align it with the title provided above”.
After completing the introduction, we would use the research tab to see what topics readers would be interested in learning about this topic. Those topics were used to guide the generation of the headlines of each subsequent section.
For the remaining sections, we asked questions of the tool, based on the headline.
For example, using our copywriting example above, if a headline read “How Do I Know If Copywriting is the Right Career for Me?”, we would ask the tool:
“What are potential advantages and drawbacks of having a career in copywriting?”. We then used the magic of human touch to make the piece flow together.
We followed those steps until the article was complete.
In both approaches, we noticed that the amount of editing and human intervention required did not enhance our processes or efficiency.
When using ReWord, we noticed a weakness was gender bias. If you are able to provide a prompt without including a specific gender as the focus, the content is much more streamlined and well-rounded. Gender-specific prompts led to heavily stereotyped responses on both sides.
Closing Thoughts
Our experimentation with AI-powered content creation tools, specifically ReWord and ChatGPT, revealed some important insights. While these tools offer innovative features and promise increased efficiency in content production, they do have limitations.
ReWord, despite its AI capabilities and user-friendly interface, didn’t prove to be significantly faster or more cost-effective than hiring a human copywriter. The editing required to bring the AI-generated content up to standard was a time-consuming process, and the dynamic link inserter did not work as seamlessly as we had hoped.
On the other hand, ChatGPT, a generalist AI tool, showed potential for quickly generating content at a minimal cost. However, the content it produced lacked consistency, often being contrite, nonsensical, or boring. It required extensive editing and attention to detail, making it less suitable for high-performance, long-term SEO content.
Ultimately, both ReWord and ChatGPT have their merits but won’t fully replace the need for human writers any time soon, especially when it comes to producing high-quality, engaging content. These AI tools can be valuable adjuncts in content creation, especially for organizations with limited resources and tight timelines, but they’re not a one-size-fits-all solution. Content quality and performance still heavily depend on human expertise and intervention.
