Generative AI, Large Language Models and the Business of Content Creation

Stories of chatbots falling in love with journalists. Actors striking to demand ownership of their digital likenesses. AI-generated authors at legacy media giants publishing questionable content. If you’re involved in the business of content creation, these aren’t just the latest eye-catching headlines; they’re the backdrop to the quality, veracity, scale and value questions that the latest round of technological advancements – collectively labeled generative AI or GenAI – pose. User expectations and the technological environment surrounding content creation are both changing rapidly. Case in point: since I’ve started writing this article, I’ve had to go back and revisit assumptions and the current landscape on a daily – and sometimes hourly! – basis.

By now, you’ve probably heard everyone you know sharing their take on GenAI in one form or another. You might even be considering implementing it in your own content creation processes – or you already have. Wherever you stand on GenAI, now is the time to be talking about its impact on content creation at a business level; so, let’s talk about it!

Exploring generative AI as a business solution

If you’re considering adding GenAI to your workflow or content creation processes, you’re likely aware that you’ve got a pretty big decision in front of you. At this point in the journey, you and your team will need to answer some fundamental questions before you dive in.

Can generative AI solve your problem?

It’s one thing to be open to integrating new technology into your content creation, but another thing entirely to define the problem you’re trying to solve and figure out whether GenAI is the right solution. Answering this question requires a deep understanding of your business processes and any inefficiencies. You’ll need to ask yourself where automation can bring the most value, and which content processes – present or future – could benefit from speed and scale. Once you’ve clearly defined the problem, you can turn your attention to solutions.

To decide whether GenAI is the right solution for your business, you need to assess whether it suits the problem. If the task at hand involves using a template to produce content at scale, personalizing existing content, or creating multiple versions of a document, GenAI could be a perfect fit. Remember, the goal isn’t just to integrate AI, but to do so in a way that brings maximum value to your business.

For most of the forty-plus years that IBISWorld has been publishing industry research, our analysis has been longform. Our team of 100+ research staff write and update thousands of industry reports that run to 30-40 pages each. Driven by user demand and enabled by improved back-end technology, we’ve spent the past two years neck deep in our digital transformation project, the Modern Research Experience.

One key part of that transformation involved our research team converting our paragraph-style analysis into list-style insights grouped together under thematic headlines. Our primary goal was to make it easier for our clients to quickly find the specific nuggets of information they really need to craft a credit analysis, valuation memo or pitch deck.

As much as anyone in product development might harbor the secret dream that every user will love every feature that we put so much thought and development effort into, we also know that constructive feedback is what’s REALLY useful. After we launched the Modern Research Experience, we dug into survey feedback and found that we had a small (but meaningful!) subset of users whose primary job to be done required paragraph-style analysis – they would copy and paste or API call our content directly into their own longform document templates. We knew it wouldn’t make sense for us to double our research team or reduce how frequently we updated our reports just to have analysts write the same content in both list form and paragraphs. This was the problem we needed to solve, so we decided to evaluate whether GenAI could be the solution.

My role as the global Head of Product Development for IBISWorld – a company that strives to be the premier provider of industry information – involves grappling with the questions that GenAI poses every day. My team and I have been exploring, experimenting with and learning about the real-world applications of GenAI, and large language models (LLMs) in particular, for over a year.

Of course, in the world of product development, learning about and rapidly small-scale testing new additions to the tech stack comes with the territory. But a couple of factors set GenAI apart from previous advancements: these tools continue to develop at a faster rate than ever before, and they’ve become easier for stakeholders to experiment with, even without any coding knowledge. The pace of iteration and learning has skyrocketed for many of us, with no sign of slowing any time soon. As the GenAI landscape continues to evolve, we're choosing to embrace change one step at a time, making sure that our processes support both our adaptability and our commitment to quality and accuracy.

The GenAI cost-benefit analysis

Like any new technology, GenAI pushes us to weigh the risks and rewards of both early and late adoption. Being an early adopter of GenAI can give your organization a competitive edge, allowing you to position your business at the forefront of innovation and reap the rewards of streamlining your processes before your competitors. However, early adopters are also the first to encounter the potential risks and challenges that emerge – and the first to bear the costs, financial or otherwise, of taking the leap.

The wait-and-see approach allows you to watch what happens when other businesses adopt GenAI, so you can learn how to translate their successes to your business and avoid the mistakes they made. But wait too long and you risk being left behind by competitors that have already had time to work through the teething issues that arise when introducing any new element to daily operations.

Finding the balance between risk and reward is key, and the question of whether the timing is right for your business can depend on factors like your industry landscape, key markets, competitors, resources and goals. It also depends on which pros you value the most, and whether they outweigh the cons for your organization. The benefits might include:

• Better efficiency: GenAI can speed up your current operations and free up valuable time for your team to focus on higher value tasks.
• Cost savings: Automating content production (or, more realistically, elements of it) can help your business produce more content without hiring more staff.
• Scalability: Using GenAI can help your business scale operations more effectively and has applications for small and large businesses and teams.
• Personalization: GenAI can personalize content for individual users, improving the customer experience.

On the other hand, GenAI can expose you to risks surrounding:

• Quality and accuracy: No GenAI model is perfect, and the content they generate can sometimes be inaccurate or lack substance.
• Data privacy: Some GenAI models feed the information you give them back into their own database, potentially exposing sensitive personal or company data to other users.
• Regulation: Copyright laws and other regulations surrounding GenAI lag behind its adoption and may pose challenges for businesses as legal frameworks catch up.
• Intellectual property: Debates about who owns AI-generated content and the ethics of training models on scraped data are unresolved and may pose risks to content ownership in the future.

After carefully weighing the pros and cons, we decided to test the capabilities of GenAI within our content process, in a project that has come to be known as Paragraph Builder. Realistically, enterprise-level GenAI solutions are still in their infancy, and we have a commitment to providing our clients accurate, human-verified analysis and forecasts, so we had a few conditions that the task needed to meet to move forward. The criteria included:

• A simple user problem: In this case, the problem was that users needed content in a different format, so we only needed to turn bullet points into paragraphs.
• No new content creation: The information had already been written and verified by analysts, so we didn’t need GenAI to create new analysis and risk the model introducing inaccurate content.
• Limited and well-structured training data: LLMs benefit from being trained on well-structured, clearly labelled datasets. IBISWorld reports are structured around a consistent, easy-to-navigate framework across our collections, making the process of training the model much more straightforward.
• A human-verified fail safe: Even trained on a verified and well-structured data set, today's GenAI outputs aren’t always perfect. By publishing both the list-style analysis and the AI-generated paragraphs, users could always flip back to the human-verified content to clarify any language errors that crept in.

With these boxes checked, it was time to work out some of the finer details.

Choosing an AI model

Like picking any supplier, software or tool, deciding on a GenAI model requires research. Consider factors like:

• Ease of use
• Integration with your existing systems
• Options for customization
• Whether the functionality meets your needs
• Ongoing costs
• Data privacy safeguards

Once we’d examined our options for GenAI models – and our engineers brought the product team’s lofty dreams of building our own back down to earth – we quickly agreed that creating this functionality on top of OpenAI’s LLM was the best fit for this project’s purpose.

Implementing generative AI for content creation

Okay, say you’ve defined your problem, made it through the cost-benefit analysis and decided that GenAI offers a worthwhile solution. Now it’s time to implement it! Your implementation process will vary depending on your use case, but it’s likely to follow a fairly standard course.

Gather data

Find, sort and clean the relevant data that your model will use as inputs. Depending on the kind of GenAI model you’re using, this might include your previous content, transactional data, user feedback, customer behavior data or sensor data.

Train the model

Once you’ve gathered your data, feed it to your GenAI model of choice. This teaches the model to recognize patterns and structures. Over time, the model will learn and improve, and eventually be able to create content that aligns with the patterns it has learned.

Test and refine

Test your model by having it create sample content. Review the results and refine the model until you reach a standard of content that you’re happy with, and that fulfils the goals you set out to achieve.

Roll out the model

Going live with your GenAI model looks different depending on what you’re doing with it. If you’re using GenAI to create marketing content, you’ll need to integrate it with other tools, train your teams to use it, implement a quality control process and then publish the content you produce.

Our process: from problem to Paragraph Builder

Because the aim of Paragraph Builder is simply to rewrite existing content, we already had our industry reports and their associated datasets close at hand, which made our data-gathering phase simple. Next, our team wrote prompts that queried the right content and returned results in line with IBISWorld’s style and tone guidelines. We ran the model across a variety of our reports and chapters, and iteratively honed the prompts to improve the responses.

Even with our best prompt engineering and OpenAI’s world-class model, the nature of GenAI as it stands means that inaccuracies sometimes creep in. Basically, an LLM is a massively scaled up version of your phone’s predictive text function. It takes huge amounts of text data and uses it to determine which words are most likely to come next. Sometimes, these models get it wrong. Our clients rely on accurate data, so our project management team audited the model’s output to assess the scope of the errors. This process gave us the confidence that the issues were infrequent, minor and manageable enough to launch, and allowed us to create informative guides for our users to explain these instances.

For us, rolling out the model looked a bit like a standard product launch. Our client support teams reached out to users to build a beta test group, while our designers and developers created the front-end interface elements that made Paragraph Builder easy to use. Marketing collaborated with our data team to create FAQ guides and communications that would educate our beta testers, and we went live!

Reviewing generative AI: did it work?

As with any new process or technology you implement in your business, you’ll need to monitor the results and make changes as necessary. Staying on top of improvements or changes in the world of GenAI is crucial to ensuring that your solution continues to meet your business’s needs in the future. Depending on your use case, you might request feedback from your users and customers, or track relevant metrics like engagement or conversions.

At the time of writing, Paragraph Builder has been available for our beta testers for just over 24 hours, so it’s too soon to know where it’s hitting the mark and where it needs more work. As we eagerly anticipate the user feedback rolling in, the team has already reaped the benefits of what we’ve learned about GenAI and integrating LLMs into content creation – benefits we’re already planning to use in our next set of discovery and experiments. Check back for Part 2 of this series in a few weeks to find out how things are shaping up!