Beginners Guide to using Generative AI as a tool for Qualitative Data Analysis
The use of Artificial Intelligence (A) and in particular Generative AI is in an exponential surge across every industry and sector. The Research and Design sectors are no different.
AI has become an integral part of our daily lives and is reshaping how businesses and institutions operate. In fact, at Indeemo we have been using various forms of AI for several years. The release of ChatGPT late in 2022 however has led to an explosion in AI awareness and adoption and it seems like every second article we read now has some reference to the disruptive potential of Generative AI.
Generative AI is a game-changing subset of AI that's transforming how we analyse qualitative data. It's not confined to one sector; its applications span across research, healthcare, finance, marketing, education and more.
Businesses, Researchers, Designers and decision-makers are turning to cutting-edge Generative AI tools to unlock valuable insights and increase productivity. In this beginner's guide to generative AI, we'll give you a quick intro to Generative AI and the use of prompts for speeding up Qualitative Data Analysis.
What is Generative AI?
Generative AI, short for Generative Artificial Intelligence, is a revolutionary subset of artificial intelligence (AI) that stands at the forefront of innovation.
Unlike traditional AI, which focuses on interpreting and processing existing data, Generative AI takes a creative leap by generating new, human-like text based on the patterns and knowledge it has acquired from vast datasets.
At its core, Generative AI employs Large Language Models (LLMs), particularly models like GPT (Generative Pre-trained Transformer), to understand, interpret, and generate text that is coherent, contextually relevant, and often indistinguishable from content produced by humans.
The most famous Generative AI tool out there is ChatGPT. It absolutely exploded in late 2022 / early 2023.
Generative AI models
ChatGPT however is just a chat interface to the underlying Generative AI models. The first Generative AI that hit the mainstream media was GPT-3.5 and this was quickly followed by GPT-4. It has again been superseded by GPT-4 Turbo with GPT5 rumoured to be arriving soon. There are now a multitude of Generative AI providers out there with different capabilities.
OpenAI is the company that is synonymous with Generative AI as it was the developer of ChatGPT. There are however a lot of competing alternatives in the marketplace such as Gemini, Anthropic, Hugging Face, Midjourney etc.
In time, we believe that Generative AI models will be as common as apps in the app store.
Generative AI, with its ability to craft text, stories, responses, and more, is a technology that can understand context, follow conversation flow, and respond to prompts with remarkable fluency. These attributes make it extremely useful for qualitative data analysis.
“If you’re new to Prompt Engineering, don’t worry! Our experts will train you in how to prompt so that you can leverage the power of this transformation in qualitative data analysis.”
What are the Benefits of Generative AI for Qualitative Data Analysis?
Generative AI's role in qualitative data analysis is transformative, offering a fresh perspective on how researchers and analysts can derive insights from unstructured data. In the past, qualitative data analysis often involved manually sifting through and manually coding extensive text-based records, survey responses, or customer feedback. This laborious and time-consuming process made it challenging to extract meaningful patterns, themes, or sentiments from large datasets.
In a research context, Generative AI is a game-changer technology. It automate and enhance the process of qualitative data analysis. It can quickly process massive volumes of text, distil key insights, and even generate new text to address specific research questions.
Firstly, think of Generative AI as your research assistant
Before we dive into the benefits of Generative AI however, let us set out our positioning on this: we believe that it will always take a human to truly understand another human.
Researchers have a remarkable skill set at uncovering nuances across qualitative datasets. They are able to pick up on subtle emotional changes as they are the ones who get to know their research participants through either a focus group, interview, or observational study.
In this framing, we believe that Generative AI will play the role of research assistant and will drastically speed up the routine jobs such as summarisation, theme extraction, translation etc. leaving the researcher with more time to get deeper into the data.
With this in mind, here's how Generative AI can revolutionise qualitative data analysis:
It speeds you up and makes you more efficient
Generative AI can process and analyse vast amounts of textual data in a fraction of the time it would take a human analyst. This means that businesses and researchers can quickly gain insights from large datasets, accelerating decision-making and research processes.
Additional Platform Capabilities
• Recruit Participants in hours
Recruit B2C and B2B participants in hours from a panel of 3 million+ Respondents.
• Speed up analysis with Generative AI
Use Generative AI prompts for summarisation, translation, thematic analysis, sentiment analysis etc. and reduce analysis time by at least 40%.
• Analyse Interviews and IDIs
Import your interviews from Zoom, Microsoft Teams or your computer, transcribe them in 27 languages and analyse them fast with Generative AI.
• Analyse Interviews and IDIs
Import your interviews from Zoom, Microsoft Teams or your computer, transcribe them in 27 languages and analyse them fast with Generative AI.
It’s a consistent and objective virtual research assistant
The clients we work with who have used Generative AI extensively tell us that it is extremely objective and consistent. Some of the researchers who have completed A/B tests (where they analyse the data themselves first and then use Generative AI as a comparison) tell us that the AI, through it’s objectivity has forced them to revisit their own analysis and has enabled them to find insights they previously missed or did not consider.
By acting as an objective virtual assistant, Generative AI ensures that every piece of data is treated objectively. This objectivity and consistency is invaluable when analysing data over time or across different datasets..
It allows you to get deeper into Qualitative Data faster
Generative AI has the capacity to quickly uncover patterns and connections within data that might not be immediately apparent at a first glance. It can explore relationships, uncover hidden trends, and generate alternative perspectives, leading to a more comprehensive understanding of the data.
As timelines and budgets increasingly shrink, having this virtual assistant to quickly get you deep into the data is extremely valuable.
It helps you analyse much more Qualitative Data
As the volume of qualitative data continues to grow in the digital age, scalability becomes a crucial consideration. Generative AI can efficiently handle large datasets, ensuring that your qualitative data analysis scales with your needs.
It's important to note that while Generative AI offers numerous advantages, it is not a replacement for human expertise. Instead, it complements human analysts and researchers by speeding up repetitive tasks, facilitating more rapid insights, and providing a fresh lens through which to view qualitative data.
In this context, Generative AI is a powerful ally for researchers, businesses, and institutions looking to unlock the full potential of their qualitative data. It speeds up the analysis process, drives informed decision-making, and opens the door to new avenues of research and discovery.
Whether you're studying consumer behaviour, customer feedback, or any other qualitative data, Generative AI is a valuable tool that's transforming the landscape of qualitative data analysis.
What are Generative AI Prompts and how can I use them for Qualitative Data analysis?
Generative AI Prompts are the specific instructions or questions you give the Generative AI model to give you your desired output. Think of prompts as the means by which you (a) give the AI context on what you are sending it and (b) tell it what you want it to help you do.
Forget “Prompt Engineering”, prompting is just a dialogue with your data.
If there’s one phrase I absolutely hate, it’s the term “prompt engineering”.
I can imagine that this was coined by a bunch of coders and executives in Silicon Valley to make what they do sound exclusive and highly technical.
Don’t get me wrong, there is a science and sometimes an art to prompting, but you do NOT need a degree in computer science or software engineering to be able to prompt like a champion.
However, to cut through the BS that surrounds prompting, approach it through this framework: think of Generative AI as your virtual assistant. Like any assistant, if you want it to do something for you, you need to give it context on the task you are asking it to perform and you need to tell what you want it to do as precisely and clearly as you can.
And once you start prompting, to really unleash its power, you need to be curious, experimental and iterative.
Here are a few top tips to get you started.
Be clear and precise with your instructions.
To create effective prompts, it's crucial to be clear and precise in your instructions. The wording, context, and details you include in your prompt directly influence the quality of the AI-generated response.
Whether you're seeking to extract insights, draft content, or answer questions, crafting well-defined prompts is key to obtaining meaningful and relevant outcomes.
A typical prompt consists of two components: the input text and the desired output. The input text (what you tell the AI and ask it to do) sets the stage by providing context, and the desired output outlines what you expect from the AI model.
By carefully structuring prompts, you guide the AI in generating text outputs that align with your objectives, making it an indispensable tool for qualitative data analysis and content creation.
What are Generative AI Tokens?
Tokens are the pieces of words that the AI uses to perform its magic. In the English language, a token is typically 3-4 characters long. Before the Generative AI model processes a prompt, what you send it will be broken down into tokens. Here are some ways to think about tokens:
1 token is approx. 4 chars in English
1 token is approx. ¾ words
100 tokens is approx. 75 words
Or
1-2 sentence is approx. 30 tokens
1,500 words is approx. 2,050 tokens
A 1 hour Zoom Interview is approx. 10,000 tokens.
Why are Generative AI Tokens important?
Tokens are also how we measure the capacity of the Generative AI models. GPT-3.5 8k for example has a limit of 8,000 tokens. GPT-4 32k has a limit of 32,000 tokens.
The total number of tokens in your prompt has implications for various aspects, including cost, response time, and whether the AI model can handle it effectively.
Each AI model has a predefined limit on the maximum number of tokens it can process in a single request. If your prompt exceeds this limit, it may need to be truncated or divided into smaller parts, potentially affecting the continuity and coherence of the generated text.
Efficiently managing token count is crucial to optimising your use of Generative AI. By being mindful of token limits, you can ensure that your prompts are well-structured and aligned with your goals, preventing any unnecessary limitations or disruptions in the generation process.
What is the impact of “Temperature” in qualitative data analysis prompts?
Temperature is a parameter that plays a vital role in controlling the randomness of the AI-generated text. It acts as a lever to adjust the creativity or determinism of the outputs from the Generative AI.
A higher temperature value, such as 0.8 or 0.9, makes the AI-generated text more random and creative., If you are looking for Shakespeare-style poetry, then this is the temperature to go with.
Conversely, a lower temperature value, like 0.2 or 0.3, makes the output more focused and deterministic. In this mode, the AI model is more likely to generate text that aligns closely with the input context, ensuring a more controlled and specific response.
When you’re doing qualitative analysis, minimising the temperature makes the AI more deterministic and less creative.
As a result, our approach at Indeemo is to minimise the temperature setting.
At Indeemo, we are constantly iterating and experimenting with prompts for qualitative data analysis. We have developed a framework to help our clients quickly get up to speed with prompting and when you work with us, we will partner with you to create effective prompts that get you up and running with Generative AI qualitative data analysis.
As we said at the start of this section, forget about “prompt engineering”. All you need to do is starting typing what you are thinking and be clear and concise when telling the Generative AI (a) what you are sending it (context) and (b) what you need it to assist you with.
The real upside of Generative AI tools
As mentioned, ChatGPT exploded into the main stream in the early part of the year and we’ve all heard stories of how it has passed the bar exam and will likely spell the end of using essays as a means of measuring what students now.
Long story short, if you are in the research industry and you are using text (e.g. open end questions in surveys) as the medium through which you understand your research participants, you can no longer be sure that the text you are reading was written by a human or AI.
In a research context, ChatGPT is the end of trust in text.
If you want rich, reliable insights that you can truly stand over in terms of authenticity and proof of origin, then you must move your qualitative research to video.
Not only is video the richest form of data collection, it allows you to stand over the origin of the data that you use to create your client reports.
Doing video research and analysis at scale
For years, clients were quite sparing in their use of video as the tools didn’t exist to help them do video analysis at scale. Sure, we had automated video transcription and keyword analysis tools, but the concept of having a virtual assistant to summarise transcripts or extract themes didn’t exist, until now.
Generative AI is going to be a boom for qualitative research and video research in particular.
it is now possible to do video research with hundreds of participants and analyse it in a fraction of the time it traditionally took.
This is why we integrated with Microsoft Teams and Zoom. We now allow you to import Focus Groups, In-depth Interviews, and even in-person sessions directly into the Indeemo Platform.
With the ability to automatically transcribe videos in over 25 languages, our platform makes it possible to do multi country, video research at scale.
Through Generative AI prompts, researchers can effortlessly summarise transcripts, surface key themes, extract timestamped quotes, and even translate videos from one language to another in mere minutes.
Eugene Murphy, our Founder & CEO, emphasises the significance of this enhancement, saying, "In an era where it is becoming increasingly difficult to trust faceless, contextless text as a reliable source of understanding, we have invested heavily into enhancing our video analysis capabilities to empower our clients to do video research faster and at scale."
Indeemo is not just a platform; it's your partner in reducing analysis time by at least 40% and scaling your video research efforts.
Let our Generative AI tools support your Qualitative Data Analysis
Indeemo is available as a self service SaaS offering under annual licence or on a project by project basis if your research is more adhoc in nature.
If you're ready to explore the future of qualitative data analysis, schedule a quick demo with us today and discover how our Generative AI capabilities can transform your qualitative data analysis.
Recent Articles
