An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Gregory Guest, PhD, MA, Emily Namey, MA, Amy O'Regan, MS, Chrissy Godwin, MSPH, and Jamilah Taylor.
Author Information and AffiliationsOnline focus groups (FGs) and individual interviews (IDIs) are increasingly used to collect qualitative data. Online data collection offers benefits (eg, geographic reach), but the literature on whether and how data collection modality affects the data generated is mixed. The limited evidence base suggests that data collected via online modalities may be less rich in terms of word count, but more efficient in terms of surfacing thematic content. There is also limited evidence on the comparative costs of online vs in-person data collection.
The purpose of this study was to compare data generated from FGs and IDIs across 4 data collection modalities: (1) face-to-face; (2) online synchronous video-based; (3) online synchronous text-based; and (4) online asynchronous text–based. We also aimed to compare participant experience and data collection costs across modalities.
We used a cross-sectional quasi-experimental design. We systematically assigned participants to 1 of the 4 modalities (according to a rolling sequence) on enrollment and randomly assigned them to either IDIs or FGs. We held constant the interviewer and question guide across 24 FGs (n = 123 participants) and 48 IDIs, conducted between September 2016 and October 2017. Participants also completed a brief survey on their experiences of data collection. A team of 3 analysts performed inductive thematic analysis of the qualitative data, generating and applying emergent theme-based codes. We also used a priori codes to tag sensitive information across modalities. Analysts were not masked to data type, but all transcripts were coded independently by 2 analysts and compared to reach final consensus coding. We operationalized data richness in terms of volume of participant data, measured by word count, and thematic content, measured by the number of thematic codes applied per modality. Using time and expense data from the study, we calculated average cost per data collection activity.
Visual (face-to-face and online video) modalities generated significantly greater volume of data than did online text-based modalities; however, there were no significant qualitative differences in the thematic content among modalities for either IDIs or FGs. Text-based online FGs were more likely to contain a dissenting opinion (P = 0.04) than visually based FGs, although this level of significance should be interpreted cautiously due to multiple comparisons. Participant ratings of data collection events were generally in the moderate to high range, with statistically significant differences in participant experience measures by modality for FGs: participants rated online video FGs lower than others on several measures. Without travel, online video data collection had the highest average costs for both IDIs and FGs; however, if estimated travel costs are included, then in-person data collection was more expensive.
Among our sample, online modalities for conducting qualitative research did not result in substantial or significantly different thematic findings than in-person data collection. We did not find that online modalities encouraged more sharing of personally sensitive information, although we observed more instances of dissenting opinions in online text-based modalities. The homogeneity of the sample—in terms of sex, race, educational level, and computer skills—limits the wider generalizability of the findings. We also did not have a geographically distributed sample, which prevented us from having actual travel expenses for the cost analysis; however, the findings from this study were largely consistent with previous comparative research.
Qualitative research refers to research that generates and/or uses non-numeric data, most typically text,1,2 and is characterized by open-ended questions and inductive probing. Given its inherent ability to allow people to respond to questions in their own words, in an open-ended way, qualitative inquiry excels at capturing individual perspectives in rich detail. Qualitative research is therefore employed across a broad range of health topics and in various settings to answer “how” and “why” type questions that are difficult to assess using closed-ended survey responses.
Two of the most common qualitative research methods are in-depth individual interviews (IDIs) and focus groups (FGs).3,4 IDIs, as the name implies, generate personal narratives from individuals. They are conducted one-on-one, last from 30 minutes to an hour, and are aimed at understanding processes and/or eliciting a participant's experiences, beliefs, or opinions on a specific topic.
Similar to in-depth IDIs, FGs use open-ended questions and an inductive probing pattern, but the group environment offers the opportunity to observe and draw on interpersonal dynamics. Ideally, FGs are constructed and conducted to take advantage of the group dynamic to stimulate discussion and a broad range of ideas. FGs usually range in size from 6 to 12 individuals (who are similar to one another in some way that is related to the research topic) and are conducted by a moderator and an assistant. FGs typically run from 1.5 to 2.5 hours and are well suited for evaluating products and programs and for gathering information about group norms or processes.5,6 One key difference between IDIs and FGs is that response independence cannot be assumed in an FG setting; therefore, the group is typically the unit of analysis when data are collected via FGs.
FGs and IDIs are employed in all fields of research, including patient-centered studies. These qualitative data collection methods are co-evolving with technology7 and are increasingly conducted online.8 An often-cited advantage of online data collection is that a researcher can collect data from individuals across multiple locations without needing to travel,9,10 reducing both time and costs.11,12 More specifically, online approaches can extend to stakeholders whose input would be lost if only face-to-face techniques were used, including populations for whom travel might be difficult. Online qualitative data collection modalities have been shown, for example, to work well with patient populations facing unique health challenges such as traumatic brain injury,13 autism,14 multiple sclerosis,15 and children with chronic conditions.16
Online qualitative data collection modalities can be categorized along 2 dimensions. One dimension refers to the nature of the communication medium: text-based or video-based. In a text-based modality, questions and responses are typed via computer. Video-based modalities use online video (with audio) technology, and questions/responses are spoken. The other dimension pertains to temporality: synchronous and asynchronous. Synchronous methods are conducted in real time through text or video-conference platforms.17 Conversely, asynchronous methods do not occur in real time; they are typically conducted through venues such as discussion and bulletin boards, email, and listservs, where a question can be posted and respondents answer at their convenience.18 Synchronous methods tend to be relatively fast-paced with a conversational back-and-forth communication flow, whereas asynchronous methods allow participants more time to consider and respond to questions. The latter are purported to generate richer and deeper data.17,19
The sampling benefits of online data collection, in terms of geographic and logistical flexibility, have been at least anecdotally documented among several patient populations. The effects of data collection modality on the information generated, however, have not been rigorously investigated. The limited number of comparative studies that exist suggest that online techniques, in general, generate a smaller volume of information—measured in terms of time and quantity of textual data20,21—than their face-to-face (ie, in-person) incarnations, and that face-to-face techniques generate “richer” data than online techniques.15,16,20 A study by Campbell and colleagues further observed that the face-to-face context “caused some participants to hold back from discussing information that they felt was too personal or potentially embarrassing”10; Woodyatt and colleagues also found slightly more discussion of sensitive topics in online versus face-to-face focus groups.22 This phenomenon—whereby participants are more likely to be open and discuss personal information online—is known as the online disinhibition effect.23
These few studies are important but lack the necessary rigor to provide a foundational evidence base. None of the studies employed an experimental design. The majority did not control for instrument or interviewer variation, had very small sample sizes, and lacked systematic and transparent analytic procedures. The 2 studies that compared richness of data, for example, did so without operationalizing the term or describing how they identified and compared this construct. It is therefore difficult to assess the validity of the comparisons. Another limitation of previous research is that studies focused on comparing only 2 data collection modalities or used only asynchronous techniques as the online context. Additionally, earlier studies failed to compare relative modality costs, an important research planning component. Finally, few of the referenced studies addressed the larger question of participant satisfaction with data collection modalities, which is critical for developing rapport and participant comfort for valid and successful qualitative inquiry.
Based on the state of the evidence, our study had 3 primary objectives:
This research was funded by PCORI, which requires reporting according to its methodology standards.24 We consider PCORI Methodology Standards 1 and 3, covering standards for formulating a research question and for data integrity and analysis, respectively, as most directly applicable to this methods-related research. Accordingly, we identified gaps in the literature and then developed a study protocol. Regarding the selection of appropriate interventions and comparators, we describe the different arms of our study (though not health/patient related) in this section, and we also define and operationalize our outcome measures. Data analysis procedures are also outlined in this section.
Given our methodological objectives, large sample requirements, and stratified participant assignment to data collection modality, we sought to define a study topic and population that was relatively ubiquitous (not too narrowly circumscribed) and could provide a wide pool of potential participants. We also wanted to generate data that could be useful in informing patient–provider interactions in an area of maternal health; after discussion with local obstetrical colleagues, we focused the topic of our data collection efforts on pregnant women's thoughts about medical risk and Zika. Our study population included women in the Research Triangle area of North Carolina over age 18 who had been pregnant between 2013 and 2016 and who were hoping to become pregnant again in the next 3 years, at the time of enrollment. Additionally, women enrolled had to have internet access and reasonable typing skills and had to agree to random assignment to either an IDI or FG and assignment to either online or in-person data collection. Analytically, we viewed sample homogeneity as a relative strength, in that it was one less potential confounder of our comparisons between modalities. We therefore did not set explicit demographic diversity criteria.
We recruited participants through a combination of local community events, magazine advertisements, radio announcements, flyers posted near establishments that serve pregnant and postpartum women, online classifieds, and online social networking groups. We also asked study participants to refer other women to the study website and recruiter. Once women contacted the study recruiter, they were screened for eligibility and provided informed consent if eligible.
Each study participant was provided an incentive of a webcam (worth about $15) and an Amazon gift card initially worth $40, and later raised to $80. (The monetary incentive was increased approximately halfway through the study, to increase enrollment rates. This increase was made after a round of data collection was completed, to keep any effect the same across arms.) An additional $30 reimbursement was offered to participants who indicated a need for childcare because they had to leave their homes for face-to-face data collection.
This was a cross-sectional, quasi-experimental qualitative study with 8 arms, distributed as per Table 1. Study research questions and data analysis methods are summarized by objective in Table 2.
Eight Study Arms by Data Collection Method and Modality.
Study Objectives, Research Questions, and Data Analysis Methods.
Randomly assigning women to modality was logistically challenging given the limited availability of the study population; instead, to limit participant selection bias among arms, we systematically assigned women to a modality and then randomly assigned the data collection method (Figure 1). As women consented to participate and were enrolled in the study, the first 15 women were assigned to the “scheduling pool” for the face-to-face modality. The next 15 women to enroll were assigned to the online video modality, and so on through each of the 4 modalities for the first round of data collection. From each distinct group of 15 women, 2 were randomly selected (according to a computer-generated sequence provided by a study statistician) to take part in an IDI. The other 13 were assigned to the focus group for that arm, an intentional over-recruitment to ensure that 6 to 8 participants from the group could attend on the same date and time. Any women who were assigned to an arm but not scheduled (due to availability conflicts) were rolled over into the next open scheduling list and rerandomized. The process was repeated for all subsequent rounds. Women were masked to their assignment until they were scheduled. We scheduled most events during usual business hours (Monday through Friday between 9 am and 5 pm), while maintaining some flexibility for synchronous events to occur on evenings or weekends, if necessary, to accommodate participants' schedules.
Systematic Assignment of Participants to Modality and Method.
The study was reviewed and approved by FHI 360's Protection of Human Subjects Committee, and verbal informed consent was obtained from all participants, individually, before initiation of data collection. Data were collected from September 2016 to October 2017.
The data collector (E.N., an experienced qualitative researcher) and the question guide were held constant across all modalities. Procedures, aside from technical connection requirements, were also kept consistent within each modality. Instrument questions were open-ended and asked in the same order, to enhance comparability.3,25 As with any qualitative inquiry, the data collector inductively probed on participant responses to seek follow-up clarifications or expansions on initial answers.3
Questions explored women's perceptions of what is “safe” and “unsafe” during pregnancy, considerations regarding preventive medical interventions along this continuum (eg, vaccines), and willingness to participate in clinical research while pregnant (see Appendix A). Before implementation, the data collection instrument was pilot-tested among an in-person focus group of 5 study-eligible, consented participants to enhance clarity and validity of questions. (These participants were excluded from later data collection events.)
At the end of each data collection event, participants completed a brief anonymous questionnaire containing several structured questions with Likert-scale response options on their perceptions of the event. We included questions on rapport, the feeling of a safe/comfortable space to share, and convenience (Appendix B). In the FG contexts, the structured questions were completed individually and independently from the group.
With permission from participants, we digitally audio-recorded all face-to-face and video-conferencing data collection activities. These audio recordings of FGs and IDIs were transcribed verbatim using a standardized transcription protocol.29,30 Transcripts for the text-based FGs and IDIs were automatically generated as part of the data collection process.
As described above, we intentionally kept key elements of the data collection process the same to minimize possible confounders. The modalities required some differences in the conduct of data collection, in accordance with best practices for each modality,8 as described below and summarized in Table 3. The duration refers to the time we asked participants to set aside for the activity; actual time averages are presented in the Results section.
Data Collection Methods by Modality.
The face-to-face modality for FGs and IDIs followed traditional qualitative data collection procedures.3,26 All face-to-face FGs and IDIs were conducted in a conference room at the study office. In-person focus groups also included an assistant who helped with greeting participants and providing refreshments.
All synchronous online activity (video and text) participants used internet-connected computers, at their homes or other convenient locations, to sign in to a private “chat room” at a designated date and time to participate in the FG or IDI. The synchronous video events involved web-connected video through this platform, along with audio over a telephone conference call line. Participants could see the moderator, other participants (for FGs), and themselves. For synchronous text-based activities, the moderator typed questions and follow-ups while participants typed their responses, all in real time. Additional time was allowed for the synchronous text-based IDIs to accommodate the delays generated by typing back and forth. FG sessions were conducted in “group mode,” where participants could see each other's responses as they were entered and respondents could type responses simultaneously. These 3 synchronous modalities are subsequently referred to as face-to-face, online video, and online chat.
Asynchronous, text-based data collection modalities used an online discussion board platform (FGs) or email (IDIs). For FGs, the moderator posted a series of 3 to 5 questions on the discussion board each day over several days. Participants could sign in at their convenience, complete the day's questions, and read and comment on each other's postings. The moderator reviewed all responses and posted follow-up questions, as appropriate, to which participants could again respond. The same procedure was followed for asynchronous IDIs, except that the medium for correspondence was email—the interviewer emailed the participant a series of 3 to 5 questions to which the participant responded in 24 to 48 hours. The interviewer's next email would contain follow-up questions on responses and the next series of new questions. Participants were prompted to complete unanswered questions before moving on to the next question. We allowed 5 to 10 days for each data collection event to be completed via this modality, depending on the pace of the participants. The asynchronous modality is subsequently referred to as email-based or online posts, for IDIs and FGs, respectively.
The qualitative data generated through IDIs and FGs were coded using an analytic strategy that integrated 2 distinct approaches to qualitative data: inductive and a priori thematic analyses. Inductive thematic analysis is a set of iterative techniques designed to identify categories and concepts that emerge from the text during analysis.27 The analytic process entails reading through verbatim transcripts and identifying possible themes. An inductive approach is exploratory in nature; themes are not predetermined before analysis but rather emerge from the data as analysis progresses.27,28 We developed an inductive codebook using a standardized iterative process.29 Emergent themes were noted as 2 data analysts (E.N. and C.G.) read through the transcripts. As the data collector was also an analyst, this process commenced after all data had been collected, to avoid influencing data collection activities. All inductive codes were explicitly defined in the codebook using the template outlined by MacQueen et al.30 Analysts (C.G., A.O., and E.N.) then used NVivo 11 (QSR International) to apply content codes to the text of each transcript. More than 1 content code could be applied to any one segment of text.
In an a priori thematic analysis, analytic themes are established before analysis and instances of those themes are sought out and coded for in the data.31,32 For this study, we created a priori codes for “sensitive/personal” themes, so that we could assess if data collection modalities vary in terms of their capacity to generate these types of themes. We defined “sensitive” disclosures as containing information about one's own experience that is highly personal, taboo, illegal, or socially stigmatized in nature, which we would reasonably expect people to be reluctant to disclose to a stranger(s). We also created a “dissenting opinion” code for FG data, to capture instances when a participant expressed an opinion opposite to an opinion expressed by another participant earlier in the discussion. We included both explicit (eg, “I disagree”) and subtle (eg, stating a different opinion without framing it as a disagreement) statements of disagreement.
Two data analysts independently coded all transcripts. Analysts performed inter-coder agreement checks on each transcript, comparing all coding and discussing any coding discrepancies. A master transcript was created to reflect the agreed-upon coding. The inductive codebook was revised, as necessary, after each successive transcript had been coded, to reflect any changes to code definitions.
To assess our first hypothesis in objective 1, that face-to-face modalities would generate richer data than online modalities, we operationalized data richness in 2 ways. First, we considered data richness in terms of the volume of information offered by the participants, operationalized as the number of words contributed by the participant(s) within each transcript. Second, we considered data richness in terms of the meaning and content of the information contributed by participants, using thematic code frequency as an indicator of thematic content. Our methods for measuring data richness for each approach are summarized in Table 4.
Operationalization of Data Richness as It Relates to Objective 1.
To assess our second hypothesis in objective 1, that online modalities would generate more sensitive disclosures than would face-to-face modalities, we assessed the data coded as “sensitive” and recoded it to reflect the nature of the unsolicited sensitive/personal themes disclosed. We summed and compared the number of unique transcripts in which sensitive themes appeared and were coded.
To assess whether participants may be more willing to offer a dissenting opinion in an online vs face-to-face FG, we focused on FG data from a question that asked about the effect of Zika on women's personal views on abortion. The responses to this question were analyzed by (a) whether any member of the group dissented from the stated opinion of others in the group who had already answered the question (dissension); and (b) number/percentage of participants choosing to abstain from answering this question, either by remaining silent or explicitly deferring response (abstention). These data were then examined by modality of focus group.
We assessed participants' experiences of data collection both quantitatively and qualitatively. Participants' responses to a series of questions with Likert-scale response options were tabulated by modality. Comments provided in an open text box associated with each question were reviewed and summarized to augment interpretation of the quantitative data.
We considered many time and cost inputs related to data collection across modalities, as summarized in Table 5. For staff-related costs, we used illustrative hourly rates; for all other nontravel costs, we used averaged actual costs as documented during the project. Note that we performed recruitment, scheduling, data collection, and data formatting in-house; we contracted online hosting platforms and transcription services. Regarding data preparation for analysis, the live-generated transcripts from the 2 online text-based modalities required some post hoc formatting. Although our project did not include travel for in-person data collection, we included estimated travel expenses based on the literature.33 For both IDIs and FGs, we assumed 4 hours of round-trip travel time. For the IDIs, we divided travel cost and time by 4 to assess per-event costs, assuming that 4 IDIs could be completed on a single trip.
Time and Cost Inputs Relevant to Data Collection for Each Modality.
We tested all outcome measures for differences by modality, separately for FGs and IDIs. For some analyses, we also considered audiovisual methods (face-to-face and online video) compared with text-based online methods, where the visual connection of the method might have been more important than whether it was online or offline. For the continuous outcome measures—word count and total thematic codes generated—we used 1-way analysis of variance (ANOVA) and Tukey honest significance tests. Further, in order to control for the number of participants per group in the word count analyses of focus group data, an analysis of covariance (ANCOVA) test was used. The ANCOVA test allowed us to examine a continuous outcome (word count) using both continuous (number of participants) and categorical (modality) covariates in our model. For the dichotomous outcomes—sensitive disclosure and dissenting minority opinion—we used a chi-square test; for the responses on a Likert scale we used a Kruskal-Wallis test.
We enrolled 171 women, who were randomly assigned to either the FG or IDI arm and systematically assigned to data collection modality (Table 6).
Number of Participants per Study Arm.
Our study sample was fairly similar across modalities: mostly white well-educated working women, in their early 30s, and within a relatively high-income bracket (Table 7). Of note, we did not find evidence that systematic allocation to modality resulted in statistically different subsamples.
Demographic Characteristics of Study Participants by Modality.
Objective 1 sought to systematically compare differences in data generated across the 4 data collection modalities: face-to-face, online video, online chat, and email/online post. The primary hypothesis predicted that data generated from face-to-face modalities would be richer than data generated from online modalities, with a secondary hypothesis stating that online techniques would elicit more sensitive/personal information than would face-to-face techniques.
Table 8 presents word-based measures of data richness by modality. Comparing IDIs, the online video and face-to-face modalities produced the richest data in terms of how active participants were in contributing to the discussion. The mean number of words contributed by participants per IDI differed significantly between audiovisual and text-based interviews: Face-to-face and online video modalities produced significantly larger numbers of words spoken by participants than did online text-based modalities.
ANOVA/ANCOVA Comparisons of Participant Word Count by Modality.
A similar but more nuanced trend held for the FGs. The face-to-face and online video modalities generated the greatest mean numbers of words spoken by participants per FG and were not significantly different from each other, after controlling for the number of participants per group. The mean number of words spoken by participants in the face-to-face modality was significantly larger than the means observed for either of the text-based modalities. There was also a significant difference in mean participant word counts between online video and online chat modalities, but not between the online video and the online message board groups.
Across the aggregate data set, 85 thematic codes were developed to categorize women's opinions and experiences (see Appendix C). The same 85 themes were present in both the IDI and FG data sets, with only small differences in thematic content, as measured by thematic code application across modalities (Table 9). Within the IDI data set, 79 thematic codes were applied in the online video IDIs, compared with 77 in the face-to-face interviews and 73 in each of the online text-based (chat and email) interviews. Among the FGs, the face-to-face modality had the highest number (80) of unique codes applied, followed by online chat (79), online posts (77), and online video (75). For both FGs and in-depth IDIs, no significant differences emerged in the mean number of codes used for each modality (P = 0.39 and P = 0.15, respectively).
Frequency of Thematic Code Application by Modality.
Variations in frequency of thematic code application were present across modalities (as indicated by shading in Appendix C), more so than which codes were applied. The clearest differences among theme/code presence across modality came from low-frequency codes (ie, codes that did not appear often in the data set). There were 10 codes in the IDI data set that were used in only 2 modalities, and 3 codes used in only 1. Where these codes were applied, they were used in only 1 to 2 interviews per subsample (representing approximately 8%-17% of the subsample). Similarly, for the FG data set, 2 codes were used in only 2 modalities and 6 codes were used in only 1 modality. Here again, the codes in question were primarily lower-frequency codes, appearing usually in only 1 focus group within the modality (representing approximately 17% of the subsample). However, 1 discernable pattern emerged in the FG data: None of the 8 codes that were used in only 1 or 2 modalities were present in the online posting FG data.
We thematically recoded data coded at the a priori “sensitive” code to reflect the nature of the sensitive disclosures (Appendix B). Topics of sensitive disclosures included the following:
We did not directly solicit information on these topics. Frequencies of all disclosures are described at the individual level for IDIs and at the group level for FGs (because there is no response independence in a group, we count only the first disclosure).
Across the IDI data, 4 of these types of sensitive disclosures were present (Table 10). Personal experience with alcohol use during pregnancy was mentioned by interviewees in all modalities, with slightly more disclosures in the face-to-face and online chat modalities. Sensitive disclosures were made by the greatest number (42%) of participants in the face-to-face IDIs.
Frequency of Disclosure of Sensitive Themes by Method and Modality.
The FG data showed less variation across modalities. Disclosures of alcohol use and medication for a mental health condition were present consistently across all modalities. Differences appeared in those disclosures that occurred rarely—exposure to secondhand marijuana smoke, personal tobacco use, and having had an abortion previously—and were spread across the 4 modalities. At least 1 sensitive disclosure was made in 5 of 6 FGs (88%) for each modality. There were no statistically significant differences in overall sensitive disclosures by modality for either FGs or IDIs.
This analysis included FGs only and looked at 1 question on abortion to see whether women might be more comfortable offering a dissenting opinion in text-based online modalities where others in the group could not see them. In both online text-based modalities (chat and discussion board posts), at least 1 participant expressed a dissenting opinion on abortion in nearly all (5 of 6) groups (Table 11). In contrast, a dissenting opinion was raised in just half of the online video groups and one of the face-to-face groups. The nonvisual, online text-based focus groups were 2.8 (95% CI, 1.2-6.8) times more likely to contain a dissenting opinion (P = 0.01) than the “visual” face-to-face and online video focus groups.
Frequency of Dissenting Opinions Within Focus Groups and Abstentions Among Participants.
We also assessed how many participants in each FG modality abstained from this question. The percentage of abstaining participants was highest (21%) in the online discussion board posts, followed by the online video FGs (18%) and online chat FGs (12%). No one abstained from offering an opinion in the face-to-face groups. The differences in the rates of participants abstaining from the question on abortion were not statistically significant.
Using a brief exit survey, we collected women's perceptions of the data collection modality in which they had participated. No significant differences were identified in participant perceptions of rapport, safe space, comfort, or convenience among the IDI sample, while varying levels of statistically significant differences were observed in participant perceptions of the same characteristics of FGs (Table 12). Across nearly all domains, women who participated in the online video FGs reported relatively lower levels of satisfaction.
Participant Perceptions of Data Collection Modality.
A majority (73%) of women who participated in a face-to-face interview felt that rapport during the interview was high, with perceptions of a high level of rapport decreasing across the modalities from online video to online chat and email (25%). This differed from the FG context, where both face-to-face and online discussion board post participants reported feeling high levels of rapport, and most of both online video and online chat participants reported moderate rapport. No participants reported feeling no rapport in any of the IDI modalities, although 3 women felt no rapport during an online video FG. One respondent said, “It was hard to build rapport online for me,” while another, who noted moderate rapport, stated that “there was a good bit of rapport, but I would say technical issues (like audio cutting in and out, video freezing) really disrupted it.”
Nearly all participants in IDIs across modalities agreed or strongly agreed that the interview environment felt like a safe space in which to talk and express their feelings. The exception was one woman in an email-based interview; she disagreed, stating, “I wasn't sure who or where these emails were going; I spoke my mind but was hesitant.” In the FGs, the reported perceptions were similar; nearly all women agreed or strongly agreed that the FG environment provided a safe space. Among the online video FG participants, 2 women disagreed. The reason provided by one pointed to the group composition and topic, rather than the modality, per se: “One respondent was strongly against abortion and made me feel really uncomfortable about discussing my own feelings.”
Nearly all IDI participants felt at least moderately comfortable answering questions across modalities; only 1 woman in an email interview (same as above) felt not at all comfortable, citing uncomfortable questions. FG participants also reported high levels of comfort across modalities, with the exception of the online video FGs, where the majority of respondents reported moderate comfort and 2 felt “only slightly comfortable.” Few comments were provided, but one woman's response suggests the discomfort came from the nature of the questions as much as the modality: “It was still challenging to share opposing views, even knowing I would likely not see these women again, and even though everyone acted respectful [sic] during the video chat.”
Relatedly, women shared their perceptions on how the modality of data collection affected their willingness to share. Women who participated in online text-based IDIs were generally split between finding that the modality made them more willing or had no effect on their willingness to share their experiences. Most women in face-to-face IDIs felt the modality made them more willing to share, while most online video IDI participants felt the medium had no effect on their willingness to share.
Within FGs, most participants in the online chat (84%) and discussion board post (76%) text-based modalities reported that the mode of communication made them more willing to share. Women in face-to-face and online video FGs were more evenly split between reporting more willingness and no effect on sharing. However, 23% of online video FG participants (and 7% of discussion board post participants) also thought the modality made them less willing to share, as summarized by one woman: “I did the online [video] focus group. If it had been just a telephone focus group, I would have been more open to sharing, but seeing the other participants made me more nervous to be open.”
Most women in all modalities reported that IDI participation was moderately or very convenient. One woman in the face-to-face sample and 2 in the online video sample felt the interview was less convenient; the only comment provided stated the participant, as a full-time working mother of a toddler, felt she had no extra time for anything. The online text-based FG participants generally reported those modalities as very or moderately convenient, while a greater proportion of face-to-face and online video FG participants found the modalities slightly to moderately convenient.
We considered many time and cost inputs related to data collection across modalities, as summarized in Table 13. For staff-related costs, we used illustrative hourly rates; for all others, we used actual costs. Regarding the data processing required to prepare data for analysis, the face-to-face and online video modalities required transcription of audio recordings, while the 2 online text-based modalities had live-generated transcripts that required post hoc formatting.
Time and Group-size Inputs for Cost Calculations.
Based on a combination of average and actual times and expenses, we calculated the total cost of data collection for each modality by method (Table 14). Among the IDIs, email-based interviews had the lowest average cost per interview, while the online video IDIs had the highest average cost per interview, with a difference of $182. For the FGs, the online video modality was again most expensive in average cost per data collection event and was $732 more costly per FG than the face-to-face focus groups (even when the number of participants in each modality are standardized).
Data Collection Costs by Modality and Method.
Advances in telecommunications technology and access provide more opportunities to conduct qualitative research. This study addressed questions about how changes in the modality of data collection might affect the data collected, in terms of thematic content and participants' willingness to discuss sensitive/personal experiences or information. We also assessed the comparative cost of different data collection modalities and participants' experience of them.
The small amount of existing literature addressing differences in online vs face-to-face qualitative data collection suggested that online techniques, in general, would generate a smaller volume of information (textual data)19,21 than their face-to-face incarnations. This was confirmed by our data for IDIs if we consider both in-person and online video modalities as “face-to-face,” as in both cases there is an audiovisual connection allowing for nonverbal as well as verbal communication and there is no need to type responses. Together, these modalities accounted for the greatest volume of text and the greatest proportion of participant text. The trends, though less clear, held for the FG sample as well, with significant differences in the total amount and proportion of participant text (after controlling for number of participants per group) between the audiovisual and text-based modalities. In both cases, the larger volume of data produced by the audiovisual modalities likely reflects the ease and speed of speech relative to typing. Consider that the average conversation rate for an English speaker is about 150 words per minute, while average typing speed is about 40 words per minute. At those rates, we might expect audiovisual (or simply audio) modalities to generate more than 3 times the amount of text as typing-based modalities. This was also why we extended the time allowed for online chat interviews—it simply took longer to get through the same questions, not because there was more discussion, but because the act of thinking and typing took longer than speaking. However, the significant difference within the FG sample between the synchronous and asynchronous text-based modalities suggests that not only typing ability but also real-time engagement with the moderator and other participants might have affected the volume of participant responses.
While our findings related to the volume of participant text are informative, the more salient measure of data “richness” relates to the content of the textual data generated across modalities, particularly since qualitative research aims to discover the meanings in participant responses rather than simply the number of words used to convey those meanings. According to our thematic code operationalization of richness, we found no significant differences in thematic content across modalities for either method (IDI or FG). Had we conducted only face-to-face or only online chat data collection, for example, the resultant thematic reports would have been nearly identical, based on the shared thematic content generated. It may be that the necessity of composing a written/typed response in online text-based modalities forces participants to condense and organize their thoughts before responding, or to be less “effusive,” resulting in reduced data volume but similar data content. Our operationalization of richness, however, does not consider the depth of discussion of particular themes. One might subjectively assess a narrative with fewer themes but greater depth as “richer” than one that superficially covers a larger number of themes. Although we did not perform subjective assessments of richness, earlier studies that have more qualitative assessed richness10,15,16,20,34,35 indicate that in-person (or online video) FGs are “richer” than text-based ones, because of the perception of more context/illustration. However, nearly all those studies also find that in-person FGs include more off-topic commentary.10,15,16,20
We also found mixed evidence of an online disinhibition effect regarding participants' sharing of sensitive/personal information across modalities. In terms of unsolicited sensitive/personal disclosures, we found no statistically significant differences across modalities for either method. Yet when presented with a sensitive topic (abortion), online text-based FG participants—those who could not see each other—offered more dissenting opinions than did participants in the visual modalities (face-to-face and online video). This aligns with data on participant perceptions of the data collection modalities, which showed that the online video FGs in particular were viewed as less comfortable and safe than the other methods. Greater proportions of women in the online text-based FG modalities, particularly the message board option, also reported that the mode of data collection made them more willing to share.
These findings may reflect general use and familiarity with some modes of communication over others. The face-to-face IDIs and FGs both scored consistently high, with participants remarking on the conversational tone and ease of interpersonal exchange, even when discussing a sensitive topic. Given the similarity between in-person face-to-face data collection and online video-based forms, in terms of the spoken conversation, visual connection, and ability to read nonverbal cues, it is perhaps surprising that online videos consistently received the lowest proportion of women reporting strong feelings of rapport, comfort, and safety. As participant feedback indicated, some of this relative discomfort was associated with the topics of discussion, but the technology (freezing video, connection challenges) also interrupted the flow of conversation, and women seemed to “warm up” to each other less when connected via video vs in person. The pre–focus group small talk that happens in a conference room did not typically happen in online video contexts; such small talk may help participants “read” one another, identify similarities and connections, and set a more relaxed tone. Online video FGs offered a chance to virtually meet the other participants (and often to see into their homes), but the technical troubleshooting and connection at the start of each FG meant that the audio wasn't joined until the group was convened and the moderator ready to start. Seeing oneself live on a computer screen among 5 to 6 others was also likely uncomfortable or distracting for some women. Conversely, the relatively anonymous online discussion post modality scored high for the FGs in terms of rapport and comfort, perhaps reflecting women's use of and familiarity with social media and posting platforms in other areas of their lives. We did not consider telephone (audio-only) data collection as part of this study because telephone-based focus groups are uncommon, but the literature suggests that the benefits of geographic reach and privacy, paired with speech, may make them a suitable option in some cases.36,37
In terms of cost to conduct, our findings generally mirror those of Rupert and colleagues,33 who found that virtual FGs do not appear to cost less or recruit participants faster than in-person groups. The highest average cost per data collection event, without travel, for both IDIs and FGs was the online video modality. For IDIs, the online asynchronous (email) modality cost least on average, while for FGs, the least costly average data collection event was the in-person modality, despite the extra cost categories of refreshments and assistant time. Online methods are often touted as less expensive because they save on the cost of researcher travel to reach a geographically distributed sample,11,12 and our data are limited in that we did not have actual travel expenses. Rather, we used estimated travel expenses to provide illustrative cost implications for data collection. Including travel raised the average per-event cost of in-person data collection considerably, making it the most expensive for both IDIs and FGs. As inputs for each study will differ, we suggest reference to Tables 5 and 14 for cost comparison calculation considerations.
As with any research, our findings come with qualifications. First, our sample was relatively homogeneous, limiting the generalizability of our findings. Given our methodological objectives (objectives 1 and 2 in particular) and stratified participant assignment to data collection modality, we viewed sample homogeneity as a relative strength, in that it was 1 less potential confounder of our comparisons between modalities. Also, the homogeneity of the sample population should be positively associated with the thematic saturation rate; groups that are alike on multiple dimensions are more likely to think in similar ways and have similar experiences, allowing for earlier thematic saturation.38 The more homogeneous sample therefore allows us to be more confident that our sample size allowed us to reach thematic saturation for our comparison of thematic content. Nonetheless, one of the potential benefits of online research is the ability to include geographically scattered populations, and our sample was geographically circumscribed. Future research with more diverse populations—in terms of socioeconomic status, race, computer/literacy, and mobility—would help to broaden the findings.
Relatedly, all study participants were required to have internet access/computer in their homes and basic typing skills, because of the systematic assignment into face-to-face or online modalities. We recognize that these eligibility criteria may also introduce bias into the study vis-à-vis generalizability—that women who have home computers and can type are not representative of the general population of women who could benefit from remote data collection. However, given the nature of the study objectives and experimental design, this limitation was unavoidable. Interpretations should be made with awareness of this limitation. Regarding the relative costs of data collection, it should be noted that we used estimated travel costs. Additionally, some researchers may rely on free or open platforms for online video- or text-based data collection to defray those costs, but such platforms potentially raise data privacy and confidentiality concerns. As the inputs for each study will differ, we suggest reference to Tables 5, 13, and 14 for cost comparison calculation considerations.
Selection of a modality for conduct of qualitative research will continue to hinge on several factors, including the type and location of the participant population, the research topic, and the project budget. A major caveat related to research population is that technology-based online approaches usually require computer and internet access, which may be barriers for populations already experiencing other access or inequity issues. Our findings on the elicitation of sensitive disclosures and expressions of dissenting opinions suggest that there may be an online disinhibition effect for nonvisual online data collection modalities, but that for some topics, the social atmosphere created by an in-person group of similar participants could work equally well. Regarding cost, online and face-to-face modalities incur different types of expenses, and priorities within the time/cost/quality resource allocation triangle will dictate which costs are the most efficient use of resources. However, most importantly, our methodological findings based on a rigorous comparative research design confirm some earlier findings16,22—that conducting qualitative research via online modalities does not result in substantial or significantly different thematic findings than conducting IDIs and FGs in person. Despite differences in interpersonal dynamics between individual interviews and focus groups, our data suggest that the effect of modality on data generated is similar across both methods. This opens opportunities for broadening the reach and inclusion of sampling to a wider geographic scope, as researcher and participant(s) need not be in the same place; it also provides opportunities to include populations with mobility or transport issues, who may not be able to travel to a study location. For researchers examining the relative strengths and trade-offs of traditional vs online qualitative data collection, we have contributed many empirical data points for consideration in research design and decision-making.
Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#ME-1403-11706) Further information available at: https://www.pcori.org/research-results/2014/comparing-interview-and-focus-group-data-collected-person-and-online
Summary of research objectives/questions and analysis methods (PDF, 539K)
Data collection guide for the RAMP-UP mixed-modality qualitative study (PDF, 341K)
Post-data collection event participant experience survey (PDF, 668K)
Abbreviated codebook for the RAMP-UP study (with number of uses per code) (PDF, 689K)
Sensitive codes used in the RAMP-UP study (PDF, 651K)
Guest G, Namey E, O'Regan A, Godwin C, Taylor J. (2020). Comparing Interview and Focus Group Data Collected in Person and Online. Patient-Centered Outcomes Research Institute. (PCORI). https://doi.org/10.25302/05.2020.ME.1403117064
The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.
This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/
Your browsing activity is empty.
Activity recording is turned off.
See more...
