Focus Groups according to Tremblay et al

Process description

The following activities describe the use of focus group methods to evaluate and refine design artifacts in the IS field. This method is adapted from traditional focus group techniques for use in design research projects. This method describes two types of focus groups: exploratory focus groups (EFG), which are used for the design and refinement of an artifact; and confirmatory focus groups (CFG), which are used for the confirmatory proof of an artifact’s utility in the field. The primary challenge is the structuring of focus groups so participants can collectively use an information systems artifact in order to provide feedback. This method describes one potential approach, in which participants collectively decide on an outcome both with and without the artifact, in order to compare decision-making strategies.

Formulate Research Problem

Description

In order to effectively define the content and focus groups, the research goals must be clearly identified. Design researchers seek to design an artifact, incrementally improve the design, and evaluate its utility and efficacy. These are two complementary, yet different, research goals. The figure illustrates the positioning of the two types of focus groups—exploratory and confirmatory—in the design research process. As discussed more fully in Hevner [2007], two forms of artifact evaluation are performed in a design research project—the evaluation of the artifact to refine its design in the design science build/evaluate cycle and the field testing of the released artifact in the application environment. We discuss the similarities and differences between exploratory focus groups and confirmatory focus groups as follows.

Further Readings

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Identify Sample Frame

Description

Three decisions are made in this step:

1. Number of each type of focus group to run
2. The desired number of participants in each group
3. What type of participant to recruit

Number of focus groups

Deciding how many focus groups to run can prove to be quite challenging. The literature states that focus groups should continue until nothing new is learned , yet deciding ―nothing new is being learned is a difficult and somewhat arbitrary task. This is especially challenging in design research. When conducting an EFG, the designers will find that there is always room for improvement of an artifact and certainly a fair amount of subjectivity in interpreting when the design of an artifact is indeed complete. There is a point where we choose to satisfy in order to move forward. For CFG, the decision that enough evidence of utility has been collected is somewhat subjective. Additionally, there is a need to balance available people and resources since focus groups can be expensive both in terms of time and money (most participants receive some sort of compensation), and expert participants may be difficult to find. In our experience, at the minimum, one pilot focus group, two EFGs, and at least two CFGs should be run. The pilot is informal (one could use students) and is used to understand timing issues and any kinks in the questioning route. A design researcher should allow for at least two design cycles and enough contrast for field test analysis. Since the unit of analysis is the focus group, it would be difficult to make a compelling argument for the utility of the designed artifact with just one CFG. In the example we outline in the later part of this manuscript, we used two CFGs .

Number of participants

Selecting group size has several considerations. It may seem simpler (and less expensive) to run fewer, larger focus groups since it takes fewer focus groups to hear from the same number of participants. Yet this could lower "sample size" since there are fewer groups to compare. Additionally, the dynamics of smaller versus larger groups are different; smaller groups require greater participation from each member, larger groups can lead to "social loafing". suggests a lower boundary of four participants and an upper boundary of twelve participants. Depending on the approach taken to demonstrate the artifact to the group (for example, whether each individual uses the artifact, versus if a moderator demonstrates it), large focus groups (more than six) could be tricky in design research since the subject matter is more complex than traditional focus group topics.

Further Readings

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Recruit Participants

Description

The identification of focus group participants is not a random selection, but rather is based on characteristics of the participants in relation to the artifact that is being discussed. A diversity of participants will potentially produce more creative ideas (and perhaps more conflict depending on topic), but segregation of participants based on skills and knowledge may provide more in-depth tradeoffs in values and success measures. In fact, research shows that bringing together groups which are too diverse in relationship to the topic of interests could result in data of insufficient depth.

For design research, the participants should be from a population familiar with the application environment for which the artifact is designed so they can adequately inform the refinement and evaluation of the artifact. Care should be taken that the participant groups are from a similar pool for both EFGs and CFGs, so that CFGs are in fact confirming a final design. Though the authors have never attempted this, an interesting approach is to use the same groups of participants twice, once to evaluate and once to confirm.

Research is mixed on whether to use pre-existing groups, though for design topics this may be advantageous since the participants have problem solved together and the focus group may approximate a realistic environment. Interaction among participants is one of the most important aspects of focus groups. For example, a group consisting of all technical experts may be very different than an expert/non-expert group. A design researcher must consider membership of the focus groups and how it aligns with the research objective early in the participant selection process. For example, if the artifact is a software requirement methodology, the group membership may consist predominately of requirements analysis experts. If the artifact is a decision aid tool, a design researcher may purposely mix different skill sets: such as systems analysis, business analysis, and context experts in order to include different aspects of the aid in the conversation.

Design researchers should strive to recruit participants that are familiar with the application environment and would be potential users of the proposed artifact. Unfortunately, in many cases such individuals are not easy to find, so plenty of time and effort should be allotted for this task. For instance, it might be possible to conduct the focus group in the evening (most participants will likely work) and offer dinner. Another good approach is to conduct the focus group at a place where the potential participants work, again enticing them with lunch or breakfast. Phone calls and e-mails should be placed at least a month before the focus groups are planned. A few days before the focus groups the participants should be reminded. Researchers should plan for a few participants to not show up, so if the goal is six people, invite eight. However, care should be taken if the no-shows upset the diversity of the focus group. For example, if both medical doctors invited are no-shows, then the group may be left with no doctors.

Further Readings

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Identify Moderator

Description

Due to the open-ended nature of focus groups, moderation can be complex, especially in social research. Several skills are important when moderating a focus group. Krueger et al. [2000] find the following skills to be highly important:

• Respect for participants, allowing all participants the opportunity to express their views
• The ability to communicate clearly, both orally and in writing
• The ability to listen and the self-discipline to control personal views
• A friendly manner and a sense of humor, and (5) the ability to involve all participants in the conversation

For design research, the moderator not only needs to have these skills, but also a clear understanding of various aspects of the designed artifact. The moderator should be familiar with the artifact and be comfortable presenting it to focus group participants.

In some cases, the moderator may be one of the artifact designers. In this case, the moderator has to be very careful not to introduce any personal bias in the presentation of the artifact (we tend to be proud of our work), particularly when conducting an EFG. It may be possible to enlist a second observer to guard against the encroachment of personal views (at least during the initial groups). This is an excellent time to receive good suggestions for improvement of the design and the designer has to be receptive to criticism and suggestions given by the participants; being careful to justify or defend the design work only in appropriate ways that do not discourage discussion and feedback.

Further Readings

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Develop and Pre-Test a Questioning Route

Description

The questioning route is the agenda for the focus group. In the questioning route you are setting the direction for a group discussion and it should closely align with your research objectives. There should be no more than twelve questions for a two-hour session. Two general principals outlined by Stewart et al. [2007, p. 61] are to order the questions from the most general to the more specific and to order the topics by the relative importance to the research agenda. Thus, the topics to be discussed are ordered by importance, and within those topics, the questions are ordered from general to specific.

For a designed artifact, this means beginning with an explanation of the motivation behind the design of this artifact, followed by a broad explanation of different scenarios on where and how the artifact could be utilized, a description of the details of the design of the artifact, training on its use, and finishing with a task where focus group participants are asked to utilize and evaluate the artifact.

For an EFG, the "rolling interview guide" is an excellent approach. With a rolling interview guide, a script is created for the first EFG but is modified for the next EFG, based on the outcome of the previous EFG. One of the advantages of this approach is that it allows for information to unfold over time as you discover more about how people would understand and use the artifact. However, it is imperative that no revisions are made to the interview guide in the CFGs, since continuous change would make comparisons across the focus groups difficult, compromising rigorous interpretation of the results.

A promising evaluation approach in design research focus groups (both EFGs and CFGs) is to create a manipulation within the focus group. Participants can be asked to collectively complete a task without the artifact and then again with the artifact. The ensuing discussion should revolve around how the artifact was used and how the completion of the task was altered by its use.

Further Readings

Stewart, D.W., P.N. Shamdasani, and D.W. Rook (2007) Focus Groups: Theory and Practice, 2nd edition, vol. 20, Newbury Park, CA: Sage Publications.

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Conduct the Focus Group

Description

Focus group sessions should be fun and stimulating for the participants and moderator. The moderator usually greets the participants as they enter and may ask them to fill out demographic information and informed consent forms (e.g., IRB forms). The participants are generally seated in a U-shape arrangement to encourage collaboration and allow space for the moderator to demonstrate the artifact. Seating arrangements are also very important. A good approach is to get to know the participants before the questioning route begins. Greeting them when they arrive is a good first step. The most assertive and expert participant should be seated next to the moderator and the least talkative directly across from the moderator. One potential risk is that an assertive participant could manipulate the conversation and the moderator will have to discreetly refocus the group.

Depending on your research protocols, focus groups may be video and/or audio taped. Generally, the participants are told they are being recorded and most institutional review boards require written consent. It is also a good idea to have an observer. The observer will not participate in the focus group, rather will take careful notes, noting in particular any strong reactions, the participants’ facial expression and general tone of any exchange between participants or between the participant and the moderator. Time management is also important when conducting a focus group. A moderator should be able to recognize when all possible issues for a topic have been covered and move on to the next topic. Pilot focus groups can help anticipate and manage the timing of focus groups.

Further Readings

Krueger, R.A. and M.A. Casey (2000) Focus Groups: A Practical Guide for Applied Research, 3rd edition, Thousand Oaks, CA: Sage Publications.

Stewart, D.W., P.N. Shamdasani, and D.W. Rook (2007) Focus Groups: Theory and Practice, 2nd edition, vol. 20, Newbury Park, CA: Sage Publications.

Bloor, M., et al. (2001) Focus Groups in Social Research, London.

Morgan, D.L. (1988) Focus Groups as Qualitative Research. Newbury Park, CA: Sage Publications.

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Analyze and Interpret Data

Description

The two design research goals for using focus groups are the incremental improvement of the design of the artifact and the demonstration of the utility of the design. For this reason, we have suggested the different focus group types of EFG and CFG. While the objectives of the two group types are very different, the methods of analyzing the focus group data from both EFG and CFG can be similar. The interpretation of the focus group discussions has many of the same challenges in demonstrating rigor that all qualitative research encounters share. Several techniques that are used for qualitative data analysis can be considered, carefully selecting those techniques that emphasize the reliability and replicability of the observations and results.

One possible approach is template analysis. Template analysis normally starts with at least a few predefined codes which help guide analysis. The first step in template analysis is to create an initial template by exploring the focus group transcripts, academic literature, the researchers’ own experiences, anecdotal and informal evidence, and other exploratory research. The contents of the discussions are also examined for the meanings and implications for the research questions. Analysts will look for common themes and variations within the transcripts that would provide rich descriptions of the participants’ reactions to design features. In template analysis, the initial template is applied in order to analyze the text, but is revised between each EFG session. Once the final template is created after the final EFG, it is used to code the CFG sessions.

Further Readings

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.

Report Results

Description

King suggests that qualitative results can be reported by creating an account structured around the main themes identified; drawing illustrative examples from each transcript as required. A similar approach can be taken when reporting focus group results. Short quotes are used to aid in the specific points of interpretation and longer passages of quotation are used to give a flavor of the original discussions. Summary tables can be very helpful, displaying both evidence and counter-evidence of the utility of the artifact by focus group. Rich descriptions can further corroborate results by using quotes from the focus group participants.

Further Readings

Miles, M.B. and A M. Huberman (1994) Qualitative Data Analysis: An Expanded Sourcebook, 2nd edition, Thousand Oaks: Sage Publications

King, N. (1998) "Template Analysis", in Symon, G. and C. Cassell (eds.) Qualitative Methods and Analysis in Organizational Research, London: Sage Publications.

Tremblay, Monica & Hevner, Alan & Berndt, Donald & Chatterjee, Samir. (2010). The Use of Focus Groups in Design Science Research. 10.1007/978-1-4419-5653-8_10.