STEP 2.a Deciding Which Data Collection Approach Is Best

Introduction

Once you have determined the scope and targets of your outcome study you are well prepared for the next stage of the process: determining data collection methods and identifying the sensitivities and special needs of particular user groups. Without question, your data collection methods will depend on the context, the goals of your study, and the resources that are available; bear in mind that different data collection methods can suit particular purposes and we strongly recommend using them in combination with one another (i.e., also known as triangulation). Below we offer detailed summaries for the primary data collection methods that we recommend for initial studies: focus groups and interviews. These summaries include definitions, how to's, and comparisons of strengths and weaknesses to help guide your choice of approaches and combinations.

We encourage librarians involved in an initial study to use primary data collection methods such as focus groups and interviews to collect as much detailed data from staff and users as possible. In addition, you may choose to use supplemental approaches, such as observation, follow-up interviews, and surveys (link to surveys file), to verify information and to discover richer outcomes as well as to collect data on professional contributions.

The at-a-glance table below provides a quick summary of each of the data collection methods to help you decide which approach might be best for you. Keep in mind that you might use a combination of approaches. You will notice that for most of our evaluation projects we used a combination of approaches to verify information and to discover richer outcomes. If you have the time and resources you will want to consider using more than one data collection method for your evaluation, as well.

For evaluating community and library services

Primary Data Collection Methods

For librarians new to outcome-based evaluation, we highly recommend using focus groups and interviews to gather detailed information from users about their interaction(s) with the service.

Focus Groups

Focus Groups Defined:

A focus group is in essence a group interview with 6-10 people of similar background. Gathering people together to share thoughts regarding questions/issues about library programs can be an efficient way of collecting substantial data in a single setting.

Focus Group How To's:

To gather data in a library setting using focus groups, consider the following:

• When conducting a focus group, which typically last 1-2 hours, we recommend a minimum of two persons serving as facilitators: one person to lead the discussion, and one person to take notes on both the content of the conversations, as well as the non-verbal communication that devices such as a tape recorder cannot capture. Noting non-verbal behavior such as nods of approval (and/or disapproval), facial and body expressions of interest (or lack thereof) can provide additional meaning to the flow of conversation during the focus group.
• Our research team frequently used both focus groups and interviews when evaluating community and library programs. In a focus group setting, for example, service providers can absorb what similar professionals are experiencing. Such opportunities often nurture thoughts that may not have occurred to them had they participated in a one-on-one interview. During a personal interview, the experiences that participants share are typically on a more personal level, and therefore the dynamics of a one-on-one interview often provide for richer data collection. See the "Interviews" (2a.1b) section for more details on using this data collection method.
• Ensure that your focus group occurs at times and in a venue that is convenient for participants, but outside the evaluated service setting, i.e., the library, to minimize biased responses known as the Hawthorne Effect (when participants give responses that they believe the investigators desire). Also, be sure to provide beverages and snacks-never underestimate the importance of food for stimulating participation and social interaction!

Focus Group Strengths:

• Excellent approach to gather in-depth attitudes and beliefs from several stakeholders at a time.
• Group dynamics might generate more ideas than individual interviews.
• Can be effectively used to focus on details regarding issues found through surveys or other data collection methods.

Focus Group Weaknesses:

• Requires staff time to set up and facilitate focus group.
• Requires staff time to identify and schedule participants for focus group.
• Requires strong facilitator to guide discussion and ensure participation by all members.
• Usually requires special equipment to record and transcribe focus group discussion.

Focus Group Examples:

Queens Borough Public Library - To gather information on how to best serve immigrant populations, we conducted focus groups with library staff, and these staff in turn administered an oral survey to library users, often in the customer's native language.
Queens focus group guide

Austin and Flint Public Libraries -To discover how after-school community technology programs impact youth, we conducted focus groups with librarians and youth participants at two public libraries. The focus group data was augmented by interviews with library and project administrators, as well as, other service providers in the area.
Austin and Flint focus group guides

Help Seeking in an Electronic World - To determine ways that public libraries utilize the Internet to provide digitized community information (CI) to their communities, we conducted focus groups with service providers, librarians, and library administrators.
Help Seeking in an Electronic World focus group guide

Peninsula Library System - To determine the impact of a community information database, we first conducted focus groups of nonprofit agency administrators who used the database.
Peninsula focus group guide

Interviews

Interviews Defined:

Interviews are question and answer sessions with an individual participant, often conducted by telephone, through face-to-face interactions or online. Generally, interviews provide a valuable opportunity to probe and explore responses, resulting in the collection of data regarding library services that is rich in content and clear in meaning. Also, in the right circumstances (where trust and anonymity are established) interviews are useful to collect more sensitive information that respondents may not wish to write down.

Interview How To's:

To gather data most effectively in a library setting using interviews, consider the following:

• Prepare an interview guide. An interview guide is an outline of the questions to ask, along with any probing questions (i.e., rephrased, reminder questions) that might help to elicit responses. The guide will help provide consistency in the administration of the interview. While it's natural to expand or revise your questions as you proceed with subsequent interviews, we suggest that you pretest your guide with a few users to "nail down" or confirm your questions so you do not have to vary them much in future (this will make data analysis MUCH easier, and is a step we 100% recommend with every approach-pretest your instruments).
• Remember to ask permission from the interviewee, as well as further consent to audio-record the interview. Inform the participant that he/she is being audio-recorded and proceed with the interview once they have granted permission. Note that different organizations might also require specific procedures for seeking informed consent.
• Emphasize the confidentiality of the interview. Explain the purpose of the interview, how participants were selected, how responses will be used, and whether they will have the opportunity to see the results of the study. Allow interviewees to ask questions about the interview.
• Use active listening skills and maintain eye contact with the participant as much as possible. If not audio-recording, then write down responses inconspicuously and try to use the participant's exact words. Ask the participant to clarify vague or ambiguous answers; probe for specifics when necessary.
• Do not be afraid of "white space." Participants need to time to digest your funny questions and form a response. Plus, some people naturally speak more slowly than others. Resist the urge to rush in and fill empty space, and do not interrupt your participants when speaking or finish their answers.
• Keep your interview guide to one page so you do not distract your participant by rustling pages.

Interview Strengths:

• Good approach to gather in-depth attitudes and beliefs from individuals.
• Personal contact with participants might elicit richer and more detailed responses.
• Provides an excellent opportunity to probe and explore questions.
• Participants do not need to be able to read and write to respond.

Interview Weaknesses:

• Requires staff time and quiet area to conduct interviews.
• Usually requires special equipment to record and transcribe interviews.

Interview Examples:

Peninsula Library System - To determine the impact of a community information database, we first conducted focus groups of nonprofit agency administrators who used the database, then we conducted followup telephone interviews of direct-service providers of these same agencies.
Peninsula interview guide

Help Seeking in an Electronic World - To determine ways that public libraries utilize the Internet to provide digitized community information (CI) to their communities, we conducted interviews with service providers, librarians, and library administrators.
Help Seeking in an Electronic World interview guide

Queens Borough Public Library - To gather information on how to best serve immigrant populations, we conducted interviews and focus groups with library staff, and these staff in turn administered an oral survey to library users, often in the customer's native language.
Queens interview guide

Austin and Flint Public Libraries - To discover how after-school community technology programs impact youth, we conducted interviews with librarians and youth participants at two public libraries. In addition, we interviewed project administrators and other service providers in the area.
Austin and Flint interview guides

Supplemental Data Collection Methods

As mentioned above, librarians can use supplemental data collection methods, such as observation and follow-up interviews, to verify and to enrich the outcomes they discover in focus groups and interviews. These complementary methods serve as valuable tools in initial as well as on-going outcome evaluation for more on Long-Term Outcome Evaluation, particularly the use of surveys, see out summary on Using Surveys (Link to IBEC URL re: Survey summary and sample instrument).

Observation

Observation Defined:

Observation entails observing and recording behavior of participants in a given situation. Observations can be a great method for obtaining data regarding participant behavior in a particular library setting. In addition, observation affords an opportunity to explore professional contributions, such as language facility or openness, in action. Observation can be considered along two dimensions: (1) degree of obtrusiveness (i.e., from totally unobtrusive to complete, active participation), and (2) whether overt (participants know they are being studied) or covert (you, the investigator observes in secret or undercover). Rule of thumb is that you do not have to ask people's permission to observe them in public places, i.e., libraries, therefore it is fine for you to conduct covert observation. Indeed, if you did ask permission or post signs, then people might behave in unnatural ways because they know they are being studied! It gets down to one's expectation of privacy and, again, different institutions and agencies will have different rules to protect their clientele.

Observation How To's:

To gather data using observation, consider the following:

• If conducting research in the library or in an environment with which they are very familiar, librarians may find that observation might add little to their understanding of what goes on in this environment. However, if, for example, they are asked to build a webpage catering to the needs of the community's immigrant populations, they might decide to ask the ESL volunteer coordinator if he/she wouldn't mind having the librarian sit in on a class or two as a means of beginning research to determine recommendations for the look, feel, and content of this new webpage.
• We recommend that you prepare an observation guide with which to record observations. Decide what behavior or types of data to focus on. For example: what types of questions are typically asked during the program? What resources are they using in class? How is the camaraderie among students? An observation guide will help to consistently record observations and to capture the data that has the most value.
• Observing an environment can be especially beneficial if librarians have the opportunity to conduct follow-up interviews and/or focus groups with some of the people that they have observed. See the "Interviews" (2a.1b) and "Focus Groups" (2a.1a) sections for more information on these data collection methods.
• It is important to keep in mind that practically all conditions under which people observe other people will be altered to some degree simply by having the evaluator in the environment as an observer. Keep this in mind when analyzing data. Be sure to note any possible influences of the observer on participants' behavior in the findings report.

Observation Strengths:

• Excellent approach to discover behaviors during library programs.
• Provides indicators of the impact of programs that might be more reliable than data gained by asking people.
• Good technique when there are observable products and outcomes.

Observation Weaknesses:

• Requires staff time to observe and record observations.
• Cannot ask questions of participants during observation.
• Follow-up interviews to verify observations may be needed.

Follow-Up Interviews

Follow-up Interviews Defined:

Follow-up interviews are valuable tools when studying the impact of community and library programming over a period of time. Generally, follow-up interviews allow evaluators the opportunity to probe and explore and/or clarify responses from initial focus groups, interviews, surveys and/or observations. Follow-up interviews can provide verification and detail for data gathered initially via some other data collection approach, as well as probe participants' behavior since last meeting (e.g., whether they used a service, subsequent needs and actions, etc). For example, we conducted follow-up interviews with online community network users approximately 10-14 days after they completed online interviews so we could probe about their use of the information that they obtained online. This was in addition to the more supplemental purpose of clarifying/expanding online survey responses.

Follow-up Interview How To's:

To gather data most effectively in a library setting using follow-up interviews, consider the following:

• Participant consent to be contacted at a specific later date is necessary. We recommend that you ask for a participant's name, contact information, and permission to be contacted during the initial focus group, interview or survey.
• The initial data collected needs to be managed effectively so that the interviewer, when conducting follow-up interviews, can provide reference to the participant's comments and responses made during the initial study.
• If the initial data collection was face-to-face, follow-up interviews do not necessarily need to also be face-to-face. Assuming the person who conducted the initial data collection will also conduct the follow-up interview, a relationship has been created with the participant and communicating via the phone may be just as effective as face-to-face in this situation.
• Interviewing is particularly beneficial in confirming and complementing any data collected through observation. While valuable data can be collected through observation (see above section), the observer often generates questions that cannot be asked during the non-intrusive observation period. Participants in an observation study typically are more at ease during an interview than in other circumstances, as they will be participating in the interview with the context already familiar to them; they were the ones being observed, and they logically assume that the interview will focus on what occurred during the observation.
• If interested in post-observation or post-interview behavior, ensure you allow adequate time for participants to carry-out any ensuing activities, but don't wait so long that participants forget what you're interested in.

Follow-up Interview Strengths:

• Excellent approach for obtaining time-sequenced data about subsequent actions, needs and thoughts.
• Good method to follow-up on users' feelings and experiences that were shared earlier using another approach.
• Provides an excellent opportunity to probe and explore questions.
• Participants do not need to be able to read and write to respond.
• Can be face-to-face or via phone or online

Follow-up Interview Weaknesses:

• Staff time to administer follow-up interviews is required.
• Contact information must be gathered in the initial data collection process.
• Special equipment to record and transcribe interviews is usually required.

Follow-up Interview Examples:

King County Library System - To determine the impact of an Asian cultural program on patrons of the King County Library System, we conducted follow-up interviews to confirm data initially gathered through a survey. On the initial survey, we asked for contact information for the follow-up phone interviews.
KCLS Follow-Up Survey

Help Seeking in an Electronic World - To determine ways that public libraries utilize the Internet to provide digitized community information (CI) to their communities, we conducted follow-up interviews with participants who had completed an online survey through one of three public library online community networks.
Help Seeking in an Electronic World Follow-Up Survey

Peninsula Library System - To determine the impact of a community information database, we first conducted interviews of nonprofit agency administrators who used the database, then we conducted followup telephone interviews of direct-service providers of these same agencies.
Peninsula Follow-Up Interview Guides