STEP 2.b Creating Open-Ended Questions Aimed at Determining Outcomes

Introduction

Throughout this Toolkit, we emphasize that library users remain the primary articulators of library outcomes. The interviews, focus groups, and observational methods discussed here enable librarians to capture detailed feedback from users themselves. Of singular importance, however, across all methods is the way in which questions are asked. In contrast to closed questions, which typically solicit a "yes" or "no" reply, open-ended questions require respondents to provide significantly more detail on their impressions of the impacts of services. Open-ended questions thus well serve librarians undertaking initial studies of services whose outcomes have yet to be identified.

Question Examples

The examples below include balanced sets of open-ended and closed questions used in our study. Designed to explore essential ingredients of outcomes, including the user's context, use of the library services, and outcomes, we offer the following sample questions as a resource to librarians embarking on a study designed to determine outcomes. For more guidance and examples of open-ended questions, see the research instruments included in the case study reports.

1. Questions that find out more about the user's context.

Public library programs and services serve an enormous variety of clienteles, from young people and families to service providers and immigrant populations. Once librarians have determined which users to target in Exercise 1, librarians, being mindful of user sensitivities identified in Exercise 2, can begin to draft a suite of questions aimed at exploring the user's context.

Example A. Teens
Teens who participate in an after-school community technology program. Find out more about their situation. Without specifically asking the why question (often considered rude or too probing), find out what motivates the teen to come to the program.

Context Questions:
Did you know how to use computers before you came?
How much had you worked with computers/Internet before you came?

What did you use them for/to do?
Do you have a computer at home that you can use?
How do you use computers now?
What do you expect to get out of <name of program>?

Example B. Agencies
Agencies that use the library's information and referral services or community network. Find out more about the clientele of each agency and how the agency serves the needs of their clientele.

Context Questions:
Please describe your audience or clientele.
How do you reach your clientele?
What makes it hard for clients to access your services or get information?

Example C. Immigrants
Remain sensitive to the collection of demographic and contact information about immigrants. Librarians can find out what motivates immigrants to come to the library by asking short, direct questions, in the immigrant's preferred language, that do not reveal personal information. Contact information should not be collected; demographic information may be collected, with caution.

Context Questions:
Why are you here today? [Interviewer Probe: What are you doing here today?]
How did you find out about the library?
Was there anything in particular (about the flyer, person, etc.) that influenced your decision to come to the library? Demographics: (Ask only if answers aren't obvious; DO NOT push customer if he/she appears uncomfortable.)
How old are you? Note gender of interviewee.
How do you describe your ethnicity?
How long have you been coming to the library?
Who in your household uses the library?

2. Questions that focus on the use of the library services.

Services are used differently by different users. People may also use the service for more than one purpose. In collecting data about use, librarians can craft a set of questions to not only identify the range of users involved with the service but also the spectrum of reasons they use the system, as seen in the questions below.

Example A. Teen use of community technology program.
Find out how the teen uses the service or some aspect of the service.

Sample questions on use:
What have you been doing today at (name of service)?
How do you describe (name of program) to other kids and your family?
If you were finished with the program, what would you remember most about it?

Example B. Agency use of community information (CI) service.
Find out how agencies use the various service components. Focus specifically on their use of the service and its component activities.

Sample questions on use:
How do you use (name of service)?
We at (name of service} call the agencies we work with "content providers." Do you consider yourself a content provider?
What does that term mean to you?
How does (name of service) work with you?

Example of use of a specialized service. The questions below are taken from a telephone interview of agency staff that requested GIS developed custom maps. The interviewee was identified by an I&R staff member.

Sample questions on use:
We understand that you requested a map of [describe the map].
Can you tell us about the situation when you asked <name of service> for this map? What were the circumstances? Can you walk us through this situation?
What kind of information did you need? What kind of assistance did you get in developing the map?
How have you used the map?
Have you had other occasions to ask <name of service> to <type of activity such as customized map>? REPEAT as necessary for additional examples.

Example C. Immigrant use of immigrant-specific services.
Immigrants use the library for a variety of purposes, including educational, social, cultural, and employment assistance. Find out how immigrants use or will use a library program, service, or resource.

Sample questions on use:
How did you use the library today?
Have you ever gone to a <class or workshop> at the library? What was it about?
What else have you done at the library?
Have you had any challenges in using the library? Can you talk about one of them?

3. Questions that focus on the outcomes of the program on participants and others.

Remember that the goal of the study is to identify and to document the consequences, both positive and negative, of the user's interaction with the service. Questions here should target the participant's response to one or more aspects of the program.

Example A. Teens.
Outcomes of the teens' projects

Sample questions that focus on outcomes:
Can you think of ways that <name of program> has helped you?
What have you accomplished?
What do you know that you didn't know before you came to the program?
Have you done anything differently because of your participation in <name of program>?

Example B. Agency interaction with CI services.
Outcomes of community information service

Sample questions that focus on outcomes:
What are the most valuable aspects about <name of library program> for [name of organization]?
How does <name of library program> help your clientele?
Can you please provide a specific example such as a success story? Other examples?
Can you provide an example of how the <name of library program> has had an impact on the work that you do?
How does the <name of service> influence your work?
Regarding the use of a specific service such as a custom map:
How did it help? How did it not help?
What were the outcomes of this help (positive and negative)?
Do you feel that <name of service> has had any impact on the relationships your organizations has with other agencies?
Are there things that you would like <name of organization> to do differently? If so, what?

Example C. Immigrants:
Outcomes of immigrant programs and services

Sample questions that focus on outcomes:
How were you helped with <X>? [Interviewer Probe: Did you learn something helpful from coming to the program or coming to the library?]
Have you learned anything important to you at the library?
Have you been able to read something that was valuable to you because you came to the library? Can you tell me more?
What other ways does the library help you?
What does the library mean to you?
Have you ever told anyone about the library? What did you tell them?

At this point in the design process, you have considered contextual factors relevant to the study, resources required for the study, and data collection methods and questions appropriate to the user group of the service.

To help you integrate these elements into a cohesive data collection plan, we offer Exercise 2: Strategizing data collection (anchor link to Exercise 2), which will provide a global view of the initial outcome study that you are about to undertake. At this point in the design process, librarians have considered contextual factors relevant to the evaluation, resources required for the evaluation, and data collection methods and questions appropriate to the user group of the service. To help librarians integrate these elements into a cohesive data collection plan, we offer Exercise 2: Strategizing data collection (anchor link to Exercise 2). This exercise will afford librarians a global view of the initial outcome evaluation study that they are about to undertake.

Using the table developed previously in Step 1: Getting Started (IBEC URL 1.c…) and this step's insights regarding data collection methods as a foundation, in the table below begin to strategize data collection methods, such as focus groups or individual interviews, that will help to document important outcomes for particular activities. A sample plan using the immigrant services example has been provided for further assistance. A sample plan using the immigrant services example has been provided for further assistance.

Data Collection: An Integral Service Element

Collecting outcome data is substantially easier when integrated fully with the service itself. For example, participants in youth community technology programs can be interviewed (or participate in focus groups) over pizza after they have completed a session or the entire program. Students are then revved up and ready to talk about the program. Focus group and interview data provide such rich qualitative data; if possible librarians should capture them on tape (but only after getting the permission of participants). Immigrants who come to a program designed to meet their needs for information about everyday life skills in America (i.e. understanding your teenager, becoming a citizen) can also be interviewed after the program. Individuals enrolled in English as a Second Language programs and those enrolled in adult literacy programs should, as well, be interviewed. Going forward, be sure to capture the stories that they tell and use the patterns to show how your library services are helping them. (We'll discuss this more in the succeeding steps).

Calling individuals who have used specialized community information services by phone may tell you how the service (and the staff) helped. The rule of thumb for telephone interviews of this sort is to keep them short and to the point. Providing people with postcards designed to collect outcome data (and including brief demographic information on the participant) that can be returned to the library may be another way of collecting outcome data. In short, in order to obtain the most appropriate data on outcomes on an on-going basis, be sure to integrate data collection with your program or service.

Once your data collection strategies have been determined and implemented, you can move on to Step 3: Analyzing Data.