STEP 1.c Preparing for Evaluation

Understanding the Importance of Context

To determine the outcomes of your service (its impact on users and people in the community) you need to determine the specific contextual components that influence the how your service can influence its users. These components include of course, the service model of the library, the set of activities or strategies that staff have devised to effectively work with those users, the users of your service, and the staff who work with them. Together, these provide the context of most services. There may be other, very important contextual components of your service such as its partnerships in the community, the demands of an external funder, the 'way' in which people interact and their needs or desires that provide the impetus to participate, etc.

Take some time and write down the key elements of the evaluation setting. This can be done through reflection or observation. Consider:

• Who comes to this setting and why?
• What actually goes on?
• What is important about your service model?
• What do the staff (and volunteers if appropriate) contribute to this service?
• What resources are particularly important?
• What resources do you lack?

Collectively these factors contribute to the scope of the evaluation and provide clues not only to the outcomes of a service but also to the amount of resources that the evaluation will require.

To help librarians determine service outcomes we have devised a four-step process, starting with an initial assessment process that frames the outcome context and continues through the process of collecting and analyzing data that result in determining specific outcomes. However, we don't stop there, nor should you. The most important part of this exercise is using what you find in a variety of ways. As you start the exercise in this section you should look ahead to the succeeding steps to get an understanding of the entire process. Once you have familiarized yourself with the whole process, you can begin to tailor the evaluation to your service.

To start the process we have developed an initial exercise, "Exercise 1: Preparing to determine program impact," (anchor link to Exercise 1, below) to help you identify the service that you wish to evaluate, its specific services and users, along with the its potential positive and negative outcomes from the perspective of its direct users and other stakeholders. Consider your responses "rough guesses" or working expectations that you will confirm in the course of your outcomes study. We provide a table that you can copy to record your responses.

1. Determine the scope of the evaluation study.

• Which service is to be evaluated?
• Which activity(s) of the service is to be studied (e.g., ESOL classes for immigrants, tutoring, conversation groups)? Which user group(s) and other stakeholders are to be included in the study?

2. For the activities listed above, potentially who is affected by them? (Think broadly about this, e.g., include families of direct users, etc.)

3. What positive (or negative) outcomes do you think are experienced by these stakeholders as a result of the activities (and may not yet been able to document)?

Think, for example, about user commentary as powerful indicators of outcomes. If you can identify them, then you are well on your way to devising an outcome study that reflects your service's impacts, so recount a story or user comment that reflects a service outcome (positive or negative; expected or unexpected) experienced by a user of the library service.

In addition, brainstorm a list of "ideal outcomes" that would help to build the strongest case for the benefits of the service. What differences does this service make in people's lives?

Be explicit about your assumptions of the outcomes that you consider likely to appear via data collection. Your data collection methods (discussed in the next step) will elicit likely outcomes in addition (and, more importantly) to ferreting out unexpected outcomes-i.e., those of which you are unaware.

The preparation exercise above provides a framework that can guide through the thought process of outcome evaluation and towards the selection of data collection outlined in Step 2: Collecting Data.

Additional tips for preparing your evaluation project:

1. Think small. Focus on one program or one aspect of your services at a time. This will help you to gather data more effectively and will provide more meaningful results.
2. Involve staff and stakeholders in the evaluation design process. Be sure that everyone who might be impacted by the evaluation has input into the goals of the project and understands their role in the evaluation process.
3. Lastly, make sure to acquaint yourself and your stakeholders on the component parts of outcome-based evaluation, including:

• Establish overall evaluation goals for a context (i.e. purpose)
• Identify key dimensions to the context that inform evaluation design
• Identify and secure resources needed to conduct evaluation
• Develop evaluation plan including timeline
• Design draft evaluation instruments
• Pre-Test draft evaluation instruments
• Finalize draft evaluation instruments
• Collect Data
• Code Data
• Analyze Data
• Confirm data analysis with inter-coder reliability testing and/or member checking where appropriate
• Write Up Findings
• Incorporate your findings on outcomes in reports for different audiences, newsletters, public presentations, etc.

For more information on preparing for outcome-based evaluation visit:

"New Directives, New Directions: Documenting Outcomes In IMLS Grants To Libraries And Museums" This site provides a general introduction to outcome-based evaluation and includes an outcome measures template for library services.

Once you have determined your primary goals for outcome-based evaluation and have determined what should be measured, you are ready to move on to Step 2: Collecting Data.