Evaluation of LSTA Grants
An important outcome of any library project is determining the impact on service to library customers. Evaluation reports provide vital information to other communities contemplating replication of the project and can serve as a springboard for additional ideas and services.
The evaluation should focus on:
- the difference in service made by an LSTA funded project
- the impact of the project on the community
- the effect on people rather than the statistical process itself
An evaluation of the project is required of every grant recipient within one month of project completion. Any costs associated with the evaluation should be included in the grant application budget.
Evaluation Plan: How will you know if it is working? Check the appropriate user group that benefits from the project. Check the evaluation method used.
All applicants must provide the following in the evaluation section of the application:
- Describe how you will know objectives or expected results have been met. Describe what is to be measured. Detail how results will be measured. Describe target benchmarks used to determine success.
- Describe evaluation methods used - ways used to measure output and outcomes. Describe how the method was determined.
- Describe data collection methodology, i.e., interviews, number gathering, journals.
- Describe the anticipated outcome or results of the project.
- Explain anticipated impact to customer service, on community. Who will benefit?
- Describe plan to share evaluation results, i.e., brochures, professional organizations, journal articles, web page.
Link all evaluation methods directly to the stated objectives and action plan. Detail the evaluation plan for the project, including a description of the questions to be asked, the information needed to answer the question, the source of the information, and instruments used to gather that information. Describe person/group responsible for the design of any instruments and procedures, the time schedule, the approach to the analysis of the information gathered, and a description of the reports that will result. Describe criteria to measure success or failure. Make sure there are adequate funds to carry out the evaluation plan.
Present how the public will be informed about what was accomplished and how experiences will be shared with others.
You MUST mark an evaluation method. The methods listed are categories listed on the federal report. The State Library must report the number of grants in each category.
Specific methods (types of evaluation) listed on LSTA federal report:
- Impact study: Relies on testimonials, stories, anecdotes, life changing impacts. Focuses on the value of the project’s effects and the ends to which the project’s means were directed.
- Outcome-based: Records the extent to which a project caused changes in a desired direction in the target population and in the cooperating organizations. A sound outcome evaluation demonstrates that changes or progress can be directly linked to program services or intervention. This type of evaluation tries to answer the question of what factors exist in the project’s environment or its history that may influence project conduct and outcomes.
- Tell It Techniques: Specifically for library evaluations developed as part of a U.S. Department of Education Library Programs contract to the University of Wisconsin - Madison School of Library and Information Studies. The State Library has copies of this planning and evaluating method. Tell It techniques include questionnaires or surveys; interviews, individual or focus groups; numbers gathering, counting use; use of existing data for comparison; observation, structured or unobtrusive; anecdotal reports of library activity; self-reporting; logs and journals.
- Other: List and describe any other evaluation methods you use. i.e., performance measurements; logic model; theory-driven; cost benefit or cost-effectiveness; expert review; replication of previous projects; relates to case studies
- Model - Provide project as model for others
When Does Evaluation Occur? Evaluation occurs throughout the planning cycle, during the project, and at the end of the project. Planning determines the evaluation method before the project starts. Pre-project benchmarks (baseline data) are necessary to measure any increase or decrease in service. The above types of evaluation methods use the summative process or the collection of data necessary for judging the ultimate success of the completed project. The primary purpose is to document the extent to which the project’s outcomes were achieved. The summative methods used (those occurring at end of project) document results and justify the program continuation or additional funding. Summative methods usually occur after the project ends to measure outcomes, outputs and impacts, but some information is often collected early in the life of a program (baseline data, test scores) to determine the relationship of different factors.
Formative methods (continual monitoring) measure how
well the project is being done and helps the project move toward its
objectives. Formative evaluations usually are conducted in the early
stages of a program and address questions about implementation and
ongoing planning. This type of evaluation typically examines process
rather than product. The primary value of formative evaluation is that
it allows adjustments to be made throughout the program, rather than at
CHECKLIST FOR REVIEW
- presents a plan for evaluating accomplishment of objectives; not whether the steps in the action plan are accomplished
- uses measurable terms to describe results
- clearly states criteria to measure success/failure
- clearly states anticipated outcomes or impacts/benefits
- includes formative evaluation methods; process for project monitoring
- describes any evaluation reports to be produced and the method of distributing the information
- clearly states who will be targeted for evaluation and how many persons will take part
- describes an evaluation method that is practical to utilize
What Are Evaluation Categories? Evaluation results can be categorized into three groups: inputs, outputs, and outcomes:
- Inputs - resources available to conduct a program, i.e., staff time, budget, volunteers
Reflects level of institutional support for program. Inputs do not totally reflect the results of the program.
- Outputs- Tend to be quantity measures of what the program did with its resources (inputs) i.e., number of hits on web page, number of people using service: used to measure end result ( i.e., increase in use), easier to collect, short-term measures
- Outcomes - Emphasize the consequences of participation in a program, often called benefits, impact, life changes, long-term measures, or success stories. Measure changes in knowledge, skills, behaviors, attitudes or conditions of life status.
All three are needed to give a complete picture of program efforts. A number of techniques used over the life of the project is best. Ultimately, outcome measures are the most important evaluation measures, since these relate directly to the overall mission of any program. However, documenting changes or measuring benefits is more difficult. An effective evaluation approach considers both quantity (counting) and quality (difference or impact). To receive a complete picture of the project, it is necessary to report quantity but also quality.