-
Planning for Evaluation
SEIR*TEC Home | About SEIR*TEC | Partners | SEIR*TEC Region | Search
-
Getting Started
Evaluation is the process of judging the value or worth of some product, project, or program. The content in this Web site is focused primarily on project evaluation. The results of evaluation help stakeholders make decisions about revising/improving or continuing/ending projects (e.g., the procedure needs improving, the project is not worthy of continued funding).
[ Top ]
Evaluation is typically conducted for three reasons: (1) to determine what type of project should be developed (needs), (2) to determine if a project has been implemented as intended (accountability), and (3) to determine the effects of a project (utility) (Rossi & Freeman, 1989).
Some suggest that evaluation may be conducted before a project is ever conceptualized (Rossi & Freeman, 1989). This type of evaluation may be termed conceptualization or needs assessment, and involves gathering data to determine gaps between the current state of affairs in some situation (e.g., technology literacy) and the desired or optimal state. The evaluation defines for stakeholders whether or not there is a gap, and if so, which are the most appropriate and pressing projects to foster and support.
RESOURCE: To assist with defining school technology needs, educators should read, "TAGLIT: A Tool for Measuring Project's Results," which describes an assessment tool that can be used to determine how technology is being used in a school. As described in the article, TAGLIT has been used to identify certain needs in schools, such as the need for professional development and training.
Evaluation is also conducted to monitor the implementation of a project. A program manager who is accountable to a funding agency may be interested in documenting internal processes and lessons learned, or a program sponsor may wish to verify that funded projects are being implemented as promised. In such cases, evaluation can help to verify that the specified target populations are receiving the treatment as promised, or the specified procedures are being carried out as indicated (i.e., "We said we would do X, and here is the evidence that we did X.").
Finally, evaluation is conducted to determine if a project has met the goals set for it or has achieved desired outcomes. This type of evaluation is the most powerful to argue for project continuation, as it provides evidence that some initiative is related to or caused a positive change (i.e., "We said the project would lead to X, and here is the evidence that the project has effected X.").
[ Top ]
As noted above, the work of evaluation can span the life span of a project from pre-conceptualization, to monitoring implemented initiatives, to judging any effects after implementation. The terms formative and summative evaluation are often used to specify when an evaluation is conducted. A formative evaluation occurs when a project is being designed or implemented to determine if ongoing processes should be revised. A summative evaluation occurs after a project has been implemented to determine if the initiative met the goals set for it and therefore should be adopted or continued as an effective treatment. Summative evaluations take two forms--effectiveness studies just after a new project has been implemented in full, and impact studies several months to years after a project has been implemented in full (longitudinal).
[ Top ]
How is Evaluation Different from Assessment?
Evaluation is related to and often confused with assessment. While assessments are used to determine how much a student knows or how effectively a student can perform some task relative to a criterion or standard, evaluations are used to determine how well a project has achieved the criteria or goals established for it. If a project's goals include "improved student achievement," however, then the evaluation would likely utilize assessment data. So while assessment and evaluation are different, evaluations often utilize assessment data to help make decisions about the value of the project under study. Evaluation is the more inclusive term; it may use assessment, but assessment does not use it.
[ Top ]
Isn't Evaluation the Same as Research?
Evaluation is related to research in that both utilize similar data collection methods and statistical analysis procedures. Evaluation differs from research in focus and involvement, however (Ohio Department of Mental Health, 2002). Research is based on existing theory and attempts to tease out the relationship between variables to add more knowledge to an existing field, while evaluation seeks to determine if a project has met the goals or criteria established for it. Adding to theory or developing a new theory is the focus of research, not evaluation. This is not to say evaluation conclusions have not been used to develop theory, just that the primary focus of evaluation is quality checking for the attainment of goals irrelevant of their contribution to theory.
Further, individuals or groups who have reviewed relevant literature, designed a project, and wish to study its effects, typically conduct research on their own intervention. An evaluator, however, is typically hired as an unbiased third party to estimate the value of some project that they probably had no part in designing.
[ Top ]
An evaluation plan represents your outline for the evaluation process. In a typical request for grant proposals (RFP), you will be asked to provide an evaluation plan. Given the large number of components that should go into an evaluation plan, the RFP page limitations for evaluation plans are often ridiculously low. Regardless, every plan that you prepare should address, either in depth or in short, each of the following components.
-
introduction provide background on the P3 being evaluated, and identify who or what is being evaluated, who is doing the evaluating, who has asked for the evaluation, and who will benefit from the evaluation findings (stakeholders) purpose indicate why the evaluation is being conducted (to identify needs, for accountability, to determine effectiveness) goals and questions (see section II for more information on evaluation questions, and models that help guide questioning) if the P3 was based on a set of goals, specify them, then provide at least one evaluation question for every goal
indicators and benchmarks (see section II for more information on data collection) provide indicators for each evaluation question, what evidence will be used to inform the questions (i.e., student achievement will increase 5% over 3 years); also, provide benchmarks for each indicator, what are the expected intervals of change (i.e., 1% increase in the first year of the program, 2% increase in the second , an 3% increase in the third) methods (see section III for more information on data collection) the design of the evaluation (e.g., case study, randomized trial, mixed methods); the sample; sources of data; instruments (e.g., student reports, teacher interviews, surveys),; a matrix showing which data or instruments will inform which evaluation questions or indicators analysis (see section IV for more information on analysis and reporting) how collected data will be processed: statistical procedures to be used, qualitative synthesis strategies to employ timeline a month-by-month plan for data collection, analysis, and reporting budget a listing of cost items and their prices
Note: a well-prepared evaluation plan can be readily converted to an evaluation report. An evaluation report should include all of the information provided in your original plan, as well as additional sections that outline results, a discussion of the results, recommendations, and limitations to the study (reliability, validity).
[ Top ]
RESOURCE: For more information, read "Tips for Writing an Evaluation Plan for a Technology Grant" by Elizabeth Byrom, SEIR*TEC Principal Investigator.
RESOURCE: For more information on and a general overview of the evaluation process, read about the three-stage process most often employed by school districts "Steps in Evaluating a School or District Technology Program" by Jeff Sun, Sun Associates.
References
Ohio Department of Mental Health. (2002). Program evaluation vs. research. [On-line]. Available: http://www.mh.state.oh.us/offices/oper/feature1.html
Rossi, P. H., & Freeman, H. E. (1989). Evaluation: A systematic approach. Newbury Park, CA: Sage Publications.
[ Top ]
Forming an Evaluation Committee
A district-wide evaluation committee is the organizing structure of the technology evaluations we have facilitated in most districts. As an initial step, districts create a technology evaluation committee composed of stakeholders from throughout the district and local community. This committee closely parallels the structure of the district's technology planning committee in that in order for the committee to be effective, it must be representative of the interests and concerns of the broad district community. It is also important that the committee not be composed entirely of individuals who are considered the technology power users in the district. Remember, the evaluation is designed to answer basic questions about technology's impact on teaching and learning. Therefore, experience in teaching and learning is considerably more important than being well-versed in technology itself. In general, the committees we work with number about 12 to 15 members and include district-level staff, a board member, and principals, as well as classroom teachers and technology specialist from all grades.
Use the Evaluation Committee Composition Matrix as a worksheet when planning your evaluation committee.
[ Top ]