1. Introduction

 

How to Use These Resources

These materials are provided to help school and district staff members who are not trained evaluators build their organizations' internal capacity to evaluate projects that apply technology in education settings. It can be used in two ways:

You can start at the beginning of this list of materials and descriptions and follow the link at the bottom of each page to each successive section to work through the steps required of a quality project evaluation. Or, if you have a specific need for, or interest in, information about evaluation materials, you can either jump to an individual section of the SEIR*TEC formative evaluation framework, or use the index provided in the Resources section.

What is Evaluation?

Evaluation is the process of judging the value or worth of a product, project, or program. Although the SEIR*TEC materials focus on formative project evaluation - intended to help project managers understand how an effort is working, so they can make decisions about revising or improving implementation - many of the concepts presented can be extended to evaluations of other things (personnel, for example) or for other purposes.

Why is Evaluation Conducted?

Evaluation is typically conducted for three reasons: (1) to determine what type of project should be developed (often referred to broadly as needs assessment), (2) to determine if a project has been implemented as intended (implementation evaluation), and (3) to determine the effects of a project (impact evaluation). There are other terms recognized by evaluators as describing the same purposes but these three terms will be used here for the sake of consistency.

Needs Assessment - Sometimes, evaluation is conducted before a project is even started (Rossi & Freeman, 1989). This needs assessment (or conceptualization) phase involves gathering data to determine gaps between the current state of affairs in a particular situation (e.g., technology integration) and the desired or optimal state. Project managers and other stakeholders can then determine whether there is a gap and if so, which are the most appropriate and pressing project activities to foster and support.

To assist with defining school technology needs, educators might consider the School Technology Needs Assessment, (STNA, say "Stenna") developed through a collaboration between SEIR*TEC and the North Carolina Department of Public Instruction." Another option is "TAGLIT: A Tool for Measuring Project's Results" - an assessment tool that can be used to determine how technology is being used in a school.

Implementation Evaluation - Evaluation is often conducted to monitor the fidelity with which the activities of a project are being implemented. A project manager who is accountable to a funding agency may be interested in documenting internal processes and lessons learned, or a project sponsor might want to verify that funded projects are being implemented as promised. In such cases, evaluation can help to verify that the specified target populations are receiving the promised services, or the sponsored procedures are being carried out as indicated (i.e., "We said we would do X, and here is the evidence that we did X.").

Impact Evaluation - Evaluation is also conducted to determine if a project has met its goals or has achieved the desired outcomes. This type of evaluation is the most powerful to argue for project continuation, as it provides evidence that a particular initiative is related to, or influenced, a positive change (i.e., "We said the project would effect X, and here is the evidence that the project has effected X.").

When is Evaluation Conducted?

The work of evaluation can span the life span of a project from pre-conceptualization, to monitoring implemented initiatives, to judging any effects after implementation.

Formative evaluation is intended to determine if or how project activities should be revised, or how resources might be reallocated, while summative evaluation is applied to decide whether a project should be adopted or continued. Both uses of evaluation findings may require shorter- or longer-term evaluation efforts, or some combination of the two. For example, a comprehensive summative evaluation might include an examination of immediate results (sometimes more specifically referred to as project effects), and a longitudinal study to determine long term impact.

The terms formative and summative evaluation are sometimes used primarily to describe when an evaluation is conducted - during or after implementation. However, this is not an entirely accurate use of these terms, which more correctly refer to the intended uses described above.

How is Evaluation Different from Assessment?

Evaluation is related to and often confused with assessment. In the context of education, assessment is the process used to determine how much a student knows or how effectively a student can perform some task relative to a criterion or standard. Evaluation is intended to determine how well a project has achieved the criteria or goals established for it. If a project's goals include "improved student achievement" however, then the evaluation would likely utilize assessment data. So while assessment and evaluation are different, evaluations often utilize assessment data to help make decisions about the value of the project under study.

Is Evaluation the Same as Research?

Evaluation is related to research in that both utilize similar data collection methods and statistical analysis procedures. However, evaluation differs from research in focus and involvement (Ohio Department of Mental Health, 2002). Research is based on existing theory and attempts to tease out the relationship between variables to add more knowledge to an existing field, while evaluation seeks to determine project fidelity of implementation and/or impact. Adding to theory or developing a new theory is the focus of research, not evaluation. This is not to say evaluation conclusions have not been used to develop theory, just that the primary focus of evaluation is quality checking for the attainment of goals rather than contribution to theory.

Further, individuals or groups who have reviewed relevant literature, designed a project, and wish to study its effects, typically conduct research on their own intervention. An evaluator, however, is often hired as an unbiased third party to estimate the value of a project, from a more distant perspective. A person working in this capacity is typically called an external evaluator, whereas someone conducting evaluation from within the organization implementing the project under examination is referred to as an internal evaluator.

The SEIR*TEC Formative Evaluation Framework

Projects that apply technology pose particular evaluation challenges. In response to this and the increased emphasis placed on determining the value of technology implementations in education settings, SEIR*TEC has developed a flexible framework for planning, implementing, and reporting on formative evaluation of technology-in-learning projects.

Next > 2. Overview: The SEIR*TEC Framework for Formative Evaluation

If you have questions or comments, contact

This page last updated 6/23/05