3. Theory

 

Explaining How Your Project Works

In the process of planning the evaluation of an after-school technology project for middle school students, an evaluator convened a focus group of project managers and board members. It quickly emerged that the various stakeholders had completely different visions of what good the project was supposed to do for participants and their families.

 

One saw it as computer skills training to prepare young people for jobs in local business, while another believed that it would provide underprivileged families with access to information capital to help lift them out of poverty. After the meeting, the project director explained to the evaluator in an aside with the evaluator that she believed the biggest benefit was that girls were provided a place to go after school, heading off behaviors that might result in pregnancy.

The implications for evaluation planning in this actual case should be clear: It could be entirely possible for one decision maker to see the project as a complete success, while another determines it to be an utter failure - based on the same evaluation findings - depending on what outcomes they expected the project was supposed to further.

Projects implementing technology in school settings may be particularly challenging to describe since it may play so many different roles in the work of teachers, administrators, or students, that may be thought of as acting on teaching and learning processes in a huge variety of ways.

Logic Maps

SEIR*TEC endorses the use of logic maps (sometimes referred to as "logic models," "causal maps," or "theory-of-action models") as powerful tools to clarify project activities and desired outcomes, and to build consensus among project managers, participants, and stakeholders.

A relatively simple introduction to logic mapping is provided in Logic Maps for Planning and Evaluation (PDF), while a more comprehensive view can be found in Building Capacity for a Technology Project Evaluation: The Role of Logic Maps (Power Point) - the latter being a presentation by SEIR*TEC at the U.S. Department of Education's Evaluation Institute (February, 2005).

Another excellent resource is Learning From Logic Models: An Example of Family/School Partnerships Programs from the Harvard Family Research Project, a brief that provides an excellent step-by-step overview of how to develop a logic map.

Developing a Logic Map

It might first be useful to examine an example of a logic map for a comprehensive educational technology project. While a logic map may vary depending on the key terms that planners agree to apply, the example logic map format illustrated here has the following features, defined as key aspects of the SEIR*TEC formative evaluation framework.

• Goals (sometimes called ultimate or long term "Outcomes") are shown at the far right of the map
• Objectives (or "Intermediate Outcomes") are one level to the left
• Arrows connect Objectives to Goals, illustrating how meeting the Objectives is expected to result in achievement of the Goals
• Strategies applied by the project are shown at the next level to the left, connected again with arrows to the Objectives that each is expected to further
• Finally, Inputs are illustrated and linked to the Strategies on which they bear

It is generally most useful to tackle creation of a logic map using the same "backward mapping" approach suggested in the description above.

1. Start by listing the Goals that the project should achieve. To the extent possible, describe these as new skills, knowledge, or attitudes for stakeholders or as changes in conditions bearing on the project - NOT as the completion of project activities.
2. Identify pertinent standards (if any) and align Goals with them - No Child Left Behind (NCLB), Enhancing Education Through Technology (EETT), state curriculum expectations, or others.
3. Define and clarify the Objectives that must be met along the way to achieving the Goals, making sure Objectives are not simply the completion of activities undertaken in the project. Redefine if necessary.
4. Define the Strategies - the broad approaches - that will be implemented in an effort to meet project Objectives. If a large number of Strategies result, it may be that either (a) some are actually activities - actions that make up Strategies - and they need to be clustered with other activities, or (b) the project is too broad or over-reaching in its definition, and should be narrowed.
5. List the Inputs that should be considered under each strategy. These are the resources available and contextual issues that bear on project implementation (e.g., budget constraints, relevant policies, curriculum requirements, or data describing stakeholders' needs).
6. Consider rearranging logic map elements vertically in their columns, putting them in some order (e.g., chronological) that makes sense to the evaluation planning team
7. Finally, work through the logic map columns to be sure that all logical linkages among elements have been illustrated

It might prove useful to start this process by using the SEIR*TEC Logic Map Template (Inspiration 7.5), completing the elements with content defined for your project.

Using the Logic Map

Importantly, a logic map is a means to an end, rather than an end unto itself. It is a conceptual tool that serves as the foundation for subsequent steps of the evaluation planning and implementation processes.

As the evaluation process unfolds, the logic map is a living document that will likely change as it is used. Project managers should regularly hold their implementation up against the logic map, to assess the degree to which project implementation aligns with it. It may be necessary to adjust the logic map to reflect changes in consensus understanding about underlying theories driving project design and implementation. It is also useful to overlay onto the logic map information about data collection and analysis, to ascertain the degree to which a complete evaluation picture is developing.

Most immediately however, the logic map should facilitate the accurate definition of project Goals and Objectives, and of the Strategies applied in attempts to achieve them.

Next > 4. Outcomes: Goals and Objectives, and Their Relationship to Strategies

If you have questions or comments, contact

This page last updated 6/23/05