NORTH CAROLINA DEPARTMENT OF PUBLIC INSTRUCTION
Technology Literacy Challenge Grant Program
Project Evaluation

 
The North Carolina Department of Public Instruction (DPI) is providing this evaluation instrument to the recipients of the States Technology Literacy Challenge Fund (TLCF) grants to help them evaluate their projects progress and impact. The intent is to provide an effective, yet relatively simple tool that will help school system leaders (a) reflect on activities to date vis à vis effective practices in project management and technology integration, (b) think about what needs to be done in order to meet project goals, and (c) consider strategies for maximizing project impact. The instrument should also provide an effective means for collecting and reporting comparable information across the projects.

The development of this evaluation tool has been a collaborative effort between NCDPI and the SouthEast and Islands Regional Technology in Education Consortium (SEIR*TEC). Leaders of the 1997 TLCF projects reviewed drafts and offered constructive suggestions for clarification and improvement. The conceptual framework and design of the instrument are borrowed from several sources, including

  • The Technology Literacy Challenge Fund grant application instructions;
  • U.S. Department of Education initiatives to collect comparable data from diverse technology projects;
  • Current trends in educational evaluation that call for multi-dimensional appraisal rather than adherence to single dimensions, such as achievement test scores;
  • Effective practices for managing grants and technology projects; and
  • The research literature on factors associated with successful instructional technology programs.

The instrument is divided into two major sections. The first applies to every project, while the second addresses the specific North Carolina TLCF instructional priority models: integration, staff development, assessment, and technical assistance. Therefore, each project must complete all of Section I and only the part of Section II that applies to their model.

Both Section I and Section II contain several principles of good practice (labeled A, B, C, etc.) and indicators of three levels of success, as described in the following scale:

  1. Minimum: The project is going forward as described in the proposal.

  2. Moderate: Project objectives are being met. Where appropriate, project strategies and activities impact additional areas of the instructional program, e.g., other schools, curricular areas, grade levels.


  4. Strong: Project strategies and activities are institutionalized throughout the school systems ongoing instructional program.


Instructions:

As you go through the form, read the three indicators for each principle and determine which one best describes where your project is as of the end of the project year. Circle the number corresponding to that indicator. In the optional "Comments" block, add any information that you think other educators will need in order to understand the status of your project. Another option is to use the empty tables at the end of each section to add principles and indicators that help you evaluate your project. Send the completed form by July 31, 1998 to:

Elsie L. Brumback, Director
Instructional Technology Division
North Carolina Department of Public Instruction
301 N. Wilmington Street
Raleigh, NC 27601-2825


Some Assurances:

One of the major purposes of this type of evaluation is to show progress and impact over time. For projects designed to take five years for full implementation, it would be unusual for the evaluation to show many 3s (strong performance) during the first year. Projects designed for one year should reach some level of institutionalization or sustainability by the end of that one-year period. In rare instances, there might be a principle that does not apply to a particular project; such cases should be explained in the Comments box. To maintain the multi-dimensional character of the evaluation, NCDPI will not attempt to collapse individual project data into a single figure, such as an average or grade. Similarly, there will be no attempt to rank projects according to evaluation data. Being able to see patterns and trends in levels of success across all the projects will be much more instructive. It will be up to the individual project leaders to see how their project compares with the others.


A Word to the Wise:

The "There is evidence that" phrase at the beginning of each section and priority area indicates that the school or district should have some form of documentation that a certain level of project success has been attained. Evidence could include anything from surveys, interviews, and classroom observational data to minutes of meetings, workshop evaluations, reports to the school board, portfolios, photographs, or multimedia presentations. This does not mean that a school or district has to put a lot of extra energy into collecting data; rather, it means that project managers should keep a file of all major project activities just in case they ever need to substantiate progress, e.g., for a legislator or auditor.
Section I: Principles and Indicators of Success for All Projects

There is evidence that:

A. The project attains the goals and meets the objectives outlined in the proposal.

1

2

3
Project activities address the objectives outlined in the proposal. Major project activities are completed, and most objectives are met as described in the proposal. The project has met its goals.
Comments:

 

 

 



B. The project supports the implementation of the school system instructional technology plan.

1

2

3
Project activities support and positively impact the school systems instructional technology plan. As a result of the project, specific objectives of the existing instructional technology plan have been accomplished. The project has moved the state of technology integration forward throughout the school system and has led to the identification of new directions and goals for the instructional technology program.

 
Comments:

 

 

 



C. Businesses and community partners are actively involved in the project.

1

2

3
The project is beginning to implement strategies for involving businesses and the community in the project. Project partners from businesses and the community are actively involved in planning and implementing the project. Old partnerships are strengthened and new ones are formed. The school system builds on the partnerships and collaborative relations to formulate new educational technology initiatives.
Comments:

 

 

 



D. The project reduces disparities in access to and utilization of technology.

1

2

3
Disparities have been identified and a plan is in place to overcome the barriers that created the disparities. Technology implementation helps reduce disparities and overcome barriers to teaching and learning. Systems are in place to anticipate and react to disparities as they arise.
Comments:

 

 

 



E. Information about the project is disseminated to other educational entities, such as schools, school districts, universities, and professional associations.

1

2

3
Information about the project is distributed throughout the school system and across the community. Products have been developed (e.g., documentation of the model, training materials, student materials) and are being disseminated. As a result of dissemination activities, other educational entities are contacting the grantee for products, information, or advice.
Comments:

 

 

 



F. The project is being implemented in a manner that enhances the likelihood that programmatic activities are sustained after federal funding ends.

1

2

3
School system leaders and project partners recognize the importance of sustaining project activities over time. School system leaders and project partners have developed plans for sustaining the projects activities after federal funding ends. School system leaders and project partners have implemented plans to sustain the project after federal funding ends.
Comments:

 

 

 



G. The project is well managed.

1

2

3
The proposal identifies project leader(s) and has outlined an organizational structure and project management strategies. Project management procedures are shared with project participants and are in operation. Project activities are on time, on target, and within budget. Project participants are satisfied with management strategies.
Comments:

 

 

 



H. Stakeholders (those who are affected by the project) are actively involved in planning and implementing technology integration.

1

2

3
Stakeholders have input into planning project activities, as stated in the proposal. The school system implements specific strategies for ensuring the input and participation of stakeholders in technology integration. The school system institutionalizes a system for maximizing stakeholder involvement in the planning and implementation of technology into teaching and learning.
Comments:

 

 

 



I. Educators have the training and technical assistance necessary for the successful completion of project objectives.

1

2

3
A plan is in place for systematic training and technical assistance to meet project objectives. Training and technical assistance strategies are developed and implemented. Project participants received training and technical assistance on site and on demand.
Comments:

 

 

 



J. As a result of the project, educators are using technology for instruction.

1

2

3
Project participants are integrating the projects technology applications into their ongoing instructional programs. Instructional technology applications associated with the project have been adopted by teachers in other curricular areas or grade levels than those anticipated originally. Instructional technology applications associated with the project are being adopted system-wide in appropriate academic areas and grade levels.
Comments:

 

 

 



K. As a result of the project, students are using technology as a tool for learning.

1

2

3
The projects target students regularly use technology for the applications described in the proposal. The scope of technology use by students has expanded beyond the original scope of the project to include other curricular areas or grade levels. All students, including those with special learning needs and those who are at risk of academic failure, use technology as a tool for learning.
Comments:

 

 

 



L. As a result of increased technology use, students demonstrate improvement in class work, assessment, attitudes, and/or behavior.

1

2

3
The project has developed strategies to document the impact of technology on student performance, attitudes, and/or behavior. As a result of project activities, the target students have demonstrated increased motivation in school work, higher performance ratings, and/or new workforce skills. The school system has evidence that project activities, strategies, or products have had a positive impact on the performance, attitudes, and behavior of students beyond those originally served by the project.
Comments:

 

 

 



M. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 

 



N. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 

 



Section II: Principles and Indicators of Success for Instructional Priority Models
Priority Area I:   Model designed to integrate technology into the teaching and learning process in a way that supports the school systems implementation of the North Carolina State Board of Educations ABCs and in keeping with President Clintons four national goals for technology.

There is evidence that:

A. A replicable model for integrating technology into teaching and learning is developed and implemented

1

2

3
An informal technology integration model is developed but has not been documented or pilot tested. A technology integration model is documented and tested. The model is implemented with the population described in the proposal. A replicable technology integration model is an integral part of the school systems on-going technology program.
Comments:

 

 

 



B. Technology is integrated into curricular areas outlined by the North Carolina ABCs.

1

2

3
Project participants are knowledgeable about the state curriculum area identified in the proposal and have initiated strategies for integrating technology into that area. Project participants have developed strategies, materials, and resources for integrating technology into the designated curricular area and are planning to expand to other areas, sites, and grade levels. Strategies for integrating technology into additional curricular areas identified by the NC ABCs are being implemented. The school system is planning to expand the strategies to address other curricular areas and grade levels, as appropriate.
Comments:

 

 

 



C. Project participants use lesson plans, instructional units, and other resources that support the integration of technology into the curricular areas designated by the North Carolina ABCs.

1

2

3
Project participants have developed or adopted lesson plans, instructional units, and other resources that incorporate technology into the curricular area and grade level(s) designated in the proposal. Most teachers of the subjects and grades identified in the proposal frequently use technology as tools for learning, as evidenced by lesson plans, instructional units, and other resources. Teachers in additional curricular areas, outlined by the North Carolina ABCs, have adopted and are using technology as tools for learning as evidenced by lesson plans, instructional units, and other resources.
Comments:

 

 



D. Teachers learn and employ innovative teaching strategies that use technology as a tool for learning.

1

2

3
Project planners identify teaching strategies that help teachers become the "guide on the side" rather than the "sage on the stage." Teachers participate in training on the strategies. Teachers experiment with and adopt student-focused teaching strategies in their classes, e.g., constructivist learning, collaborative work groups, student research projects. As a result of the project, teachers fluidly employ a variety of effective teaching strategies, according to the needs of the students and the nature of the subject matter.
Comments:

 

 



E. Administrators provide opportunities for teachers to learn and experiment with technology applications.

1

2

3
Principals and central office staff begin to implement ways to encourage and support the use of technology by project participants, e.g., arranging time, providing resources, and allowing freedom for experimentation. Principals and central office staff implement strategies that encourage and support teachers use of technology, e.g., time for learning and access to state of the art equipment. Mechanisms are in place systemwide to support the use of technology by all educators. Access to technology resources, training, and technical support is equitable.
Comments:

 

 



F. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 



G. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 




Section II: Principles and Indicators of Success for Instructional Priority Models

Priority Area II: Technology staff development training model for teachers designed to implement the technology competencies outlined in the School Technology Users Task Force Report.

There is evidence that:

A. A replicable model for teacher training is developed and implemented.

1

2

3
An informal professional development model is developed but has not been documented or pilot tested. A professional development model is documented and tested. The model is implemented with the teacher population described in the proposal. A replicable professional development model is an integral part of the school systems on-going technology program as well as the systemwide professional development program for all teachers.
Comments:

 

 

 



B. Professional development activities meet the identified needs of teachers.

1

2

3
The school system conducts a formal needs assessment of the teachers involved in the project and designs professional development activities based on the results. Each year, the school system conducts formal needs assessments of all teachers (e.g., survey, focus groups) and collects other needs data throughout the year (e.g. workshop evaluations). Each school in the system institutionalizes annual formal and informal needs assessments and uses the results to update the professional development component of the school improvement plan.
Comments:

 

 

 



C. Professional development activities are offered at times and places that enhance participation by teachers.

1

2

3
The school system offers professional development activities for project participants at a variety of times and places, e.g., after school, on weekends, and during the summer. The school system tries and evaluates innovative ways of enhancing participation in professional development activities while acknowledging the demands on teachers time. The school system has evidence of teacher satisfaction with the technology training program, such as increased participation by all teachers.
Comments:

 

 

 



D. Professional development activities are on-going and just-in-time.

1

2

3
Workshops on technology applications and instructional strategies are offered throughout the year. The requisite technology is in place for teachers to apply new skills and strategies (i.e., teachers dont receive training before they have access to the technology). Project participants engage in professional growth activities that are timed to coincide with classroom activities and instructional units. All teachers in the system have access to professional development activities that are timed to coincide with classroom activities and instructional units.
Comments:

 

 

 



E. Teachers learn and apply the North Carolina Technology Competencies.

1

2

3
Project participants are beginning to apply the Technology Competencies in their teaching and in professional activities. Project participants and their peers have developed skills identified in the NC technology competencies and are demonstrating them in their teaching. Teachers throughout the system have developed skills identified in the Technology Competencies and are demonstrating them in their teaching.
Comments:

 

 

 



F. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

 

    
Comments:

 

 

 



G. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 

 




Section II: Principles and Indicators of Success for Instructional Priority Models

Priority Area III: Performance based assessment model that measures the impact that technology makes on student achievement and is consistent with the four expected results established by the School Technology Commission.

There is evidence that:

A. A replicable performance based assessment model is developed.

1

2

3
An informal model is developed but has not been documented or pilot tested. A performance based assessment model is documented and tested. The model is implemented with the population described in the proposal. A replicable performance based assessment model is an integral part of the school systems on-going technology program.
Comments:

 

 

 



B. The assessment model is multi-dimensional.

1

2

3
Project planning and initial activities are based on input from several groups and sources of information. The assessment model contains a variety of sources of information and perspectives on the impact of technology, i.e., goes beyond scores on standardized achievement tests. The assessment model is an interwoven collection of several strategies, each of which addresses a different target audience, issue, or expected outcome to measure the impact of technology.
Comments:

 

 

 



C. Refinements to the assessment model are based on information from the piloting activities.

1

2

3
A process has been developed for refining the assessment model based on project findings. Effectiveness and efficacy data have been gathered and used to refine the assessment model or its components. A system is in place for periodic revision of the assessment model, based on findings from existing activities.
Comments:

 

 

 



D. The model establishes a process for documenting at least one of the four expected benefits identified in the North Carolina Instructional Technology Plan (NCITP):

  • Student learning/achievement
  • Student workforce readiness
  • Teacher productivity
  • Cost-effectiveness

1

2

3
The project identifies which of the expected benefits of the North Carolina Instructional Technology Plan will be addressed by the assessment model. Project activities include data gathering and analysis to determine the impact of technology implementation on at least one of the NCITP areas. As part of the assessment model, a process exists to document the benefits of implementing technology to at least one of the NCITP areas.
Comments: (Indicate which of the expected benefits the project addresses.)

 

 

 



E. Documentation of project activities includes references to research and the professional literature supporting the impact of technology on the teaching/learning process.

1

2

3
The project includes plans to use research or current literature to develop or revise activities in developing the model. Findings from similar projects and published research have been reviewed and incorporated into the assessment model. Documentation of the assessment model describes activities that incorporate existing research findings as well as states parallels to or extends findings published in current literature.
Comments:

 

 

 



F. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

 

    
Comments:

 

 

 



G. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 

 




Section II: Principles and Indicators of Success for Instructional Priority Models

Priority Area IV: Creative, replicable model for providing technical assistance directly to classroom teachers in the integration of technology into their day-to-day instructional program.

There is evidence that:

A. A replicable model for technical assistance is developed.

1

2

3
An informal technical assistance model is developed but has not been documented or pilot tested. A technical assistance model is documented and tested. The model is implemented with the population described in the proposal. A replicable technical assistance model is an integral part of the school systems ongoing technology program.
Comments:

 

 

 



B. Technical assistance is designed to meet the identified needs of teachers and staff.

1

2

3
The technical assistance model is based on input from the teachers and staff of the sites to be served. Teachers and staff complete a yearly needs assessment for technical assistance that is used to determine support to be provided. Teachers and staff routinely assist in determining technical assistance activities that will best meet their needs.
Comments:

 

 

 



C. The scope of work for the technical assistance providers is realistic and flexible.

1

2

3
Data are gathered to determine realistic job expectations of the technical assistance providers. Technical assistance plans are designed accordingly. Technical assistance providers have a high level of job satisfaction. A process for maintaining realistic expectations of technical assistance providers is an integral part of the local school system technology plan implementation.
Comments:

 

 

 



D. Technical assistance is readily available and on-going.

1

2

3
Technical assistance activities have been designed to optimize resources and time, with input from those being served. Technical assistance activities are being implemented in all the designated schools. Technical assistance strategies deemed successful during the project are part of the ongoing school budget and personnel allotments.
Comments:

 

 

 



E. Information is available that the technical assistance is beneficial to teaching and learning.

1

2

3
Project planners have developed, but not yet implemented, a system for gathering information on the benefits of technical assistance. The project collects and analyzes data to document the benefits to teachers and students. School system leaders and project participants use the benefits of technical assistance to further technology support efforts in the schools.
Comments:

 

 

 



F. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments:

 

 

 



G. Additional principles and indicators: ________________________________________
______________________________________________________________________

1

2

3
 

 

 

 

 

 

    
Comments: