Educational Affairs Office: Program Evaluation

PROGRAM EVALUATION (Advance scientific knowledge and understanding of Earth, the Solar System, and the Universe...)

Programmatic Usage of Evaluation

INDICATOR Is this a quality education program?

SUPERIOR Program is assessed by education professionals to determine customer needs, unique NASA contribution, scientific accuracy, NASA education guidelines, requirements of successful teachers, best classroom practices, program impact, replicability and sustainability.
Program goals, objectives, implementation, and potential impact on target group(s) are well-articulated. Thoughtful and informed alignment between outcomes and program activity.
Program assumptions are tested against a review of literature, educational data or reports, lessons learned from similar efforts, and a knowledge base of successful practices. Characteristics of (documented) successful programs are reflected in the program.

GOOD Program is assessed by NASA or education professionals to determine NASA contribution, NASA guidelines, scientific accuracy, requirements of teachers, current classroom practices, program uniqueness and creativity, possible long-term usage.
Program goals, objectives, implementation, and potential impact on target group(s) are clearly articulated. Thoughtful relationship between outcomes and program activity.
Program assumptions are reviewed against educational reports, similar program efforts, or a knowledge base of successful practices. Characteristics of (documented) successful programs are reflected in the program.

ACCEPTABLE Program is reviewed by teachers and NASA officials for scientific accuracy and novelty, with some attention to successful classroom practices.
Program goals, objectives, implementation, and potential impact on target group(s) are evident, if not well-articulated. Some relationship between outcomes and program activity, sometimes unanticipated
Program assumptions are related to personal experience(s) of development staff, and generalized to larger populations.

NEEDS REVISION Generalization of impact from limited samples of NASA- or space-attentive customers.
Confusion between program activity and program impact.
Program assumptions are based on expectations of the formal education system.

Innovation

INDICATOR Has this program or product been done before, or done well?

SUPERIOR All program elements are both non-standard and non-traditional. The program "goes where no one else has gone before."

GOOD Most program elements are both non-standard and non-traditional.

ACCEPTABLE Some program elements are both non-standard and non-traditional.

NEEDS REVISION Program elements are standard and traditional

Educational Value

INDICATOR What is the educational value of this program or product?

SUPERIOR Program, activity, or materials accurately reflect best practices by educators in current educational standards or professional literature

GOOD Good alignment between program practices or materials as recommended in current educational standards or professional literature

ACCEPTABLE Some alignment between program practices or materials as recommended in current educational standards or professional literature

NEEDS REVISION No alignment between program practices or materials as recommended in current educational standards or professional literature

Internal

INDICATOR How should our programs be judged for technical accuracy and merit?

SUPERIOR Program, activity, or materials extensively evaluated by institutionally representative panels for accuracy, educational value, and scientific merit against established standards

GOOD Program, activity, or materials evaluated by representative panels for accuracy, educational value, and scientific merit against established standards

ACCEPTABLE Program, activity, or materials evaluated by ad hoc individuals or panels, but not against established standards or criteria

NEEDS REVISION Program, activity, or materials evaluated by only a few individuals, not evaluated against established standards or criteria, or not evaluated at all

External

INDICATOR What is NASA's Education Division's role?

SUPERIOR Program, activity, or materials extensively evaluated and highly rated by NASA Education Division

GOOD Nominal evaluation by NASA Education Division, no rating given

ACCEPTABLE No evaluation by NASA Education Division or no rating given

NEEDS REVISION Program not shared with NASA Education Division

Professional Qualifications

INDICATOR How should we choose an educational program evaluator?

SUPERIOR Program, activity, or materials extensively evaluated by an independent evaluator or agency with nationally recognized expertise in an area of educational (science education, instructional technology, professional/staff development, etc.)

GOOD Program, activity, or materials evaluated by an independent evaluator or agency with some recognized expertise in an educational area

ACCEPTABLE Program, activity, or materials evaluated by an independent evaluator or agency with some experience in an educational area

NEEDS REVISION No independent evaluator, or evaluator(s) outside their area of expertise

Educational Usage of Evaluation

INDICATOR What are the educational uses of evaluation studies?

SUPERIOR Evaluation builds capacity of staff, affects resource allocation, informs supporters and contributors of progress, supports high-quality instructional programs, and promotes quality access to students and teachers.

GOOD Informs supporters and contributors of progress, affects resource allocation, and supports high-quality instructional programs

ACCEPTABLE Informs supporters and contributors of progress. Users of evaluation may or may not make premature judgments, or may not consider the evaluation design or evaluation instrument.

NEEDS REVISION Misuse of evaluation studies; policies made on assumptions of the study, or misinterpretation of the data or conclusion(s).

Content: Richard Alvidrez
Web Developer: Kristy Kawasaki

INDICATOR	Is this a quality education program?
SUPERIOR	Program is assessed by education professionals to determine customer needs, unique NASA contribution, scientific accuracy, NASA education guidelines, requirements of successful teachers, best classroom practices, program impact, replicability and sustainability. Program goals, objectives, implementation, and potential impact on target group(s) are well-articulated. Thoughtful and informed alignment between outcomes and program activity. Program assumptions are tested against a review of literature, educational data or reports, lessons learned from similar efforts, and a knowledge base of successful practices. Characteristics of (documented) successful programs are reflected in the program.
GOOD	Program is assessed by NASA or education professionals to determine NASA contribution, NASA guidelines, scientific accuracy, requirements of teachers, current classroom practices, program uniqueness and creativity, possible long-term usage. Program goals, objectives, implementation, and potential impact on target group(s) are clearly articulated. Thoughtful relationship between outcomes and program activity. Program assumptions are reviewed against educational reports, similar program efforts, or a knowledge base of successful practices. Characteristics of (documented) successful programs are reflected in the program.
ACCEPTABLE	Program is reviewed by teachers and NASA officials for scientific accuracy and novelty, with some attention to successful classroom practices. Program goals, objectives, implementation, and potential impact on target group(s) are evident, if not well-articulated. Some relationship between outcomes and program activity, sometimes unanticipated Program assumptions are related to personal experience(s) of development staff, and generalized to larger populations.
NEEDS REVISION	Generalization of impact from limited samples of NASA- or space-attentive customers. Confusion between program activity and program impact. Program assumptions are based on expectations of the formal education system.

INDICATOR	Has this program or product been done before, or done well?
SUPERIOR	All program elements are both non-standard and non-traditional. The program "goes where no one else has gone before."
GOOD	Most program elements are both non-standard and non-traditional.
ACCEPTABLE	Some program elements are both non-standard and non-traditional.
NEEDS REVISION	Program elements are standard and traditional

INDICATOR	What is the educational value of this program or product?
SUPERIOR	Program, activity, or materials accurately reflect best practices by educators in current educational standards or professional literature
GOOD	Good alignment between program practices or materials as recommended in current educational standards or professional literature
ACCEPTABLE	Some alignment between program practices or materials as recommended in current educational standards or professional literature
NEEDS REVISION	No alignment between program practices or materials as recommended in current educational standards or professional literature

INDICATOR	How should our programs be judged for technical accuracy and merit?
SUPERIOR	Program, activity, or materials extensively evaluated by institutionally representative panels for accuracy, educational value, and scientific merit against established standards
GOOD	Program, activity, or materials evaluated by representative panels for accuracy, educational value, and scientific merit against established standards
ACCEPTABLE	Program, activity, or materials evaluated by ad hoc individuals or panels, but not against established standards or criteria
NEEDS REVISION	Program, activity, or materials evaluated by only a few individuals, not evaluated against established standards or criteria, or not evaluated at all

INDICATOR	What is NASA's Education Division's role?
SUPERIOR	Program, activity, or materials extensively evaluated and highly rated by NASA Education Division
GOOD	Nominal evaluation by NASA Education Division, no rating given
ACCEPTABLE	No evaluation by NASA Education Division or no rating given
NEEDS REVISION	Program not shared with NASA Education Division

INDICATOR	How should we choose an educational program evaluator?
SUPERIOR	Program, activity, or materials extensively evaluated by an independent evaluator or agency with nationally recognized expertise in an area of educational (science education, instructional technology, professional/staff development, etc.)
GOOD	Program, activity, or materials evaluated by an independent evaluator or agency with some recognized expertise in an educational area
ACCEPTABLE	Program, activity, or materials evaluated by an independent evaluator or agency with some experience in an educational area
NEEDS REVISION	No independent evaluator, or evaluator(s) outside their area of expertise

INDICATOR	What are the educational uses of evaluation studies?
SUPERIOR	Evaluation builds capacity of staff, affects resource allocation, informs supporters and contributors of progress, supports high-quality instructional programs, and promotes quality access to students and teachers.
GOOD	Informs supporters and contributors of progress, affects resource allocation, and supports high-quality instructional programs
ACCEPTABLE	Informs supporters and contributors of progress. Users of evaluation may or may not make premature judgments, or may not consider the evaluation design or evaluation instrument.
NEEDS REVISION	Misuse of evaluation studies; policies made on assumptions of the study, or misinterpretation of the data or conclusion(s).