Evaluating Change in Student and Teacher History Content Knowledge
The Teaching American History grant program (TAH) was established primarily to raise student achievement by improving teachers' knowledge and understanding of and appreciation for traditional U.S. history. Rigorous evaluation is required to determine the extent to which teacher knowledge and understanding of and appreciation for U.S. history has increased and to correlate this increase with increased student performance as a result of TAH program efforts.
The American Institute for History Education (AIHE) has worked with over 60 school districts nationwide to tailor TAH professional development training to local needs while meeting substantive history standards. AIHE's 35 Liberty Fellowship programs blend in-depth history content, including lectures by top university professors, history education specialists' viewpoints, historical research projects, and on-site field-study tours, as well as diverse digital and traditional resources and proven classroom practices.
Evaluation Solutions of Grant Evaluation, Inc. (ES/GEI) has designed and carried out evaluations for numerous AIHE (and other) grants, which provide an opportunity to examine the impact of the TAH program on a large number of students and teachers. Evaluation findings clearly indicate that the Liberty Fellowship program is succeeding in raising teacher knowledge, understanding, and appreciation of U.S. history, and preliminary evidence indicates it is bringing out a corresponding increase in student history knowledge and appreciation levels.
Five approaches are taken to evaluating changes in teacher knowledge, understanding, and appreciation.
1. Teacher perceptions and practices are evaluated through surveys—one given annually and another after each TAH session. The annual baseline survey records teacher classroom practices and perceived student engagement levels. The surveys record teacher perceptions on whether TAH sessions increased teacher content knowledge, along with their integration of new material, and their classes' use of new materials or places (field studies). Statements indicating increased knowledge and use of new material were verified with objective testing. On the baseline test several questions were compiled to indicate overall appreciation, interest, and knowledge levels and first-year data was compared with final-year data.
Results: AIHE colloquia and summer institutes increased teacher depth of understanding of the topics covered and provided skills or knowledge that teachers could apply to other areas of history or teaching, according to 95% of the 503 teacher responses.
2. Teachers are given standard questions related to the periods being studied, and the standard questions and the overall test grade are compared through pre- and post-tests—before and after the year's TAH sessions. Standard questions are drawn from released national test items (NAEP, SAT, State Exams, AP, national polls and surveys) and from AIHE professors and sessions.
Results: On the national standard questions teacher knowledge gain averaged 29%.
Challenges and Lessons Learned
There are no real national uniform assessments or standards for teacher or student U.S. history knowledge, understanding, and appreciation, although existing exams can provide useful material for questions. Generating a test using national standard questions requires aligning the history content of the questions with the professional development content and state and local standards. Selection of national, rather than local, questions tests broader historical themes and ideas—as the specific facts covered in professional development sessions vary widely from professor to professor, even when lecturing on the same era or content.
A major issue is now also being factored in. Teacher participants indicate that TAH sessions encourage a shift in historical thinking skills, indicated by a stronger ability to correctly respond to certain types of questions—such as comparing and contrasting, or identifying alternate historical interpretations. National questions that involve these skills can help establish TAH impact on both content familiarity and historical thinking skills.
Each teacher test also collects information on college history course background and on grade taught, as these factors have been found to correspond with trends in teacher performance on content tests.
A main limitation is that the tests generally must be abbreviated to fit into a short time slot and more in-depth essays are difficult to administer. A solution to this may be in year three to schedule a full hour on the TAH program agenda and offer incentives (rewards) for teacher assessment participation.
3. Teachers' standard test and question scores after exposure to the TAH program are compared with a match group of teachers who were not exposed to the TAH program.
Results: In a preliminary study, TAH teachers scored 15% higher than match groups of teachers not in the program.
Challenges and Lessons Learned
The match group above is based on a limited sample and match group recruitment is one of the major challenges in obtaining suitable match group participants. Most of the project area history teachers are in the TAH program, leaving a limited number to match from the same region. In addition, teacher non-participation in TAH may correlate with factors that also impact performance such as less-motivated teachers, or overburdened teachers, or perhaps teachers who have doctorates in history and do not see a need for TAH participation.
Several tactics have been employed to recruit match group participants. TAH participants who recruit a match participant were offered a place in a high-odds incentive drawing together with the recruit. Recruits could be from any school district. Key matching criteria include teacher history education, experience levels, grade taught, professional development history, and certification levels.
The advantage of evaluating more than one TAH program leads to the potential of using baseline testing of new first-year participants as matches for other groups after exposure. This matches teachers with similar interest and motivation to be in the program.
Match teachers also can be targeted, although it generally requires incentives or personal contacts to obtain cooperation. Teachers are offered access to their results if desired, although teacher responses are kept strictly confidential from others—even TAH project directors and school administrators.
4. Teachers are tested for content knowledge before and after TAH sessions with questions related to each professor's lecture or the TAH session readings and content. At times the post-test is given later in the year to assess longer-term knowledge retention.
Results: In examining dozens of content sessions involving nearly 1,000 teachers in numerous states, this year the average increase in teacher content knowledge scores as a direct result of these sessions was 44%. Most of the content questions are written by AIHE professors who teach the sessions. Increases included a 45% gain in urban areas, a 44% gain in suburban areas, and a 37% gain in more rural areas.
Challenges and Lessons Learned
It is essential that adequate time be scheduled on the agenda at each TAH session being evaluated. Lecturers are provided a question development guide and asked to provide content questions and lecture objectives. Incentives that work well to generate professor-lecturer responses are email Amazon book certificates. In addition, national standard questions related to the content area are applied for comparison.
Long- and short-term knowledge gain evaluations can be acquired by altering the time period between pre- and post-tests. If the same lecturer is used for multiple TAH groups, the pre- and post-results can be used for matching or cross checks.
Teachers are more enthusiastic if tests are anonymous and ES uses instant response clickers in place of paper tests with live PowerPoint graphic results given on post-tests.
5. Evidence of changes in teacher knowledge is examined through pre- and post- and comparison group analyses of lesson plans. ES/GEI is developing a specialized lesson plan analysis rubric focusing on state standards and historical thinking skills. A classroom observation rubric is also being developed.
Several approaches are being undertaken to measure changes in student content knowledge and several alternative methods are being planned.
1. History students are surveyed to ascertain their interest, appreciation, and knowledge, and their classroom practices. These TAH student results are compared with matched comparison groups each year.
Results: Compared with non-TAH students, the TAH students were 30% more likely to agree that history lessons engaged students more this year than last; 20% more agreed that their interest in history increased over time; and 24% more agreed that they contrasted historical times and events in class.
2. Students are given a standard test with questions related to the periods being studied. Both the standard questions and the overall test grade are compared with a match group of non-TAH students. Standard questions are drawn from released national test items (NAEP, SAT, State Exams, AP, national polls and surveys) and from AIHE professors and sessions. Questions with national result data provide further comparison data for contrast with the TAH group.
Results: Last year TAH students scored eight percent to 30% higher on the content knowledge test than match groups of students not in TAH programs. On one of the most complete quasi-experimental match group designs involving more than 2,600 students, the TAH students' test scores were 30% higher than the non-TAH comparison group.
Challenges and Lessons Learned
Generally the same issues and solutions apply to recruiting student groups as to recruiting match teachers not in a TAH program. Very similar issues also apply to developing the student test from national and custom-designed assessment items. These tests must be completed in less than one class period. Coordinating tracking and shipping of student tests also required close communication with project directors. Teachers are provided a test packet for each class with directions and a prepaid return mailer. Confidentiality is guaranteed and no individual student is identifiable.
Match classes also can be targeted, although it generally requires incentives or personal contact to obtain cooperation. Teachers are offered access to their class results if desired, although these are kept confidential from others—even TAH project directors and school administrators.
3. State test scores are being compiled and compared where appropriate.
4. Alternative performance measures, such as read/think-aloud interviews and essay and project assessments, are planned on a sampling basis to provide additional evidence.
Evaluating changes in student and teacher history knowledge, understanding, and appreciation is challenging, but the results can be rewarding. AIHE teachers were asked to identify the impact that their participation would have on their students. The projected effects went far beyond higher standardized test scores and included greater student engagement in history, gains in higher-order historical thinking skills, and stronger ability to relate history to current events. Evaluation must encompass measurement to confirm if these TAH goals are reached. So far, ES/GEI evaluations have indicated that the TAH program has greatly contributed to salient increases in student achievement.