In the library of an Oakland, CA, middle school, four 8th-grade American history teachers are gathered around a table. A doctoral student in U.S. History, the school librarian, and two staff members from the Oakland Unified Teaching American History (TAH) Grant's professional development project join them. The teachers have come from three different schools to observe a lesson on the Fourth Amendment they have planned together.

They wanted students to understand the ideas, rights, and controversy embedded in the dry language of the Constitution. The lesson began in a dramatic fashion. The teacher who was teaching the lesson arranged for a campus security guard to walk into the classroom and search the backpacks of three students. The students had agreed before class to participate in the simulation. After the search, students in the class were asked to write a brief response explaining whether they thought the search was legal. A discussion of this question followed. Then the students read and tried to rewrite the Fourth Amendment in their own words.

Reading and understanding the Amendment proved, as the teachers anticipated, a challenge to many of the students in this class, which included a number of second-language students. At one point the teacher asked, "What do you think they mean by the term effects?" As the teachers had predicted the students had difficulty in explaining how the term was used in this context.

After this introduction, the teacher passed out the Supreme Court case, T.L.O. v. New Jersey (1985), which asked what rights students have against search and seizure if they are on school grounds. (The court ruled they don't have the same rights as individuals outside the authority of the school.) Finally, students were asked to revise what they had written at the beginning of the period on whether or not the search was legal.

Initially, the group of teachers clustered around the table was certain that the lesson was successful; students seemed to understand that there were limitations to their Fourth Amendment right to not be searched. Then, the student writing samples were passed out to each teacher. After reading what students wrote the mood at the table changed, for it became clear that the students had gained, at best, only a limited understanding of the Fourth Amendment and how it is applied in a variety of situations and contexts. They did not see how the Fourth Amendment had been applied to the T.L.O. case and argued that school officials had no right to search their belongings without a warrant or probable cause.

As this finding emerged, teachers began to reconsider the design of their lesson—what would they do differently next time? When the lesson was taught again, it benefited from this close examination of instruction and student learning. Indeed, to help the students better understand how the Fourth Amendment had been applied, the teachers refined the lesson to provide a greater focus on the actual court ruling and reasoning in T.L.O., as well as looking closely at additional significant Fourth Amendment cases in American history.

This brief example of teacher collaboration illustrates one aspect of the Oakland TAH program. This collaborative process is known as lesson study.

The Oakland Unified School District has received two TAH grants. Each project had the goal of increasing teacher content knowledge of American history and connecting classroom teachers and professional historians. But, working to increase teacher content knowledge of American history was just a first step towards achieving the main goal of the project—increasing student knowledge, understanding, and achievement in American history. This goal raises the fundamental question we sought to answer in our projects:

"How can the enhanced historical knowledge gained by teachers find its way into their lessons and thus increase student knowledge and understanding of American history?"

From long experience in professional development, it is clear the most challenging aspect of this work is helping teachers make connections between what is learned in a workshop and what happens in their classrooms. Lesson Study helped us meet our goal of strengthening that connection.

As described above, "Lesson Study" provides an ongoing method to examine, refine, and improve instruction. The process is quite basic. A group of 3–5 teachers meet to plan a lesson on a specific historical topic and identify what important information, ideas, and concepts they want the students to understand. The lesson is then taught in one group member's classroom, while the other teachers and project staff observe. After the observation the group members and project staff meet to analyze, with a focus on student talk and writing and on how successful the lesson was in achieving the instructional goals they set for themselves. Based on this discussion, the teachers then refine and/or revise the lesson before it is taught in another group member's classroom.

It is important to note two very important details about what is accomplished in the planning phase:

1. The teachers develop a student question for the lesson. The student question guides the selection of materials and activities that will help students develop thoughtful and accurate responses. It also identifies what student words and work will be the focus of the teachers' analysis of the lesson's effectiveness.

2. The teachers also develop a research question for themselves about the teaching and learning of American history. This both focuses the lesson planning and the gathering of data, and gives the lesson importance beyond the immediacy of its topic and teaching.

Below are some examples of how past lesson study groups connected their student and teacher questions.

A fifth grade lesson focused on Chinese immigration through family photos:

Student question: What can we learn from this picture about the experiences of Chinese immigrants?
Teacher question: Can students use an immigration story to understand a larger historical movement? (A focus on making generalizations and inferential thinking.)

A fifth grade lesson focused on the experiences of slaves and questions of freedom. The class had read the historical novel A Picture of Freedom: The Diary of Clotee, a Slave Girl, by Patricia C. McKissack:

Student question: Which characters in the book have the most and least amount of freedom?
Teacher question: Can students develop a nuanced understanding, through multiple perspectives, of freedom at this time and place in American history?

An eighth grade lesson on Nat Turner:

Student question: Was Nat Turner's Revolt a success?
Teacher question: How can we help students understand that it is possible to tell different stories and come to different conclusions about the same event?

An 11th grade lesson on Populism:

Student question: How successful was the Populist Party?
Teacher Question: How can we teach students to use evidence to support their argument?

We found that Lesson Study, both through its promise and its implementation addressed a genuine need among history teachers for a systematic way of learning about how to improve instruction, but it was not without its challenges. Lesson study takes time, a scarce resource for teachers. It requires meeting after school and finding and locating resources for a lesson. It requires an understanding among members that by investigating a lesson they might come to different answers and understandings about how best to increase student knowledge and understanding. Indeed, a number of lesson study groups were not able to overcome these challenges. Some teachers showed up late to meetings, or didn't show up at all. Some teachers planned extensively while others in the group did not contribute an equal share. Also, lesson study requires a stance towards teaching and collaboration that is often at odds with how teachers work together at school sites. A successful lesson study develops a lesson that is seen by group members as "our" lesson, rather than the lesson of the teacher who is going to teach it first.

So why would teachers want to continue with lesson study? The answer can be found in what teachers believed to be beneficial besides the opportunity to collaborate. Not only did lesson study address a need, but it helped meet the need. Over three quarters reported that they actually learned something new about their teaching—something that was revealed to them through the lesson study process. There was the learning of new content as lessons were developed and materials selected. Indeed, a number of groups chose to focus their lessons on topics they had not taught in depth before, such as Nat Turner and slave rebellions, McCarthyism and the Cold War, or the Fourth Amendment in American history.

There were also new specific instructional strategies designed to help students learn and understand more about American history. For example, groups focused on how to help students read difficult primary source documents, move from specific historical details to generalizations about a time or place, or see a specific event through the multiple perspectives of the time. And there was new learning around the big instructional questions teachers framed for themselves. "Can fifth grade students develop a nuanced understanding of freedom and slavery?" "How to help students use historical evidence to develop and support an historical argument?"

To support lesson study through TAH activities, our efforts have focused on linking lesson studies with historians' presentations. This allows us to provide resources (documents, activities, readings, etc.) that support teachers through the lesson planning phase so they can focus on the lesson analysis portion. The analysis portion, framed by a teacher research question, is often the part of teaching that teachers are unable to make time for as they try and meet the demands of moving through their American history curriculum.

When we first started with lesson study we didn't stress this aspect enough, even though we understood the research nature of the process. We learned that having a teacher question helped immensely in focusing on student understanding as a means for evaluating the lesson's success and instructional meaning. For lesson study to be truly successful, it should help teachers improve the instruction of a particular lesson, inform their instruction beyond that one lesson, and influence future instructional decisions and choices. Why else spend that much time on one lesson?

Lewis, Catherine, "Lesson Study: A Handbook for Teacher-Led Instructional Change," (Research for Better Schools, 2002).

Stigler, James and Hiebert, James, "The Teaching Gap: Best Ideas from the World's Teachers for Improving Education in the Classroom," (New York: Free Press, 1999).

Oakland Unified School District TAH project website.

What should Teaching American History (TAH) evaluation programs evaluate? Of course, the most obvious answer would be that they should evaluate the success of the programs. But what constitutes success? This is a much more challenging question.

Our team of researchers at the University of Maryland has been conducting evaluations of TAH programs since first-round grants were vetted. This fall we begin evaluating the fifth of these programs here in Maryland. As we began our evaluation work, we conceptualized the question of measuring success around trying to understand knowledge growth among the history teachers who participated in these programs. After all, it seemed to us, that's what these programs were fundamentally designed to do—enhance the knowledge of participants in order to better prepare them to teach history. Here again, we encountered tough questions: What does it mean to enhance teachers' historical knowledge? What do we mean when we say knowledge? And how do you measure gains?

Drawing from a growing body of research in history education, we conceptualized that knowledge as of three tightly interwoven types: (a) foreground substantive knowledge, (b) background substantive knowledge, and (c) procedural or strategic knowledge.

We defined foreground substantive knowledge as ideas and understandings of what happened in the American past, engaged in by whom, for what reasons, and to what end results. This form of knowledge is what we typically read about in American history books—accounts of what happened and what they meant. Background substantive knowledge turns on ideas historical investigators impose on an unruly, broadly temporalized past in order to corral its unwieldy nature and give it some meaning useful to readers. Ideas such as historical significance, causation, change over time, chronological sweep, evidence, and historical contextualization make up concepts of the background type. Procedural or strategic knowledge involves using background concepts together with cognitive processes in order to arrive at foreground substantive understandings. Being able to ask historical questions, to seek out and assess sources as evidence for making claims, to know how to evaluate the validity and reliability of sources, and to build interpretations require strategic knowledge.

To assess change in teachers' knowledge of the three types, we created a complex instrument that we could use in a time-series design. This meant that we could administer the instrument before teachers began the TAH program and again after they had completed it, or at various intervals along the way to the end of the three-year funding cycle. This allowed us to measure baseline knowledge against changes brought about by the program's intervention elements. It also allowed us to ask TAH program directors to solicit comparison group teachers to take the assessment so that we could compare scores between participants and nonparticipants in a quasi-experimental design. This has proven workable and productive, although it sometimes has been difficult to get comparison group teachers to return to take the assessment a second time.

Our most significant challenge involved figuring out what sort of items to create to measure these differing types of knowledge. The assessment needed to be relatively efficient to administer, repeatable without practice effects, reasonably reliable, and high in construct validity. We settled on a rather heavy reliance of multiple, forced-choice items in each of the knowledge types. However, because history is an ill-structured knowledge domain (meaning that problems worth studying can be defined in multiple ways with varying interpretive results), we turned the multiple-choice items effectively upside down. By this I mean that, instead of positing only one correct answer to the items, we offered three possibilities with only one distractor of the four being patently incorrect.

With considerable effort, we structured the three remaining acceptable distractors into a descending order from most-to-least acceptable and weighted them. This structure has allowed us to disaggregate item scores to show the direction of movement in teachers' responses (towards stronger or weaker knowledge) and to map the multiply-interpretive, ill-structured nature of history domain knowledge onto the items themselves.

To augment these items, we constructed a DBQ-style essay we ask teachers to write. We purposely chose events about which a variety of interpretations are possible based on conflicting testimony provided in the four documents teachers read and on the basis of which they are asked to craft their responses. We score these essays using a complex 21-point rubric that has five key categories (e.g., contextualizes interpretation, assesses the status of sources used). This single essay, we have found, is the most knowledge-sensitive element of the assessment and correlates highly with the three types of knowledge the multiple-choice items measure.

We also borrowed from the research literature in educational psychology to design two additional scales that we include in the instrument—interest and epistemological stance. We know from the research literature that if an intervention program does not elicit interest from participants, their knowledge is unlikely to change. We also know from a different research literature that to think historically in ways that enable deeper historical understandings, teachers need to conceptualize history as an interpretive domain, ill-structured in its problem spaces, and prone to regular revision.

To understand history as such, those who investigate it and apply its forms of knowledge need to work from a set of criteria for what counts in making sense of the past. Assuming that history falls from the sky, authorless and ready-made, tends to cognitively handcuff teachers, especially when facing conflicting testimonies from the past. The epistemology scale attempts to measure changes in teachers' understandings of the bases and warrants for historical knowledge and correlates them with other items on the assessment.

This instrument—called the HKTA for Historical Knowledge and Teaching Assessment—produces a rich array of powerful data. It sheds considerable light on what teacher participants know, can do with what they know, and how their ideas change (or not) across the programs' durations. Most importantly, results provide project partners with feedback on the strengths and weaknesses of the interventions and ways they can go about making changes as the programs evolve in growing participants' knowledge of American history.

We have learned many things from using this assessment tool. Because of its complexity and number of scales, we have struggled to keep its length reasonable so it can be administered in a relatively short timeframe. We have found that after about an hour's duration, teachers begin to tire (although generally they take the assessment in good spirit and sometimes seek out their personal scores which we release only to them on individual request). Given the richness of ideas and constructs we are trying to sample—so as to provide sound feedback to project partners—this creates tradeoffs for us that we have had to manage carefully. Rich data collection has to be weighed against economies of efficiency in assessment administration time.

The epistemology scale has created additional concerns. The items presented in Likert-scale format have a tendency to be prone to social-desirability item-selection effects. To date we have been reluctant to release this scale's outcomes because we are still sorting out how validly it measures epistemological stances among teachers.

The most important learning aspect of administering the assessment has come when we report out data to project partners. As I noted, the HKTA exposes both strengths AND weaknesses in the TAH programs. Though this is as intended, we have found it frequently difficult to communicate weaknesses to partners who invest much energy in producing powerful programs.

The struggle here often turns on helping historians, who operate as content experts, to understand what the assessment tells us about what it means to transfer that content into history lessons for pre-collegiate students. This is a language historians are understandably least familiar with. In particular, the assessment reveals gaps between the efforts of the historians and that of the pedagogy experts assigned to the projects. Such gaps can be delicate observations to convey. It has helped that the various scales on the HKTA generate data useful for the purpose of strengthening these connections.