Our goal throughout Peopling the American Past—a Teaching American History (TAH) project partnership of seven small and rural Virginia school districts and George Mason University—was to support teachers in developing TAH skills, especially in teaching effectively with primary sources. To accomplish this goal with each of three teacher cohorts we provided feedback on classroom observations and on their written work.

Our intention was to keep feedback collaborative, something that was done with, not to, teachers. Our project team—project director, academic program director, and master teachers—was interested in fostering growth, not evaluating. Whether speaking in person with a teacher after observing her class, emailing teachers our thoughts on their lesson topics, or conducting extensive phone conferences on unit outlines, we framed our feedback so that teachers would want to use it to improve their work. This meant that we had to listen carefully to our teachers, notice if our goal was being achieved, and change our approach when necessary.

In our first year we required teachers to create an entire lesson plan that we then observed them teach whenever it fit into their curricula. We naively asked them to do it all: integrate TAH content knowledge, select high-quality online primary sources, use engaging teaching strategies, apply the Virginia Standards of Learning, and follow a prescribed template. And we sought to assess all this in our feedback. This approach had problems. While some taught their lessons in the first two months of school, others taught theirs as late as April or May, when our observation feedback was too late to be used in writing their units. Also we noticed that with all these requirements, teachers did not focus on what we considered most important: teaching effectively with primary sources.

So for our second and third years we pared down our requirements. Instead of the entire lesson plan we asked teachers first to create primary source infusions—select one or two viable primary sources and incorporate or infuse these into an existing lesson. Each did this twice during September and October, and the project director, often joined by the academic program director, observed one class. Then we scheduled an informal conversation, first asking the teacher what she liked about the lesson and next what she might change, and we responded too. At each point we provided specifics about what we noticed. The project director provided written feedback in an email.

Here is an example from a middle school lesson on the Transcontinental Railroad:

I especially liked:
Your use of the Library of Congress website Railroad Maps, 1828—1900. This is an ambitious choice and there's a lot going on there. It was clear that you preselected some relevant maps to share with your students.

I wonder if:
You might, as we discussed, first model using one of the maps yourself as a primary source. Then you could ask students to work in pairs on a different preselected map or two and see if they could find their way around their map. The NARA website has a bunch of useful questions for examining maps, but don't feel compelled to use all of them—select and adapt.

Even veteran teachers appreciated this approach, many commenting that they rarely had colleagues or supervisors observing and discussing with them how they taught history. When we offered an optional second observation, many teachers asked us to return.

Next we asked teachers to use what they learned from observed infusions to write their unit. Instead of the first-year requirement of a seven- to ten-lesson unit, we asked for a two- to three-lesson mini-unit built on a few high-quality primary sources and a short historical background essay to contextualize these sources. Instead of waiting to give our major feedback until teachers completed their rough draft as in year one, we provided extensive feedback when it had maximum impact: right after teachers wrote their outline.

Each outline included a background essay, an annotated list of resources, and the primary sources to be used. Teachers got written feedback on their outlines from our project team, with our academic program director focusing on their historical narrative and sources, and our master teachers suggesting effective teaching strategies to meet the developmental needs of their students. The project director then scheduled individual phone conversations with teachers who were equipped with their outline, our comments, and Internet access to connect them to the websites we recommended. From here, teachers wrote their rough drafts, again received written feedback especially on their teaching strategies, completed their final drafts, and presented their mini-units to their colleagues and the project team.

This approach was definitely labor intensive and some teachers required more patience and persistence than others. Still, in the end we were fully satisfied with our revised feedback process.

One of the most unique and rewarding aspects of Teaching American History (TAH) grants is the opportunity they provide for K–12 teachers to work with college-level historians. Most classroom teachers find themselves isolated from the academic world, and tied to curricula and state standards that often take the complexity and richness out of history. TAH grants provide opportunities for teachers to enrich their historical understandings by connecting them with university professors and current research on historical topics. These connections are extremely interesting and valuable to teachers, but the content and knowledge exchanged in these interactions are not always easily translated into classroom practice. Teachers need ongoing support and structure to help bridge the gap between seminars they attend and their own classrooms.

In the Foundations of U.S. History grant in Loudoun County, VA, we have continually adapted our program to address those needs and maximize the benefits of the relationship between teachers and historians who work with the program. In our first year of the grant, we provided a two-week summer course, with 12 different historians presenting half-day seminars on their areas of expertise. We also included two separate workshops on teaching strategies. During the year, we followed up with our teachers by providing additional workshops, book discussions, and film seminars with various historians, as well as assignments that required teachers to incorporate primary sources into their teaching. This "parade-of-presenters" model had certain strengths. Teachers were exposed to a wide array of historical interpretations and approaches. Like their own students, teachers have varied learning styles; the range of presentation styles from individual historians addressed some of those differences. Readings and workshop materials selected by the professors also introduced teachers to varied sources and perspectives.

In reflecting on our first year, though, we recognized some weaknesses. While teachers demonstrated improved content knowledge about the topics we covered in the institute, they were still missing some of the big picture ideas and themes that tied those topics together. With the "parade" approach, teachers didn't have the ongoing interaction with historians who presented or the opportunity to ask questions that would help make connections with later presentations. Even though we included a session on historical thinking skills, teachers continued to have difficulty articulating what those skills are and how to develop them in their students. In classroom observations and in reading teachers' primary source activities and curriculum units, we noticed that many still struggled with integrating primary source analysis and historical context in thoughtful and constructive ways. We realized that we hadn't provided enough opportunities to fuse content and strategies. We decided we wanted to develop more consistency in the program and provide more of an opportunity for teachers to benefit from a closer collaboration with historians.

For the second year of our grant project, we created a Lead Historian role, which was filled by a George Mason University professor, Dr. Christopher Hamner. Instead of two new speakers each day of our Summer Institute, Hamner gave several lectures, interspersed with a handful of guest professors. With this model, Hamner was able to build connections and trace themes among the lectures and historical periods. We also identified key components of historical thinking that Hamner emphasized in his presentations, so that teachers would have an opportunity to practice and reinforce these skills before trying to teach them to their own students. Hamner modeled the use of primary sources and helped teachers see how they could use a few sources to get students to think about historical issues in a critical way, a way that both addressed and went beyond the standards for which teachers are responsible.

The teachers connected with Hamner not just because his lectures on historical topics were engaging, but because he shared his own experiences and thoughts on teaching. He described techniques he's used with his own students, and he modeled for them reflective teaching practices in examining the results of using various strategies with his students. He made it clear that the challenges he faces with graduate and undergraduate students are not unlike the challenges experienced by K–12 teachers. In those two weeks, the teachers developed a relationship with an academic historian. They learned new content, and they gained confidence in their ability to teach history more effectively based on his model. By inviting several guest presenters, we still maintained some of the valuable benefits of bringing multiple perspectives and interpretations to the discussion.

Subsequent to the Summer Institute experience, Hamner continued his involvement with the teachers in the program. During the year, he led three book discussions, and in those discussions continued to emphasize some of the core themes and ideas from the summer. As teachers developed lesson plans and primary source activities, he provided feedback and helped teachers shape their historical understandings and teaching strategies.

Evidence from our observations and evaluation measurements indicates that the lead historian model has had a positive impact on teachers' knowledge and practices. In an assessment at the end of the Summer Institute, teachers demonstrated significant improvement in content knowledge and historical thinking skills as compared to their performance on a pre-assessment. Through classroom observations and reading teachers' reflections, I've had the opportunity to see many of the skills being put to use. Teachers are selecting high-quality primary sources, they are asking more thoughtful analysis questions, and they are helping students focus on big ideas and themes, rather than just lists of details and "essential information" required by their standards.

In addition to the measurable gains in knowledge and skills, we've noticed a less tangible, but equally impressive, increase in teachers' enthusiasm and confidence as their relationship with Hamner evolved. While part of this can be attributed to his personality, I believe this is a replicable model. Teachers benefit from this kind of investment, interest, and feedback from a college-level professor. It adds to a sense of feeling valued for what they do, and it encourages them to aspire to a certain type of thinking and teaching.

The benefits are also reciprocal; Hamner has expressed that his work in the Summer Institute was the most rewarding teaching he's done. It required him to reflect on and evaluate his own teaching practices, and he also benefited from the suggestions and strategies presented by teachers. He's used his experiences with TAH to reshape his own college-level teaching over the past semester.

As we prepare to launch our third cohort, we're still improving the program to build on the successes of the lead historian model. Throughout the Summer Institute, we've built in times to discuss teaching strategies in connection with the content that is being taught, instead of in isolated seminars. We've used Hamner's input to develop new assignments that focus more closely on historical research and primary analysis. We have added additional opportunities for teachers to meet individually with Hamner to discuss their culminating projects throughout the year, and believe that these discussions are important opportunities for teachers to develop their thinking on historical topics. We also plan to include Hamner in some classroom observations. In addition to providing feedback to teachers, this will provide him an opportunity to be more closely connected with the realities of teachers' classrooms.

Our model demonstrates some of the benefits of collaboration, as well as some of the challenges that continue to face us all as history educators. Changing teaching practices takes time and continued support. Sitting and listening to a professor's lecture can be interesting and engaging, but that rarely translates directly into substantial change in K–12 classrooms. It is not an automatic process. Teachers need the opportunity to revisit and reinforce the skills and content they have learned. They need feedback and support as they incorporate primary sources and develop analysis tools. This requires a sustained commitment from historians and teachers. The resources provided by the TAH grant help to support that relationship and help make possible real change in both K–12 and post-secondary history education.

What should Teaching American History (TAH) evaluation programs evaluate? Of course, the most obvious answer would be that they should evaluate the success of the programs. But what constitutes success? This is a much more challenging question.

Our team of researchers at the University of Maryland has been conducting evaluations of TAH programs since first-round grants were vetted. This fall we begin evaluating the fifth of these programs here in Maryland. As we began our evaluation work, we conceptualized the question of measuring success around trying to understand knowledge growth among the history teachers who participated in these programs. After all, it seemed to us, that's what these programs were fundamentally designed to do—enhance the knowledge of participants in order to better prepare them to teach history. Here again, we encountered tough questions: What does it mean to enhance teachers' historical knowledge? What do we mean when we say knowledge? And how do you measure gains?

Drawing from a growing body of research in history education, we conceptualized that knowledge as of three tightly interwoven types: (a) foreground substantive knowledge, (b) background substantive knowledge, and (c) procedural or strategic knowledge.

We defined foreground substantive knowledge as ideas and understandings of what happened in the American past, engaged in by whom, for what reasons, and to what end results. This form of knowledge is what we typically read about in American history books—accounts of what happened and what they meant. Background substantive knowledge turns on ideas historical investigators impose on an unruly, broadly temporalized past in order to corral its unwieldy nature and give it some meaning useful to readers. Ideas such as historical significance, causation, change over time, chronological sweep, evidence, and historical contextualization make up concepts of the background type. Procedural or strategic knowledge involves using background concepts together with cognitive processes in order to arrive at foreground substantive understandings. Being able to ask historical questions, to seek out and assess sources as evidence for making claims, to know how to evaluate the validity and reliability of sources, and to build interpretations require strategic knowledge.

To assess change in teachers' knowledge of the three types, we created a complex instrument that we could use in a time-series design. This meant that we could administer the instrument before teachers began the TAH program and again after they had completed it, or at various intervals along the way to the end of the three-year funding cycle. This allowed us to measure baseline knowledge against changes brought about by the program's intervention elements. It also allowed us to ask TAH program directors to solicit comparison group teachers to take the assessment so that we could compare scores between participants and nonparticipants in a quasi-experimental design. This has proven workable and productive, although it sometimes has been difficult to get comparison group teachers to return to take the assessment a second time.

Our most significant challenge involved figuring out what sort of items to create to measure these differing types of knowledge. The assessment needed to be relatively efficient to administer, repeatable without practice effects, reasonably reliable, and high in construct validity. We settled on a rather heavy reliance of multiple, forced-choice items in each of the knowledge types. However, because history is an ill-structured knowledge domain (meaning that problems worth studying can be defined in multiple ways with varying interpretive results), we turned the multiple-choice items effectively upside down. By this I mean that, instead of positing only one correct answer to the items, we offered three possibilities with only one distractor of the four being patently incorrect.

With considerable effort, we structured the three remaining acceptable distractors into a descending order from most-to-least acceptable and weighted them. This structure has allowed us to disaggregate item scores to show the direction of movement in teachers' responses (towards stronger or weaker knowledge) and to map the multiply-interpretive, ill-structured nature of history domain knowledge onto the items themselves.

To augment these items, we constructed a DBQ-style essay we ask teachers to write. We purposely chose events about which a variety of interpretations are possible based on conflicting testimony provided in the four documents teachers read and on the basis of which they are asked to craft their responses. We score these essays using a complex 21-point rubric that has five key categories (e.g., contextualizes interpretation, assesses the status of sources used). This single essay, we have found, is the most knowledge-sensitive element of the assessment and correlates highly with the three types of knowledge the multiple-choice items measure.

We also borrowed from the research literature in educational psychology to design two additional scales that we include in the instrument—interest and epistemological stance. We know from the research literature that if an intervention program does not elicit interest from participants, their knowledge is unlikely to change. We also know from a different research literature that to think historically in ways that enable deeper historical understandings, teachers need to conceptualize history as an interpretive domain, ill-structured in its problem spaces, and prone to regular revision.

To understand history as such, those who investigate it and apply its forms of knowledge need to work from a set of criteria for what counts in making sense of the past. Assuming that history falls from the sky, authorless and ready-made, tends to cognitively handcuff teachers, especially when facing conflicting testimonies from the past. The epistemology scale attempts to measure changes in teachers' understandings of the bases and warrants for historical knowledge and correlates them with other items on the assessment.

This instrument—called the HKTA for Historical Knowledge and Teaching Assessment—produces a rich array of powerful data. It sheds considerable light on what teacher participants know, can do with what they know, and how their ideas change (or not) across the programs' durations. Most importantly, results provide project partners with feedback on the strengths and weaknesses of the interventions and ways they can go about making changes as the programs evolve in growing participants' knowledge of American history.

We have learned many things from using this assessment tool. Because of its complexity and number of scales, we have struggled to keep its length reasonable so it can be administered in a relatively short timeframe. We have found that after about an hour's duration, teachers begin to tire (although generally they take the assessment in good spirit and sometimes seek out their personal scores which we release only to them on individual request). Given the richness of ideas and constructs we are trying to sample—so as to provide sound feedback to project partners—this creates tradeoffs for us that we have had to manage carefully. Rich data collection has to be weighed against economies of efficiency in assessment administration time.

The epistemology scale has created additional concerns. The items presented in Likert-scale format have a tendency to be prone to social-desirability item-selection effects. To date we have been reluctant to release this scale's outcomes because we are still sorting out how validly it measures epistemological stances among teachers.

The most important learning aspect of administering the assessment has come when we report out data to project partners. As I noted, the HKTA exposes both strengths AND weaknesses in the TAH programs. Though this is as intended, we have found it frequently difficult to communicate weaknesses to partners who invest much energy in producing powerful programs.

The struggle here often turns on helping historians, who operate as content experts, to understand what the assessment tells us about what it means to transfer that content into history lessons for pre-collegiate students. This is a language historians are understandably least familiar with. In particular, the assessment reveals gaps between the efforts of the historians and that of the pedagogy experts assigned to the projects. Such gaps can be delicate observations to convey. It has helped that the various scales on the HKTA generate data useful for the purpose of strengthening these connections.

"These documents are too hard for my students."
"But this topic is never on the Regents."
"My Assistant Principal doesn’t want me to spend time on that"

When we began our Teaching American History (TAH) grant, our effort to immerse participants in recent social history was often met with friendly, but firm resistance. Indeed, the refrains we heard from middle and high school teachers in New York City echoed the very gap between university-based historians and K-12 history educators that our TAH program was designed to bridge.

While most teachers expressed personal interest in our lectures, readings, and primary documents, they were uncertain about the utility of such content for their students. During our first summer institute, when teachers were asked to create curriculum, many retreated into familiar territory, developing activities far afield from what we had covered at the content-rich school year seminars.

While eager to foster the intellectual growth of participants, changing what happens in classrooms is where the rubber hits the road. In this case, the rubber represented nearly a million dollars in federal funding and huge investments in time and energy from professional developers, school administrators, and teachers.

With generous input from our program evaluator Dr. William J. Tally and our colleague Frank Poje, a veteran teacher, we set out to close the gap between what teachers learned in our professional development and what they taught their students. Over the past several years, we’ve learned many lessons, but two stand out:

First, articulate clear historical understandings—big ideas and questions about U.S. history—that will engage both you and teachers over the course of three years. Good historical understandings help us think like historians and make connections among past events and between past and present.

Second, provide a variety of compelling documents that are accessible to students. A good mix of documents can reveal a range of viewpoints, present different types of evidence, or pose a historical problem to be investigated.

Previously, we had organized seminars around a loosely-defined theme (e.g. foreign policy) and set of related topics (e.g. Westward Expansion, 1890s Imperialism, World War II), yet struggled to choose from the volume of available materials. As history educators Nikki Mandell and Bobbie Malone have observed, "Knowing that a class must study the American Civil War or Ancient Rome does not, by itself, tell us what students should know about that topic."

We decided to reshape our foreign policy year to focus more narrowly on “America Goes to War.” Specifically, we wanted to examine:

• the ideas that have led the nation into war and have shaped how the U.S. has fought its wars
• the perspectives of other nations and people that change and enhance our understanding of war
• the impact of war abroad on U.S. society and politics at home

During our World War II seminar, for example, we explored the 1st and 3rd of our historical understandings by juxtaposing official articulations of why Americans needed to mobilize for war with the actual experiences of soldiers and homefront workers. Readings on Roosevelt’s "Four Freedoms" speech, along with posters and newsreel footage, conveyed the ideals the U.S. government communicated to motivate support for the war: the need for racial equality and unity; the emphasis on femininity among women who took industrial jobs; the value of democracy and freedom for people around the globe.

In contrast, Dr. Roscoe Brown, a veteran of the famed African American army air unit the Tuskegee Airmen, gave a compelling talk about his experiences in the segregated military, while the documentary Rosie the Riveter Revisited conveyed the voices of “real life Rosies” whose lives bore little resemblance to government propaganda messages about women performing war work.

When it came time for teachers to create activities during the summer, we equipped them with a set of primary documents designed to support our historical understandings. Already armed with the historical understanding about the impact of war abroad on U.S. society at home, teachers analyzed:

• a photograph of black and white workers leaving a Beaumont, Texas, shipyard in 1943
• "Beaumont to Detroit: 1943,"a poem by Langston Hughes decrying wartime violence against African Americans
• a "United We Win" government poster promoting interracial cooperation on the job
• a Pittsburgh Courier editorial explaining their "Double V" campaign for victory over fascism abroad and victory over racism at home

Embracing these documents, teachers designed activities through which their students can explore gaps between idealized propaganda messages and the lived experiences of diverse Americans, and the ways that support for the war effort became linked to African American demands for full equality.

As one participant shared, "I am more determined than ever to make the time to use documents and aim for a richer experience for my students. Simply covering the material may get you to the end but does not usually yield much understanding or generate much passion from anyone involved."

Nikki Mandell and Bobbie Malone, Thinking Like a Historian: Rethinking History Instruction (2007), 24.
Rosie the Riveter Revisited (Clarity Films, 1984)
John Vachon, "Workers leaving Pennsylvania shipyards, Beaumont, Texas," photograph, June 1943, Farm Security Administration, Office of War Information Collection, Library of Congress.
Langston Hughes, "Beaumont to Detroit: 1943," from The Collected Poems of Langston Hughes (Alfred A. Knopf, Inc., 1994).
Alexander Liberman (photographer), "United We Win," poster (Washington, D.C.: Government Printing Office for the War Manpower Commission, 1943); from National Archives, “Powers of Persuasion: Poster Art from World War II"
"The Double V Campaign," Pittsburgh Courier, 28 March 1942, 6.