1 Introduction

The transformation of education in university departments is a pressing issue. Research suggests that the success of a given intervention is dependent on the local context, and so change efforts should attend to the culture in which the change is embedded (Kezar, 2013; Schein, 2010). Because departments tend to have a stable culture, we argue that a department is a sensible scale to focus change efforts (Corbo et al., 2016; Ngai et al., 2020; Quan et al., 2019; Reinholz et al., 2017).

Our project cultivates department-level changes in university departments through Departmental Action Teams (DATs) (Corbo et al., 2016; Reinholz et al., 2017). DATs consist of faculty, students, and staff in a single department working on a broad-scale issue related to undergraduate education. External facilitators with expertise in education, organizational change, and facilitation support DAT members in enacting departmental change and growing as change agents. DATs have chosen to focus on a variety of issues including curricular changes (e.g., improved curricular alignment) and cultural changes (e.g., improving recruitment and retention of women and underrepresented minority students). While our project does not prescribe the specific changes DATs make, our long-term goal is that the DAT and the DAT’s department make progress toward the following six Core Principles (Quan et al., 2019):

1. Students are partners in the educational process.
2. Work focuses on achieving collective positive outcomes.
3. Data collection, analysis, and interpretation inform decision-making.
4. Collaboration among group members is enjoyable, productive, and rewarding.
5. Continuous improvement is an upheld practice.
6. Work is grounded in a commitment to equity, inclusion, and social justice.

These principles guide our facilitation of DATs and describe what we consider to be an ideal departmental culture.

The DAT Project is an NSF-funded project spanning four years. Two components of the DAT Project are running DATs (17 over two campuses) and refining the DAT model. The DAT Project team is comprised of researchers and facilitators. All authors of this paper have contributed to research and/or facilitation of DATs. One result of the DAT Project is a refined DAT model and a Theory of Change (TOC) for the DAT model. The DAT TOC articulates how our project activities (i.e., DAT formation, facilitation) lead to our long-term goal (departmental culture aligned with our Core Principles). Information about implementing DATs can be found at our website (www.dat-project.org) and in our book Facilitating Change in Higher Education: The Departmental Action Team Model (2020). Within this paper, we present a description of our process of developing our TOC for two purposes:

1. To allow others doing change work to follow similar processes for developing their own Theories of Change and
2. To give others the opportunity to apply aspects of our TOC to their own work.

Through creating opportunities for others to engage with and critique our work, we hope to improve upon our model and add nuance to our field’s understanding of institutional change.

1.1 Development of the DAT TOC

Our motivation for developing a TOC was to better understand, evaluate, and improve upon the DAT model. Similar to Malachowski et al. (this volume), our Theory of Change linked our specific project activities to our project outcomes. Articulating our assumptions about why we thought our intervention would lead to certain outcomes allowed us to critique and ultimately refine our practice (Connolly & Seymour, 2015; Weiss, 1995). Externalizing our TOC also enabled us to empirically test these assumptions and better understand the impact of our model (Anderson, 2006; Vogel, 2012). Finally, having a TOC helped us as facilitators to track and evaluate DATs’ progress.

The development of the TOC began with the entire project team identifying our long-term outcome for departments with DATs (Taplin & Rasic, 2012). Drawing on our collective experiences facilitating DATs, we then backwards-mapped the intermediate outcomes that were necessary to reach this long-term outcome. After several team meetings, a subteam formed to refine our TOC diagram and write explanatory text. These components have undergone several revisions since we initially began TOC development in 2017.

Although the first three authors spent the most time reflecting on and developing the DAT TOC, we frequently engaged with others during this process. For example, other DAT facilitators and researchers provided feedback on the modifications made to the TOC and referred to specific experiences with DATs that led them to question or confirm parts of the TOC. We also solicited feedback from several people outside of the DAT project. An organizational change expert who has developed TOCs for other projects helped us to formalize our intermediate outcomes. Versions of the DAT TOC were presented at several conferences, where change agents in discipline-based education research fields commented on the TOC’s utility and logic.

We used the DAT TOC to inform facilitation and research. The TOC was used to guide facilitator moves, and facilitators regularly reflected on which TOC outcomes the DATs were making progress towards. This served to check the content of the outcomes as well as the relationships between the outcomes. The TOC was also used as a research tool to code the meeting minutes of DATs. This process resulted in better-delineated outcomes that eliminated redundancies uncovered by coding and provided evidence that the TOC could be a valuable research tool.

1.2 The DAT TOC

To develop the DAT TOC, we completed a guided activity from The Community Builder’s Approach to Theory of Change: A Practical Guide to Theory Development (Anderson, 2006) based on “Project Superwoman,” a program to support women who are survivors of domestic violence. The Project Superwoman TOC is depicted in Figure 1. While this TOC provided a good starting point, we found that we had to modify some of its structures. Departmental change is a complex process, and the DAT TOC had to reflect that complexity.

The basic structure provided in the Project Superwoman example included these components:

• A set of outcomes, represented by the boxes in the diagram
• Relationships among the outcomes, represented by the arrow in the diagram
• Narrative text, which describes the assumptions, justifications, and indicators for the outcomes and their relationship

In the Project Superwoman example, nearly all of the outcomes referred to the same stakeholder (the women in the program) and there was one outcome identified as the Long Term Outcome (LTO). The outcomes were all related via pre-conditions; if one outcome is a pre-condition for another, then achieving the latter outcome depends on the achievement of the former outcome.

In the following sections, we define the major components of the DAT TOC so that its structure can be interpreted. We primarily focus on deviations in the basic structure from the Project Superwoman example, and we describe their rationale and relationship to our project. To illustrate these components, we refer to specific elements of the DAT TOC (Figures 2-5).

2 The Outcomes

Each box in the DAT TOC, whether standalone or embedded in another box, represents an outcome that we hope to achieve as part of our change process. The content of the outcomes was informed by literature on organizational change and high functioning teams as well as our experience working with DATs. During our revision process, we examined the outcomes for redundancy.

2.1 Defining Stages, the Long-Term Outcome (LTO), and Intermediate Outcomes

The first major deviation in the DAT TOC is that rather than constructing one TOC to encompass the entire DAT model, we instead divided the TOC into three stages. The stages describe processes before a DAT is formed (Stage 1, see Figure 2), while a DAT is running (Stage 2, see Figures 3 and 4), and after the DAT has ended (Stage 3, which is still under revision as of the writing of this manuscript). We decided to do this because each stage represents a distinct, relatively independent period in the life cycle of a DAT. We have also constructed an overview of the entire TOC to see the stages combined (Figure 5).

The DAT project’s LTO, “The department is supported by its members in making sustainable, positive, iterative changes that are aligned with the Core Principles,” is the final outcome in Stage 3 (and therefore of the TOC as a whole). The LTO underwent several revisions during our development process. For example, an earlier version was “The department has made positive and sustainable change in undergraduate education.” We changed this language to reflect our realization that our goal is not solely for change to happen in a department, but that department members continue to drive change even after a DAT has ended. We also realized that while the Core Principles are infused into our facilitation of DATs, it is also essential that alignment of department culture with the principles be an explicit piece of the final outcome.

Stages 1 and 2 of the TOC also have a final outcome (or outcomes), which we refer to as intermediate outcomes. We view these as accomplishments in their own right and also as the outcomes that are necessary for the subsequent stage to begin. For example, the Stage 1 intermediate outcome is Outcome 4, “A DAT forms in the department,” which is necessary for the DAT to proceed in Stage 2. The Stage 2 intermediate outcomes define the state that the department (Outcome 7), the DAT (Outcome 8), and the DAT’s members (Outcomes 9 and 10) need to have reached in order for departmental change to continue to happen outside the formal DAT context in Stage 3.

2.2 Defining the Stakeholders

Stakeholders play many roles in change efforts, from guiding the effort as change agents to providing feedback and support as allies. The DAT model requires the interaction of multiple stakeholders, and thus we explicitly identify which stakeholder is associated with each outcome through the use of language (the stakeholder is always the subject of sentences in the TOC diagram) and color:

• Navy = Individual department members (in Stage 1) or DAT members (in Stage 2)
• Teal = The DAT’s department or its leadership
• Light blue = The DAT as a unit
• Orange = Facilitators
• Beige = Stakeholders external to the department, such as administrators

These constitute all of the stakeholders who are essential to the DAT model. We are careful to make these distinctions because they help a user of the TOC identify the stakeholders who are relevant in their context and to clearly specify who is achieving what outcome.

In some cases, more than one stakeholder plays a role in an outcome (e.g., Outcome 1, “Facilitators, department members, and department leadership communicate about having a DAT in the department”). To indicate this, we use multiple colors to reflect the multiple stakeholders: orange to represent facilitators, navy to represent individual department members, and teal to represent departmental leadership (Figure 2).

An additional role of the intermediate outcomes is to demarcate when stakeholders enter or leave the change process. For example, the DAT as an entity enters the TOC in Outcome 4 for Stage 1, so this is the first time that an outcome is colored light blue. Similarly, there is no intermediate outcome in Stage 2 associated with the facilitators because their role has finished by the end of Stage 2.

3 The Relationships

In addition to clearly defined outcomes, the TOC defines the relationships among the outcomes and uses a variety of visual representations to distinguish them. Our current understanding of these relationships emerged and shifted during the development process, as we articulated the logic of how one outcome impacts the others based on our experience facilitating DATs and relevant literature. This logic is spelled out in the narrative text that accompanies the TOC. While we have converged on Stages 1 and 2 of the TOC, we view the relationships among outcomes as subject to revision, based on future data.

3.1 Defining Pre-Conditions and “Sufficiency”

The basic relationship between two outcomes is a pre-condition and is represented by an arrow between two outcomes. For example, Outcome 2, “There is external support for a DAT to form,” is a pre-condition for Outcome 3, “All stakeholders are ready to form a DAT,” because without external support for DATs (e.g., from administrators or funders), stakeholders will be unable to form a DAT. Thus, there is an arrow in the TOC pointing from Outcome 2 to Outcome 3.

Most of the outcomes in the TOC do not describe discrete events but rather ongoing processes (i.e., they are more like progress bars than ready lights). Thus, implicit in our TOC is the concept of “sufficiency.” For example, sufficient progress must be made on Outcome 5, “Facilitators support DAT members in creating change and developing as change agents,” before progress can be made on Outcome 6, “The DAT engages in its work,” because without the work of the facilitators, the DAT cannot proceed. Moreover, just because progress on a subsequent outcome has begun, progress on earlier outcomes need not stop. It is possible (and often necessary) to continue making progress on previous outcomes even as later outcomes are progressing. In this example, the DAT will continue working while the facilitators continue to support the DAT and its members.

3.2 Defining Co-Conditions

There were many situations in which we found the pre-condition concept to be insufficient to describe the relationship between two outcomes, because we recognized that each outcome could be a pre-condition for the other. In this situation, we refer to those outcomes as co-conditions. We describe this relationship further in the next section.

3.3 Defining Sub-Outcomes

Finally, we recognized that some outcomes could be productively divided into discrete components that were important enough to represent in the TOC in their own right. We refer to these components as sub-outcomes. The sub-outcomes that comprise an outcome are equivalent to the outcome in the sense that achieving all of the sub-outcomes is equivalent to achieving the outcome.

Because we wanted to represent both the sub-outcomes and the larger outcome in the TOC, we developed a “box within a box” structure. For example, Outcome 6D, “The DAT builds a positive relationship with the department,” has three sub-outcomes (“The DAT regularly communicates progress, outcomes, and successes to the department,” “The DAT cultivates department allies for its work,” and “The DAT seeks department input for its work”; see Figure 3). Achieving these three sub-outcomes is what we mean by achieving the outcome of a positive relationship between the DAT and the department.

There are three variations of the sub-outcome structure:

1. The sub-outcomes have a pre-conditional relationship, in which case they are connected by an arrow (for example, the sub-outcomes of Outcome 6C, “The DAT engages in a change effort”).
2. The sub-outcomes have a co-conditional relationship, in which case there is no arrow connecting them (for example, the sub-outcomes of Outcome 6D, “The DAT builds a positive relationship with the department”).
3. The sub-outcomes are not dependent on each other (although they may still interact), in which case we put them in a numbered list (for example, the sub-outcomes of Outcome 6A, “DAT members and facilitators co-create the DAT’s culture”).

Any of these can be nested or combined with any of the others. For example, Outcome 6B, “DAT members grow as change agents” (which is a sub-outcome of Outcome 6, “The DAT engages in its work”) consists of two sub-outcomes related to DAT members increasing capacities relevant to change and DAT members increasing feelings related to change. These sub-outcomes are co-conditions: increasing their capacity helps DAT members feel more like change agents, while feeling more like a change agent will motivate them to increase their capacities. Each of these sub-outcomes consists of four sub-outcomes which are neither pre-conditions nor co-conditions of each other. For example, there are four capacities that DAT members can build, and increasing each is an outcome in its own right. However, none of the capacities are required for the growth of the others (although they may impact each other).

3.4 Defining Cycles

Because relationships in a TOC focus on which outcomes are necessary for other outcomes to proceed, it is not possible to create closed cycles in a TOC using just pre/co-conditional relationships. However, the work that DATs do to accomplish their goals for departmental change is cyclic and iterative. To address this issue, we developed a DAT change cycle (Figure 4) that visually represents the temporal cycle that DATs engage in to create change. This cycle is embedded in the sub-outcome of Outcome 6C, “The DAT engages in change cycles for outcomes at multiple scales.” We do not define the number of cycles the DAT needs to go through before this outcome has been sufficiently achieved to move to the next outcome because that will be highly dependent on the context of the department.

4 Implications

How the TOC is enacted depends on the departmental context. For example, what counts as sufficient for one department may be insufficient for another. In that sense, the TOC is providing an abstract, idealized view of the conditions that must be met for a DAT to succeed, but what those outcomes look like in practice and how they are achieved will vary given the departmental context.

Our Theory of Change has implications beyond the DAT model. The DAT TOC demonstrates the complexity of departmental change efforts, especially those that intend to enact change equitably and sustainably. Change efforts are enacted by people who are embedded within communities, so it is important to pay attention to people’s capacities within the local context. It is not enough to just focus on enacting change; groups should also work on building relationships with the department, growing individuals’ ability to make change, and functioning well as a team. The interconnectedness of these components illustrates why complex change can sometimes be slow. Change is ultimately an iterative process that involves many moving parts.

Our TOC can be a helpful tool for researchers and facilitators who are interested in cultivating change. By including multiple components and inventing ways of describing relationships between components, our model presents a more nuanced way of describing change than most TOCs. For those engaged in change efforts already in progress, we hope our TOC draws their attention to aspects of change they may not have considered but could address. For those who have already engaged in change efforts, we hope our tool can help them analyze the successes and challenges of their work. It is our intention that this TOC can help the field move forward with a deeper understanding of how to enact change in higher education.

5 Acknowledgements

There are a number of people whose contributions made this work possible. We would like to thank Stephanie Chasteen, Noah Finkelstein, Andrea Beach and members of DATs for their feedback. We would also like to thank Oath Agency for their design work on the TOC. This material is based upon work supported by the National Science Foundation under Grant No. 1626565.

6 About the Authors

Courtney Ngai is a Research Scientist in the Institute for Learning and Teaching (TILT) at Colorado State University.

Joel C. Corbo is a Senior Research Associate in the Center for STEM Learning at the University of Colorado Boulder.

Gina M. Quan is an Assistant Professor in the Physics & Astronomy Department at San José State University.

Karen Falkenberg is the Director of Teaching Excellence and a Team Development Specialist at The Institute for Teaching and Learning (TILT) at Colorado State University.

Christopher Geanious is an Instructional Designer with The Institute for Teachiing and Learning at Colorado State University.

Alanna Pawlak is a Postdoctoral Researcher in the Center for STEM Learning at the University of Colorado Boulder.

Mary E. Pilgrim is an Assistant Professor in the Department of Mathematics and Statistics and Associate Director of the Math and Stats Learning Center at San Diego State University.

Daniel Reinholz is an Assistant Professor in the Department of Mathematics and Statistics at San Diego State University.

Clara E. Smith is Diversity & Inclusion Training & Development Specialist in the Human Resources Department at the University of Colorado Boulder.

Sarah Wise is Research Faculty in the Center for STEM Learning at University of Colorado Boulder.

7 References

Anderson, A.A. (2006). The community builder’s approach to theory of change: A practical guide to theory development. The Aspen Institute Roundtable on Community Change.

Connolly, M. R., & Seymour, E. (2015). Why theories of change matter [WCER Working Paper No. 2015-2]. https://wcer.wisc.edu/docs/working-papers/Working_Paper_No_2015_02.pdf

Corbo, J. C., Reinholz, D. L., Dancy, M. H., Deetz, S., & Finkelstein, N. (2016). Framework for transforming departmental culture to support educational innovation. Physical Review Physics Education Research, 12(1), 010113. https://doi.org/10.1103/PhysRevPhysEducRes.12.010113

Kezar, A. (2013). How colleges change: Understanding, leading, and enacting change. Routledge.

Malachowski, M., Osborn, J. M., Karukstis, K. K., Kinzie, J., & Ambos, E. L. (this volume). “Scaffolding research into undergraduate STEM curricula and cultures: An emerging model for systemic change.” In K. White, A. Beach, N. Finkelstein, C. Henderson, S. Simkins, L. Slakey, M. Stains, G. Weaver, & L. Whitehead (Eds.), Transforming Institutions: Accelerating Systemic Change in Higher Education (ch. 4). Pressbooks.

Ngai, C., Corbo, J. C., Falkenberg, K., Geanious, C., Pawlak, A., Pilgrim, M. E., Quan, G. M., Reinholz, D. L., Smith, C., & Wise, S. B. (2020). Facilitating Change in Higher Education: The Departmental Action Team Model. Glitter Cannon Press.

Ngai, C., Pilgrim, M. E., Reinholz, D. L., Corbo, J. C., & Quan, G. M. (2020). Developing the DELTA: Capturing cultural changes in STEM departments. CBE—Life Sciences Education, 19(2), ar15. https://doi.org/10.1187/cbe.19-09-0180

Quan, G. M., Corbo, J. C., Finkelstein, N. D., Pawlak, A., Falkenberg, K., Geanious, C., Ngai, C., Smith, C., Wise, S., Pilgrim, M. E., & Reinholz, D. L. (2019). Designing for institutional transformation: Six principles for department-level interventions. Physical Review—Physics Education Research, 15(1), 010141. https://doi.org/10.1103/PhysRevPhysEducRes.15.010141

Schein, E. H. (2010). Organizational culture and leadership. John Wiley & Sons.

Taplin, D., & Rasic, M. (2012). Facilitator’s source book: Source book for facilitators leading theory of change development sessions. ActKnowledge.

Reinholz, D. L., Corbo, J. C., Dancy, M., & Finkelstein, N. (2017). Departmental action teams: Supporting faculty learning through departmental change. Learning Communities Journal, 9, 5–32.

Vogel, I. (2012). ESPA guide to working with theory of change for research projects. Isabel Vogel in association with LTS/ITAD. https://www.espa.ac.uk/files/espa/ESPA-Theory-of-Change-Manual-FINAL.pdf

Weiss, C.H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community-based initiatives for children and families. In J. P. Connell, A. C. Kubisch, L. B. Schorr, & C. H. Weiss (Eds.). New Approaches to Evaluating Community Initiatives: Vol. 1. Concepts, methods and contexts (pp. 65–92). Aspen Institute.