Interim Annual Report
Effective Implementation Cohort
Executive Summary
The Effective Implementation Cohort (EIC) represents provider-district partnerships learning how to implement coherent instructional systems for middle school mathematics. Specifically, the learning partnerships’ goal within the EIC is to increase districts’ capacity to implement a high-quality middle-grade math curriculum to accelerate learning for students experiencing poverty, Black, Latino/a, and/or English Learner (EL)-Designated students. As the learning partner for the EIC, we are striving to learn and share what districts need to have in place for effective implementation, what aspects of implementation at the district and school level most benefit priority students, and what makes site-wide implementation successful.
In this interim report, we present results from multilevel modeling of district demographics and indicators, a teacher survey, a student survey, and district administrative data including student achievement scores. We also present results from descriptive analyses of key enabling conditions. In this executive summary, we also incorporate findings from qualitative interviews with coaches, executive sponsors, and providers, which are elucidated more fully in the 2023 EIC qualitative analysis report and the EIC Data Use Output. We describe key takeaways from the triangulation of findings.
Highlights Include:
- Examination of how individual factors from student-level data predict students’ experiences, engagement, and beliefs related to mathematics; perceptions of teachers’ culturally relevant pedagogy; demographics; and math achievement, including descriptive analyses of how race/ethnicity relates to students’ math proficiency.
- How teachers’ perceptions of the math curriculum (i.e., appropriateness, feasibility) and self-efficacy for math instruction and cultural pedagogy impact student outcomes.
- How district-level demographics and enabling conditions impact implementation, as indicated by students’ experiences, engagement, and beliefs related to mathematics, as well as their achievement.
- Exploratory “deep dive” analyses examining how student and teacher factors predict math experiences, expectations, beliefs, and achievements from three districts in which teacher and student data sources were linked.
- Descriptive analyses regarding how the set of key enabling conditions varies across priority contexts.
- Qualitative findings that provide additional insight into key enabling conditions.
- Qualitative findings related to how grantees use or do not use leading and lagging measures to identify challenges and make improvements to systems of support.
- Qualitative findings regarding factors not identified in established measures that are critical to identifying challenges and making improvements.
Our analysis had several goals:
- Explore how student, teacher, and district characteristics relate to students’ math experiences and beliefs by using multilevel modeling.
- Investigate how student, teacher, and district characteristics relate to student math achievement.
- Examine how three LEA contextual factors (i.e., governance structure, leadership stability, and locale) relate to key enabling conditions, including district-level measures.
- Examine how those key enabling conditions changed from the 2021-2022 to the 2022-2023 school year.
- Engage in qualitative analyses of key learning questions and enabling conditions.
Student beliefs, engagement, and experiences analyses included 19,307 middle school students who completed a survey in Spring 2022; 532 teachers who completed a survey in Spring 2022; and district implementation teams who also completed measures of enabling conditions throughout the 2021-2022 year. Math achievement data were provided by districts, and analyses included 83,002 students. In addition, exploratory “deep dive” analyses were completed with 6,686 students and 220 teachers from three districts that provided additional data identifiers (not provided by all districts) that allowed for students to be connected to their own individual teachers.
Students
How do student characteristics predict student beliefs, engagement, and experiences with math?
How do student characteristics predict math achievement?
Student demographics relate to all outcomes.
*These findings held even in models accounting for both teacher and district factors.
Math Enjoyment
Students who identified as Black or who were included in the group classified as Another Race (students who identify as Asian, Native Hawaiian or Pacific Islander, American Indian or Alaska Native, Mixed Race, or More than One Race) reported significantly higher agreement with liking and enjoying math than White students. No differences were found between White students and students who identified as Hispanic, or between students with different levels of English language proficiency.
Male students reported significantly higher agreement with liking and enjoying math than female students.
Math Self-efficacy
Students who identified as White reported significantly higher agreement with feeling confident in solving math problems and performing math-related work than their peers who identified as Black or Hispanic.
Students who were proficient in English reported higher feelings of confidence in math than students who had limited English language proficiency in the exploratory “deep dive” group.
Male students reported significantly higher agreement with feeling confident in math than female students.
Engagement in Math
Students who identified as White reported significantly higher agreement with feeling engaged in math (i.e., feeling motivated and displaying participation in math class) than students who identified as Black or Hispanic. No gender differences or English language proficiency differences were found regarding engagement.
Growth Mindset
Students who identified as White reported significantly higher agreement with the belief that math abilities can be developed and improved through work, practice, strategies, and input from others than students who identified as Black, Hispanic, or another race.
Students who were proficient in English reported significantly higher agreement with having a growth mindset than students who had limited English language proficiency in the exploratory “deep dive” group.
Female students were significantly more likely to report higher agreement with this belief than male students.
Math Achievement
Student math achievement data reflect that 44% of Asian students, 35% of White students, and 30% of students of another race were at or above proficiency in math, whereas only 19% of Hispanic students and 9% of Black students were at or above proficiency in math in 2021-2022. In total, only 20% of all EIC students were categorized as at or above proficiency in math.
Students in the Another Race group (encompassing students who identified as Asian, Native Hawaiian or Pacific Islander, American Indian or Alaska Native, Mixed Race, or More than One Race) scored higher on state end-of-year mathematics assessments than White students.
White students scored higher on state end-of-year mathematics assessments than students who identified as either Hispanic or Black.
Students who were proficient in English scored higher on state end-of-year mathematics assessments than students who were identified as having limited English language proficiency.
Gender did not significantly predict student math achievement.
Even when accounting for district-level nesting, students' perceptions of teachers’ culturally relevant pedagogy were also related to their math experiences.
Students who thought that their teachers used more culturally relevant examples reported significantly higher agreement with liking and enjoying math and significantly higher agreement with feeling engaged in math. However, they also reported lower agreement with the belief that math abilities can be developed and improved through practice, strategies, and input from others.
Students who held perceptions that their teachers had greater respect for their cultural backgrounds were significantly more likely to report higher agreement with: liking and enjoying math, feeling engaged in math, having higher beliefs that math abilities can be improved and developed through practice, and having confidence in math.
Teachers
How does teacher perception (e.g., feasibility, acceptability, and appropriateness) of the math curriculum influence implementation?
What conditions maximize likelihood of successful implementation across contexts relevant to priority students?
How does teacher perception (e.g., feasibility, acceptability, and appropriateness) of the math curriculum influence implementation?
When accounting for grouping, no teacher-response factors (averaged up to the district level) accounted for any of the student outcomes, including math achievement. This finding held true for both the cohort-wide analyses and the exploratory “deep dive” analyses in which teachers could be linked to their own students.
However, in the exploratory “deep dive” analyses, measures of variability related to achievement suggested that 45% of variability in achievement outcomes was due to students’ grouping by teacher. After modeling, substantial variability (27%) still remained related to grouping due to factors not accounted for by this model. This could reflect differences in important constructs not included in this model (i.e., fidelity), or could also reflect pre-existing differences between groups (i.e., teachers with more or fewer advanced students).
Notably, this analysis was completed on the 2021-2022 student and teacher data, when 60% of districts were in the first year of implementation. It will be important to continue to examine these relations as districts in partnership with their professional learning Providers continue their implementation efforts into subsequent years. Furthermore, even those districts that had begun implementing the curriculum were not yet receiving intensive supports to enhance their implementation, which may also contribute to further findings in this area in future years.
Findings from the qualitative analysis provided some insights into ways that districts were continuing to grow these areas in 2022-2023, including using specific strategies to work on feasibility (e.g., allocating time for teaching the curriculum, providing supports such as professional development and coaching, providing help related to appropriate pacing).
An identified facilitator of implementation was providing time for teachers to deepen their content knowledge and understanding of the curriculum. Coaches in particular noted the importance of providing that time, as well as access to resources or training, for teachers to engage in learning.
What conditions maximize likelihood of successful implementation across contexts relevant to priority students?
Similarly to the teacher-response data, no district-level contextual factors (district demographics or indicators of enabling conditions including district capacity and district teaming) significantly predicted any of the student outcomes, including math achievement. Again, these findings still reflect the 2021-2022 year in which most districts were in the first year or early phases of implementation, so this may change as districts - in partnership with their professional learning Providers - continue their implementation supports over time.
In both the cohort-wide and the exploratory “deep dive” analyses, a few contextual effects emerged related to student characteristics (e.g., the proportion of students of a certain race; the average student-perception rating of teachers’ use of cultural examples or expression of cultural respect). These findings also can be interpreted in tandem with the individual-level effects of the same variables, which help to understand how a student’s own characteristic interplays with the district characteristic. These findings are as follows:
Students’ perceptions of teachers’ cultural respect and math engagement.
While the individual finding suggests that students’ engagement increases as they perceive their teacher’s cultural respect to increase, there was also a significant negative contextual effect which suggests that district-wide, as cultural respect increases, student engagement goes down. Combined, the pattern appears to show that if a student’s teacher is approximately on par with the district average, there is predicted to be no effect on the student’s engagement, regardless of whether the student perceives the teacher to frequently or infrequently show cultural respect. If the student perceives their teacher to be more respectful than the district average, then the student is predicted to be more engaged, whereas if the student perceives their teacher to be less respectful than the district average, students are less engaged.
Limited English language proficiency and achievement.
Again, while the individual finding suggested that students who were proficient in English scored higher on achievement tests than students who had limited English language proficiency, a contextual effect emerged suggesting that having a higher proportion of students with limited English language proficiency in the district also predicted greater achievement. This finding emerged in the student-predictor model, and should be interpreted with particular caution as it did not remain significant in the model where other teacher-level predictors were included, though it was also significant in the model with district-level predictors. This finding may also reflect a statistical artifact, as achievement scores were standardized by district using z-scores; therefore, it might be that the overrepresentation of students with limited English language proficiency in specific districts (up to 82.4%) shifted the distribution for those populations.
Use of cultural examples and enjoyment, math self-efficacy, and growth mindset.
In the exploratory “deep dive” analyses, a significant and positive contextual effect emerged related to the use of cultural examples for three outcomes. These analyses linked students directly to their math teachers rather than to their district. For each, these findings suggest that when teachers are on average rated by their students as using more examples reflective of students’ culture, their students’ enjoyment, math self-efficacy, and growth mindset are expected to increase. No individual effect was found for enjoyment or self-efficacy, suggesting this effect may be more meaningful at the contextual level. Perplexingly, the individual effect for growth mindset was contrary, as noted above: when a student rates their own teacher as higher in using cultural examples, their agreement with having a growth mindset is predicted to be lower.
Qualitative findings also highlighted some of the factors that are likely to maximize implementation.
Providers, Executive Sponsors, and Coaches identified the following as facilitators of implementation:
- Consistent messaging that clarifies expectations, sets the tone for the work, and increases buy-in and involvement from administrators.
- Celebrating impacts and/or highlighting the work.
- Providing non-threatening opportunities and time for teachers to deepen their content knowledge and understanding of the curriculum via coaching walk-throughs and professional development.
Providers, Executive Sponsors, and Coaches also identified important barriers to implementation:
- Teachers’ mindsets regarding students’ abilities and/or the teachers’ own desires to adopt a new curriculum.
- Turnover among teachers and leaders such as principals and Central Office administrators.
Enabling Conditions
How does the set of key enabling conditions vary across priority contexts?
How do the qualitative findings provide additional context related to key enabling conditions?
Enabling Conditions: The enabling conditions being examined to support implementation of the high-quality math curriculum include district teaming, LEA executive sponsor engagement, communication, fit and feasibility, implementation planning, and measurement planning. District capacity for implementation and school leadership were also explored.
District Teaming and Relationships
In general, relationships between Executive Sponsors and Providers were noted to be positive on both sides. Providers reported some challenges in relationships, including emphasizing the importance of frequent and open bidirectional communication. Coaches also highlighted the importance of building safe spaces and personal relationships with those they coach through various strategies.
LEA Executive Sponsor Engagement
Executive sponsors facilitated implementation by being visible leaders of the work and engaging in thoughtful conversation and data review across levels (i.e., Central Office, schools, etc.). Providers and Coaches worked closely with Executive Sponsors to plan for implementation, focusing on sustainability.
Communication
Executive Sponsors often took the lead role in communication, supported by Providers and Coaches. Coaches reported strong internal communication within their teams; they also worked to support bi-directional communication with teachers and Providers by developing supportive relationships and respecting them as professionals.
Communication challenges were also noted, including difficulties meeting with school leadership; and the need to be intentional, consistent, timely, and compelling when communicating with critical perspective groups around the importance of the EIC work.
Fit and Feasiblity
Providers and Executive Sponsors mentioned several strategies to assess and ensure fit of the curriculum including offering professional development (to increase learning around integrity indicators used in classroom walkthroughs); working with teachers to provide opportunities for students to “do the math and be the mathematicians”; encouraging district, administrator, and teacher buy-in; and promoting alignment of the EIC work within existing district priorities. In terms of feasibility, District Teams ensured sufficient allocated time for teaching the curriculum and receiving other supports needed for successful implementation.
Implementation Planning
Many District Teams adjusted the original implementation plan or informally made changes to implementation strategies. District Teams relied on the plan to lead discussions and keep the implementation goals on track. Some plans were incorporated into data dashboards for easier reference to individual areas of the plan.
One strategy to ensure sustainability was identifying key people, roles, and responsibilities within the grant and developing structures and processes for retaining these resources beyond the grant funding period.
Measurement Planning
Providers emphasized the importance of data in implementation work and regularly met with Executive Sponsors, District Teams, and Coaches to review data and discuss findings. All groups noted that they had access to multiple types of data from multiple sources, tended to triangulate data for interpretation, made use of the data for a multitude of purposes, and were cognizant of the need to tailor messages when sharing results based on the targeted audience.
Two additional learning questions related to measurement were answered in the EIC Data Use Output. These questions addressed how grantees use or eschew measures related to the instructional system and to implementation, as well as what factors do not exist in established measures that are critical to identifying challenges and making improvements. Key findings showed that:
- The most frequently used data were actionable data that helped identify where to make changes. Characteristics of data that were identified as useful included data aligned with the questions district teams were interested in answering (relevant data); data that were easily accessed, collected, and understood; and valid and reliable data. Promising approaches to data practices included centering data as an integral activity to implementation work, with a focus on data triangulation.
- Notable data challenges included district teams disseminating data results to families and finding the time to analyze information in a timely and meaningful way.
- Data needs included: opportunities to conduct more in-depth, timely, and targeted analyses aligned with a stated purpose; interpreting trends for action; creating more meaningful stories based on available data; and creating data matrices and intentional data plans for a variety of audiences. Limitations around data collection (e.g., reliability, validity, feasibility of data) were also raised.
Overall, no strong patterns emerged regarding district demographic factors (governance structure, leadership stability, or locale) as predictors of enabling conditions.
Promisingly, examinations of changes in enabling conditions across school years 2021-2022 and 2022-2023 showed consistent increases, though many did not reach statistical significance, likely due to the small number of districts. In particular, factors related to the districts’ Competency (i.e., use of strategies to develop, improve, and sustain educators’ ability to implement an effective innovation as intended) including training, coaching, and fidelity were trending toward or reached statistically significant increases from 2021-2022 to 2022-2023.
Implications & Next Steps
When examining proficiency categories (not proficient, approaching proficiency, proficient, or exceeding proficiency), overall, student proficiency in math was low, particularly for priority students. No racial/ethnic group exceeded 44% of students at or above proficiency in 2021-2022, and the total percentage of students in the EIC sample at or above proficiency was 20%. This highlights the need for high-quality math instruction for all students. In particular, the extremely low rates of proficiency for students who identified as Black and Hispanic, as well as the analyses that showed that students who identify as Black, Hispanic, or as having limited English language proficiency are predicted to have lower scores on year-end state mathematics assessments, demonstrate the importance of approaches that focus on priority student populations.
Furthermore, findings suggested that male students report higher enjoyment and liking of math and more confidence in their math abilities than female students. In contrast, female students reported having higher belief that their math abilities can be developed and improved through work and practice (i.e., growth mindset). Although no differences were found on engagement in math class or in math achievement, these results suggest the importance of finding ways to foster math enjoyment and liking for girls in math to increase their positive affect toward the subject.
Although analyses completed on the 2021-2022 data did not show significant impacts of teacher- and district-level indicators on student outcomes, this reflected a period of early implementation. Examinations into change over time on key enabling conditions suggest that small increases, including some trending toward or reaching statistical significance, are seen across enabling factors (e.g., organizational leadership, competency, and data systems for decision-making). Improvements in these key early indicators at the district level could, over time, contribute to changes in long-term student outcomes.
In particular, increases were trending toward or reaching statistical significance in the area of Competency, which included training, fidelity, and coaching. Although growth represents a significant increase from the prior year, it does not mean that these areas are at expected levels. For example, although there was a significant increase in capacity for coaching; districts, on average, grew from reporting approximately 27% of the total possible score for capacity for coaching on the District Capacity Assessment (DCA) to 48% of the total score, suggesting there is still much more room for growth. Typically, a score of 60% on the DCA is associated with improvements in fidelity or integrity of the selected practices (Kloos et al., 2022).
Student demographic predictors and perceptions of their teachers’ cultural pedagogy remained significant predictors in all models, even after accounting for teacher- and district-level factors. The generally positive and significant impact of students’ perceptions of their teachers’ use of cultural pedagogy strategies highlights how critically important providing professional development in culturally responsive teaching will be to improving student outcomes.
Exploring and leveraging strategies that are facilitating math enjoyment for our Black students may also help support their engagement, growth mindset, self-efficacy, and achievement in mathematics.
Significant attention and improvements are needed to support our students who identify as Hispanic given their lower reports of growth mindset, engagement, and self-efficacy in mathematics than their non-Hispanic peers, as well as their lower achievement scores.
Providers and their districts, within their implementation plans, will need to tailor their supports and strategies in order to address and mitigate barriers; leverage facilitators; and attend to various identified data needs, such as creating meaningful data stories from feasible and reliable data collection efforts.
We acknowledge several limitations that impact the interpretation of the findings such as limited sample representation, small sample size at the district level, loss of nuance in cohort-wide analyses due to teachers’ data not being directly linked to their students’ data, and control of potential confounding variables.
Next Steps
After the 2022-2023 student survey, teacher survey, and achievement data become available in October 2023, further investigation is warranted to examine how these relations unfold as implementation progresses.
Furthermore, when data on fidelity and the quality of professional learning for 2022-2023 also become available, it will be important to also include those factors within teacher models to better understand the role that they play in student outcomes and their relations to other implementation constructs. This will be especially important since the exploratory “deep dive” model for achievement suggested substantial variability at the teacher level that was unexplained by current predictors.
Download the full report
The information above along with the document at the link below together comprise the Full Interim Report.
The Effective Implementation Network, a project within the National Implementation Research Network (NIRN), is housed within the FPG Child Development Institute at the University of North Carolina at Chapel Hill.
Built with Shorthand