EIC Formative Report
Brief Summary
The goal of the Effective Implementation Cohort (EIC) is to increase districts’ capacity to implement a high-quality middle-grade math curriculum to accelerate learning for students experiencing poverty, Black, Latino/a, and/or English Learner (EL)-Designated students. The current report presents results from a student survey, a teacher survey, and a report on the cohort’s use of implementation strategies. Highlights include:
- In general, students who reported that their teachers used more culturally relevant examples in lessons and/or were more respectful of students’ cultural backgrounds held more positive beliefs about math and reported more positive experiences with math.
- Findings related to student demographics were mixed. For example, students who identify as Black or Asian reported higher agreement with liking and enjoying math than White students; however, students who identify as White reported higher agreement with feeling motivated and participating in math than students who identify as Black.
- When examining how district contextual factors relate to student and teacher outcomes, stable leadership emerged as an important predictor, and was associated with students’ positive math experiences and beliefs; higher teacher ratings of acceptability, appropriateness, and feasibility of the curriculum; and higher teacher confidence in delivering math instruction, including through use of cultural pedagogy strategies.
- Overall, implementation strategies were closely linked to the individual providers, with each provider likely to use the same strategies with each of their partner districts.
Full Executive Summary
The Effective Implementation Cohort (EIC) represents provider-district partnerships learning how to implement coherent instructional systems for middle school mathematics. Specifically, the learning partnerships’ goal within the EIC is to increase districts’ capacity to implement a high-quality middle-grade math curriculum to accelerate learning for students experiencing poverty, Black, Latino/a, and/or English Learner (EL)-Designated students. As the learning partner for the EIC, we are striving to learn and share what districts need to have in place for effective implementation; what aspects of implementation at the district and school level most benefit priority students, and what makes site-wide implementation successful.
In this formative report for the EIC, we present analyses from a teacher survey, a student survey, and a report of specific implementation strategies in use to examine:
- Students’ experiences, engagement, and beliefs related to mathematics, perceptions of teachers’ culturally relevant pedagogy, and demographics
- Teachers’ perceptions of the curriculum and professional learning support
- Implementation strategies being used by provider and LEA dyads
Our analyses had several goals.
First, we explored how student perceptions of their teachers and student demographic relate to their math experiences and beliefs.
Second, we investigated how three LEA contextual factors (governance structure, leadership stability, and locale) relate to each of the three areas above.
Third, we aimed to gain a clearer understanding of variations in implementation strategies used.
Student Beliefs, Engagement, and Experiences
19,620 middle school students completed a survey in Spring 2022 and findings indicated:
How do student demographics relate to their math experiences?
Math Enjoyment
- Students who identify as Black or Asian reported significantly higher agreement with liking and enjoying math than White students.
- Male students reported significantly higher agreement with liking and enjoying math than female students.
Engagement in Math
- Students who identify as White reported significantly higher agreement with feeling engaged in math (i.e., feeling motivated and displaying participation in math class) than students who identify as Black, American Indian, more than one race, or another race.
- Non-Hispanic students reported significantly higher agreement with feeling engaged in math than students who identify as Hispanic.
Math Self-efficacy
- Students who identify as White reported significantly higher agreement with feeling confident in solving math problems and performing math-related work than their peers who identify as American Indian, Pacific Islander, more than one race, or another race.
- Black and White students did not significantly differ in their ratings of confidence in math.
- Students who identify as Asian reported significantly higher agreement with feeling confident in math than White students.
- Non-Hispanic students reported significantly higher agreement with feeling confident in math than their peers who identify as Hispanic.
Growth Mindset
- Students who identify as White reported significantly higher agreement with the belief that math abilities can be developed and improved through work, practice, strategies, and input from others than students who identify as Black, American Indian, or another race.
- Male students were significantly more likely to report higher agreement with this belief than female students.
How do students’ perceptions of teachers’ culturally relevant pedagogy relate to their math experiences?
58.1% of students report that teachers “never” or “once in a while” include examples of student’s different cultures, backgrounds, or families in lessons.
Students who thought that their teachers use more culturally relevant examples reported significantly higher agreement with liking and enjoying math (β = .078, p < .001) and significantly higher agreement with feeling engaged in math (β = .051, p < .001). However, they also reported lower agreement with the belief that math abilities can be developed and improved through practice, strategies, and input from others (β = -.117, p < .001).
Students who held perceptions that their teachers have greater respect for their cultural backgrounds were significantly more likely to report higher agreement with: liking and enjoying math (β = .118, p < .001), feeling engaged in math (β = .171, p < .001), and having confidence in math (β = .125, p < .001).
How do district contextual factors relate to students’ math experiences?
District contextual factors were associated with differences in students’ beliefs that math abilities can be grown and developed, math enjoyment, math engagement, and confidence in math.
Although patterns of results varied slightly across the four outcomes, in general, students reported more positive experiences and beliefs in districts with stable leadership.
Teacher Perceptions of Implementation Constructs
532 teachers completed a survey in Spring 2022 and findings reflected:
How do district contextual factors relate to teachers’ perceptions of implementation?
Teachers from urban districts reported slightly higher quality of professional learning services received. No differences were found by LEA governance structure or leadership stability.
Overall, ratings of acceptability, appropriateness, and feasibility of the curriculum were highest for teachers in districts with elected superintendents; with stable leadership; and in urban locales.
Teachers in districts with stable leadership and teachers in rural districts were more likely to say they have higher confidence in delivering math instruction, including through the use of cultural pedagogy strategies.
Implementation Analysis
57 implementation strategies were identified from prior literature (e.g., Fixsen & Blasé, 2020; Powell et al., 2015) and observation of any additional strategies in use. Strategies were grouped into three categories: data supports (9 strategies), implementation supports (45 strategies) and communication (3 strategies). NIRN implementation specialists identified strategies being used by each Provider-LEA dyad based on review of multiple data sources.
Implementation strategy mapping analyses revealed the following:
What is the overall picture of strategy usage across districts?
- There is a wide range (13 – 47) in how many strategies (N = 57) are used.
- Districts who are using more data support strategies are also likely to be using more implementation support strategies and more communication strategies.
How are providers using strategies across districts?
- While providers may use different strategies from other providers, each provider is likely to be using the same implementation strategies with each of their partner districts.
- Provider participation in NIRN’s Deep Dive supports is associated with their partner districts using more strategies specifically related to data supports.
How do district contextual factors relate to the use of implementation strategies?
- Districts with appointed superintendents used more total strategies than those with elected superintendents.
- Rural districts were using more implementation support strategies than suburban or urban districts.
- When looking at each of the 57 specific strategies individually, there were also some differences in use of particular strategies by context.
Implications
Further investigation into the quality and intention of culturally relevant strategies such as inclusion of diverse examples within math lessons are necessary.
- The negative association between the inclusion of examples of students’ different cultures and backgrounds in lessons and students’ beliefs that math abilities can be developed or improved may be explained by the use of stereotypes within those examples that cement students’ ideas of their ability based on ethnicity and race.
- The positive associations between the inclusion of diverse examples in lessons and enjoyment and engagement in math may indicate benefits of students seeing themselves represented.
Significant attention and improvements are needed to support our students who identify as Hispanic given that they reported lower agreement both with feeling engaged in math and with feeling confident in solving math problems compared to their non-Hispanic peers.
Exploring and leveraging strategies that are facilitating math enjoyment for our Black students may also help support their engagement in math and belief that math abilities can be improved and developed.
Strategies to develop enabling leadership and to support stabilization during transition are needed given the role of leadership within teacher’s perceptions of the acceptability, appropriateness, and feasibility of the selected curriculum and their levels of confidence in their math instruction.
Support may be needed for Providers to tailor implementation strategies to meet the unique need and local context of partner districts. This may also highlight the importance of initial district-provider match to ensure that districts are getting the support they need for their specific local context.
Common areas of growth for Providers around underutilized implementation strategies may need to be explored.
Next Steps
These findings warrant additional analyses to better explain the relationships among implementation readiness constructs, district and school level coaching strategies, school level leadership, teachers constructs, and student experiences, beliefs and engagement in mathematics. Specific areas of investigation may include:
- How do LEA-level implementation readiness constructs correlate or predict teacher perceptions of implementation and their self-efficacy?
- How does building-level leadership impact teachers’ perceptions of the acceptability, appropriateness, and feasibility of the curriculum?
- What are the relationships between the implementation strategy of coaching and other implementation strategies with teachers’ self-efficacy in mathematics instruction and culturally relevant pedagogy?
- What are the relationships between cultural sensitivity and teachers’ self-efficacy in math instruction and cultural pedagogy?
These additional analyses will support the identification of additional inferences and meaningful conclusions. We acknowledge several limitations that impact the interpretation of the findings such as limited sample representation and control of potential confounding variables.
For a copy of the full Formative Report for the Fall of 2022, click below.
Built with Shorthand