Annual Interim Report
Year 1 (2021-2022)
Descriptive analyses of quantitative and qualitative data were collected for various indicators of the six readiness constructs within the EIC in the 2021-2022 academic year using various methods (e.g., interviews, surveys, observations) with district executive sponsors, coaches, and provider staff. Below you will find the key takeaways and recommendations from the NIRN team. For more information on the EIC visit our website and the Learning Agenda.
*Summaries from Teacher and Student Outcomes are forthcoming in the near future*
Overall Key Takeaways
- Growth in the development of district teams to be accountable for implementation is present among many Provider-LEA dyads within the cohort; however, communication within teams and with their communities is an area of need.
- Professional learning is a strength, but variability in the provision of coaching is present in the availability of coaching, who is providing and receiving coaching, and processes and procedures to support coaching.
- Access to and use of data by teams to consistently inform ongoing implementation and monitoring progress is an area of need in general. A specific area of need is the collection and use of fidelity or integrity data for the use of the curriculum and delivery of instructional practices.
- Competing initiatives were not typically identified beforehand and often are being identified/addressed when impacting implementation at the instructional staff level.
- Covid-19 continues to impact time and capacity as well as cause disruptions within local contexts; interrupted learning; staffing and sub-shortages; LEA transitions; and provider transitions.
Themes by Readiness Construct
District Teaming
Linking teams at district and school levels serve as the structure to drive implementation as well as serve as the facilitator for bi-directional communication. Creating and monitoring linking team functioning is a proactive approach to implementation.
Overall team functioning is reported as strong in terms of representation, attendance, processes, use of time, positive relationships, and accountability.
LEA Executive Sponsor Engagement
Establishing a culture of collective/collaborative leadership has emerged as an area of needed support. Creating roles and responsibilities of a DIT, including procedures for decision-making, can begin to break through power dynamics. The use of the implementation team survey assists with lifting the realities of the interworking of teams and the leadership.
Overall, data indicated that executive leaders work with their lead partners to support an effective partnership, have the necessary decision-making authority, use effective bi-directional communication with district leadership, team, staff, and others, can speak to and answer questions about math implementation, and ensure that priority students are centered when making decisions including resource allocation.
A consistent area of need across all Provider-LEA dyads is the ability of the provider and district team to cultivate the role of LEA sponsor as a champion and use data and information to inform planning.
Diminishing executive sponsor engagement and participation has been seen to hinder work when further complicated by turnover at the provider level.
Power differentials observed within teams with active and engaged executive sponsors have been observed to limit collaborative decision-making and the use of information and input from team members.
Communication
Communication and engagement of critical perspectives within the district and its community is a significant area of need across the cohort. Efficient and practical communication methods and processes are needed internally within the team and with schools to support planning and problem solving as well as engagement of broader community perspectives.
Communication is overall the lowest readiness construct in comparison to performance on other readiness constructs. Communication messages being used are coherent, but challenges are seen within efficient processes between providers, district teams, and schools, as well as identifying critical perspectives internal and external to the district, planning for differentiated messages and mechanisms for bi-directional communication for specific audiences, and measuring the effectiveness of communication.
Assessing Fit & Feasibility
A process for determining the fit and feasibility of the chosen curriculum is a component of implementation that occurs throughout a project. The selection of a curriculum should be a collaborative exercise to ensure the various levels of the
the educational system is considered when determining the usability of a curriculum.
Most districts selected their high-quality math curriculum before engaging in the EIC. They did not engage in a comprehensive fit and feasibility assessment, which can and should continue through implementation.
The majority of providers and districts continue to focus on how well the curriculum addresses their district’s specific needs and level of evidence. Less consideration is given to factors related to the fit of the program in terms of alignment with other initiatives/programs, capacity, and available supports, such as coaching and a well-defined engagement process for optimizing readiness among staff. As such, most districts struggled with identifying competing priorities and mitigating actions for those priorities.
Majority of districts also continue to struggle to engage critical perspectives or stakeholders in identifying what changes are needed based on assessments of fit and feasibility within their local context.
Implementation Planning
Professional learning overall is an implementation strength. Coaching is a weaker implementation component across the project and one with the most significant variability.
Overall, providers and districts consistently use their developed theories of action to guide their implementation efforts, centering priority students when planning and ensuring equitable resource allocation.
Areas of need within implementation planning across providers and districts are continuing to progress in their planning among competing priorities, using data for planning and monitoring progress, planning for competency development activities beyond professional learning, and shared accountability for the co-creation of implementation strategies.
Primary audience for professional learning has been teachers and is an overall strength among providers and districts. Professional learning for others, such as coaches and principals, varies across dyads. The COVID-19 pandemic limited professional development time, and strategies for addressing this included onboarding new staff given turnover and incentivizing professional development on Saturdays for teachers. An area of need across the cohort related to professional learning is to
determine how and when differentiated and cohesive professional learning provided to all the essential areas of the system (e.g, coaches, educators in particular areas, building leaders, district leaders) can be provided.
Coaching is an ongoing area of need across the cohort. Some dyads are starting coaching either through LEA-existing positions or newly created positions. Some Providers include coaching within their service delivery plan, and others do not. There are instances where additional providers have been contracted to facilitate the coaching in an LEA. In addition, only a few providers and districts have plans for developing local capacity to provide coaching.
Coach listening sessions highlighted four essential items that would be helpful:
- a “thought partner” to bounce ideas off of and get feedback from
- dedicated time to meet with teachers
- additional training or resources on the curriculum
- coaching professional development.
Planning for additional implementation sites and implementers is varied. In some but not all instances, data, and planning for readiness and implementation activities is being used to support expanding implementation within additional schools and
classrooms.
Measurement Planning
Data systems and a data sharing process has emerged as an area of needed support. Support is needed in data access, data visualization, and structures for data sharing
throughout the implementation pathways.
Overall, providers and districts include different types of data (process, programmatic, and impact) measures in their measurement planning and have measurable indicators
using these various types of data from various sources (e.g., students, teachers, principals, coaches).
An area of growth across most providers and districts is having and using a process for reviewing and using data for planning and decision-making purposes.
Variability exists among providers and dyads regarding access to the varied data types across district team members and between providers and districts. In some instances, both the provider and all-district team members have access to the varied types of data being used. In other cases, the provider does not have access, or only certain district team members have access to the data. Furthermore, coaches report the need for better access to data, disaggregated and longitudinal data, to support effective coaching of teachers.
Value for implementation data (process, programmatic) beyond student outcome data is needed within various Provider-LEA dyads. In some cases, overemphasis and usage of student outcome data only is present.
EIC Resources
District Recommendations
District Teaming
- Continue to recognize teaming structures as a priority by mitigating the effects of staffing shortages and needs by offering multiple ways to participate in team discussions and decisions, incentivizing alternative team times, and developing
flexible teaching schedules or calendars to allow for planning. - Ensure an environment conducive to active engagement (executive decision-making authority, access to current data) with the activities expected of an implementation team (differentiated from leadership or advisory team), including accommodations for a variety of virtual participation methods and documentation within job descriptions and/or professional growth plans.
- Develop materials (e.g., project description, meeting agreements, video overviews of materials) for onboarding new team members.
- Assign a mentor to support new team members.
- Support the development of school-based teams to support implementation of the math initiative with direct communication linkages to the district team for timely planning and problem-solving.
LEA Executive Sponsor Engagement
- Address turnover of the executive sponsor and leaders by reviewing or developing selection and support processes that prioritize leadership skills that address adaptive issues such as team stress, trauma, flexibility, etc.
- Consider identifying a point and backup executive sponsor on a team with decision-making authority to mediate the impact of schedule conflicts and turnover.
- Ensure needed FTE (time, regular schedule, executive decision-making authority, access to current data) for activities expected of an executive sponsor and documentation within the job description and/or professional growth plan.
- Consider including the implementation champion role within the executive’s performance plan and assessment.
Communication
- Expand the reach of communication efforts beyond the immediate community. Consider who could be impacted by the information or have unique perspectives. Identify those (stakeholders) that could recognize bias in the information and be critical thought partners.
- Provide clarity on how communication flows between teaming structures (school/campus, district to purveyor, or other external support teams) along with developing a communication protocol that can be frequently reviewed and
modified with changing contexts and priorities. - Ensure communication is clear, concise, elicits desired emotion, relevant to the specific target audience, and addresses concerns.
Assessing Fit & Feasibility
- When assessing fit and feasibility, take into consideration the need to address essential instruction that was interrupted and the impact on the scope and sequence as well as vertical articulation among grade levels.
- Instructional best practices and implementing the curriculum as intended should be communicated and messaged as a priority by leadership.
- Revisit capacity, fit, and supports available for new and selected curricula (i.e., Hexagon Tool) to ensure needs continue to be met in changing contexts and to identify what to build upon to ensure effective implementation.
- Be willing to pause or take actions needed to de-implement existing programs, not in alignment with current needs or priorities.
Implementation Planning
- Continue creative ways to address staffing and substitute shortages as well as delays in professional learning through alternative days/times with incentives.
- Work with providers to ensure coaching supports are in place for all staff including new and temporary staff (e.g., long-term substitutes, alternative licensure candidates) to compensate for shortages and limited staff selection.
- Continue to prioritize key levers such as professional learning and coaching and use data to target action planning.
- Use data and criteria to determine the expansion of implementation (e.g., additional teachers and/or schools).
Measurement Planning
- Ensure data collection continues and systems are in place to address barriers or gaps in access to data by team members and staff.
- Establish a co-constructed and aligned prioritized data calendar (to inform agendas as to when teams can expect specific data sources to use, as well as data that is expected for the purposes of any grant-related research). Often the same data can be used for different purposes.
- Integrate implementation data (e.g., training effectiveness, coaching data, fidelity data) into existing or plans for data visualizations/dashboards of student outcome data. Data visualizations should include data on the system and support for teacher outcomes (knowledge, mindset, beliefs, practice)
and student outcomes (beliefs, engagement, experience, and academic performance).
Provider Recommendations
District Teaming
- Support the team in structures and processes that may be interrupted with the transition of members.
- Prioritize a regular meeting schedule (virtual as needed) with clear roles and responsibilities and expected data sources (calendar) the team may expect to have at each meeting to engage in improvement cycles.
- Support the district team to build capacity within school teams to address differentiated needs among schools.
LEA Executive Sponsor Engagement
- Establish a frequent brief check-in to ensure visible support, ask-answer questions, spot potential issues early, respond efficiently, and provide accurate positive feedback.
- Connect with new district leaders to provide information during times of
transition.
Communication
- Assist in keeping communication protocols in place and develop feedback loops to ensure all voices are heard.
- Embed “communication” as a regular standing agenda item.
- Identify key point/backup contacts with the district team and a common place to document important updates, notes, and decisions to review regularly.
Assessing Fit & Feasibility
- Address the potential need for adaptations to the curriculum based on interruptions to instruction and subsequent changes in data collection and analysis.
- Leverage internal capacity or partnerships with researchers/higher education to provide information related to any updated evidence of programs used in similar educational disruptions (e.g., post-natural disasters) as well as usability and supports available for curriculum.
- Update or provide recommendations or guidance regarding conditions for need, fit, and capacity of districts to implement in changing contexts.
- Explicitly define and provide criteria as to what is meant by integrity or fidelity for the delivery of math practices and use of the curriculum.
Implementation Planning
Adjust and be flexible with timelines for professional learning and meeting schedules.
Ensure plans are in place to address new and temporary staff training on the selected curriculum.
Consider how coaching services will be provided and transferred to local ownership over time for sustainability purposes.
Measurement Planning
- Adjust and be flexible with timelines for data collection and analysis.
- Keep data-driven decision-making a priority.
- Support co-created data collection and use calendars with LEAs aligned to stage-based measurement expectations outlined in RFP.
- Model value of different sources of data (e.g., qualitative data, voices of implementers) to inform implementation practice and research.
- Support the development of accessible and user-friendly data dashboards/visualizations of data to support monitoring of implementation.
- Ensure data is a standing agenda item.
Funder Recommendations
District Teaming
- Reinforce in RFPs specific budget lines and expectations around regular district
team meetings and use of formative data to mark progress and barriers for
expected implementation activities. - Support the district by recognizing that transitions in team membership may
delay certain aspects of the project. - Reinforce in RFPs the development of action items for anticipated or
“predictable” barriers as well as sufficient funding for evidence-based practices
for learning essential skills.
LEA Executive Sponsor Engagement
- Connect with intermediaries and executive sponsors to support problem
solving and prioritization. - Provide clarity in RFPs about the roles, functions, and expectations of executive
sponsors; ensure time is allocated in RFP for those responsibilities to be
covered.
Communication
- Ensure that regularly reporting methods are still in place, but flexibility is
granted to the content based on available information. - Engage the community and continue to provide positive messaging on the
support being provided. - Include expectations and rationales for including a concrete communication
plan within RFPs (e.g., provide suggested templates).
Assessing Fit & Feasibility
- If not within provider/intermediary scope of expertise, ensure RFP includes
expectation of measuring implementation fidelity at district level with an
operationalized definition. - Incentivize RFPs to encourage program developers or intermediaries to develop
fidelity checklists/measures (or to partner with applied researchers) to develop
ways to measure emerging implementation fidelity progress.
Implementation Planning
- Ask about potential barriers in the implementation plan and how assistance can be provided.
- Recognize that timelines and schedules may need to be adjusted.
- Develop RFPs with a stage-based framework indicating differentiating key
implementation activities emerging at different times of implementation. - Consider budget lines specifically geared towards providing/supporting
development of coaching and coaching systems anytime training is identified. - Consider funding research on new teacher development (post-grad or alternative licensure) through mentoring and coaching (i.e., development of evidence-based models for interning or ‘residency’) for mathematics instruction specifically with priority populations.
Measurement Planning
- Be patient and flexible while awaiting data and results.
- Illustrate a stage-based approach within RFP to create the space and expectation for different types of work and data to come “on-line” at different points within the project on the way to impacting distal outcome measures.
Built with Shorthand