Response to MTSS/SEL Article
Click on a topic below to go directly to that section.
Pilot Program Overview
Building Blocks for Excellence
Define Student Excellence
Develop Learning Opportunities
Develop Student Assessments
Plan for a CBA
Implement a CBA
Evaluate a CBA
Limitations and Directions for Future Research
Lessons Learned Click HERE for a “pdf” version of this Webpage
Pilot Program Overview
This webpage discusses the article, “A Grant Project to Initiate School Counselors’ Development of a Multi-Tiered System of Supports Based on Social-Emotional Data” by Karen Harrington, Catherine Griffith, Katharine Gray and Scott Greenspan, The article was published in The Professional Counselor, Volume 6, Issue 3, pages 278-294, a Special Issue on “School Counselors and a Multi-Tiered System of Supports Cultivating Systemic Change and Equitable Outcomes.”
The article reviews a pilot program that used the CBA and Protective Factors Index/Report Card as a framework for a three-year Federal education grant. The program was conducted as part of an Elementary and Secondary School Counseling Program (ESSCP) grant from the U.S. Department of Education between October 2012 and August 2015. It was conducted in four elementary schools in a diverse urban-suburban school district in the northeastern United States. Prior to the grant, the district had no elementary school counselors.
The grant had two federally mandated priorities: a) increase the student-to-school counselor ratio at the elementary level and b) reduce the number of disciplinary incidents in the elementary student population. In addition, the grant defined complementary goals associated with implementing an elementary counseling program, expanding the role of school counselors, collecting and using social-emotional and behavioral data in conjunction with academic data, and improving the district’s MTSS through the effective use of data-based decision making.
The Construct-Based Approach (CBA), articulated in our book Achieving Excellence in School Counseling through Motivation, Self-Direction, Self-Knowledge and Relationships was used as the research-based framework for designing and implementing the elementary school counseling program and developing and using the Protective Factors Index to gather data on students’ social-emotional and behavioral development.
Two studies were conducted on this grant program. An Evaluation Study analyzed the critical program components, steps taken and changes required to successfully achieve the results specified in the grant proposal. A Confirmatory Factor Analysis was conducted to determine whether the Protective Factors Index (PFI) developed as the assessment tool for collecting social-emotional and behavioral data was reliable. Both studies concluded that the PFI was “promising” as a universal social-emotional data collection approach and that additional research should be conducted to verify this conclusion.
Because the grant program’s development and implementation process aligned so closely with the six building blocks for a CBA, this webpage discusses how the CBA as a whole is likewise a promising approach to implementing school counseling programs that uses social-emotional and behavioral data, in conjunction with academic data, to determine students’ developmental and learning needs, and decide on appropriate interventions to support their academic achievement and well-being. The Evaluation Study is discussed on this webpage. The Factor Analysis is discussed on a separate webpage (Click HERE or follow the link in the Sidebar to your left).
Building Blocks for Excellence
A CBA is a learner-centered approach to improving school counseling programs, counselor practice and student outcomes. The CBA Website provides structured processes, clearly-defined protocols and easy-to-use tools to help develop and deliver a CBA. This grant program provides an excellent example of how one school district successfully used CBA building blocks to design and implement a new elementary school counseling program where none had previously existed.
This webpage illustrates how, by utilizing the CBA’s Building Blocks for Excellence, positive results can be achieved. Each building block will be explored in terms of what the Grant Team and members of the school community did to achieve success. It will provide insights into what areas are most critical to focus on, and the types of actions you will need to take. A CBA can be used to create a new program or be integrated into your existing school counseling program. A CBA is not intended to replace your current program, only to improve it.
Define Student Excellence
A CBA is a learner-centered program with three primary purposes: a) enhance school counselors’ capacity to support student’s development and learning, b) help students achieve at their highest potential and c) ensure that the design, delivery and evaluation of the counseling program are implemented with fidelity. A CBA is concerned with achieving excellence, primarily in terms of students’ academic achievement and well-being, but also in terms of school counseling program quality, organizational effectiveness and counselor practice.
A CBA defines student excellence in terms of research-based constructs strongly related to students’ academic achievement and well-being. For the new elementary school counseling program, student standards were developed in terms of Protective Factors that reflected social-emotional and behavioral competencies characteristic of proficient and engaged learners. Program excellence was defined in terms of meaningful learning opportunities offered to students and targeted interventions delivered through the district’s Multi-Tier System of Supports (MTSS). Organizational excellence was defined in terms of the efficiency, productivity and effectiveness of its support infrastructure. Excellence in counselor practice was defined in terms of competencies that characterized exemplary profession school counselors.
“Research-base” describes the body of literature that demonstrates what programs and practices work and the extent to which they achieve their expected results. The Grant Team adopted the four research-based CBA constructs articulated in our book as foundational filters for designing the grant program. They were selected due to a review of more than 50 years of research on student learning and achievement. Using the four constructs offered a research-base that demonstrated strong linkages to student achievement in areas that are within the purview of school counselors and school counseling programs. A brief description of the four constructs is provided below:
- Motivation: the forces that compel action and direct the behavior of individuals.
- Self-Knowledge: the understanding people have about their own abilities, values, preferences, and skills and is a necessary precondition for effective self-regulation.
- Self-Direction: being able to identify one’s own life directions, to make academic choices consistent with these directions, and to connect classroom learning to their life goals.
- Relationships: the ability to establish and maintain productive, collaborative, social relationships with teachers and peers.
The Grant Team, with ongoing input and feedback from constituent groups within the school community, re-articulated the four constructs as Protective Factor indicators that aligned social-emotional and behavioral attributes with the constructs, thus ensuring a continuous link to the research-base. The Protective Factor indicators evolved into a Protective Factor Index (PFI) which served as the primary assessment tool for collecting social-emotional and behavioral data about students’ non-academic development and learning. The PFI indicators will be discussed in the “Develop Student Assessments” section later in this document.
Implications of the Research-Based Constructs for the Grant Program
The importance and value of the four CBA constructs has been established by research. The Grant Team concluded that because these constructs have such a strong link to students’ academic achievement and well-being, that by using them as primary filters for designing the elementary school counseling program would have the greatest potential for achieving substantive results in terms of students’ improving their learning processes and learner outcomes.
A CBA is Results-Based
A CBA is also results-based. The Grant Team defined results in terms of the grant, student performance, program quality, organizational support and counselor practice. Results can be viewed as “expected results” which are defined at the beginning of any process, and as “actual results” which are measured and determined at the end of a process. Expected results for the grant are discussed below. Actual results are discussed in the section on “Evaluate a CBA” later in this document.
Expected Results for the Grant
The grant was expected to meet two federally-mandated priorities: a) increase the student-to-school counselor ratio and b) reduce the number of discipline-related incidents among elementary school students as indicated and documented on Office Discipline Referral (ODR) forms. The grant also identified complementary results to be achieved that focused on building a support infrastructure. More specifically, the district expected to achieve these results by implementing a new elementary school counseling program:
- Identify social-emotional and behavioral competencies strongly related to student achievement and well-being that were within the purview of school counselors.
- Develop the infrastructure for collecting meaningful social emotional indicators to use, along with academic data, in the district’s Multi-Tier System of Supports (MTSS).
- Expand the role of school counselors in determining and delivering interventions.
- Build the capacity of school counselors to become leaders in data-driven discussions about student success (data-based decision making).
- Create positive system-wide change in which school counselors serve as leaders in using data as a tool for supporting students’ social-emotional, academic, and behavioral development.
We will return to these expected results for the grant in the “Evaluate a CBA” section later in this document to summarize the actual results.
Expected Student Results
In general, the grant sought to improve students’ learning processes and learner outcomes by achieving the results specified for the grant, and by the way school counselors’ supported student development, learning and achievement. The grant also articulated specific results students were expected to achieve. These took the form of the indicators in the Protective Factors Index (PFI). The indicators articulated what was expected of students.
Student progress in relation to each indicator was rated by teachers at the end of each grading period (three times a year) and the results input into the district’s student information system (SIS) wherein they were capable of being juxtapositioned with students’ academic data. The ratings were incorporated into the district’s new standards-based report card so that academic, social-emotional and behavioral data was all available on each student’s report to enable counselors, teachers and data teams to make more accurate and informed decisions regarding students’ educational progress and pathways.
The importance of making academic, social-emotional and behavioral data available in one place, and on an ongoing basis, cannot be understated. It provided educators with an opportunity to see the complex relationship of these three types of data, how they impacted each other, and provided a more in-depth understanding of each student’s strengths and weaknesses. This deeper understanding of students enabled them to better determine which interventions would be most effective in addressing students’ learning needs.
The PFI indicators can, and should, be viewed as student standards of excellence. Just as the content areas all have academic standards which students are expected to achieve, the counseling program has social-emotional and behavioral standards which students are expected to achieve. Data regarding student progress in all three are needed to make informed decisions about students’ educational progress and pathways.
Students are expected to become proficient in each indicator as indicated on the PFI section of the student report cards. Here are the CBA constructs and primary PFI indicators which serve as student standards for social-emotional and behavioral development.
- Engages in class activities
- Exhibits positive and optimistic behavior
- Demonstrates and eagerness to learn
- Demonstrates perseverance in completing tasks
- Identifies academic strengths and abilities
- Identifies things he/she is interested in learning
- Demonstrates the ability to self-regulate actions and emotions
- Demonstrates resilience after setbacks
- Makes productive use of classroom time
- Works collaboratively in groups of various sizes
- Seeks assistance when necessary
- Respects and accepts authority
- Forms respectful, equitable relationships with peers
In addition, a PFI Scoring Rubric with measurable competency statements related to each indicator was created to guide teachers in the rating process. This helped ensure consistency in scoring across classrooms and grades. The competency statements should also be considered as results students are expected to achieve.
As noted above, a CBA is also concerned with excellence in other areas of school counseling. These are briefly discussed below.
The Grant Team defined results they expected to achieve in terms of the quality of the school counseling program. One important program result was the delivery of meaningful learning opportunities design to help students achieve the PFI standards and associated competencies in the PFI Scoring Rubric. These are discussed in the “Develop Learning Opportunities” later in this document.
A second type of results associated with program quality involved the MTSS set of interventions. The determination of students’ needs had to be linked to PFI standards and age and developmental appropriateness. Tier placements and the specific types of interventions needed to be determined by reviewing academic, as well as social-emotional and behavioral, data. School counselors needed to play a prominent role in data discussions and the determination and delivery of MTSS interventions.
It is important to remember that even though a well-articulated set of student and program results are essential to a successfully delivered school counseling program, that having these define cannot guarantee success. A well-designed and implemented support infrastructure is required. Critical components of a reliable support infrastructure, as discussed later in the section on “Implement a CBA” include critical areas such as roles and accountabilities, policies and protocols, integrated data management systems, fluid communication networks and professional development.
Counselor Practice Results
Results were also defined for school counselor performance. The Grant Team emphasized that building human capacity was required for a successful grant program. Many of these results can be gleaned from the types of professional development that were offered to counselors and other members of the school community. In particular, counselors were trained in the effective use of data-based decision making and in becoming leaders in data discussions. This is further discussed in the “Implement a CBA” section.
Develop Learning Opportunities
The grant’s student standards (PFI indicators) focused on social-emotional and behavioral competencies. Once student standards have been defined, it is the responsibility of educators to provide meaningful learning opportunities to help students understand what is expected of them, how to achieve what is expected of them, and how they will be assessed.
As part of their learning experiences, students become aware of the importance of becoming proficient learners and developing as mature and responsible individuals capable of regulating their emotions and behaviors and contributing to the well-being of their classmates and members of their school community.
The Grant Team provided a meaningful set of learning opportunities that were transformational in nature. In keeping with its desire to be research-based, the Team selected curriculum programs that research has determined to be evidence-based. Curriculum offerings were delivered under the title “Success Classes.” School counselors also created their own curriculum (10 lessons per grade) which was based on evidence-based practices (e.g., Zones of Regulation, Second Step). Success Classes were delivered to all students. Other learning opportunities included:
- Summer Boot Camp Transition Program: a series of four free week-long summer boot camps that was open to all district fifth graders. Curriculum content and activities were drawn from two evidence-based programs, Student Success Skills and The Real Game. Topics included perseverance, organizational skills, and study strategies that are critical to success in middle school.
- Behavior Improvement Plans (BIPs) : plans developed by school counselors for students in need of intensive behavioral support in the classroom. The counselors also coordinated referrals with the special education department and/or other mental health professionals when needed.
- Character Trait of the Month: a school-wide intervention focused on the development of positive qualities such as respect, honesty, and courage.
- Parent/Guardian Newsletter: a newsletter compiled and distributed by the counseling department with suggestions on home-based activities to enhance the development of social emotional competencies. Topics included informing parents/guardians of the Character Trait of the Month, suggesting ways to initiate conversations with their children about what they are doing in school, and recommending books to read with their children.
- Expo: an annual event held at the school at the end of each school year. Parents/guardians were invited to attend and see artifacts their children had created in their Success Classes.
Curriculum was delivered in three settings which correspond to the district’s MTSS program: a) Tier 1: all students, Tier 2: small groups and c) Tier 3: intensive interventions primarily with individual students. MTSS is further discussed in the “Implement a CBA” section later in this document.
As is obvious from the description of the learning opportunities, students were provided a rich variety of opportunities to learn what was expected of them.
Develop Student Assessments
Building Block 1 (Define Student Excellence) was used to develop the standards and competencies students were expected to achieve. For the grant, these consisted of the PFI indicators and associated competencies in the PFI Scoring Rubric. Building Block 2 provided opportunities for students to learn what was expected of them and how they could achieve the standards. Building Block 3 provides the means to measure students’ progress, proficiency and achievement.
Major Changes Required Related to Assessing Students
Major changes related to student assessment data needed to be made to create a data-rich and data-decisive environment that maximized the potential for effectively using data to determine students’ needs, determine appropriate interventions and assess the impact of delivering the interventions. Critical changes included:
- Student assessments determine the progress students are making toward the results they are expected to achieve. Student results (standards and competencies) against which student progress could be measured) had to be defined.
- The PFI and Scoring Rubric needed to be developed as assessment tools. The rubric was organized into three levels (K-1st, 2nd-3rd, and 4th-5th grades) to ensure developmental and age appropriateness, and to help ensure accuracy and consistency in teacher ratings across classrooms and grades.
- A new and expanded electronic ODR form needed to be created to replace the manual form that was in use.
- Data collection and processing protocols needed to be changed to ensure that required social-emotional and behavioral data could be efficiently and accurately gathered in a timely fashion.
- The district’s student information needed to be upgraded to allow for inputting, processing and reporting of social-emotional and behavioral data. Expand he technology platforms to handle the new data requirements and ensure data literacy among members of the school community.
- The student report, needed to be changed to incorporate the PFI indicators and students’ teacher ratings across the three grading periods.
[BT1Dash]The role of school counselors needed to be expanded, including becoming official members of data teams and leaders in data discussions.
- Data-based decision making needed to become the methodology everyone uses to make informed decisions.
- Counselors, teachers and other key personnel needed to be trained in how the new changes operated and what was expected of them in terms of their roles and accountabilities. Specifically, school counselors needed to be trained in data-based decision making and in how to become leaders in data discussions that involved determining students’ need and deciding on appropriate interventions. Parents/guardians also needed to be trained in understanding the new report card and how they could help their children in their social-emotional and behavioral development.
As is obvious from this long list of changes required to assess students based on academic, as well as social-emotional and behavioral data, is a complex but worthwhile process. It should be remembered that these changes were made over a three year period and continue to be refined even though the grant has ended.
Plan for a CBA
There is no substitute for effective front-end planning that includes the following processes: goal setting, action planning, plan implementation, progress monitoring, timely corrective action, evaluation and using evaluation results to update and/or create new plans. Although the article did not specifically address the planning component of the grant, it is obvious from the results achieved that significant attention was dedicated to planning activities.
For example, planning was involved in developing the grant proposal and outlining the primary strategies and action steps that would be required to successfully complete the grant program. In order to comply with federal requirements, goals needed to be defined along with how the results would be measured. Required annual status and final evaluation reports relied heavily on using what was planned to conduct the evaluations.
In addition, the number of processes to be implemented and products to be produced required significant planning. For example, professional development opportunities for school counselors and teachers, processes for collecting social-emotional and behavioral data, inputting of data, updating the student information system, delivery of curriculum to students, meetings to attend (e.g., MTSS, RTI, data teams), an annual Expo and four summer boot camps to run, plus production of a parent/guardian newsletter all needed to be planned and implemented in a timely fashion in order to ensure the success of the grant program.
Implement a CBA
The “Deliver a CBA” module on our website divides the implementation effort into two main areas: a) delivering challenging school counseling learning opportunities to students and b) establishing a reliable support infrastructure. The learning opportunities delivered by school counselors as part of the grant program are discussed in the “Develop Learning Opportunities” section earlier in this document. These multi-faceted opportunities, delivered in response to PFI and ODR data, were delivered in the MTSS three-tier intervention framework to help students mature and act responsibility throughout their social-emotional and behavioral development.
Establish a Reliable Support Infrastructure
Just as MTSS is a system of supports for students, a reliable support infrastructure is a system of supports for the school counseling program and the existence of a school counseling environment conducive to student achievement, development and success in school and life. What makes it reliable is that the supports are consistent and can be depended on to help the counseling program achieve its desired results. A support infrastructure has multiple components, each contributing to the overall stability and effectiveness of the school counseling program. The following components, emphasized on our Website, are discussed below in terms of how they were used in the grant program.
Roles and Accountabilities
Roles and accountabilities are critical to the success of any program. The grant program and school district had many stakeholders and constituent groups that interacted to design, plan and implement the elementary school counseling program. Each constituent group, and individuals with each group, had specific roles they were expected to perform, and the success for which they were held accountable.
The expected results of the grant program could not be fully achieved unless everyone was aware of, and embraced, their roles and held themselves accountable for delivering on what was expected of them. Without this level of awareness and commitment, program activities quickly devolve into loss of direction and an ongoing need for crises management. The Grant Team ensured clarity in role definitions, nurtured collaborative interaction and decision making, and honored input and feedback from all stakeholders.
A primary goal of the program was to expand the role of school counselors by getting them assigned as official members of data teams, trained as experts in data-based decision making and leaders in data discussions, and recognized as essential participants in discussion involving students’ needs and appropriate MTSS interventions. These results were accomplished and school counselors are now recognized as having valuable insights and expertise involving data-based decisions and decisions regarding appropriate interventions.
Policies and Protocols
Well-considered policies and clearly articulated protocols were equally as important as clearly-defined roles and accountabilities in the successful implementation of the grant program. Policies are the rules that govern the school counseling environment. Protocols are the processes involved in implementing the policies. Some policies are approved by the district’s school committee or mandated from the State or Federal level. A second category of policies consists of beliefs and ways of doing things that have the power of a policy because they have been established as common and accepted practice.
As an example of an approved policy, the district approved a new policy which mandated that counseling groups be formed based on issues identified in the data and no longer simply by teacher request or anecdotal evidence. An example of the second category is the collection of social-emotional and behavior data for use in the MTSS. This process was not adopted as policy but became commonly accepted practice, therefore having the power of a policy in daily practice.
Protocols consist of processes conducted in concurrent and sequential steps. MTSS, RTI, data-based decision making and decisions about appropriate interventions are all protocols, and in any systemic reform initiative, need to be reviewed and changed to make them more efficient and increase their capacity to achieve positive results.
Two major changes that required updated protocols were collecting social-emotional data via the PFI and behavioral data via the ODR. In the case of the PFIs, no protocol existed and needed to be created. In the case of the ODR, data had been collected manually and needed to be converted to a protocol that allowed the data to be entered into the district’s SIS and available electronically along with social-emotional and academic data.
Correct protects, the proper way to do things, are housed in organizational structures such as data teams. Reform initiatives often require a restructuring of these units. A prime example in the grant program is the need to restructure the data teams to make school counselors official team members. This one change allowed the team to benefit from counselors’ experiences and insights which in turn help improve the district’s MTSS. The expanded types of data collected allowed for a more comprehensive, in-depth examination of the relationship between academic, social-emotional and behavioral factors influencing student outcomes. The MTSS became more nuanced and allowed the district to respond to a wider range of issues related to students’ development and learning needs.
The introduction of school counselors to data teams changed the way the team processed data. Social-emotional and behavioral data, in conjunction with academic data, allowed school counselors to continuously monitor students’ social-emotional and academic progress. It also enabled counselors to easily evaluate if their interventions were creating the desired impact by giving them the ability to access this data in real time.
Information Management System
The efficient production, processing and distribution of information are fundamental to the success of any endeavor. Information must be produced in meaningful formats, organized and disseminated in a timely fashion to those who need it to make informed decisions. Information-related processes must be well-designed and effectively managed.
Our website discusses the need for a school counseling data management system (SCDMS) to manage all the information needed to fully implement a comprehensive school counseling program. A SCDMS is a sub-system of the district’s more comprehensive information management system. The SCDMS must be integrated with other sub-systems such as the district’s SIS and data processes required by the MTSS.
Information is a system and therefore must be approached systematically. The district’s information system prior to the grant was not capable of producing and generating the data needed to implement the grant. Systemic changes needed to be made. For example, the district did not systematically collect social-emotional data. The PFI was developed as an assessment tool, along with protocols for collecting the data, entering it into the SIS and generating reports (e.g., student report cards, responses to queries that extracted information from the SIS for data analysis purposes). As noted above, the ODR process also needed to be revised. Both these changes required adding new functionality to the SIS to accommodate the data and make it available.
The information system also required changes in how data was used. Adding new functionality to the SIS made it possible to house academic, social-emotional and behavioral data in one location and generate reports that allowed people to explore the relationship of these three critical types of data, and make more informed decisions about addressing students’ needs based on the expanded database available. A critical change that rendered changes in the SIS to benefit students was the inclusion of school counselors on data teams so they could contribute their expertise and insights about students’ social-emotion and behavioral development to data team members and help them make more informed decisions about appropriate interventions.
Fluid Communication Network
An important part of an integrated information system is a fluid communication network. Vital information needed to make informed decisions must be produced and get to the people who need it. The flow of information must be fluid and arrive when it is needed to maximize the potential for using the data to benefit student learning. Upgrading the SIS allowed real-time access to vital information by a wide variety of members of the school community. For example, PFI data which had not been available became available three times a year at the end of each grading period, and any time during the year once it was posted by querying the SIS.. ODR data, which had not been available electronically, was updated weekly always available.
This access, along with attendance data updated daily, allowed for much faster and more accurate decisions about placement of students in the MTSS tier structure and which interventions would be most effective. This data also provided ongoing information on the impact of delivering the interventions by reviewing changes in the data collected over time.
Relevant information in a timely fashion is required for effective decision making processes. The Grant Team ensured that opportunities were made available to constituent groups within the school community to provide input and feedback relating to decisions the Grant Team needed to make. For example, the Team wanted to ensure that each PFI indicator reflected competencies considered relevant and sensitive by staff and families. A representative group of school counselors, teachers from each grade level, a teacher of English Language Learners, a special education teacher, and the principals from each elementary school reviewed the 15 original PFI items for developmental appropriateness and cultural sensitivity.
The group expressed concerns about two indicators under the Self-Knowledge construct (“Identifies personal feelings” and “Identifies personal strengths and abilities”). The review group felt that these indicators involved attributes valued more by the dominant culture, and that with the diverse student population, some families might consider benchmarking their children against culturally-specific standards was not fair. As a result, these two items were omitted from the grant version. This ability to have fluid processes to gather important perspectives helped make the PFI sensitive to local interests and concerns.
Another example of gathering information from constituencies involved the rating scale teachers used in the PFI assessment tool. In Year 1, teachers had two responses to choose from: “struggling” or “on target.” In Year 2 teachers requested four possible responses: “meets standard,” “progressing towards standard,” “emerging,” and “not meeting standard” in order to provide a more specific response resulting from their observation of students’ behavior. It is critical to the success of a CBA and the PFI that ongoing input and feedback be gathered and honored.
In systemic change, everyone needs to be trained, primarily because so many things are structured and/or done differently. The Grant Team realized that the various stakeholders in the grant process had informational and skill-related needs, and that these needs had to be addressed in order to successfully implement the grant program.
Primary beneficiaries of the professional development program were school counselors and teachers. Building human capacity for change became an ongoing priority. This was needed to ensure that individuals and groups in the school community could perform their roles and accountabilities efficiently and productively. Monthly professional development workshops were provided to school counselors. Topics included:
- Data-based decision making
- Data analysis and Excel graph generation
- Evidence-based practice
- MTSS tiered interventions
- Report generation and queries from the SIS
- Determination of student issues from review of academic, social-emotional and behavioral data
- Aggregated and disaggregated trends in student performance and behavior
- Leading data discussions in data team meetings
- Data process monitoring
- Evaluating the impact of delivering interventions on students’ development, learning and achievement
Teachers received training in data-based decision making and use of the teacher-rated PFI assessment tool and Scoring Rubric.
Evaluate a CBA
Expected Results vs. Actual Results
The grant process began with defining the results to be achieved and then developing action plans to achieve them. Building Block 6 focuses on evaluating the extent to which the expected results were achieved and what next steps would be required to improve the implementation process. To accomplish this required a clear understanding of what the expected results meant in terms of actual changes, in short how the program at the end of the process looked compared to where it began.
As discussed in the “Define Student Excellence” section earlier in this document, different types of results were identified as being critical to the successful implementation of the grant. These included: a) results for the grant, b) student results, c) program results, d) organizational results, and e) school counselor performance results. The Grant Team was required by the U.S. Department of Education to conduct ongoing evaluation, submitting annual status reports and a complete evaluation at the conclusion of the grant.
This section will examine the actual results achieved in the four elementary schools during the three-year grant cycle, and additional results that impacted systemic changes at the district level. It will answer the all important question: “Did the grant achieve what it set out to accomplish?”
Actual Results for the Grant
Determining actual results begins with the expected result statements defined at the beginning of the process. For example, the grant set out to accomplish the following:
- “Increase the student-to-school counselor ratio.” As the district had no elementary counselors prior to the grant, this federal priority was achieved.
- “Reduce the number of disciplinary-related incidents (as indicated in documented Office Discipline Referral (ODR) forms.” As a result of the grant, there was a significant reduction in the number of incidents.
- “Identify social-emotional and behavioral competencies strongly related to student achievement and well-being that were within the purview of school counselors.” This was achieved by basing the elementary school counseling program on the four research-based CBA constructs and development of the Protective Factors Index and Scoring Rubric.
- “Develop the infrastructure for collecting important social emotional indicators to use along with academic data in a Multi-Tier System of Supports (MTSS).” This was achieved through use of the PFI as the assessment tool to collect social-emotional data three times a year at the end of each grading period. The data, along with disciplinary-related behavioral data, was entered into the district’s student information system and was reported, along with academic data, on student report cards. The data collection and processing infrastructure created a single electronic source of academic, social-emotional and behavioral data with easy and real-time access to school counselors, teachers, data teams and anyone involved in the MTSS.
- “Expand the role of school counselors in determining and delivering interventions”. As a result of the grant, school counselors were made official members of school-based data teams responsible for evaluating student data and determining appropriate MTSS interventions. As official members of the data teams they were able to make significant contributions to data discussions about students and offer their insights and expertise regarding appropriate interventions.
- “Build the capacity of school counselors to become leaders in data-driven discussions about student success (data-based decision making).” School counselors were trained in data-based decision making and in how to become leaders in data discussions. By offering their expertise in addressing issues related to social-emotional and behavioral development, they were able to help data team members and other educators come to a deeper understanding of how academic, social-emotional and behavioral data interact, and how to apply that understanding to make more informed decisions about how to best support student development and learning.
- “Create positive system-wide change in which school counselors serve as leaders in using data as a tool for supporting students’ social-emotional, academic, and behavioral development.” The success of the grant program at the elementary level had a positive effect on the secondary level as well which also expanded the role of the school counselor and more fully embraced data-based decision making processes. The district is now exploring how to customize and implement the PFI and Scoring Rubric in the middle and high schools.
- The grant also contributed to helping the district overcome an “under-performing” designation by the State Department of Education. The State mandated an annual Accelerated Improvement Plan (AIP). A number of elements from the grant program were embedded in the annual reports. The positive impact also reaches beyond the district. The State Department of Education, in recognition of the success of the program, now recommends that other school systems visit the district to learn how they can benefit from a CBA and PFI approach to designing, delivering and evaluating comprehensive school counseling programs.
Limitations and Directions for Future Research
The primary intent of the grant program was to implement a new elementary school counseling program and achieve the results discussed herein, rather than to conduct a formal experimental study. Therefore, it cannot at this time be established with certainty whether the implementation of the grant initiatives and subsequent positive outcomes share a causal relationship. In addition, it is not at this time possible to know the specific elements of the program that yielded the most positive results. For now, however, it is good to know that this initial program evaluation showed positive results from all grant-related activities.
Future research involving an experimental study will be needed in which: a) outcomes are compared to similar schools that did not receive similar resources; and (b) there are outcome measures in place for each grant initiative. Further, additional studies will be needed that expand these efforts to students and schools in different regions, grade levels and with a higher number of participants.
This evaluation concluded that the PFI is a promising new instrument for measuring social-emotional behaviors in elementary classroom, but noted that additional research is needed to validate its use as a universal brief screener for elementary schools, or for middle and high schools where less face-to-face contact between students and the teachers who rate them may impact the instruments reliability and validity.
Further research will be necessary to establish the most applicable social-emotional learning theories to school settings, including research of the Construct-Based Approach (CBA) to school counseling which is still in its early stages.
We set out to demonstrate the power and potential of using the six CBA Building Blocks for Excellence as a substantive framework for improving school counseling programs, counselor practice and learner outcomes. The graphic below identifies some of the most important lessons learned from this grant program that can help you develop and successfully deliver a CBA in your school and district.
The CBA and PFI can help improve the support we provide to students’ academic achievement and well-being, especially in the areas of social-emotional and behavioral development. A CBA can help make school counseling programs essential to student support and school improvement initiatives and will help school counselors to become recognized for their expertise, insights and contribution to student success.
Whatever aspects of a CBA you wish to pursue, always remember that positive change can be guaranteed when the effort is approached by everyone as a community of dialog, reflection and collaboration.