“Evaluate a CBA” Overview
Click on a topic below to go directly to that section.
Welcome to “Evaluate a CBA”
Research vs. Evaluation
Four Types of CBA Evaluation
Data-Based Decision Making
Three Key Evaluation Questions
School Counseling Data Management
School Counseling Accountability Reports
Links to CBA Evaluation Templates
Welcome to “Evaluate a CBA”
A discussion regarding evaluation of a CBA begins with an understanding of what a school counseling program is and the results that can be expected from delivering it. The school counseling program is a coherent subsystem of a school that includes: 1) an organized set of interventions, prevention-oriented activities and services that are designed to achieve student-learning goals, and 2) professional decision-making structures that facilitate the planning, management and use of counseling program components.
Evaluation is essential to the successful delivery of a CBA. In order to accomplish this, counselors need to be able to determine what is working, what is not working, and how to continuously improve the program and its outcomes. CBA program results are defined in vision and mission statements and in various plans (e.g., strategic, annual, counselor-supervisor agreements) that are developed to define and guide the delivery process. These plans identify the results to be achieved, actions to be taken, ownership of action steps, a timetable for completion, and how success will be measured. CBA evaluation processes are systematic and well-constructed, and generate meaningful data to understand what is working and what needs to be improved.
CBA program evaluation is like going to a doctor’s office to find out what can be done to address health-related problems, in our case the health of the school counseling program and school community. The doctor conducts tests to gather evidence of what might be the causes of problems, forms a diagnosis and prescribes specific things to do to address the problems. The problem may be minor (e.g., a minor cut) and require only a simple solution like a band aid. At the other end of the spectrum, major surgery or radical treatment may be needed if the problem is life-threatening. Program evaluation, in addition to diagnosing problems that are treatable can also identify policies and protocols that are no longer functioning, cannot be resuscitated and need to be laid to rest. Three factors influencing this approach are: a) the diagnosis must be accurate, b) the prescribed course of action must have the capacity to resolve the problem, and c) what is prescribed must be followed as instructed.
The “Evaluate a CBA” section is intended to help you get started on establishing evaluation processes and instruments that will allow you to assess the status of your current counseling program, gather input and feedback for constituents groups involved in the program, generate meaningful data needed to analyze and continuously improve your program, and report the accomplishment and challenges of your program to others.
Program evaluation is addressed in Chapter 10 of our book Achieving Excellence in School Counseling through Motivation, Self-Direction, Self-Knowledge and Relationships, and in Construction Site 6 of our CBA Toolkit. This section is not intended to replicate these materials. Rather the contents of this section are intended to reflect our thinking since the publication of these works. Templates included on the Website that were provided in the book or toolkit are a refinement of the original versions. Not all templates in the CBA Toolkit are included here. Over time, additional tools and templates related to CBA program evaluation will be added to this section.
A link with ordering information for our book and toolkit, published by Corwin in 2014 and 2015 respectively, is provided in the Side Bar.
Research vs. Evaluation
Concise definitions of key terms are critical to understanding program evaluation, otherwise the discussion suffers from a lack of clarity and confusion. The first two terms it is helpful to define are “research” and “evaluation.” The graphic below identifies distinguishing characteristics of these two terms.
School counseling research is typically conducted by members of the higher education community, national school counseling organizations and research centers, and/or government agencies (e.g., via grants). As noted, research studies cost a lot of money, take a lot of time to complete and are very rigorous in their methodology.
For examples of how researchers approach school counseling, visit the “Ronald H. Fredrickson Center for School Counseling Research and Evaluation (http://www.umass.edu/schoolcounseling/), and review any school counseling research journal. Or, use search engines to find websites addressing school counseling research studies and results. Examples of research can include research briefs, meta-studies and program/outcome evaluations. Some addition research-focused websites are listed in the “Free Resources” module.
Evaluation, on the other hand, is conducted at the local level primarily by districts, schools and individual counselors. Since this website is intended primarily to help counselors at the local level get started in developing and delivering a CBA, this section will focus on evaluation rather than research. See the “Defining Student Excellence” section in the “Develop a CBA” module for a discussion of the research-base for a CBA and how it relates to school counseling.
Four Types of CBA Evaluation
The next set of definitions further unpacks the notion of evaluation into four primary types of evaluative processes that are required to determine program status, make improvements and achieve positive change. The graphic below identifies the four types of CBA evaluation.
CBA is results-based, that is, specific outcomes are expected to be achieved as a result of implementing a CBA school counseling program. CBA evaluation is a set of processes that determine the extent to which the specified results have been achieved and whether the counseling program has been delivered with fidelity. The four types of CBA evaluation are discussed below.
The mission of a CBA school counseling program is to ensure that students, as a result of participating in the program, are highly-motivated, self-directed learners who are knowledgeable about themselves and others, engaged in meaningful relationships, and developing as contributing members to society and the well-being of our world. This is accomplished by delivering a counseling program based on four research-based constructs that are strongly related to students’ academic achievement and well-being: motivation, self-direction, self-knowledge and relationships. Assessments are used to determine student progress, proficiency and achievement in relation to CBA student standards based on the four constructs.
See the “Develop Student Assessments” section in the “Develop a CBA” module for a discussion on the purpose and importance of assessing students and suggestions on types of student assessments that can be used as part of your CBA school counseling program.
The purpose of evaluating program effectiveness is to determine whether specific components and/or the whole CBA school counseling program is being fully implemented and achieving the specified program results. School counselors evaluate their counseling program from two perspectives: program component evaluation and whole program evaluation.
CBA Program Component Evaluation
CBA program component evaluation focuses on specific components of the school counseling program which are determined to be in need of improvement. Reasons for evaluating specific components can vary. For example, specific curriculum activities can be evaluated to determine their impact on students’ social-emotional development, or the impact of particular activities on students’ behavioral development as evidenced in Office Disciplinary Referrals (ODRs). An anti-bullying program can be evaluated to determine its impact on negative interactions students exhibit as they relate to each other.
Specific professional development opportunities as part of the school counselor professional development program, or the entire school counselor professional development program as one component of the overall school counseling environment, can be evaluated to determine to extent to which these opportunities are improving counselor practice and increasing their capacity to help students learn and achieve.
CBA component evaluation is an ongoing process, focusing on a prioritized list of components, activities and interventions for which data are required to determine need and appropriate ways to address the need. In all these examples, specific aspects, rather than the whole school counseling program, are the focus of the evaluation process.
A popular name for a methodology that focuses on educator-driven evaluation processes at the school level is called “action-research.” Although we feel that this is an important process in which school counselors can participate, the name does not align with our distinction between “research” and “evaluation.” This methodology is aligned with our definition of “CBA program component evaluation” and does not align with our definition of research. For our discussion of a CBA, therefore, our use of “CBA Component Evaluation” incorporates methodological considerations described in “action-research” literature.
CBA Whole Program Evaluation
Evaluation of the design and delivery of the whole school counseling program must also be conducted. For example, school counseling programs are an “every child initiative.” This differentiates school counselors from other professional support personnel (e.g., social workers, school psychologists) who only see some of the students. School counselors use a programmatic approach and deliver core learning opportunities to all students, primarily through a school counseling curriculum. This is a major factor in ensuring that school counseling programs are preventive in nature rather than relying only on interventions necessitated by emerging in individuals and sub-populations of students.
Whole program evaluation examines whether the program has been delivered to all students. In addition, counseling programs are designed to focus on students’ metacognitive and social-emotional development. Outcomes to be evaluated are whether these needs are defined and whether the activities and interventions that are applied in response to these needs have had a positive effect. Whole program evaluation is rigorous, time-consuming and costly, and is conducted every five to seven years to determine if the program is achieving its desired results and whether major changes are needed. It is not intended to identify short-term tactical maneuvers necessitated by temporary problems.
Program evaluation is based on multiple sources of data and information. The evaluation process is time consuming but worth the effort by generating useful information for program improvement. Here, more global measures of gains in competence and accomplishments are most useful. For CBA programs, student standards reflect the ultimate student-learning outcomes. Grade level competencies are the measureable indicators that these outcomes are being achieved. The percentages of students achieving competency in motivation; self-direction, self knowledge and relationship skills are indicators that a school or district-level program is achieving its expected student-learning outcomes.
Examples of whole program evaluation can include a longitudinal study of data related to student proficiency and achievement, delivery of an entire CBA school counseling curriculum, evaluation of the effectiveness of a complete CBA planning cycle, evaluation of outcomes of responsive services, and evaluation of the impact of the total professional development program. Results from evaluating all major components of the school counseling program are analyzed collectively so that required improvements can be identified and incorporated into the district’s school counseling strategic planning processes.
Organizational outcomes are related to a reliable support infrastructure that includes components such as: a) clearly-articulated roles and accountabilities, b) clearly-defined policies and protocols, c) a fluid communication network, d) professional development, and e) a community of dialog, reflection and collaboration. The extent to which key individuals and organizational units understand and embrace their roles and fulfill their responsibilities needs to be evaluated. In addition, the level of support by school and central office administrators, school committees and community-based organizations are all critical areas of support that need to be examined.
Planning is also a critical organizing function that requires evaluation. See the “Plan for a CBA” section in the “Deliver a CBA” module for a discussion of the importance of front-end planning, an understanding of a complete school counseling planning cycle and planning templates. CBA plans should clearly reflect the results of both program component and whole program evaluations. In other words, CBA plans should use the evaluation data to develop action steps that will reinforce program strengths and successfully address areas the data shows are in need of improvement. CBA component evaluation provides an ongoing feedback loop to monitor essential processes and outcomes and take corrective action as needed in a timely fashion.
There is no substitute for professional development in the implementation of new ideas and approaches to successfully delivering a CBA school counseling program. In systemic change, everyone needs to be trained/retrained. It is critical that the professionals who directly impact or are impacted by the CBA program have their professional needs identified and met. It is important to ensure the professional development opportunities are focused, relevant and meaningful.
Professional development is one of the most important components of a CBA. It provides an ongoing infusion of relevant knowledge, appropriate skills and guidance that improves program quality and counselor practice. The processes that comprise professional development activities must be continuously monitored and evaluated in the short-term to ensure that this vital vehicle for program improvement is achieving its desired results. We produce data on students so that adjustments to their learning plans can be made frequently and consistently. We need to do the same for the professionals who are responsible for the successful delivery of the school counseling program.
It is critical to determine deficiencies in professional development as they occur, and rectify them as soon as possible, so as to ensure continuously improving professional development that produces needed changes in the delivery of the CBA. Professional development is also a critical importance in the counselor performance evaluation system as additional training that supports the improvement of counselor practice is required in areas the evaluation process identifies as requiring improvement.
School Counselor Performance
Finally, there is a need to evaluate school counselor performance to determine proficiency levels as measured against counselor role definitions and job responsibilities. Such measurement activities point to areas in need of improvement and ensure that ways to ensure the needed improvement are scheduled.
School counselor performance evaluation is also an integral part of evaluating a CBA. Whereas CBA component and whole program evaluations are critical to determining program efficacy and whether the program is being delivered with fidelity, counselor performance evaluations help ensure highly-qualified professional school counselors to deliver the CBA and achieve its intended results.
The purpose of school counselor performance evaluation is to improve counselor practice and hence the quality of the school counseling program. Performance evaluation processes evaluate school counselors based on how well they perform their role and the activities required to help students achieve the CBA student standards and how well they contribute to implementing the counseling program with fidelity.
School counselors are expected to fully embrace the role of a professional school counselor, acquire and effectively apply relevant knowledge, develop and utilize appropriate skills, exhibit attitudes and behaviors that lead to success, display personal initiative and engage in collaborative interactions. They are expected to produce evidence that demonstrates their competence and proficiency.
Data-Based Decision Making
Data is information about a topic of inquiry that we organize in order to conduct analysis and make informed decisions regarding what is being studied. A CBA utilizes a Data-Based Decision Making (DBDM) approach in both student assessment and program evaluation. DBDM enables school counselors to determine need, select appropriate responses to the need, and determine the impact of implementing those responses.
An example of using DBDM in student assessments is to collect and analyze social-emotional data on students using the Protective Factors Index (PFI) which is discussed in the “Develop Student Assessments” section in the “Develop a CBA” module. The data is used to determine which students need what types of interventions to address issues in their social-emotional development. Based on the data, students may be placed in appropriate interventions in a district’s Multi-Tiered System of Supports (MTSS) in programs such as RTI or PBIS). Student progress can be determined by reviewing data collected in these programs. The PFI was developed specifically for the purpose of collecting social-emotional data related to the CBA’s four foundational constructs of motivation, self-direction, self-knowledge and relationships.
An example of using DBDM in program evaluation to gather and review data needed for school counselors to develop preventive learning opportunities for students (e.g., a school-wide need for lessons on conflict resolution. The data helps establish the nature of the problem and the types of topics and lessons that would be required to successfully address the need. Data can then be gathered while the school-wide learning opportunities are being delivered and analyzed to determine their impact and decide on next steps.
School counselors need to be proficient in the collection of data and in data-based decision making processes. We need to be leaders in data discussions and how to use data to select, deliver and evaluate the impact of appropriate activities and interventions.
Data has intrinsic value. It provides valuable information to shape school counselor practice and help us to better serve students. For this reason alone, data is important. But data is also important because counselors need it to communicate their contribution to student development and achievement.
There are still far too many people outside the counseling department who do not understand what counselors do. Some who influence program and budget decisions have negative perceptions about school counselors. Data provides us with a way to promote school counseling by providing compelling evidence that what we do has a positive impact on students’ academic achievement and future success.
Three Key Evaluation Questions
Another way of generating meaningful information that can be used in the evaluation process is to answer three key evaluation questions. The graphic below displays the questions that can help determine that the school counseling program is being delivered with fidelity.
The questions focus on three critical areas of a CBA that must produce positive results:
- The focus on student achievement helps determine the extent to which the program helps students achieve the CBA student standards, and its effectiveness in doing so.
- The focus on customer satisfaction helps determine the extent to which the needs of those involved in and impacted by the school counseling program are satisfied with the program’s goals, operation, performance and results. Critical constituent groups to be queried include students, parents/guardians, counselors, teachers, administrators).
- Finally, the focus on evidence-based practice helps determine the extent to which the program is supported by research and employs evidence-based interventions and practices that have been shown to lead to optimal outcomes for students. Program evaluation also examines whether the results claimed for the school counseling program are accurate and substantiated by data.
These are valid questions to answer in both program component evaluation and whole program evaluation. In both instances, the evaluation process is a systematic and well-planned collection of data from multiple sources that yield valuable information for program improvement. This process also yields valuable advocacy information since it provides answers to the question: “What is it that school counselors do that is of significant value to students and the school community?”
School Counseling Data Management
Fully implemented school counseling programs generate and report data to demonstrate the impact of implementing the program’s activities and interventions on student achievement and school improvement. Data requirements can be complex; therefore the data need to be managed in a systematic way. This can be accomplished by establishing a systematic approach to school counseling data management.
Requirements for the collection, processing, reporting and analyses of data, and measures for determining the extent to which specified results are achieved, must be incorporated into CBA strategic and annual planning processes. Ideally, an annual CBA data plan should be developed and ready to implement by the beginning of the school year. Otherwise, essential data processes have a way of slipping through the cracks and the quality of the data gathering process is compromised.
Data collection is central to both program component and whole program evaluation. Data gathered for component evaluation are used in the short-term to improve the CBA activities and interventions. This data is also organized and maintained to allow for a longitudinal understanding of how individual activities have impacted the overall quality of the program when an in-depth analysis of the entire school counseling program is conducted.
Data management can be a daunting task, given all the data-related activities required in schools. Gathering more and more data is often construed as an add-on that further burdens overworked counselors and takes valuable time away from students. The demands for data to demonstrate accountability are increasing. It is critical that we develop effective and efficient ways to approach data related to school counseling. The approach must be systematic and focus on the production of meaningful data (data that can be used to drive decisions). Such an approach must be a central component of a CBA school counseling program.
School Counseling Accountability Reports
School counselors are expected to demonstrate their accountability in terms of achieving the specified results for the school counseling program, and demonstrating their proficiency in fulfilling their role descriptions and job responsibilities. School counseling accountability reports serve the purpose of providing compelling evidence that the counselors are holding themselves responsible by substantiated their accomplishments by data and communicating an understanding of the challenges involved in achieving some results and additional steps that can be taken to overcome them.
Two basic types of accountability reports are important in a CBA: a) an annual school-based “School Counseling Accountability Report” produced at the end of each school year by the school counseling department and b) individual counselor accountability reports that demonstrate how they are fulfilling the requirements of the district’s support staff/school counselor performance evaluation system.
A template for the school-based accountability report is provided below in the “Links to CBA Evaluation Templates” section below. A template for “School Counselor Performance Evaluation Report” is under development.
Links to CBA Evaluation Templates
Several templates are provided to help you evaluate a CBA. A “pdf” file containing Instructions, and a MS Word file (docx) for you to input your information, are provided for each Template. Follow the links below to access the Instruction and Template files.
CBA Readiness Survey
This survey is designed to help you determine the extent to which you are ready to implement a CBA in your school and district. Survey results are used in both the planning and program evaluation processes. Action steps can be developed to reinforce or strengthen processes you have already met. They can also be written to improve processes which require further development.
High School Student Program Review Survey
It is important to know what students perceive about the school counseling program and its impact on their learning, achievement and lives. The data gathered can be used to identify strengths and weaknesses of the program and take corrective action to make improvements related to student needs. This template gathers perception data from high school students.
CBA Meeting Minutes
Meeting minutes are a valuable source of information that can be used in the CBA evaluation process, especially in terms of a chronological record of issues discussed, decision made, measures applied and results achieved.
School Counseling Accountability Report
Counselors are responsible for demonstrating their progress toward and achievement of expected results for the school counseling program.CBA evaluation data establishes the accomplishments and challenges of the CBA school counseling program. Once data is collected and analyzed, it needs to be reported. This template allows school counselors to generate a report on an annual basis that provides compelling evidence of counseling program accomplishments substantiated by data.