Program Evaluation

Monitoring the success of your school-based telehealth (SBTH) program allows you to track and measure achievement of the program goals. Ongoing process evaluation and quality improvement efforts focused on the program model, workflow, and processes are adjusted to ensure the program is effective and efficient. Evaluation plans will evolve as the program matures and utilization increases.

Key Considerations

Back to Top

Before designing your evaluation plan, reflect on meaningful measures with funders, institution, community partners, and program leaders. Consider your program design to assist with shaping the evaluation plan by revisiting the following questions with your evaluation staff member or team: 

  1. What needs are we trying to meet?  
  2. What is the goal of the program? 
  3. What scope of services are we providing?

Defining outcomes relevant to the program, stakeholders, and stage of maturity is an integral part of SBTH program development (see Assessing Needs and Readiness and Program Models for details). In the earliest stages of development, identify the needs the program seeks to address and define them in a quantifiable way.1 This sets the stage for further defining process and outcome metrics—outlined below in Figures A and B—that are most relevant to partners (hyperlink to Partnerships) such as students, caregivers, providers, school officials, payers, and other funders. These metrics of success aid progress tracking and guide the development, trajectory, and growth of the SBTH program. Figure A illustrates examples of such metrics during each stage of SBTH program maturity.

  • Starting with Stage 0, define the program in a measurable way.  
  • Moving into Stage 1, the pilot phase, workflow and process measurement refinement take place. As your SBTH program becomes operational, the collection of process metrics such as enrollment and utilization data can improve marketing and school engagement practices.  
  • Once the program reaches the anticipated visit volume, progress to Stage 2 to measure student, provider, telepresenter, and caregiver experience and satisfaction. As the program matures, monitor the quality of care significant to stakeholders. This may include technical quality (e.g., quality of connection, wait time) or clinical quality (e.g., decrease in ADHD symptoms, symptoms associated with childhood trauma). Both metrics are essential to explore as you establish the program and increase utilization.  
  • Finally, as the program reaches maturity and serves a larger quantity of students, adopt a population health approach in Stage 3 to help your SBTH team determine the cost-effectiveness and value-based outcomes in the health and educational space. Applying this type of framework can support SBTH programs in matching their metrics and evaluation approaches with the scope and scale of their programs.

Once you establish the program and the evaluation metrics, it is crucial to engage in continuous quality improvement. Figure B provides a framework for developing ongoing assessment measures of operational effectiveness, including continuous program evaluation and monitoring process measures to shape improvement decisions. One option to measure partners’ experiences is via a Net Promoter Score (NPS) survey. Based on a single question, the NPS asks, “How likely is it that you would recommend our program to a friend or colleague?” Participants’ answers range from 0 (not at all likely) to 10 (extremely likely).

Since technical quality and efficiency of care are essential to all parties engaged in telehealth, identifying an efficiency metric can further refine your SBTH program’s workflow and increase participant engagement. For example, consider tracking the time from the appointment request to the provider’s connection. Likewise, you can measure technical reliability by the number of times a visit could not occur due to equipment and/or software failures. Another important measure is tracking the frequency that workarounds are used, to ensure a successful telehealth visit as part of the technical reliability measure. Finally, as with SBHCs, a crucial part of the continuous quality improvement process is identifying diverse funding streams to support the program’s sustainability.

 

Examples from the Field

Back to Top

Students in 7th grade or above can complete the telehealth patient satisfaction survey. Keep this in mind when planning where to implement telehealth services. For example, if the program is offered in an elementary school, student survey data will be lacking. In the case of Michigan Schools, workflow improvements were designed with the medical director and staff input and then placed into policy format for consistency. When an additional clinic was brought on in year two, identified issues were avoided by revisiting lessons learned. All new employees were required to complete a thorough orientation and policy review.  

Incorporating the online risk assessment or post-visit satisfaction survey into the initial workflow was identified in the Continuous Quality Improvement Process (CQIP) in Michigan Schools. In replicating this model, it is recommended to bring in these required surveys early in your process making them part of the routine of care. These pieces help gain revenue through process improvement. 

The MUSC SBTH program began in 2013 with three schools and now supports over 100 schools. The program’s goal is to partner with the medical home to ensure students receive the care they need to remain healthy and able to learn. This SBTH program provides acute and chronic care management. All students in a partner school are eligible for services, provided the caregiver has completed a consent form to participate in the program.   

When a student presents to the school nurse with a chief complaint, the nurse completes her regular assessment. When the nurse determines that a provider should evaluate the student, they can offer SBTH as an option. Once connected, the visit takes place as it would in a regular clinic setting. The telehealth cart incorporates peripheral devices including a digital stethoscope, otoscope and exam camera. The school nurse serves as the telepresenter and operates the technology with the guidance of the connecting provider. After a thorough history and physical exam, the diagnosis is made and the provider sends prescriptions to the local pharmacy. If the caregiver is not present or participating in the visit by phone or video, they are contacted afterward. A visit summary note is then sent to the medical home. Students then return to class unless excluded based upon school district guidelines. 

MUSC employed a population health approach to determine the associated rates of emergency department (ED) use among students with asthma in their Williamsburg County STBH program. Medicaid claims data analysis determined that rates of ED use among all students with asthma and access to the program significantly decreased throughout the SBTH program’s existence. This outcome emphasizes that the association extends beyond the students directly served, indicating a likely “spillover effect” to all students in the county.Notably, there was an approximate 21% reduction in ED visits among students with asthma in the first two years of the program and a 35% reduction in the third year.

Resources Spotlight

Back to Top

References 

  1. Garber, K., Wells, E., Hale, K., & King, K. (2021). Connecting Kids to Care: Developing a School-Based Telehealth Program. The Journal for Nurse Practitioners. https://doi.org/10.1016/j.nurpra.2020.12.024 
  2. Harvey, J., Ford, D., King, K., Beeks, R., Kruis, R., Simpson, K., & McElligott, J. (2021). Telehealth Outcomes Research: Show Me the Data. Telemedicine and e-Health. http://doi.org/10.1089/tmj.2020.29048.abstracts 
  3. Medical University of South Carolina (2020). Telehealth Service Implementation Model (TSIM™): A Framework for Telehealth Service Development, Implementation, and Sustainability. Retrieved from: https://muschealth.org/-/sm/health/telehealth/f/tsimsummarycoedocument2021.ashx?la=en 
  4. Reichheld FF. (2003). The one number you need to grow. Harvard Business Review. Retrieved from: https://hbr.org/2003/12/the-one-number-you-need-to-grow 
  5. Bian J, Cristaldi KK, Summer AP, et al. Association of a School-Based, Asthma-Focused 
  6. Telehealth Program with Emergency Department Visits Among Children Enrolled in South Carolina Medicaid. JAMA Pediatrics. Sep 9 2019;173(11):1041-8. doi:10.1001/jamapediatrics.2019.3073