Using Digital Dashboards to Implement and Evaluate Programs

In response to the higher rates of suicide among farmers and limited access to healthcare and mental health services, the Farm and Ranch Stress Assistance Network (FRSAN) was developed to address the rising mental health crisis in farming communities. This publication explores the need for mental and behavioral health services for farmers and rural communities and identifies the challenges in coordinating complex programming and implementation of services. The NexGen Program analysis creates a framework that helps to determine the most effective mental health intervention programs, packages the suite of tools together, and promotes the adoption of improved programs to meet the unique needs of agricultural producers and their allies. Missouri’s FRSAN evaluation program uses a continuous improvement evaluation approach, implementing an educational decision-oriented model to identify strengths and limitations in curricula, instruction, and delivery. Unique to this approach is the use of digital dashboards to streamline and define processes that create a simple path forward to support large multidisciplinary teams and multiple mental health training programs.

A Case Study of How to Use Digital Dashboards to Improve Program Implementation and Evaluation University of Missouri Team Identifies Challenges and Solutions in Coordinating Programming and Implementation of Mental Health Services

By Amy Patillo, Ed.D., Matthew Pezold, M.S. and M.A., and Karen Funkenbusch, M.A., University of Missouri

ATTRIBUTION

A Case Study of How to Use Digital Dashboards to Improve Program Implementation and Evaluation

ISBN: 978-1-955687-15-7

Publish Date: July 27th, 2022

Citations for this publication may be made using the following:

Patillo, A., Pezold, M., and Funkenbusch, K. (2022). A Case Study of How to Use Digital Dashboards to Improve Program Implementation and Evaluation (1 st ed). Kansas City: Extension Foundation. ISBN: 978-1-955687-15-7.

Producer: Ashley S. Griffin

Peer Review Coordinator: Rose Hayden-Smith

Editorial Support: Rose Hayden-Smith and Maria Pippidis

Evaluation Support: Paulina Velez Rocateli

Technical Implementer: Rose Hayden-Smith

Welcome to Managing Evaluation for Farm Stress Programs , a resource created for the Cooperative Extension Service and published by the Extension Foundation. We welcome feedback and suggested resources for this publication, which could be included in any subsequent versions. This work is supported by New Technologies for Agriculture Extension grant no. 2020-41595-30123 from the USDA National Institute of Food and Agriculture. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the U.S. Department of Agriculture.

For more information please contact:

Extension Foundation c/o Bryan Cave LLP One Kansas City Place

1200 Main Street, Suite 3800 Kansas City, MO 64105-2122 https://impact.extension.org/

TABLE OF CONTENTS

Attribution .............................................................................................................................................. 2 Table of Contents..................................................................................................................................... 3 About the Authors ................................................................................................................................... 4 Part One - Introduction................................................................................................................... 6 Abstract................................................................................................................................................... 6 The Challenge .......................................................................................................................................... 6 Part Two – Evaluating the Effectiveness of Mental Health Programs in Missouri ............................. 9 NCFRSACi: Audience Served ..................................................................................................................... 9 Unique Value Proposition ...................................................................................................................... 10 Why is NCFRSACi Important in Missouri?................................................................................................ 10 Justification Statement .......................................................................................................................... 13 Framework ............................................................................................................................................ 17 Evaluation Model................................................................................................................................... 19 Part Three – The Digital Dashboard: DotDigital ............................................................................ 21 Principles of CIPP Utilized in Digital Dashboard ....................................................................................... 21 Principles Implemented ......................................................................................................................... 22 Design Principles ............................................................................................................................................................... 23 Best Practices ........................................................................................................................................ 24 Highlights of Key Dashboard Features .................................................................................................... 25 Programming Links ........................................................................................................................................................... 27 MU Extension Programs ................................................................................................................................................... 28 Facebook ........................................................................................................................................................................... 28 Favorite Links .................................................................................................................................................................... 29 Part Four – Summary.................................................................................................................... 31 Part Five – References, Supporting Documents/ Appendices ......................................................... 32 References ............................................................................................................................................ 32 Supporting Documents and Appendices ................................................................................................. 32 Connect With Us! ......................................................................................................................... 35 University of Missouri Extension ...................................................................................................................................... 35 Show-Me Strong Farm Families ........................................................................................................................................ 35

ABOUT THE AUTHORS

Amy Patillo, Ed.D.

Dr. Amy Patillo serves as a Labor and Workforce Development Specialist with the University of Missouri Extension. Amy is a passionate strategist and practitioner committed to tackling Missouri’s workforce issues through collaboration with state, regional, and community workforce partners. As a Field Specialist, she works across the state to coordinate the University’s high priority commitments and identifies projects where Extension can grow Missouri’s workforce. Amy strives to address new workforce challenges with innovation and transformative business solutions. She earned a Master’s in Communication Sciences and Disorders and a second Master’s in Educational Administration from Missouri State University, as well as a Specialist degree and Doctorate degree in Educational Leadership and Administration from St. Louis University.

Amy can be reached via email at patilloa@missouri.edu

Matthew Pezold, M.S., M.A.

Matthew Pezold serves as a Labor and Workforce Development Specialist with the University of Missouri Extension in the Kansas City and Urban West Region. Matt strives to provide education, training, and programs that increase the state’s workforce skills while providing relevant data that helps equip organizations to make informed and data-driven decisions. As an educator, he strives to create High-Tech, High-Touch, inclusive learning environments and opportunities for his students. He earned a Master of Science in Agriculture and Applied Economics, from the University of Missouri and a Master of Arts from Saint Louis University.

Matt can be reached via email at pezoldma@missouri.edu

Karen Funkenbusch, M.A.

Karen Funkenbusch is an Extension instructor in Occupational Therapy, School of Health Professions, at the University of Missouri and Nutrition, Health, and Family state specialist. Funkenbusch coordinates statewide leadership for agricultural health and safety, a University of Missouri Extension program serving the health, safety, wellness, and quality of life needs of agricultural workers and their families. Her research emphasis includes health, safety, wellness for farmers, ranchers, families, and youth.

Current research topics include Ergonomics for Women Tools and Equipment Selection, Tractor Technology Enabling Digital Agriculture Research and Education, Mental Health First Aid, Youth, Adult, Face-to-Face and Virtual, Virtual Reality (VR) 360, Tractor Safety, Development of Colorectal Rectal Cancer (CRC) Screening Education Materials for Agricultural Workers, and Interpersonal Isolation on Mental Health in Rural and Agricultural Communities. She recently served as an Extension New Technologies for Agriculture in Extension (NTAE) Fellows to assist with program evaluation of the North Central Farm and Ranch Stress Assistance Center initiative (NCFRSACi) in Missouri. Karen serves as the national lead for the Agricultural Safety and Health Extension Community of Practice (CoP) and, on various multi-state and national interdisciplinary teams, boards, councils, committees, and organizations that promote injury prevention and cultivates health strategies to safely improve the lives of rural communities. She is active with the International Society of Agricultural Safety and Health (ISASH) board of directors, NORA Agriculture, Forestry and Fishing Sector Council, National Center for Farmworker Health Advisory Committee, External Advisory Committees for the Great Plains Center for Agriculture Health and Central States Center for Agricultural Safety and Health, and Missouri Show Me Farm Safety Committee and is an active member of NCERA 197, ASABE, Epsilon Sigma Phi, UMEA, MAEP, NACAA, NARMH, and ARHP professional organizations.

Karen can be reached via email at funkenbsuchk@missouri.edu

Credit for author and publication images: University of Missouri

Part One - Introduction

ABSTRACT

In response to the higher rates of suicide among farmers and the limited access to healthcare and mental

health services, the Farm and Ranch Stress Assistance Network (FRSAN) was developed to address the rising

mental health crisis in farming communities. This publication serves the dual purpose of explaining the

necessity of mental and behavioral health services for farmers and rural communities and identifies the

challenges in coordinating complex programming and implementation of services. The NexGen Program

analysis creates a framework that helps to determine the most efficacious mental health intervention

programs, packages the suite of tools together, and promotes the adoption of improved programs to meet

the unique needs of agricultural producers and their allies. Missouri’s FRSAN evaluation program uses a

continuous improvement evaluation approach, implementing an educational decision-oriented model to

identify strengths and limitations in curricula, instruction, and delivery. Unique to this approach is the use of

digital dashboards to streamline and define processes that create a simple path forward to support large

multidisciplinary teams and multiple mental health training programs.

THE CHALLENGE

The New Technologies for Ag Extension Project Accelerator (NTAE) NexGen Program Evaluation simplifies

and streamlines the evaluation process for the University

of Missouri (MU) Extension’s mental health and suicide

prevention program. The program is part of the multi-state

North Central Farm and Ranch Stress Assistance Center

initiative (NCFRSACi) supported with funds from the United

States Department of Agriculture National Institute of Food and

Agriculture (USDA-NIFA) under project number (USDA-NIFA

Award #2020-70028-32728).

Given complex evaluation and program delivery challenges,

this publication outlines a strategy used by the University of Missouri (MU) Extension to manage and

evaluate complex initiatives to coordinate easy access to program assets and reduce project reporting

limitations when working with large, multi-disciplinary teams. The publication highlights the development

processes and tools created to help a large, multidisciplinary team improve the collection and reporting of

impacts and outcomes of the NCFRSACi training and education programs.

In Missouri, family farms represent two-thirds of the total land acreage, making an $88.4 billion economic impact (Missouri Department of Agriculture 2021) in the state and supporting nearly 400,000 jobs. Agriculture has a rich history and is the livelihood of agriculture producers and their families. The development of mental and behavioral health services is a response to the economic challenges farmers experience.

MU Extension, affiliates and partners responded by taking concrete steps to create a supportive pathway by:

 Creating networks of community support (local stakeholders) and connecting us to farmers and

ranchers who experienced stress, anxiety, and depression, to assist in sharing mental health tools

(Mental Health Tool Kit) and behavioral health resources.

 Investing in training, Extension delivered and adapted Mental Health First Aid (MHFA), Q.P.R.

Gatekeeper (Question. Persuade. Refer.), Taking Care of You (TCoY), Tai Chi, and Weathering the

Storm in Agriculture: How to Cultivate a Productive Mindset programs in rural communities.

 Providing free behavioral counseling and teletherapy services to farmers, ranchers, and farm

families experiencing stress, anxiety, and depression.

 Serving the agriculture community through the Iowa Concern helpline that employees trained

telephone responders providing referral for additional assistance.

 Creating an online network, Show Me Strong Farm Families (@Show-Me Strong Farm Families), to

quickly share mental health tools and resources and workshops with rural communities and

producers experiencing emotional and financial stressors.

The program draws from the expertise of a large team of multi-disciplinary Extension educators and mental

and behavioral health professionals, which includes specialists in:



Human Development



Nutrition, Health and Family



Agriculture and Environment



Agronomy



Workforce Development



Livestock



4-H Youth Development



Behavioral Health Counselors

 Lincoln University, Missouri’s 1890 Land Grant Institution

The multi-disciplinary team approach is a critical design element that allows MU Extension to connect mental health and behavioral health expertise with farmers and their allies who have a high level of hesitancy towards receiving these services. These experts are uniquely positioned to help because they are well established and familiar with serving rural communities.

While providing a suite of mental and behavioral health services is an effective method to address the

needs of unique participants, it creates a complex system to document and evaluate as external evaluators.

Concurrent with program delivery across disciplines, the complexity of the process includes:

 Reporting on the delivery of multiple mental and behavioral health programs;

 Coordinating data collection with a large multi-disciplinary team of certified instructors;

 Collaborating and supporting instruction and instructors dispersed geographically across the state of Missouri; and

 Utilizing a large number of data and digital platforms to coordinate data collection and expand access to the tools and resources for program implementation.

Part Two – Evaluating the Effectiveness of Mental Health Programs in Missouri

This section provides information about NCFRSACi and its audiences, a brief NexGen program analysis, and why mental health programs are critically needed in rural Missouri.

NCFRSACI: AUDIENCE SERVED

The Farm and Ranch Stress Assistance Network

Key target audiences for this project are: - producers (operators and farmworkers) - farm youth and families - allies of agricultural producers (agribusiness

(FRSAN) Program is an established network that

connects individuals who are engaged in farming,

ranching, and other agriculture-related occupations to

stress assistance programs. FRSAN expands stress

management and mental health resources and

services to agricultural producers, advocates, and

professionals and healthcare providers)

stakeholders who support agricultural producers. The

key target audiences for this project include

individuals engaged in production agriculture with

specific emphasis on:

 Producers (male and female operators, farmworkers);



Farm youth and farm families; and

 Allies of agricultural producers (specifically agribusiness professionals and healthcare providers).

NC-FRSAN provides professional agricultural behavioral health intervention, supports training programs,

telephone helplines, resource hub, clearinghouse website, webinars, support groups, and professional

behavioral health services for producers, agriculture-related occupations, and individuals who support

producers (Figure 1).

Figure 1. North Central Farm and Ranch Stress Assistance Network Resources

UNIQUE VALUE PROPOSITION

The NexGen Program Evaluation assesses the NCFRSACi’s mental health services efficacy in alleviating the

mental health challenges that agricultural producers and their allies are facing, so that a comprehensive

suite of producer-centric mental health services can be developed and implemented in the North Central

Region (NCR) Cooperative Extension Association.

WHY IS NCFRSACI IMPORTANT IN MISSOURI?

Evaluating the effectiveness of the mental healthcare program s that are provided to Missouri’s farmers,

ranchers, and other agricultural production workers who are experiencing anxiety, depression, substance

misuse, and death by suicide is critical to improving the effectiveness of mental health programs.

While stress, mental health concerns, and substance misuse issues run high, many NCR communities are in

mental health professional shortage areas, which impedes NCR agricultural producers, particularly rural

ones, from obtaining adequate care. In areas where mental healthcare is available, stigma, cost, and

scheduling delays often discourage agricultural producers from seeking help. Left untreated, mental health

conditions such as depression, anxiety, and substance use can lead to reduced quality of life, absenteeism,

and presenteeism in work contexts, higher rates of agricultural-related injury, suicidal ideation, and

premature death due to suicide, heart disease, cancer, and respiratory diseases.

The NexGen Program analysis creates a framework that helps to determine the most efficacious mental

health intervention programs, packages the suite of tools together, and then promotes the adoption of the

improved programs across the NCFRSACi partner institutions. Identifying and removing barriers to access,

data, and information is key to improving impacts and outcomes. The internal and external barriers that

were addressed are found in Figure 2 (below) and include:



Lack of accessible data sources;

 Limited evidence of the quality of mental health interventions/opportunities designed for agriculture producers;

 Lack of access to information and coordination of resources; and

 Challenges to integrating mental health care within rural community-based health care programs.

Figure 2. Original Data Collection and Evaluation Process

Prior to the creation of the digital tools developed to coordinate and evaluate the program, each component

of the program was recorded and documented individually. Figure 2 illustrates the dispersed method for

collecting and sharing data, information, and resources which were located on multiple digital platforms. For

example, marketing materials, some individual program data, and a schedule of activities were maintained

in Microsoft Teams. The monthly and annual reports were kept in the University of Illinois-Urbana Box

account. Pre - and Post-surveys for programs could be found in Qualtrics, only existed in paper format, or

were housed on a national learning management system with restricted access. Additionally, information to

access farmer resources could be found on the North Central Farm and Ranch Stress Assistance web page,

the University of Missouri Extension website, and social media.

Centralizing these resources in one location helped the larger team to be more aware of the many tools and resources available and helped team members better connect individuals to the services and information they needed.

The NCFRSACi is currently piloting several programmatic innovations that are designed to mitigate barriers

to mental health care access across Missouri.

Routinely assessing mental health programs

and establishing measurement-based care

is essential to expanding quality mental

health care into more rural communities.

The NCFRSACi in Missouri is working to

stratify quality measures to better address

potential gaps in services for producers with

the long-term intent to continue to adapt

and expand services to other vulnerable

populations. Evaluating the training

components is essential to building capacity

for quality improvement across the state.

The program analysis requires coordination of data sources across multiple mental health program

interventions, engagement of agricultural producers and their allies, evaluation of MU Extension’s NCFRSACi

providers, and identification of best practices and efficacy of NCFRSACi mental health programs, education,

and training.

JUSTIFICATION STATEMENT

Agricultural producers throughout the United States (US)

and Missouri are in crisis. The mental health crisis in rural

“…with just 3.7% of the

Missouri is more challenging than in urban communities,

recommended supply filled,

due to a unique set of external challenges faced by farmers

Missouri faces the largest

and ranchers. Recent studies, media coverage and

congressional action validate the need to address mental

shortage of behavioral health care

health outcomes in rural communities. Contributing to

providers in the U.S.”

mental health challenges on the farm, recent extreme

weather events, economic pressure and foreign trade

policies have combined to produce a series of lean years for agricultural producers not seen since the farm

crisis of the 1980s.

While the U.S. economy has experienced quarter-over-quarter growth since the end of the recession in

2008-2009, the farm sector has experienced six periods of recession. Missouri has nearly 16,000 fewer

family farms today compared to 20 years ago, and bankruptcies are on the rise. Bushel prices have fallen by

47% since 2012, while farm income and credit conditions have deteriorated.

Pervasive stigma and geographic barriers to accessing mental health care can prevent those in need of

behavioral health services from seeking help for stress, anxiety and depression. Each of Missouri’s 99 rural

counties is a designated Mental Health Professional Shortage Area (HPSA), and with just 3.7% of the

recommended supply filled, Missouri faces the largest

shortage of behavioral health care providers in the

“Between 2003 and 2017, the

U.S. The prevalence of depression among Midwestern

farmworkers is estimated to be more than 45%.

suicide rate among rural

Furthermore, the rate of heavy alcohol use among

Missourians grew by 78%...”

full-time workers in agriculture is 9.4%, higher than

the overall rate of 8.7% among all other full-time

workers. NCR producers face financial-, weather-, and

time-related stressors as well as stressors related to

complex governmental regulations. Net farm income has fallen 30.5% below its peak in 2013. Financial

circumstances are associated with depression and anxiety among NCR agricultural producers and, in a

national poll, more than 80% of farmers and farmworkers indicated that financial issues, farm or business

problems, and the fear of losing their farm had a strong negative impact on their mental health. Suicide is

growing at an alarming rate in rural Missouri. Between 2003 and 2017, the suicide rate among rural

Missourians grew by 78%, and throughout the last decade, their hospital emergency department visits for

suicide attempts or ideation increased 177%. Rural men between the ages of 35 and 44 have triple the

statewide suicide mortality rate.

Accor ding to Health Research and Service Administration (HRSA), every one of Missouri’s 99 rural

counties has a shortage of mental health professionals (Figure 3). In addition, there are 57 rural

counties in Missouri without a licensed psychologist or psychiatrist, leaving large swaths of mental

health deserts throughout rural areas of the state (Figure 4). These geographic barriers to access

result in many rural populations forgoing care altogether or depending on hospital emergency rooms

and other nontraditional services for their behavioral health needs.

Farming households are disproportionately affected by shortages of mental health professionals in

rural areas, with farmers being three times as likely to live in a Mental Health HPSA. With 266 Mental

Health HPSAs in 2019, Missouri has the fifth-highest number of designated Mental Health HPSAs in the

country. And with just 3.7% of the need for mental health professionals met in the state, Missouri has

the largest shortage in the country. (Reidhead et al. 2020 ).

Editor’s Note: For additional information see: Reidhead, M., Medows, J., Dothage, K., Funkenbusch, K.,

Davis, C., Williams, S., Gowdy, R. N. & Greimann, A. (2020, February) Growing Stress on the Farm: The

Expanding Economic and Mental Health Disparities in Rural Missouri. Centers for Disease Control Stats of

the State - Suicide Mortality by State; Centers for Disease Control Wonder database; Missouri Institute of

Mental Health Report.

Figure 3. Mental Health Professional Shortage Areas and Clinics in Missouri, 2019

Figure 4. Psychologists & Psychiatrists Per 1,000 Residents by County in Missouri

Personal and environmental risk factors also contribute to increased risk of adverse mental health among

NCR agricultural producers. Individuals may be at increased risk of mental health conditions due to factors

such as gender, education, decision-making authority, legal status, and a sense of self-efficacy or mastery

(i.e., that one has control over their life). Farm families, including children and spouses, tend to experience

agricultural stressors acutely and immediately, resulting in increased risky behavior by youth on farms,

developmental problems among adolescents, reduction in spousal perception of marital quality, and marital

instability (Reidhead et al. 2020).

In response to strong demand, Missouri expanded access to the suite of mental and behavioral health programs, resulting in the need to compare outcomes across the state in order to learn what programs are most effective in Missouri. This expansion of services created the need to further validate the program impacts and outcomes.

Missouri’s suite of mental health programs is not producer -centric and is designed to address specific mental

health challenges, effectively treating different symptoms and requiring varying commitments of resources

and personnel. The NexGen Program Evaluation determines the mental health care programs that produce

the desired outcomes and help to create wrap-around producer-centric programs that can be utilized by

Missouri mental health care providers and NCR partner institutions. The NexGen Program Evaluation

developed an operationalized product that can be effectively implemented across the state based on the

impacts and outcomes demonstrated in Missouri.

The NexGen Program Evaluation:

 Supports partner institutions to increase effectiveness in addressing the mental health challenges

that producers in the region are facing.

 Addresses the mental health challenges of producers to improve their quality of life, as well as to

help reduce the following:

absenteeism;

•

work presenteeism;

•

rates of agricultural-related injury;

•

suicidal ideation; and

•

• premature death due to suicide, heart disease, cancer, and respiratory diseases.

 Ensures an inclusive mental health support network that expands local communities’ capacity to

intervene and disrupt the negative effects that poor mental health is creating for producers and

their families. The NexGen comparative analysis team’s objective is to accelerate the creation of this

supportive network across the NCR.

FRAMEWORK

The NCFRSACi’s program evaluators in the NCR i dentified challenges in collecting and reporting impacts and

outcomes as a result of insufficient processes and difficulty accessing Mental Health First Aid participant data

using their online learning portal. Additionally, the Program included a number of multi-state, multi-

disciplinary teams, and a program with numerous instructional deliverables.

In response to these complex program evaluation challenges the Missouri Program evaluation team focused

on establishing and streamlining a data collection, data reporting, and data processing system.

The Missouri Program was carefully assessed to identify the program evaluation approach that was actively

being used, to identify how data was being collected, how participant data was being accessed, where data

collection was stored, and identify the data collection methods used by instructors and the online registration

platforms. Assessment of the program evaluation approach identified the following challenges:

 The suite of mental health and suicide prevention education and training curricula were evaluated

using different software platforms and different evaluation tools.

 The number of different program offerings - as well as two funding sources running simultaneously -

created the need to track participant scholarships, and funding specific deliverables across multiple

programs and timelines.

 The instructors delivering the training were in different disciplines, working under specific guidance

from education directors. Each discipline collected participant and evaluation data differently.

 Access to program data, including participant demographics, evaluation of instructors, and

evaluation tools was limited by the individual's role. For example, instructors had access to

evaluation of the instructors teaching performance, but frequently did not have access to the

surveys completed by participants.

 Impact measures, both quantitative and qualitative, were being collected in different locations

across disciplines, within university reporting systems, and across multiple online platforms that

were not readily accessible to the program administration team. This challenged the program

evaluation team to identify multiple points of data collection located across multiple software

platforms.

 Overall, it was important to map the evaluation process that was in use and provide information,

access, and consolidation of data evaluation methods and tools.

 The program evaluation approach was focusing on outreach, relevance, and delivering a successful,

targeted program, while overlooking opportunities to demonstrate short, mid-term and long-term

impact and efficiency.

The primary focus of the Missouri program evaluation team was to:

 Ensure outreach is reaching the intended stakeholders;

 Ensure the target audiences needs are met;

 Ensure the evaluation outcomes are usable by stakeholders;



Demonstrate accountability to funders;

 Provide high quality mental and behavioral health products that are cost effective, equitable, and

feasible; and

 Provide continued program improvement and implementation across diverse settings.

EVALUATION MODEL

Program evaluation is a critical component of good decision making and is essential to effectively

communicating with stakeholders (funding agencies, policymakers, and instructors). Missouri’s FRSAN

evaluation program uses a continuous improvement evaluation approach, implementing an educational

decision-oriented model to identify strengths and limitations in curricula, instruction, and delivery. This

approach improves program effectiveness and plans for future implementation with targeted audiences by

concentrating time and resources towards project delivery and high-impact deliverables rather than an

emphasis on program objectives as the priority. FRSAN evaluation deploys a standard evaluation process to

identify, obtain, and deliver high-quality information to assess best practices and analyze deliverable

impacts.

The CIPP (Context, Input, Process, Product) Model focuses on effective evaluation of program

implementation and assessing critical components of the Program to drive program improvement by

evaluating implementation and short-term results (Stufflebeam 2015 ). The CIPP Evaluation Model (Figure 5)

is a comprehensive framework that looks at four areas of program evaluation: Context Evaluation, the

overall goals and mission of the program; Input Evaluation, the program plans and resources; Process

Evaluation, the activities and components of the comprehensive approach; and the Product Evaluation,

focusing on the outcomes and objectives (Poorvu Center for Teaching and Learning 2021 ). The Model

provides guidance for “executing a sound, practical evaluation and assuring that findings will be applied in

processes of program improvement and accountability” (S tufflebea m 2015 ).

Figure 5. CIPP Evaluation Model

Source: Poorvu Center for Teaching and Learning (2021). CIPP Model, Yale University.

The CIPP Evaluation Model Checklist provides a tool for applying the CIPP Model to assess projects and

programs. The purpose of the checklists (Stufflebeam 2015 ) are to :

1. Help evaluators carry out sound evaluations and evaluation plans;

2. Assure the evaluation generates timely information to execute deliverables for targeted audiences;

3. To evaluate and report program quality, cost-effectiveness, integrity, feasibility, safety, equity,

significance, and lessons learned;

4. Help assure evaluation findings will be used for the purposes of program improvement and

accountability; and

5. Help avoid pitfalls that could impair the evaluation’s success.

Part Three – The Digital Dashboard: DotDigital

This section provides information about the digital dashboard we created using DotDigital. This dashboard

provided a responsive program evaluation approach that helped with program implementation and

improvement

PRINCIPLES OF CIPP UTILIZED IN DIGITAL DASHBOARD

1. Process Evaluation is collected via the Master Schedule and Qualtrics Survey Collection . The program improvement approach relies on the continuous monitoring, tracking, and evaluation of

ongoing mental health programming, participant demographics, and program feedback which allows

for a more robust evaluation beyond the end of the program summary report.

2. Context Evaluation brings together multiple dashboard components linked directly to discipline

specific processes. In addition to providing information that is easy to share with stakeholders, the

approach provides timely feedback about needs, resource allocation, program improvements,

improved communication and collaboration, and program outcomes.

3. Product Evaluation dashboard elements identify data touchpoints using multiple digital platforms where data collection occurs . This approach provides for ongoing programmatic evaluation, reduces

the cost of evaluation and allows the team to better capture the most meaningful impacts and

outcomes.

4. Input Evaluation is captured by centralizing program resources and tools allowing the larger team

to identify gaps and additional needs to be addressed both geographically and programmatically .

This is a systematic approach that ensures the evaluation focuses on identifying key issues and

questions to be addressed with stakeholders and implementing strategies that best serve rural and

agricultural communities.

The continuous improvement model necessitated a responsive program evaluation approach, resulting in the

creation of a digital dashboard for comprehensive program implementation and improvement, allowing

instructors to identify and serve unmet mental health needs in communities. Using a digital dashboard

streamlined program implementation processes, data collection, and access to information and materials

necessary to assure successful program delivery.

PRINCIPLES IMPLEMENTED

One of the primary objectives for the program was to provide solutions to complex evaluation processes that

support program implementation and project management across several disciplines, and to support the

unique needs of each instructor.

In developing this process , the team focused efforts on building and supporting trust and transparency

amongst a large multiple disciplinary team which involved nearly 50 individuals within the University of

Missouri System (UM System). After a discovery period of active listening the evaluation team identified

bottlenecks in the workflow processes. In response, the team created a workflow and data management

process that centralized the points of access to digital platforms. This strategy helped the team better

understand how to improve data tracking for reporting deliverables, eased the challenge of fiscal

reconciliation across the UM System and improved data collection methods and reporting process.

DotDigital is a digital marketing and communication tool available at the University of Missouri; it was

utilized to create digital dashboards to support the program. However, Google Docs and Microsoft Suite tools

could be used to create a digital dashboard. DotDigital worked well for the program because it is simple to

use, and easy to customize to meet the needs of the entire team, eliminating the need to rely on the

Information Technology department or a webmaster to modify the dashboard. The key components of the

dashboard are illustrated in Figure 6 and the design principles utilized throughout the development process

are highlighted below.

Figure 6. Farm Stress Dashboard (DotDigital Creator View)

Design Principles

 Increased transparency The FRSAN Program provides a suite of resources and training to support

the mental wellbeing of farmers and ranchers. This dynamic and comprehensive approach creates a

complicated process, which the Program team sought to improve by clarifying the processes and

increasing access to the most current data.

 Visualizes workflow process The digital dashboard provides a visually intuitive way to

understand the entirety of the programming process and it allows users a simple way to navigate

between the digital portals and tools that are central to efficiently delivering programs and reporting

impacts and outcomes.

 Flexibility Designing a process that was user friendly and responsive to the needs of the Program

team informed the development of the dashboard. The entirety of the dashboard is controlled by

the external evaluation team, which allows the team to quickly refine and improve the dashboard to

better meet the needs of the entire team.

 Centralized access to information The external evaluation team worked to centralize key

planning documents for event planning and reporting, as well as digital marketing and resource

distribution, which is now accessed via the Farm Stress Dashboard. For example, key program

objectives, registration codes, and training videos are only one click away. A second example

includes the creation of a scholarship tracker which helps to ensure that the program team meets

its goals, and that budget resources are allocated as proposed.

 Standardization of scheduling and data collection Centralizing access to the suite of platforms

improved efficiency. For example, centralizing the scheduling process streamlined the planning and

event tracking process for five separate intervention programs allowing the program to efficiently

track progress. Similarly, standardization of the data collection process across platforms greatly

reduces the time spent reporting, organizing, and cleaning the data, freeing up more time to report

impacts and outcomes.

 Real time data Managing the goals, objectives, and the budget requires access to timely

information, and improved data collection and planning processes helped to increase the entire

team's access to real time data. This was key because the team has to closely monitor events to

ensure that they are reaching the target audiences, and to ensure that it fully leverages the

scholarships that cover the participants registration costs.

 Centralizing access to online portals Creating the dashboard allows the team to centralize the

workflow process, so that the entire team can access the suite of digital platforms via a single

dashboard. This centralization allows all team members to efficiently access the tools necessary for

event planning and reporting, digital marketing and distribution, as well as online instruction and

collaboration.

BEST PRACTICES

The program evaluation team quickly identified the need to centralize information so that all team members

could easily access the tools needed for delivering mental health services. The evaluation team felt it was

extremely important to provide rapid and responsive data reporting to support the larger team, resulting in

the creation of a digital platform developed, designed, and deployed by the evaluation team, so as to not

depend on the time and talent of information technology

The Dashboard delivers these best practices for program evaluation

professionals.

The digital platform ultimately generated complete



Increased transparency

transparency for the project, which was a welcomed and



Visualization of workflow process

unintended benefit of the tool. The most beneficial aspect

of the digital dashboard approach was the ability to



Flexibility

visualize the workflow of multiple teams working



Centralized access to information

simultaneously across the suite of mental health programs.



Standardized scheduling and data collection (improves efficiency)

Ultimately, this process accomplished the primary goal to

standardize program evaluation and data collection using a



Real time data

field-tested evaluation model and standardizing data



Centralized access to online portals

collection.

The digital dashboard is a responsive tool allowing the program to centralize access to information, data, and

platforms. Another advantage is that dashboards are easily adapted to the needs of the program team and

are easily customized to the specific needs of other projects. Dashboard features were strategically designed

to address evaluation objectives. The scheduling feature and registration process helps the team track

outreach to target audiences and ensure financial accountability. Improving program outcome and impact

measures ensures the target audience's needs are being met. Centralizing program information allows the

team to communicate outcomes to stakeholders and deliver a suite of mental health products that are cost

effective, equitable, and accessible in rural communities.

HIGHLIGHTS OF KEY DASHBOARD FEATURES

Included are screenshots of the Farm Stress Dashboard. Figure 7 represents the entirety of the dashboard.

From this Dashboard the team can navigate and access all the tools and resources necessary to provide

program delivery for five separate mental health programs. In fact, this dashboard allows the team to

manage two grants that simultaneously provide programing and funding to support the mental wellbeing of

farmers, ranchers, and their community allies.

Want to learn more? Watch the below video or follow this link a dashboard for external facing audiences.

Learn more about the Farm Stress Dashboard https://www.youtube.com/watch?v=ud3geTsDrSk

Run time: 10 minutes

VIDEO

The digital dashboard is a responsive tool allowing the program to centralize access to information, data, and platforms. Learn more about its features on the following pages.

Figure 7: Farm Stress Dashboard

Figure 7 highlights the key features of the dashboard, as demonstrated in Figure 6. Starting at the upper

right corner are four navigation icons, which are hyperlinks to other program dashboards and other data

hubs.

This great feature creates a simple linkage between dashboards that allows the Principal Investigator and

the External Evaluators to navigate to other internal facing program dashboards. Using DotDigital, the team

is able to efficiently manage four additional grants and a resource page for a specific Extension Department.

The resource page is an example of an external use of a dashboard that is used to connect external

stakeholders to workforce data tools, reports, webpages, social media, and archived newsletters. The next

key element in the “Favorites Dashboard” section includes key planning documents, such as registration and

fiscal codes, registration instructions, and the key program deliverables.

Figure 8. Favorites Dashboard

Programming Links

This sect ion includes the “Programming Links” which allow the team to directly access the “Master

Schedule ,” an Excel tracking document that simultaneously tracks the planning and progress of five separate

mental health programs, as well as the remaining participant training scholarships that are available for each

program.

As seen in Figure 9, a scholarship tracker - created within an Excel spreadsheet - was essential to improve

the tracking of program deliverables and managing the scholarships allocated in the budget. As seen below,

this tracker tracks the expected scholarships that have been allocated for that program, as well as the actual

number of “seats” that have been purchased by the two relevant projects. These numbers are automatically

updated each time an event is scheduled within the “Master Schedule.” This tool helps the team to en sure

that scholarships are not overspent and tracks the progress towards completing key deliverables.

Figure 9. Scholarship Tracker

Also found under the Programming Links section is the “Teams'' link that takes team members directly to the

Microsoft Teams platform. This is where the larger collection of program material and official documents are

stored, and where team members can directly communicate with other team members via a messaging

platform. The “ MyExtension ” link takes team members directly to an event scheduling and reporting portal,

the first step in planning most workshops and training. (This is not the same for all of the mental health

programs). The “Qualtrics” link takes team members directly to one of the data collections tools, and most of

the pre- and post- evaluation surveys are accessed directly via Qualtrics.

MU Extension Programs

This section includes the official “MU Extension Programs” which are managed via the Farm Stress

Dashboard. Each of these links takes team members directly to the MU Extension website where the relevant

Farm Stress programs and resources are listed, marketed, and where individuals can access the online

registration system.

Facebook

This section includes direct links to the official Facebook pages which are used to market and to share Farm

Stress resources. Centralizing the point of access allows instructors to quickly navigate to the social media

pages to confirm marketing posts, and quickly access online materials that can be directly shared on

Facebook pages of MU Extension County Extension, Regional Extension, and other sponsoring organizations.

Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32 Page 33 Page 34 Page 35

impact.extension.org