CTSI Evaluation and Continuous Improvement (CI) and Quality and Efficiency (QE) Functions

The CTSI is an innovative infrastructure to support and advance research, education, and training in clinical and translational sciences.  Our integrated “Cores” and “Functions” work together to facilitate your needs.  This quarter we present the CTSI Evaluation and Continuous Improvement (CI) and Quality and Efficiency (QE) Functions. Consultations with both Functions are available to you in support of your clinical and translational research.

CTSI Evaluation and Continuous Improvement (CI) Function

CTSI has a strong commitment to rigorous evaluation and improvement that is supported through the Evaluation and Continuous Improvement (CI) Function. The goal of this Function is to facilitate a data driven culture and optimize performance of all CTSI Functions, such that the Hub is audit-ready and report-ready, bolstered by informed decision-making process. The evaluation approach the Function uses is underpinned by the Logic Model as outlined by NCATS.

The Evaluation and CI Function has specific aims and objectives that were developed to guide the Function’s operational and implementation plan, ranging from: ensuring policy driven evaluation and CI methodology; ensuring standardization, optimization, and validation in CTSI evaluation and CI process and milestones; and ensuring optimal compliance with NCATS’s Common Metrics Initiative (CMI).

The Evaluation and CI Function team has adopted several evaluation standards based on UNDP (United Nations Development Program). These standards are:

• Independent – Evaluation system must be free of conflict of interest.
• Intentional – Evaluation rationale and the decisions to be based on it should be clear from the outset.
• Transparent – Meaningful consultation with stakeholders is essential for the credibility and utility of the evaluation.
• Ethical – Evaluators must have professional integrity, respect the rights of institutions and individuals to provide information in confidence, and be sensitive to the beliefs and customs of local social and cultural environments.
• Impartial – Removing bias and maximizing objectivity.
• Of high quality – Evaluations should meet minimum quality standards as defined.
• Timely – Evaluations must be designed and completed in a timely fashion to ensure the usefulness of the findings and recommendations.
• Used – Evaluation is a management discipline that seeks to provide information to be used for evidence-based decision-making process.

A mix of both qualitative and quantitative methodologies are used to support diverse evaluation needs of CTSI Functions. A range of instruments, such as: Function-specific evaluation plans, Clinical Research Appraisal Inventory (CRAI), Graduates Tracking Survey System (GTSS), Annual CTSI Satisfaction Survey, and various events and programs evaluation tools, are deployed to monitor and evaluate various CTSI programs, services, and initiatives.

Highlights of Evaluation & CI Year 1 Activities

In the first year of CTSI 3.0 implementation, the Evaluation and CI Function had accomplished various evaluation and CI related milestones. Highlights of such actives are:

• Completed Evaluation & CI Function policy and standard operating procedure
• Established standards & guiding principles for the Evaluation & CI Function
• Completed evaluation plans for each of the 18 CTSI functions (collaboratively with the Functions)
• Standardized, and documented in standard operating procedures, the compilation and reporting processes of the requirements of the Common Metrics Initiative.
• Submitted numerical data and Turn the Curve (TTC) Plans for the Common Metrics Initiative (2019)
• Completed analyses of the 2020 CRAI and CTSI Community Science Café Program surveys
• Updated and distributed the 2020 CTSI Annual Satisfaction Survey
• Completed analyses of the responses from the 2020 CTSI Annual Satisfaction Survey

Quality & Efficiency (Q&E) Function

In collaboration with other CTSI Functions’ Leadership, the goal of the Quality & Efficiency (Q&E) Function is to facilitate the development of verifiable and sustainable process improvements within the CTSI enterprise service areas by identifying and addressing current and potential performance improvement opportunities.

Continuous Improvement Cycle

The Q&E Function has specific aims and objectives that were developed to guide the Function’s operational and implementation plan, ranging from: innovating processes, systems, and tools to increase the quality and efficiency of our Hub’s Function performance; completing periodic quality assurance, performance, and efficiency assessments; and preparing our CTSA Hub’s  Q&E Report

The Q&E Function uses various well-known Lean and Six Sigma tools, such as: DMAIC Six Sigma process, cause & effect diagrams, process mapping, discrete event simulation, PDCA (Plan-Do-Check-Act), and 8- Disciplines, to support various quality and efficiency projects in CTSI Functions.

Highlights of Q&E Year 1 Activities

In the first year of CTSI 3.0 implementation, the Q&E Function had accomplished various quality improvement and efficiency works. We highlighted these activities and accomplishment below:

• Developed standard operating procedure and overall policy for the Q&E Function
• Developed Evaluation Plans for the Q&E Function
• Completed baseline for the Pilot, IRB review, Clinical Trials Office (CTO), and Adult Translational Research Unit (A-TRU) processes.
• Completed “barriers to research” analyses as an outcome of the CTSI Annual Satisfaction Survey
• Developed proposals for remedial actions to reduce IRB review Turn Around Time (TAT)

Accomplishments

Using continuous improvement and industrial engineering methodology, the Q&E Function has developed an innovative approach for ranking and prioritizing “barriers to research”. This approach utilizes a multi-criteria model based on effect, cost to mitigate, and strategic impact to CTSI operations.

This approach was outlined in a scientific peers-reviewed paper which was published in Journal of Clinical and Translational Science (Grabenstetter, D., Rashid, R. and Whittle, J., “Using a process improvement approach to identifying barriers to research in a CTSA hub environment”, Journal of Clinical and Translational Science, DOI: 10.1017/cts.2020.522, December 2020.)

Integration of Evaluation & CI and Q&E Functions with other CTSI Functions and Partners

The Evaluation & CI and Q&E Functions are inherently integrated with all CTSI Functions through its activity in assessing performance level, identifying improvement opportunities, and facilitating the efforts for resolution.