D159 Task 1 Evaluation Measure Set Example

Evaluation Measure Set: Outcome, Process, Structure and Balancing Measures for a Patient Blood Management Program

Student Name

Leavitt School of Health, Western Governors University

D159: Evidence-Based Measures for Evaluating Healthcare Improvements, Task 1

Course Instructor

Month Day, Year

Evaluation Measure Set: Outcome, Process, Structure and Balancing Measures for a Patient Blood Management Program

Purpose

This measure set will be used to evaluate a patient blood management program on the adult medicine and general surgery services of a composite 350-bed community hospital. The program revises the red cell order set, adds an ordering alert that appears when the patient's hemoglobin is already above the restrictive threshold, educates clinicians and nurses and sends monthly feedback reports. Measures are organized using Donabedian's model of structure, process and outcome (Donabedian, 1988), with balancing measures added to watch for harm the program could create elsewhere. Each measure is written out fully so two people pulling it would get the same number.

Primary Outcome Measure

Measure 1: Percentage of red cell units transfused above threshold. Numerator: red cell units transfused to in-scope inpatients when the most recent hemoglobin, resulted within 24 hours before the order, was 8.0 g/dL or higher. Denominator: all red cell units transfused to in-scope inpatients. In scope: adults 18 and older on the medicine and general surgery services. Excluded: patients with a massive transfusion protocol activation, active bleeding documented as the indication, intensive care, cardiac surgery, oncology and obstetric patients. Period: calendar month. Data source: blood bank transfusion records linked to the laboratory hemoglobin results in the electronic health record. Baseline: 41% (12 months before launch). Target: 20% or less within 12 months of go-live. The measure uses 8.0 g/dL as the line because guidelines recommend a restrictive threshold of 7 g/dL for most stable adults and allow 8 g/dL for patients after orthopedic surgery or with cardiovascular disease (Carson et al., 2023), so transfusion at or above 8.0 g/dL is outside guidance for nearly all in-scope patients.

Secondary Outcome Measures

Measure 2: Red cell utilization rate on the two services (units divided by patient days, times 1,000), monthly, from blood bank and census data. Baseline: 58 units per 1,000 patient days. Target: a 20% reduction within 12 months.

Measure 3: Percentage of transfusion orders for a single unit, monthly, from the electronic order record. Baseline: 42%. Target: 80%.

Process Measures

Measure 4: Percentage of red cell orders placed through the revised order set with an indication and hemoglobin value recorded, monthly, from the order record. Target: 95%. This shows whether the tool is being used at all.

Measure 5: Alert override rate, calculated as the number of alert firings where the clinician proceeded with the order divided by all alert firings, monthly, from the alert log, with override reasons summarized. No fixed target at first; the rate will be interpreted alongside Measure 1 and reasons will guide alert redesign.

Measure 6: Percentage of in-scope nurses and ordering clinicians who have completed the program's education, reported quarterly from the learning management system and department attendance records. Target: 90%.

Structure Measures

Measure 7: Presence of the program's structural elements at each quarter: blood management nurse coordinator in post; order set and alert active; feedback reports delivered on schedule. Reported as yes or no by the transfusion committee. These measures confirm that the program exists as designed before its results are judged.

Balancing Measures

Measure 8: Rapid response calls with anemia or hypotension documented as a contributing factor, per 1,000 patient days, monthly, from the rapid response team log with chart review of flagged calls. Any increase above the baseline range will prompt case review.

Measure 9: Thirty-day readmissions for patients who received a transfusion or had a nadir hemoglobin below 8.0 g/dL during admission, quarterly, from the hospital's readmission data.

Measure 10: Time from order to transfusion start for red cells ordered under the new process, monthly, from blood bank and administration timestamps, to confirm that the new order steps are not delaying transfusions that are needed.

Balancing measures are included because restricting a therapy could, in principle, cause harm through under-treatment or delay; the trial evidence behind current guidelines found that restrictive thresholds did not worsen outcomes that matter to patients in most groups studied (Carson et al., 2023), but the hospital needs its own data to confirm this in its patients.

Cost Measure

Measure 11: Estimated avoidable transfusion cost, quarterly, calculated as the reduction in units transfused above threshold compared with baseline multiplied by the agreed cost per unit of $525. The method was agreed with the finance department before launch so the figure is credible to leadership.

Interpreting Change Over Time

Most measures in the set are monthly counts or rates, and a single month can move up or down by chance. The set therefore specifies, before any data arrive, how change will be judged: each monthly measure will be plotted on a run chart with its baseline median, and a real change will be declared only when the pattern meets accepted run chart rules, such as a shift, in which at least six points in a row fall on one side of the median, or a trend of at least five points moving in one direction (Perla et al., 2011). This protects the program from two common errors: celebrating a single good month and abandoning a working change after a single bad one. Measures with small monthly counts, such as rapid response calls linked to anemia, will be reviewed case by case as well as charted, since a run chart of very small numbers can hide an important individual event.

Targets were chosen to be ambitious but grounded. The primary outcome target of 20% matches the transfusion committee's existing internal goal and is close to what other hospitals achieved with similar decision support. Process targets are set high because the tools are built into the ordering workflow, so low use would itself signal a problem worth investigating.

How the Set Works Together

The set is designed to answer three questions. Was the program delivered? Structure and process measures show whether the coordinator, tools and education are in place and used. Did practice change? The primary outcome and single-unit rate show whether transfusions above threshold and multiple-unit orders fell. Did anything get worse? Balancing measures show whether patients were harmed or care was delayed. If the primary outcome does not improve, the process measures will show whether the cause was poor uptake of the tools or a design problem, which require different fixes. The data collection and reporting plan that follows sets out how each measure will be gathered, analyzed and shared.

References

Carson, J. L., Stanworth, S. J., Guyatt, G., Valentine, S., Dennis, J., Bakhtary, S., Cohn, C. S., Dubon, A., Grossman, B. J., Gupta, G. K., Hess, A. S., Jacobson, J. L., Kaplan, L. J., Lin, Y., Metcalf, R. A., Murphy, C. H., Pavenski, K., Prochaska, M. T., Raval, J. S., . . . Pagano, M. B. (2023). Red blood cell transfusion: 2023 AABB international guidelines. JAMA, 330(19), 1892-1902. https://doi.org/10.1001/jama.2023.12914

Donabedian, A. (1988). The quality of care: How can it be assessed? JAMA, 260(12), 1743-1748. https://doi.org/10.1001/jama.1988.03410120089033

Perla, R. J., Provost, L. P., & Murray, S. K. (2011). The run chart: A simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1), 46-51. https://doi.org/10.1136/bmjqs.2009.037895