Unlike in many areas of medical practice, measuring the relevant outcomes to assess performance in anesthesia services is difficult. Assessments of anesthesia services are impeded by additional levels of complexity aside from those that already affect benchmarking in other specialties. These difficulties largely lead to imprecision, as some of the methods used to assess effectiveness lack granularity and consistency.
Because anesthesia services are heavily entwined in the work of the surgeon, attributing a particular outcome to differences in quality of anesthesia services is difficult. That is, a systematic decline in perioperative outcomes could just as easily be attributed to a decline in performance during surgery as to a decline in performance of anesthesia team. The two are too tightly intertwined to measure varying outcomes under a constant level of surgical quality but varying anesthesia quality.
In addition to the noise that surgery introduces into an outcomes-focused evaluation of anesthesia, measuring mortality and serious morbidity as an indicator of anesthesia effectiveness is an intrinsically imprecise method. The rates of both mortality and serious morbidity that are tied to the use of anesthesia have fallen to such low levels that tracking them will only identify the very worst cases, and in fact will tend to identify cases where some exogenous event is responsible for the outcome.
A lack of serious review has stalled an empirically-informed consensus on benchmarking in anesthesia services. A study conducted by Guy Haller and his colleagues examined a total of 108 indicators used to measure the clinical effectiveness of anesthesiology, 57% of which were derived from outcomes of care while most of the remainder (42%) were derived from the process of administering care. Of those 108 indicators, only 40% had been validated by any measure beyond an expert opinion.
Prior to Haller’s and his colleagues’ study, no review had been conducted specifically intended to evaluate indicators of effectiveness for anesthesia services. The most closely related study published prior to this point identified 37 indicators of effectiveness among 27 articles, but of those only 2 such indicators related directly to anesthesia. Furthermore, those two indicators focused on outcomes, specifically death and other adverse events after surgery, which are inherently imprecise ways to assess the effectiveness of anesthesia.
At least in part, this dearth of systematic research examining indicators of effectiveness for anesthesia stems from a mismatch between the methods of academic research and the tendencies of healthcare providers in codifying their metrics. Most studies of benchmarking in healthcare have drawn upon the published literature on the subject, while Haller’s study utilized user’s manuals, web sites, and other such resources in order to gather a set of indicators that they could review. Measures of effectiveness for anesthesia tend to be described in these types of references rather than academic research.
Inconsistencies in how different providers characterize quality of anesthesia also creates challenges in constructing valid benchmarks. Some research groups will report adverse events only when they warrant a legal or disciplinary remedy, while others characterize adverse outcomes as events that result in mortality or severe morbidity. This inconsistency between research groups compounds the challenges of using outcome-focused metrics to assess the quality of services.
Additionally, different research groups define the perioperative period differently. Some such groups consider the perioperative period to terminate once the patient is discharged, while others consider the period to continue for 24 hours or more after the fact. This variation introduces further uncertainty into the benchmarking process by complicating the relationship between anesthesia and surgery. Not only are the two interlinked, but the exact nature of that linkage is not consistently documented across research groups.
Furthermore, the collection of data is inconsistent across research groups. Some rely upon voluntary reporting of outcomes while others conduct studies that examine all incidents within a given interval of time. This discrepancy introduces the potential for bias, in particular for an asymmetric bias, as voluntary reporting of incidents is likely to be systematically different from exhaustive studies over a finite period of time.
The challenges of measuring anesthesia effectiveness suggest that more process-focused indicators are needed to reliably benchmark this type of care and more robust studies are needed to validate those indicators.
References
https://academic.oup.com/bja/article/109/1/80/236985#2749964
https://academic.oup.com/bja/article/85/2/271/1745481#91202663
