Bonnie: An Open Source Clinical Quality Measure Testing Tool

Bonnie is a new open source software tool that MITRE has developed and released in April 2014 that allows Meaningful Use (MU) Clinical Quality Measure (CQM) developers to test and verify the behavior of their CQM logic.  The goal of Bonnie is to reduce the number of defects in CQMs by providing a robust and automated testing framework. Bonnie allows measure developers to independently load measures that they have constructed using the Measure Authoring Tool (MAT). Loading the measures into Bonnie converts the measures from their Extensible Markup Language (XML) eSpecifications into executable artifacts and measure metadata.

Bonnie Dashboard Page

The measure eSpecification format that Bonnie loads is Health Quality Measure Format (HQMF) XML. The HQMF specification provides the metadata and logic that describe the specifics of calculating a CQM. Bonnie can load the HQMF describing a measure and programmatically convert the HQMF specification into an executable format that allows calculating the measure directly from the specification. 

The measure metadata loaded into Bonnie is then used to allow developers to rapidly build a synthetic patient test deck for the measure using the clinical elements defined during the measure construction process. By using measure metadata as a basis for building synthetic patients, developers can rapidly and efficiently create a test deck for a measure. 

Once a CQM has been loaded into Bonnie, a user can inspect the measure logic and then build synthetic test records and set expectations on how those test records will calculate against a measure. This capability to build synthetic test patient records, set expectations against those records, and calculate the measures using those patient records provides an automated and efficient testing framework for CQMs. 

Using the Bonnie-supported CQM testing framework allows measure developers to more clearly understand the behavior of the measure logic, validate that the measure logic encodes their intent, and allows for multiple iterations of measure updates to be validated against a test deck. 

Bonnie Measure Page

Additionally, the development of a test deck as part of measure development provides benefits after the measures are finalized. The test deck build as part of measure development can be used to demonstrate the intent of the measure though the use of patient examples included in the test deck. Furthermore, the test deck provides systems that implement the measures with a means to validate the development of their systems. This is provided in the form of a base set of synthetic patient records with known expectations for calculating against the implemented measures. Finally, the test deck could be used as a basis for the test deck used as part of the Meaningful Use certification program. 

Bonnie has been designed to integrate with the nationally recognized data standards used by the Meaningful Use program for expressing CQM logic for machine-to-machine interoperability. This provides enormous value to the CQM program and federal policy leaders and stakeholders: this software tool verifies that the new and evolving standards for the Meaningful Use CQM program are tractable and can be implemented in software.   

Additionally, Bonnie was designed to provide an intuitive and easy-to-use interface based on feedback from the broader measure developer community. A key goal of the Bonnie application is to deliver a user experience that provides an efficient and intuitive method for constructing synthetic patient records for testing and validating CQMs. 

The Bonnie software is freely available via an Apache 2.0 open source license. The Meaningful Use program makes all or parts of the Bonnie software available for inspection, verification, and even reuse by other government programs or federal contractors. 

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2014.

Conceptual Hospital Dashboard Design

I have been neglecting the "design" aspect of this blog for some time.  To re-engergize the need for more engaging content, to put some eye-candy out... and have some fun on a late Friday afternoon… I thought I would share a shelved design for a dashboard that I developed for a COO at a notional hospital:

Notional Hospital COO Dashboard Design

I have made several assumptions when coming up with this mockup.  The first is that quality and quality metrics would be of utmost importance to this user.  Maybe pay-for-performance programs have been adopted based on Clinical Quality Measures (CQMs), and quality issues have an immediate impact on the bottom line of the organization.  I included both a kiviat visualization for showing targets against measured results for CQMs, and a longitudinal trend for those CQMs over time.

With the disclaimer that I am not a clinician; I made some assumptions that knowing the types of procedures that are being performed, and when those procedures deviate from expected norms is something else that a COO would track.

Lastly, I am showing a design technique that Stephen Few will sometimes use to provide a non-white background to reduce eyestrain.  Having met him several years ago, I am a fan of Stephen's work.  I always like the muted background color that he tends to include in his works.

I welcome the feedback.

Kudos if you "get" the names of the ten physicians.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

Applying Cost To Clinical Quality Measures

With the Kamira research project that I am leading, our team was able to instrument the Meaningful Use (MU) Stage 2 Clinical Quality Measure (CQM) logic with the procedural logic for calculating the results of the CQMs via the popHealth project.  What this popHealth CQM calculation software provided the Kamira team was the clinical building blocks of the CQM logic, which we were able to marry with publicly available claims data hosted by CMS based on de-identified real-life claims records.

With the CQM calculation land this collection of claims records, we were able to asses ranges of the cost associated with addressing the MU Stage 2 CQMs.  With this, we were able to provide a guess for how much cost would be introduced into the healthcare system if a provider were to attempt to address the numerator logic of an MU Stage 2 CQM.

Consider the MU Stage 2 CQM "NQF 0062: Diabetes Urine Protein Screening".  This CQM measures the percentage of patients 18-75 years of age diagnosed with diabetes who had a some form of urine screening during the measurement period.  However, if you look at the conditional logic of this CQM, you can see that having the dialysis procedure would 

Procedural numerator logic for "NQF 0062: Diabetes Urine Protein Screening"

When we applied the CMS claims data, you can see the wide range of costs associated with this particular CQM's numerator logic, spanning microalbumin testing, ACE inhibitor, and access to dialysis:

Numerator costs on NQF 0062 - Diabetes Urine Protein Screening

As a non-clinician, the first two tests in the numerator logic make sense to me (microalbumin lab tests).  They appear to be in the "spirit" of this CQM, measuring if the provider has performed the microalbumin lab test for kidney damage.  However, the Kamira automated cost assessment highlighted that the vascular access for dialysis is clearly a much more expensive procedure to address this numerator logic than the lab test.  We didn't have any claims data on the kidney transplant from our sample set, but I can only speculate that the cost for that claim would be in the six figure range.

Looking purely at the CQM logic, and not applying any clinical perspective on this CQM, the vascular access for dialysis procedure is viewed as equivalent to the microalbumin lab test, at least within the scope of in this CQM logic.  Both are equal when assessing if the provider is performing the best quality of care for their population of diabetic patients.  Again... a kidney transplant is also in that same category as being semantically equivalent to the microalbumin lab test!!!

On the flip side, it appears to me that vascular access for dialysis is meant to address a known clinical problem, whereas the microalbumin lab test is designed to only collect and present information to a provider.  This difference in the clinical purpose of the clinical activities is not considered in the CQM logic.  Again, the CQM logic views both as equivalent when measuring the quality of care that a provider is applying to their population of patients.

Based on the Kamira automated CQM cost analysis, my recommendation is to groom the logic of "NQF 0062: Diabetes Urine Protein Screening", and re-position the dialysis procedure probably belongs in the exception or exclusion logic, vs. the current numerator logic.  There are some additional opportunities to apply this cost consideration on all Meaningful Use Stage 3 CQMs.  By applying these cost metrics to the MU Stage 3 CQMs, at minimum, the policy makers could discuss the impact of cost with the numerator logic clinical data.  Additionally, these conversations could potentially lead to streamlining and pruning the MU Stage 3 CQM logic to avoid noisy and high variance healthcare costs within the logic of one CQM.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.