Bonnie: An Open Source Clinical Quality Measure Testing Tool

Bonnie is a new open source software tool that MITRE has developed and released in April 2014 that allows Meaningful Use (MU) Clinical Quality Measure (CQM) developers to test and verify the behavior of their CQM logic.  The goal of Bonnie is to reduce the number of defects in CQMs by providing a robust and automated testing framework. Bonnie allows measure developers to independently load measures that they have constructed using the Measure Authoring Tool (MAT). Loading the measures into Bonnie converts the measures from their Extensible Markup Language (XML) eSpecifications into executable artifacts and measure metadata.

Bonnie Dashboard Page

The measure eSpecification format that Bonnie loads is Health Quality Measure Format (HQMF) XML. The HQMF specification provides the metadata and logic that describe the specifics of calculating a CQM. Bonnie can load the HQMF describing a measure and programmatically convert the HQMF specification into an executable format that allows calculating the measure directly from the specification. 

The measure metadata loaded into Bonnie is then used to allow developers to rapidly build a synthetic patient test deck for the measure using the clinical elements defined during the measure construction process. By using measure metadata as a basis for building synthetic patients, developers can rapidly and efficiently create a test deck for a measure. 

Once a CQM has been loaded into Bonnie, a user can inspect the measure logic and then build synthetic test records and set expectations on how those test records will calculate against a measure. This capability to build synthetic test patient records, set expectations against those records, and calculate the measures using those patient records provides an automated and efficient testing framework for CQMs. 

Using the Bonnie-supported CQM testing framework allows measure developers to more clearly understand the behavior of the measure logic, validate that the measure logic encodes their intent, and allows for multiple iterations of measure updates to be validated against a test deck. 

Bonnie Measure Page

Additionally, the development of a test deck as part of measure development provides benefits after the measures are finalized. The test deck build as part of measure development can be used to demonstrate the intent of the measure though the use of patient examples included in the test deck. Furthermore, the test deck provides systems that implement the measures with a means to validate the development of their systems. This is provided in the form of a base set of synthetic patient records with known expectations for calculating against the implemented measures. Finally, the test deck could be used as a basis for the test deck used as part of the Meaningful Use certification program. 

Bonnie has been designed to integrate with the nationally recognized data standards used by the Meaningful Use program for expressing CQM logic for machine-to-machine interoperability. This provides enormous value to the CQM program and federal policy leaders and stakeholders: this software tool verifies that the new and evolving standards for the Meaningful Use CQM program are tractable and can be implemented in software.   

Additionally, Bonnie was designed to provide an intuitive and easy-to-use interface based on feedback from the broader measure developer community. A key goal of the Bonnie application is to deliver a user experience that provides an efficient and intuitive method for constructing synthetic patient records for testing and validating CQMs. 

The Bonnie software is freely available via an Apache 2.0 open source license. The Meaningful Use program makes all or parts of the Bonnie software available for inspection, verification, and even reuse by other government programs or federal contractors. 

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2014.

Conceptual Hospital Dashboard Design

I have been neglecting the "design" aspect of this blog for some time.  To re-engergize the need for more engaging content, to put some eye-candy out... and have some fun on a late Friday afternoon… I thought I would share a shelved design for a dashboard that I developed for a COO at a notional hospital:

Notional Hospital COO Dashboard Design

I have made several assumptions when coming up with this mockup.  The first is that quality and quality metrics would be of utmost importance to this user.  Maybe pay-for-performance programs have been adopted based on Clinical Quality Measures (CQMs), and quality issues have an immediate impact on the bottom line of the organization.  I included both a kiviat visualization for showing targets against measured results for CQMs, and a longitudinal trend for those CQMs over time.

With the disclaimer that I am not a clinician; I made some assumptions that knowing the types of procedures that are being performed, and when those procedures deviate from expected norms is something else that a COO would track.

Lastly, I am showing a design technique that Stephen Few will sometimes use to provide a non-white background to reduce eyestrain.  Having met him several years ago, I am a fan of Stephen's work.  I always like the muted background color that he tends to include in his works.

I welcome the feedback.

Kudos if you "get" the names of the ten physicians.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

Review: CardioChek PA Blood Meter

I have been working for the Office of the National Coordinator for Health Information Technology (ONC), on the popHealth and Cypress projects for several years now.  Both projects are based around Clinical Quality Measures (CQMs).

By education, I am a physicist and engineer, and not a clinician.  However, I have repeatedly noticed the importance of lipid profiles within the context of the logic of several of the Meaningful Use CQMs.  Further, in the very recent past, these metrics had been considerably high for me; a male in my late 30's.  Between my work looking at CQMs emphasizing the importance of just measuring lipid profiles, and my own personal warning signs associated with this health metric, I thought it would be good to collect some "hands on" experiences with the collection of the data for this metric.

When my department at MITRE had some additional overhead resources available at the end of our fiscal year, I purchased a portable blood testing device that could provide me with my own lipoid profile information (total cholesterol, HDL cholesterol, LDL cholesterol, and triglycerides).

I eventually picked the CardioChek PA blood meter device.  Interestingly, this was not the first CardioChek blood device that I purchased.  I originally found the consumer CardioChek (no "PA") device on Amazon.com.  The consumer device lists at about $125.  That seemed reasonable, but the device requires three separate strips to measure total cholesterol, HDL cholesterol, and triglycerides (you can calculate the LDL cholesterol from those three).  While that may not sound bad, I quickly found out that it involved giving myself at least two, sometimes three, pricks with lancets to draw enough blood for the three separate tests.

Additionally, I purchased this device for my department at work, and felt that an experience getting three pokes with a needle might not go over well with my colleagues, resulting in some future retribution via office pranks.

To solve the problem of collecting a full lipoid panel from one drop of blood, I purchased these PTS panels that appeared nice in the sense that they have the ability to drive the calculation of multiple readings from a single sample of blood.

Each box comes with one lipid panel MEMo chip that can be inserted
into a CardioChek PA device and 15 single use lipid profile panels

Unfortunately, I quickly discovered that these really nice 3-in-1 lipid profile panels are not compatible with the $125 consumer CardioChek device.  To support multiple readings from a single drop of blood would require the more expense CardioChek PA device, that runs at close to $700.

Being the end of the fiscal year, the additional money was a little easy to come by.  I went back to our finance staff and purchased the more expensive, clinical-grade device.  I also picked up some supporting medical equipment like gloves, lancets, pipettes, and band aids.

CardioChek PA blood testing device, several tests
with some additional medical equipment

To take your lipid profile, you need one lipid panel test chip, called a MEMo Chip by the manufacturer, and one lipid panel test strip.  The MEMo chip contains lot-specific calibration and other information needed to properly perform testing.  The lot-specific information is presumably associated with the 15 test panels that come with the package.  I would not recommend mixing and matching test panels with different MEMo chips, because of this prior calibration by the manufacturer.

There is also guidance around always storing the unused test panels at a temperature between 68-80˚F.  I could see this temperature requirement as a challenge for some home/consumer users.  Lastly, there is an expiration date on the test panels.  For all the tests I have, the expiration date, is less than one year from now, a little under 8 months of viable shelf time.

You can see the relative size of the test panel and MEMo chip in the picture below.  You only need one strip for a test.  I just flipped one test strip over to show the single wide channel where you deposit your blood, and the three openings for the device sensor to read the total cholesterol, HDL cholesterol, and triglycerides.

Lipoid panel test chip and two lipoid panel test strips

It appears that the CardioChek PA device has the ability to independently test up to four different metrics from the single sample of blood.  While I haven't found any tests that use a full four metrics from one sample, I am still happy that the manufacturer (presumably) recognized this need for multiple tests from a single sample.

CardioChek PA sensor

Collecting and depositing your sample for the device is relatively easy for an individual, non-clinician.  I suggest you get 2 paper towels, and a small band aid before you get started.  If you are unfamiliar with using lancets, they are small and cheap medical implements used for capillary blood sampling (no vein or arties). A lancet includes a spring-loaded needle.  When used, it pops out and makes a very small puncture in your skin, allowing a few drops of blood to appear on your skin over the next ~10 seconds.  They are single use and disposable.

You can see how they are fairly straight-forward to use for the blood sampling that you can then collect with pipettes.

Lancet

About one drop of blood after lancet met my finger
with small pipette in background

After my first two attempts to use the device for a full lipid profile, the device would eventually display "TEST ERROR" on the screen.  Needless to say, I was disappointed to think that I had put over $1K into this exercise, and had no data to show.  As it turns out, my finger was not providing enough blood to generate the full lipid profile.  There is some documentation provided with the device that makes this association from the "TEST ERROR" message.  However, I just can't understand why the manufacturer didn't make this more intuitive for users.

On my third attempt, when I applied a liberal amount of blood on the sample channel, the device worked fine.  I feel that the accuracy of the device is very high.  The readings it provided were all within 10% of measurements that I had collected from my Primary Care Provider (PCP) the previous week.  For me, the time between depositing your blood on the strip until the results are displayed ranged from 40 to around 60 seconds on three test runs.

This problem that resulted in multiple "pokes" with the lancet makes for an amusing story at my expense.  However, I can't emphasize enough; the "TEST ERROR" message really means "Not enough blood".  This is an opportunity for improvement in this product for first time users.

On this topic of human-computer interaction, I was also disappointed with the CardioChek PA user interface.  For $700, the interface feels like bleeding-edge late 1980s technology.

CardioChek PA user interface

For a $700 device, I would think that the manufacturer could easily upgrade the resolution and include color at a modest increase in manufacturing costs.  Ideally, this would include some longitudinal data on changes associated with the data.  See my suggested illustration below, again based on Juhan Sonin's designs for a HealthCard for the patient (consumer).

My updated lipid profile with sparkline visualizations

On the positive side, the CardioCheck PA is worth purchasing if you think you will be frequently taking your lipid profile at home.  It appears very accurate on the measurement results.  Once you learn the interface and how to correctly provide enough of a blood sample, the device works well.

My biggest issue with this product is the variance in the cost of the CardioChek PA device at $700, versus a slightly more limited consumer CardioChek device at $125.  I feel that this price point makes the CardioChek cost-prohibitive for most consumers (patients).

I would grade the CardioChek PA device a B- for the purposes of home users.

Somewhat related to where this may go in 5-10 years, I was able to identify an amazing illustration, showing various ranges for numerous blood metrics:

Reference ranges for blood tests

Knowing that a single drop of blood could theoretically yield all these metrics makes for some interesting ideas about how the consumer could have access to these metrics on a daily basis, at their home.

Another interesting opportunity for this market would be to introduce a blood sensor without the embedded interface that could communicate with an iPhone, similar to how the Withings BP Cuff works.  I have that Withings device at home, and plan on developing a review of that device later.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2012.