Complexity and Certification of MU Stage 1 Eligible Professional Clinical Quality Measures

While working in the Clinical Quality Measure space on the two open source projects popHealth and Cypress, I have observed trends in the adoption of EHR vendors of various Clinical Quality Measures (CQMs).  For a little background on the role of CQMs in the Meaningful Use program, read the beginning of the entry that I wrote about Applying Kiviat Visualization to Meaningful Use Clinical Quality Measures. 

For Meaningful Use Stage 1, there are minimal requirements by EHR vendors to support the 6 Core and Core Alternate CQMs, and any 3 of the remaining 38 Meaningful Use Stage 1 CMS.  This allows for some malleability by the commercial EHR vendors to select CQMs based on either their ability to implement CQM logic in their product or based on customer demand for specific CQMs.

The Office of the National Coordinator for Health Information Technology (ONC) hosts the Certified HealthIT Product List (CHPL pronounced "CHaPeL") service.  You can view the list of certified products through the CHPL web interface.  Compiling the results for the EHR products against various Meaningful Use Stage 1 CQMs, there are some interesting results:

Meaningful Use Stage 1 Ambulatory Clinical Quality Measures:
Adoption by EHR vendors from data collected via the CHPL service

In addition to the work I am leading via Cypress, I am also leading a research project to assess the quality of Clinical Quality Measures called "Kamira".  The Kamira project can provide metrics on the quality of CQMs.  For early 2013, this has included automated Cyclomatic Complexity calculation of the CQM algorithmic logic based off of the JavaScript code that the Cypress and popHealth projects use to calculate the CQM results.

If you then compare the results of the CQMs that were tested and certified by EHR vendors against the complexity score of the CQMs, you can see a weak correlation between the two.  You can download the full file with the results here.

Meaningful Use Stage 1 Ambulatory Clinical Quality Measures:
Adoption by EHR vendors from data collected via the CHPL service
compared against Cyclomatic Complexity Analysis of the CMQ logic

It's worth noting that the correlation here is weak, but there does appear to be a trend toward vendors opting to implement the less complex CQMs in their products when they have some latitude to choose.

FYI, the ranges for the CQM complexity (the colored diamonds) are:

• 1-10 Very Simple/Low Risk (green)
• 11-20 Nominal/Moderate Risk (yellow)
• 21-50 Complex/High Risk (orange)
• >50 Untestable/Extreme Risk (red)

These metrics are somewhat arbitrary, but I picked ranges for CQM complexity from a Carnegie Mellon paper that had a good number of citations, so I think that the values and thresholds are fairly defensible.

Where this work might have a few vulnerabilities is that I am 100% certain that EHR vendors do not use complexity as their only consideration when selecting CQMs to implement in their products.  For instance some of the red, highly complex CQMs which were in the middle when it came to adoption by EHR vendors are cardiac CQMs.  From my perspective, it's a safe assumption that some of these EHR vendors were going to bite bullet and implement the cardiac CQMs regardless of the complexity associated with them because there is more demand in the marketplace from providers that need the cardiac CQM results, vs. say the behavioral health CQMs.  However, I think that CQM developers need to start tracking complexity of CQMs as they are developed for MU Stage 3 or beyond.

Lastly, the Kamira project just launched last week.  I plan on posting the MU Stage 2 complexity results for the Eligible Professional CQMs in the coming weeks.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.