Meaningful Use Stage 2 Clinical Quality Measure Analysis Against Live Patient Data

The Meaningful Use Stage 2 Clinical Quality Measures (CQMs) reference a wide variety of clinical data elements.  As part of the Kamira research project that I am wrapping up, we jointly developed a paper that describes the results of a study with the Massachusetts eHealth Collaborative (MAeHC).  The purpose of the paper was to determine data elements which are present in a large operational patient population and to describe the impact of missing data elements as components to the CQMs.

The full paper is publicly available on the Kamira site here.

To highlight, some of the results of the study determine the presence or absence of each of the clinical codes used in MU2 CQM data elements using a large set of patient data (500k patient records covering 5M care events) collected by MAeHC.  The results were used to determine the impact on the MU2 CQMs at two levels:

• Clinical data elements were ranked according to the intersection of their component clinical codes and the codes present in the patient data 
• CQM populations were classified according to the ranks of their component clinical data elements. 

The results of this analysis were then used to determine which CQMs are likely to work well with the patient data and which require data that is not present.

Some key findings of the paper include:

• The majority of the clinical codes used in value sets referred to by the MU2 CQMs are not found in patient data. Very few of the SNOMED (0.03%) and LOINC (2%) codes used in MU2 CQMs are found in the patient data.
• Many value sets do not contain any codes that are present in the patient data: across all measures over half of the MU2 CQM value sets contain no codes. Such value sets are referred to as non-intersecting.
• There are many measure populations containing only non-intersecting value sets. Across all measures, nearly 60% of distinct denominators and 27% of numerators reference only data that is not present in the patient records. Such measures will, by definition, report 0/0 or 0/n results.

Below is an illustration taken from the paper that details code occurrence of what is available in Meaningful Use Stage 2 for ICD-9, LOINC, SNOMED, CPT, CVX and RxNorm. Each bar represents all of the codes from the respective code system defined in Meaningful Use Stage 2 for the Eligible Professional CQMs.  The blue segments of each bar show the percentage of codes that are present in the system.

Another compelling illustration is the value set intersection results.  A value set contains one or more codes from one or more clinical vocabularies for a clinical concept, like "Diabetes". Given the code occurrence results above it is possible to compute the intersection of each Meaningful Use Stage 2 CQM value set with the patient data as follows: I=P/T where I is the intersection, P represents the number of codes in the value set that are present in the data and T represents the total number of codes in the value set.

Below, each bar represents the total number of value sets for either both Eligible Hospital and Eligible Professional CQMs (All), just the Eligible Hospital (EH) CQMs, or Eligible Professional (EP) CQMs.  Each bar is subdivided to show the proportion of value sets with different levels of intersection.

A key takeaway from this paper is that the majority of the codes used in the value sets referred to by the Meaningful Use Stage 2 Eligible Professional CQMs are not found in the operational patient data.

Only finding a minority of the codes would not necessarily be a problem... value sets may contain many codes and provided one of those codes is present in the patient data, the corresponding data element will be also be satisfied.  Unfortunately, the value set intersection results show that many value sets contain no codes actually present in the patient data. Across all measures over half of the MU2 CQM value sets contain no codes.  The primary reason for non-intersecting value sets is choice of code sets: 73% of the non-intersecting value sets include only some combination of SNOMED-CT, LOINC and ICD-10 codes.  All of these are scarce in the live/operational patient data.

Additionally, the non-intersecting value sets are not distributed evenly across all measure populations where their impact would be lessened.  Instead, there are many CQM populations containing only non-intersecting value sets.  Across all measures, nearly 60% of distinct denominators and 27% of numerators reference only data that is not present.  Such CQMs will, by definition, report 0/0 or 0/n results… every time.  Yikes!!!

Admittedly, this is only one sample point of data using a single practice.  However, it is data from coded clinical source used by a national leader in healthcare information technology.  If this issue is present in other operational EHR systems (as I strongly suspect) the only way to address this problem moving forward in MU Stage 3 will be to re-consider the required clinical codes that need to be capture in the EHR systems used operationally... or refine the translations from clinical concepts to clinical codes.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

popHealth v2.4 Design for Meaningful Use Stage 2

As some of you know, I have been leading the open source popHealth project at MITRE for the past 4 years.  The MITRE popHealth team is currently working for the Veterans Health Administration (VHA).  Our work is focused on using popHealth as the software service allow the VHA to meet Meaningful Use Stage 2 certification for Clinical Quality Measures (CQMs).

Since we hadn't been working on popHealth in any substantial level in most of 2013, so we identified some problems for ongoing operations and maintenance (O&M) of the popHealth software when we started to put more and more engineers back on the project.  The concerns that the popHealth team had was the software that manages the User Interface (UI) presentation layer in the browser was becoming too complex and convoluted.  This had the potential for introducing difficulty when a non-MITRE contractor will eventually manage the popHealth software in an "O&M" mode for the VHA.  This will happen when our team completes the development task and moves off of this project.

What the popHealth team opted to do was to clean up the JavaScript and HTML so that there was an Application Programming Interface (API) that could provide JavaScript Object Notation (JSON) for the popHealth data needed for the UI directly to the browser.  This is a cleaner software design that also allows user to forgo the existing popHealth UI if they need to provide just the raw CQM data.

In the process of refactoring the popHealth web application infrastructure, the team opted to introduce some tweaks to the popHealth presentation layer but still maintain the existing interaction model.  What we decided to do was only slightly evolve the existing popHealth branding and look and feel, so that is is more maintainable from an engineers perspective, but does not introduce a radically different interaction model require re-training for all our existing users.

The popHealth v1.4 user interface that was aligned with the Meaningful Use Stage 2 program is below:

popHealth v1.4 Dashboard Design for Meaningful Use Stage 1

In the process of refactoring the popHealth web application infrastructure, the popHealth design team opted to introduce some tweaks to the presentation layer but still maintains the established L&F and interaction model.  The latest updated designs for popHealth v2.4 follow:

popHealth v2.4 Dashboard Design for Meaningful Use Stage 2

Changes in the evolved popHealth v2.4 design include:

• Eliminated confusing links for the "parameters" of each CQM on the right; now, there is just a link to access the patients.
• Reduced the number of colors and focused on a more muted and simplified color pallet 
• Eliminated the single horizontal bar visualization and replaced it with two stacked bars for the fractions
• Identified the exception and exclusion population with muted gray bars on the stacked bar charts
• Enhanced the performance rate fraction with a visual radial circle around the fraction value

If anyone has opinions on the latest designs, the team welcomes any and all constructive feedback.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.