Meaningful Use Stage 2 Clinical Quality Measure Analysis Against Live Patient Data

The Meaningful Use Stage 2 Clinical Quality Measures (CQMs) reference a wide variety of clinical data elements.  As part of the Kamira research project that I am wrapping up, we jointly developed a paper that describes the results of a study with the Massachusetts eHealth Collaborative (MAeHC).  The purpose of the paper was to determine data elements which are present in a large operational patient population and to describe the impact of missing data elements as components to the CQMs.

The full paper is publicly available on the Kamira site here.

To highlight, some of the results of the study determine the presence or absence of each of the clinical codes used in MU2 CQM data elements using a large set of patient data (500k patient records covering 5M care events) collected by MAeHC.  The results were used to determine the impact on the MU2 CQMs at two levels:

• Clinical data elements were ranked according to the intersection of their component clinical codes and the codes present in the patient data 
• CQM populations were classified according to the ranks of their component clinical data elements. 

The results of this analysis were then used to determine which CQMs are likely to work well with the patient data and which require data that is not present.

Some key findings of the paper include:

• The majority of the clinical codes used in value sets referred to by the MU2 CQMs are not found in patient data. Very few of the SNOMED (0.03%) and LOINC (2%) codes used in MU2 CQMs are found in the patient data.
• Many value sets do not contain any codes that are present in the patient data: across all measures over half of the MU2 CQM value sets contain no codes. Such value sets are referred to as non-intersecting.
• There are many measure populations containing only non-intersecting value sets. Across all measures, nearly 60% of distinct denominators and 27% of numerators reference only data that is not present in the patient records. Such measures will, by definition, report 0/0 or 0/n results.

Below is an illustration taken from the paper that details code occurrence of what is available in Meaningful Use Stage 2 for ICD-9, LOINC, SNOMED, CPT, CVX and RxNorm. Each bar represents all of the codes from the respective code system defined in Meaningful Use Stage 2 for the Eligible Professional CQMs.  The blue segments of each bar show the percentage of codes that are present in the system.

Another compelling illustration is the value set intersection results.  A value set contains one or more codes from one or more clinical vocabularies for a clinical concept, like "Diabetes". Given the code occurrence results above it is possible to compute the intersection of each Meaningful Use Stage 2 CQM value set with the patient data as follows: I=P/T where I is the intersection, P represents the number of codes in the value set that are present in the data and T represents the total number of codes in the value set.

Below, each bar represents the total number of value sets for either both Eligible Hospital and Eligible Professional CQMs (All), just the Eligible Hospital (EH) CQMs, or Eligible Professional (EP) CQMs.  Each bar is subdivided to show the proportion of value sets with different levels of intersection.

A key takeaway from this paper is that the majority of the codes used in the value sets referred to by the Meaningful Use Stage 2 Eligible Professional CQMs are not found in the operational patient data.

Only finding a minority of the codes would not necessarily be a problem... value sets may contain many codes and provided one of those codes is present in the patient data, the corresponding data element will be also be satisfied.  Unfortunately, the value set intersection results show that many value sets contain no codes actually present in the patient data. Across all measures over half of the MU2 CQM value sets contain no codes.  The primary reason for non-intersecting value sets is choice of code sets: 73% of the non-intersecting value sets include only some combination of SNOMED-CT, LOINC and ICD-10 codes.  All of these are scarce in the live/operational patient data.

Additionally, the non-intersecting value sets are not distributed evenly across all measure populations where their impact would be lessened.  Instead, there are many CQM populations containing only non-intersecting value sets.  Across all measures, nearly 60% of distinct denominators and 27% of numerators reference only data that is not present.  Such CQMs will, by definition, report 0/0 or 0/n results… every time.  Yikes!!!

Admittedly, this is only one sample point of data using a single practice.  However, it is data from coded clinical source used by a national leader in healthcare information technology.  If this issue is present in other operational EHR systems (as I strongly suspect) the only way to address this problem moving forward in MU Stage 3 will be to re-consider the required clinical codes that need to be capture in the EHR systems used operationally... or refine the translations from clinical concepts to clinical codes.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.