Meaningful Use Stage 2 Clinical Quality Measure Analysis Against Live Patient Data

The Meaningful Use Stage 2 Clinical Quality Measures (CQMs) reference a wide variety of clinical data elements.  As part of the Kamira research project that I am wrapping up, we jointly developed a paper that describes the results of a study with the Massachusetts eHealth Collaborative (MAeHC).  The purpose of the paper was to determine data elements which are present in a large operational patient population and to describe the impact of missing data elements as components to the CQMs.

The full paper is publicly available on the Kamira site here.

To highlight, some of the results of the study determine the presence or absence of each of the clinical codes used in MU2 CQM data elements using a large set of patient data (500k patient records covering 5M care events) collected by MAeHC.  The results were used to determine the impact on the MU2 CQMs at two levels:

• Clinical data elements were ranked according to the intersection of their component clinical codes and the codes present in the patient data 
• CQM populations were classified according to the ranks of their component clinical data elements. 

The results of this analysis were then used to determine which CQMs are likely to work well with the patient data and which require data that is not present.

Some key findings of the paper include:

• The majority of the clinical codes used in value sets referred to by the MU2 CQMs are not found in patient data. Very few of the SNOMED (0.03%) and LOINC (2%) codes used in MU2 CQMs are found in the patient data.
• Many value sets do not contain any codes that are present in the patient data: across all measures over half of the MU2 CQM value sets contain no codes. Such value sets are referred to as non-intersecting.
• There are many measure populations containing only non-intersecting value sets. Across all measures, nearly 60% of distinct denominators and 27% of numerators reference only data that is not present in the patient records. Such measures will, by definition, report 0/0 or 0/n results.

Below is an illustration taken from the paper that details code occurrence of what is available in Meaningful Use Stage 2 for ICD-9, LOINC, SNOMED, CPT, CVX and RxNorm. Each bar represents all of the codes from the respective code system defined in Meaningful Use Stage 2 for the Eligible Professional CQMs.  The blue segments of each bar show the percentage of codes that are present in the system.

Another compelling illustration is the value set intersection results.  A value set contains one or more codes from one or more clinical vocabularies for a clinical concept, like "Diabetes". Given the code occurrence results above it is possible to compute the intersection of each Meaningful Use Stage 2 CQM value set with the patient data as follows: I=P/T where I is the intersection, P represents the number of codes in the value set that are present in the data and T represents the total number of codes in the value set.

Below, each bar represents the total number of value sets for either both Eligible Hospital and Eligible Professional CQMs (All), just the Eligible Hospital (EH) CQMs, or Eligible Professional (EP) CQMs.  Each bar is subdivided to show the proportion of value sets with different levels of intersection.

A key takeaway from this paper is that the majority of the codes used in the value sets referred to by the Meaningful Use Stage 2 Eligible Professional CQMs are not found in the operational patient data.

Only finding a minority of the codes would not necessarily be a problem... value sets may contain many codes and provided one of those codes is present in the patient data, the corresponding data element will be also be satisfied.  Unfortunately, the value set intersection results show that many value sets contain no codes actually present in the patient data. Across all measures over half of the MU2 CQM value sets contain no codes.  The primary reason for non-intersecting value sets is choice of code sets: 73% of the non-intersecting value sets include only some combination of SNOMED-CT, LOINC and ICD-10 codes.  All of these are scarce in the live/operational patient data.

Additionally, the non-intersecting value sets are not distributed evenly across all measure populations where their impact would be lessened.  Instead, there are many CQM populations containing only non-intersecting value sets.  Across all measures, nearly 60% of distinct denominators and 27% of numerators reference only data that is not present.  Such CQMs will, by definition, report 0/0 or 0/n results… every time.  Yikes!!!

Admittedly, this is only one sample point of data using a single practice.  However, it is data from coded clinical source used by a national leader in healthcare information technology.  If this issue is present in other operational EHR systems (as I strongly suspect) the only way to address this problem moving forward in MU Stage 3 will be to re-consider the required clinical codes that need to be capture in the EHR systems used operationally... or refine the translations from clinical concepts to clinical codes.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

popHealth v2.4 Design for Meaningful Use Stage 2

As some of you know, I have been leading the open source popHealth project at MITRE for the past 4 years.  The MITRE popHealth team is currently working for the Veterans Health Administration (VHA).  Our work is focused on using popHealth as the software service allow the VHA to meet Meaningful Use Stage 2 certification for Clinical Quality Measures (CQMs).

Since we hadn't been working on popHealth in any substantial level in most of 2013, so we identified some problems for ongoing operations and maintenance (O&M) of the popHealth software when we started to put more and more engineers back on the project.  The concerns that the popHealth team had was the software that manages the User Interface (UI) presentation layer in the browser was becoming too complex and convoluted.  This had the potential for introducing difficulty when a non-MITRE contractor will eventually manage the popHealth software in an "O&M" mode for the VHA.  This will happen when our team completes the development task and moves off of this project.

What the popHealth team opted to do was to clean up the JavaScript and HTML so that there was an Application Programming Interface (API) that could provide JavaScript Object Notation (JSON) for the popHealth data needed for the UI directly to the browser.  This is a cleaner software design that also allows user to forgo the existing popHealth UI if they need to provide just the raw CQM data.

In the process of refactoring the popHealth web application infrastructure, the team opted to introduce some tweaks to the popHealth presentation layer but still maintain the existing interaction model.  What we decided to do was only slightly evolve the existing popHealth branding and look and feel, so that is is more maintainable from an engineers perspective, but does not introduce a radically different interaction model require re-training for all our existing users.

The popHealth v1.4 user interface that was aligned with the Meaningful Use Stage 2 program is below:

popHealth v1.4 Dashboard Design for Meaningful Use Stage 1

In the process of refactoring the popHealth web application infrastructure, the popHealth design team opted to introduce some tweaks to the presentation layer but still maintains the established L&F and interaction model.  The latest updated designs for popHealth v2.4 follow:

popHealth v2.4 Dashboard Design for Meaningful Use Stage 2

Changes in the evolved popHealth v2.4 design include:

• Eliminated confusing links for the "parameters" of each CQM on the right; now, there is just a link to access the patients.
• Reduced the number of colors and focused on a more muted and simplified color pallet 
• Eliminated the single horizontal bar visualization and replaced it with two stacked bars for the fractions
• Identified the exception and exclusion population with muted gray bars on the stacked bar charts
• Enhanced the performance rate fraction with a visual radial circle around the fraction value

If anyone has opinions on the latest designs, the team welcomes any and all constructive feedback.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

Applying Cost To Clinical Quality Measures

With the Kamira research project that I am leading, our team was able to instrument the Meaningful Use (MU) Stage 2 Clinical Quality Measure (CQM) logic with the procedural logic for calculating the results of the CQMs via the popHealth project.  What this popHealth CQM calculation software provided the Kamira team was the clinical building blocks of the CQM logic, which we were able to marry with publicly available claims data hosted by CMS based on de-identified real-life claims records.

With the CQM calculation land this collection of claims records, we were able to asses ranges of the cost associated with addressing the MU Stage 2 CQMs.  With this, we were able to provide a guess for how much cost would be introduced into the healthcare system if a provider were to attempt to address the numerator logic of an MU Stage 2 CQM.

Consider the MU Stage 2 CQM "NQF 0062: Diabetes Urine Protein Screening".  This CQM measures the percentage of patients 18-75 years of age diagnosed with diabetes who had a some form of urine screening during the measurement period.  However, if you look at the conditional logic of this CQM, you can see that having the dialysis procedure would 

Procedural numerator logic for "NQF 0062: Diabetes Urine Protein Screening"

When we applied the CMS claims data, you can see the wide range of costs associated with this particular CQM's numerator logic, spanning microalbumin testing, ACE inhibitor, and access to dialysis:

Numerator costs on NQF 0062 - Diabetes Urine Protein Screening

As a non-clinician, the first two tests in the numerator logic make sense to me (microalbumin lab tests).  They appear to be in the "spirit" of this CQM, measuring if the provider has performed the microalbumin lab test for kidney damage.  However, the Kamira automated cost assessment highlighted that the vascular access for dialysis is clearly a much more expensive procedure to address this numerator logic than the lab test.  We didn't have any claims data on the kidney transplant from our sample set, but I can only speculate that the cost for that claim would be in the six figure range.

Looking purely at the CQM logic, and not applying any clinical perspective on this CQM, the vascular access for dialysis procedure is viewed as equivalent to the microalbumin lab test, at least within the scope of in this CQM logic.  Both are equal when assessing if the provider is performing the best quality of care for their population of diabetic patients.  Again... a kidney transplant is also in that same category as being semantically equivalent to the microalbumin lab test!!!

On the flip side, it appears to me that vascular access for dialysis is meant to address a known clinical problem, whereas the microalbumin lab test is designed to only collect and present information to a provider.  This difference in the clinical purpose of the clinical activities is not considered in the CQM logic.  Again, the CQM logic views both as equivalent when measuring the quality of care that a provider is applying to their population of patients.

Based on the Kamira automated CQM cost analysis, my recommendation is to groom the logic of "NQF 0062: Diabetes Urine Protein Screening", and re-position the dialysis procedure probably belongs in the exception or exclusion logic, vs. the current numerator logic.  There are some additional opportunities to apply this cost consideration on all Meaningful Use Stage 3 CQMs.  By applying these cost metrics to the MU Stage 3 CQMs, at minimum, the policy makers could discuss the impact of cost with the numerator logic clinical data.  Additionally, these conversations could potentially lead to streamlining and pruning the MU Stage 3 CQM logic to avoid noisy and high variance healthcare costs within the logic of one CQM.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

Thoughts on Clinical Quality Measure and Clinical Decision Support Convergence

I have been tracking several discussions for the past year to consider the convergence of Clinical
Quality Measures and Clinical Decision Support within Electronic Health Record systems.  As a quick refresher on the two types of healthcare reports:

Clinical Quality Measurement (CQM)

• Typically post-event reporting on treatment, usually via a quarterly or annual basis
• Longer term impact on healthcare quality of a population of patients

Clinical Decision Support (CDS)

• Guide clinical choices during treatment now
• More direct and immediate impact on the quality of care being provided to a patient

CQMs and CDS are clearly very closely related disciplines.  In fact, they are two sides of the same coin.  They both consider what actions providers take with the "hand they are dealt" with their patients.  For both types of reports, data in input via electronic clinical data, and both are implemented within EHR systems.  Additionally, the logic of a CDS rule and a CQM report are indeed very similar.

With CQMs, you are reviewing actions taken in the past to help educate and influence decisions in the future.

With CDS, you are working in real-time... in the present.

To put this in another context, with CQMs you may be viewing data that may have led to the death of a patient.  The goodness or badness has been performed by the provider or healthcare team.  With CDS, you have the potential to actually kill a patient by taking the wrong action.

This is the part of the CDS space that I personally find troubling to work with, since there is a much more real risk to harming a patient with either a bug in CDS logic or with ambiguity in standards that could be used to express CDS

Consider a Warfarin CQM that measures “Did you perform appropriate warfarin dosing?”  You could apply this CQM to a population of cardiac patients and consider the results in aggregate to see if over the course of a year.  Now consider a CDS that determines "What is the lethal dose of warfarin dosing that I am providing this patient... right now?!?”

For me, addressing the second scenario become really serious very quickly.  For healthcare providers, I do not think that they are phased with either domain, since they deal with life and death scenarios all the time.

But again, for me, these are the scenarios and domains where I lose sleep after work.

Several considerations have been discussed to expand the Health Quality Measures Format (HQMF) XML standard that is used for expressing the CQM logic for the Meaningful Use program.  The HQMF is an HL7 standard for representing a Clinical Quality Measure (CQMs) in XML.  The HQMF standard is a declarative approach to defining procedural CQM reporting logic.  What the HQMF standard has going for it, is (regrettably) being named the authoritative source for expressing the logic for the Meaningful Use Stage 2 CQMs.

The HQMF implementation of CQMs defines the data elements referenced by the logic, and associates them to value sets using object identifiers (OIDs).  These OIDs can then be used with the National Library of Medicine's (NLM's) Value Set Authority Center (VSAC) to translate clinical concepts into various operational clinical codes.

For those really interested in using the HQMF, you can download the HQMF files for all the Meaningful Use Stage 2 Clinical Quality Measures from the CMS site here.

Having worked with the HQMF XML standard indirectly for the past 2 years on both the popHealth project, and for the past year the Cypress project, my team has repeatedly expressed concern over the standard due to ambiguity and difficulty in ensuring that the HQMF standard is properly implemented in an EHR.  For our team, our challenge was consuming HQMF XML and turning it into procedural logic for Cypress and popHealth.

Based on the challenges my team has faced, I am very cautious about any considerations to re-use the HQMF standard for CDS support, since I feel it does a poor job expressing even the CQM logic.  My primary concern with the plans on converging CQMs and CDSs logic is the current declarative approach used for CQMs which is the HQMF XML standard.

What I feel is needed is a procedural definition of both CQM and CDS logic.  The reason for this need for procedural expression of the logic is that it allows for more rigorous testing, and also reduces the ambiguity associated with EHR systems have to implement their own parser to convert the declarative HQMF definition of either a CQM or CDS into their own procedural set of software instructions.

Consider the following procedural JavaScript logic built into popHealth v1.4 for the Meaningful Use Stage 1 CQM, NQF 0421 Adult Weight Screening:

function() {
  var patient = this;
  var measure = patient.measures["0421"];
  if (measure == null)
    measure={};

  var day = 24 * 60 * 60;
  var year = 365 * day;
  var effective_date = <%= effective_date %>;

  var measurement_period_start =  effective_date - (1 * year);
  var latest_birthdate = latestBirthdayForThisAge(65, measurement_period_start);
  var earliest_encounter =        measurement_period_start - year;

  var population = function() {
    var correct_age = patient.birthdate <= latest_birthdate;
    return (correct_age);
  }

  var denominator = function() {
    return inRange(measure.encounter_outpatient_encounter,
                   earliest_encounter, effective_date);
  }

  var numerator = function() {
    return weight_numerator(measure, 22, 30);
  }

  var exclusion = function() {
    return weight_exclusion(measure, earliest_encounter, effective_date);
  }

  map(patient, population, denominator, numerator, exclusion);
};

By having a single procedural definition for both CQMs and CDS, we should be able to have a higher level of confidence in the performance of the EHR systems that have to implement either a CQM or CDS.  What could then be done, is test and certify EHR systems ability to process the procedural definition of the standard, and then have a higher level of confidence that the any new CQM expressed in procedural logic... as long as it were syntactically well formed... would work with an EHR system.

The benefit to this type of approach is accelerated velocity in development, testing, and deployment of new CQMs and new CDS reports/rules.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

The Four Different Types of Clinical Quality Measures

Clinical Quality Measures (CQMs) have been a significant part of my work for the past 3 years.  My first foray into the CQM space was with the popHealth project, which is a reference implementation of the CQM logic that a provider or EHR vendor could use or incorporate into a software project.  popHealth was entirely focused on a family of CQMs called "proportion-based".

While the "proportion-based" CQMs get the majority of the attention with the healthcare community, it was not until a year working on popHealth that I discovered that there are a total of four different types of CQMs that affect how the CQM logic is implemented and reported.

The four different classifications of CQM logic are:

Proportion

• This is the type of CQM that most individuals are familiar with when referring to the Meaningful Use program.  
• These types of CQMs are routinely referred to as the "Numerator/Denominator" CQMs.  I recently wrote about the exception and exclusion logic, but it is worth noting that those CQM reporting characteristics are only applied to proportion-based CQMs.
• Usually, the proportion-based CQMs are a positive measurement of quality, meaning that usually, the higher the value of the Numerator/Denominator proportion, the better you are doing as a healthcare provider
• Example: "What percentage of women over the age of 45 and under the age of 65, who have had an outpatient encounter in the past 2 years, have had a mammography screening?"

Continuous Variable

• These CQMs are usually applied in the hospital CQM domain.  
• These types of CQMs measure the average time 
• Example: "What is the average time for Emergency Department (ED) admission until either Discharge or Admission to Inpatient Hospitalization?"

Episode of Care 

• These CQMs assess each distinct ‘encounter’ between a patient and a provider, during a a measurement period.
• A single patient can contribute to numerous considerations of the CQM result if they had numerous encounters
• Examples: "Did the provider measure the patient’s blood pressure during a particular episode?" or "Were heart attack patients discharged with an Rx for Aspirin during a particular episode?"

Longitudinal

• These CQMs take into account complete patient record with focus on a ‘measurement period’
• Examples: "Have patients who turned 2 years old during the measurement period received all required vaccinations on schedule?" or "Have diabetic patients received 2 foot exams during the measurement period?"

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.