Bonnie: An Open Source Clinical Quality Measure Testing Tool

Bonnie is a new open source software tool that MITRE has developed and released in April 2014 that allows Meaningful Use (MU) Clinical Quality Measure (CQM) developers to test and verify the behavior of their CQM logic.  The goal of Bonnie is to reduce the number of defects in CQMs by providing a robust and automated testing framework. Bonnie allows measure developers to independently load measures that they have constructed using the Measure Authoring Tool (MAT). Loading the measures into Bonnie converts the measures from their Extensible Markup Language (XML) eSpecifications into executable artifacts and measure metadata.

Bonnie Dashboard Page

The measure eSpecification format that Bonnie loads is Health Quality Measure Format (HQMF) XML. The HQMF specification provides the metadata and logic that describe the specifics of calculating a CQM. Bonnie can load the HQMF describing a measure and programmatically convert the HQMF specification into an executable format that allows calculating the measure directly from the specification. 

The measure metadata loaded into Bonnie is then used to allow developers to rapidly build a synthetic patient test deck for the measure using the clinical elements defined during the measure construction process. By using measure metadata as a basis for building synthetic patients, developers can rapidly and efficiently create a test deck for a measure. 

Once a CQM has been loaded into Bonnie, a user can inspect the measure logic and then build synthetic test records and set expectations on how those test records will calculate against a measure. This capability to build synthetic test patient records, set expectations against those records, and calculate the measures using those patient records provides an automated and efficient testing framework for CQMs. 

Using the Bonnie-supported CQM testing framework allows measure developers to more clearly understand the behavior of the measure logic, validate that the measure logic encodes their intent, and allows for multiple iterations of measure updates to be validated against a test deck. 

Bonnie Measure Page

Additionally, the development of a test deck as part of measure development provides benefits after the measures are finalized. The test deck build as part of measure development can be used to demonstrate the intent of the measure though the use of patient examples included in the test deck. Furthermore, the test deck provides systems that implement the measures with a means to validate the development of their systems. This is provided in the form of a base set of synthetic patient records with known expectations for calculating against the implemented measures. Finally, the test deck could be used as a basis for the test deck used as part of the Meaningful Use certification program. 

Bonnie has been designed to integrate with the nationally recognized data standards used by the Meaningful Use program for expressing CQM logic for machine-to-machine interoperability. This provides enormous value to the CQM program and federal policy leaders and stakeholders: this software tool verifies that the new and evolving standards for the Meaningful Use CQM program are tractable and can be implemented in software.   

Additionally, Bonnie was designed to provide an intuitive and easy-to-use interface based on feedback from the broader measure developer community. A key goal of the Bonnie application is to deliver a user experience that provides an efficient and intuitive method for constructing synthetic patient records for testing and validating CQMs. 

The Bonnie software is freely available via an Apache 2.0 open source license. The Meaningful Use program makes all or parts of the Bonnie software available for inspection, verification, and even reuse by other government programs or federal contractors. 

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2014.

ICD 10 vs ICD 9 Code Format Structural Differences

ICD is the World Health Organization's (WHO's) International Classification of Diseases (ICD) and Related Health Problems and is the international standard diagnostic classification system, and is the tenth revision of the ICD.  The ICD is the coding system which physicians and other healthcare providers currently use to code all diagnoses, symptoms, and procedures recorded in hospitals and physician practices.  

It is a big deal.

Today, the Department of Health and Human Services (HHS) just published an updated rule for the adoption of ICD-10 code sets.  HHS is requiring that all HIPAA covered entities must be ICD-10 compliant by October 1st, 2015.  This newly updated compliance date is meant to be firm and not subject to any change.  This is the third time that ICD-10 has been delayed, so I strongly suspect that this new deadline will be met by next fall.

Last year, I put together a visualization of analysis of the ICD-10 coding landscape.  Below is a simple primer explaining the distinction between ICD-9 and ICD-10, demonstrating structurally the difference in the two coding systems.  In particular, the ICD-10 code set has been expanded from five positions (first one alphanumeric, others numeric) to up to seven positions. The codes use alphanumeric characters in all positions, not just the first position as in ICD-9.

ICD-9 vs ICD-10 coding

Some other interesting artifacts of ICD-9 vs ICD-10 that I discovered today:

• As of the latest version, there are ~68,000 codes in ICD-10, as opposed to the ~13,000 in ICD-9.  More specifically, there are nearly 5 times as many diagnosis codes in ICD-10 than in ICD-9 and there are nearly 19 times as many procedure codes in ICD-10 than in ICD-9.  Yikes!
• The new code set provides a significant increase in the specificity of the reporting, allowing more information to be conveyed in a code.  To support this, the terminology has been modernized and has been made consistent throughout the code set.  There are codes that are a combination of diagnoses and symptoms, with a claim that fewer codes need to be reported to fully describe a condition.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2014.

Visualization of ICD-10 Code Counts

This past week I have been working in the bowels of the QRDA Category 1 XML for the popHealth project that we are deploying for the Veterans Health Administration (VHA).  In the process of working with the QRDA Category 1, I had to resuscitate some of my Ruby and REXML skills that had atrophied in the past year.

This weekend, I wanted to shakeout some of my technical skills in a cleaner environment and downloaded the XML for the full set of ICD-10 codes from the CMS site.

Why ICD-10?  It is the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD) by the World Health Organization (WHO).  ICD-10 provides a hierarchy of structured codes for diseases, symptoms, findings, complaints, social circumstances, and external causes of injury/diseases.  The big national issue related to ICD-10 is that it will be required for expressing claims data to the Center for Medicare and Medicaid Services (CMS) starting on October 2014.

The current state-of-the-practice for capturing this coded data in Electronic Health Record systems is (IMHO) still ICD-9, the predecessor to ICD-10.  One of the biggest differences between ICD-9 and ICD-10 is the fidelity of data that can be captured in ICD-10.  In particular, there are over 68,000 distinct codes in ICD-10 as opposed to the roughly 13,000 in ICD-9.

Working with the XML file provided on the CMS site that details the ICD-10 code hierarchy, I wanted to see if I could convert the data into a format that would allow me to visualize the code counts into a D3.js example.  I figured it was good to exercise some XML knowledge outside of the complexity of the QRDA Category 1 XML.  Further,I wanted to learn a little more about the structure of the ICD-10 codes.

It is worth noting that the CMS ICD-10 XML is surprisingly easy to understand for the purposes of enumerating the full set of codes and the hierarchy.  The QRDA Category 1 XML… not so easy to understand.

What I did was to load the ICD-10 XML hierarchy into a simple Ruby program via REXML.  I created a aggregate count in a hash table of the second-level codes in the ICD-10 hierarchy by traversing the XML file.  I had to do this only at the second-level of the ICD-10 hierarchy because the sheer number of third-level ICD-10 codes broke the D3.js visualization examples.  To explain this a little more, the hierarchy of an example diabetes code down that the fourth level in ICD-10 follows:

E00-E89: Endocrine, nutritional and metabolic diseases
  |-> E08 Diabetes mellitus due to underlying condition
    |->E08.2 Diabetes mellitus due to underlying condition with kidney complications
      |->E08.22 Diabetes mellitus due to underlying condition with diabetic chronic kidney disease

So for the illustration of ICD-10 code counts, I stopped aggregating at just the second level of the hierarchy and count/aggregate codes from the third and forth levels.  Each tiny square in the illustration below represents the counts of just the second level of the ICD-10 space of roughly 68,000 total codes.

Once I had the counts of individual ICD-10 codes aggregated at second-level of the ICD-10 hierarchy, I exported a JSON file that could work with the D3.js example that I picked.  Below is a thumbnail (admittedly... illegible) of the ICD-10 code counts transformed with the D3.js treemap example.

Visualization of second level ICD-10 code counts

If you want to try and download a higher resolution image of the ICD-10 codes and actually read more of the details, click here.  HEADS UP… it is gianormous.

With the illustration, starting from left-to-right and then top-to-bottom, the sections in the ICD-10 data set that coincide with the colors in the illustration as follows.  The only confusing item is the last "chapter" from ICD-10 is the gray box in the bottom left "Factors influencing health status and contact with health services".  I think the D3.js code had to try and fit that section into the illustration.

• A00-B99: Certain infectious and parasitic diseases
• C00-D49: Neoplasms
• D50-D89: Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism
• E00-E89: Endocrine, nutritional and metabolic diseases
• F01-F99: Mental, Behavioral and Neurodevelopmental disorders
• G00-G99: Diseases of the nervous system
• H00-H59: Diseases of the eye and adnexa
• H60-H95: Diseases of the ear and mastoid process
• I00-I99: Diseases of the circulatory system
• J00-J99: Diseases of the respiratory system
• K00-K95: Diseases of the digestive system
• L00-L99: Diseases of the skin and subcutaneous tissue
• M00-M99: Diseases of the musculoskeletal system and connective tissue
• N00-N99: Diseases of the genitourinary system
• O00-O9A: Pregnancy, childbirth and the puerperium
• P00-P96: Certain conditions originating in the perinatal period
• Q00-Q99: Congenital malformations, deformations and chromosomal abnormalities
• R00-R99: Symptoms, signs and abnormal clinical/laboratory findings, not elsewhere classified
• S00-T88: Injury, poisoning and certain other consequences of external causes
• V00-Y99: External causes of morbidity
• Z00-Z99: Factors influencing health status and contact with health services

If you are interested, you can access the JSON file with the second level code counts from my GitHub repository that I setup.

Further, you could use this JSON with several other data hierarchy examples off the D3.js site if you are interested.  They use the same JSON format for representing the data, and you should be able to just drop the JSON that I created into that HTML if you tweak the name of the file in the examples and set your width and height of the demo to about one thousand times greater than what is provided since the about of data is so large.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.