Bonnie: An Open Source Clinical Quality Measure Testing Tool

Bonnie is a new open source software tool that MITRE has developed and released in April 2014 that allows Meaningful Use (MU) Clinical Quality Measure (CQM) developers to test and verify the behavior of their CQM logic.  The goal of Bonnie is to reduce the number of defects in CQMs by providing a robust and automated testing framework. Bonnie allows measure developers to independently load measures that they have constructed using the Measure Authoring Tool (MAT). Loading the measures into Bonnie converts the measures from their Extensible Markup Language (XML) eSpecifications into executable artifacts and measure metadata.

Bonnie Dashboard Page

The measure eSpecification format that Bonnie loads is Health Quality Measure Format (HQMF) XML. The HQMF specification provides the metadata and logic that describe the specifics of calculating a CQM. Bonnie can load the HQMF describing a measure and programmatically convert the HQMF specification into an executable format that allows calculating the measure directly from the specification. 

The measure metadata loaded into Bonnie is then used to allow developers to rapidly build a synthetic patient test deck for the measure using the clinical elements defined during the measure construction process. By using measure metadata as a basis for building synthetic patients, developers can rapidly and efficiently create a test deck for a measure. 

Once a CQM has been loaded into Bonnie, a user can inspect the measure logic and then build synthetic test records and set expectations on how those test records will calculate against a measure. This capability to build synthetic test patient records, set expectations against those records, and calculate the measures using those patient records provides an automated and efficient testing framework for CQMs. 

Using the Bonnie-supported CQM testing framework allows measure developers to more clearly understand the behavior of the measure logic, validate that the measure logic encodes their intent, and allows for multiple iterations of measure updates to be validated against a test deck. 

Bonnie Measure Page

Additionally, the development of a test deck as part of measure development provides benefits after the measures are finalized. The test deck build as part of measure development can be used to demonstrate the intent of the measure though the use of patient examples included in the test deck. Furthermore, the test deck provides systems that implement the measures with a means to validate the development of their systems. This is provided in the form of a base set of synthetic patient records with known expectations for calculating against the implemented measures. Finally, the test deck could be used as a basis for the test deck used as part of the Meaningful Use certification program. 

Bonnie has been designed to integrate with the nationally recognized data standards used by the Meaningful Use program for expressing CQM logic for machine-to-machine interoperability. This provides enormous value to the CQM program and federal policy leaders and stakeholders: this software tool verifies that the new and evolving standards for the Meaningful Use CQM program are tractable and can be implemented in software.   

Additionally, Bonnie was designed to provide an intuitive and easy-to-use interface based on feedback from the broader measure developer community. A key goal of the Bonnie application is to deliver a user experience that provides an efficient and intuitive method for constructing synthetic patient records for testing and validating CQMs. 

The Bonnie software is freely available via an Apache 2.0 open source license. The Meaningful Use program makes all or parts of the Bonnie software available for inspection, verification, and even reuse by other government programs or federal contractors. 

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2014.

Visualization of ICD-10 Code Counts

This past week I have been working in the bowels of the QRDA Category 1 XML for the popHealth project that we are deploying for the Veterans Health Administration (VHA).  In the process of working with the QRDA Category 1, I had to resuscitate some of my Ruby and REXML skills that had atrophied in the past year.

This weekend, I wanted to shakeout some of my technical skills in a cleaner environment and downloaded the XML for the full set of ICD-10 codes from the CMS site.

Why ICD-10?  It is the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD) by the World Health Organization (WHO).  ICD-10 provides a hierarchy of structured codes for diseases, symptoms, findings, complaints, social circumstances, and external causes of injury/diseases.  The big national issue related to ICD-10 is that it will be required for expressing claims data to the Center for Medicare and Medicaid Services (CMS) starting on October 2014.

The current state-of-the-practice for capturing this coded data in Electronic Health Record systems is (IMHO) still ICD-9, the predecessor to ICD-10.  One of the biggest differences between ICD-9 and ICD-10 is the fidelity of data that can be captured in ICD-10.  In particular, there are over 68,000 distinct codes in ICD-10 as opposed to the roughly 13,000 in ICD-9.

Working with the XML file provided on the CMS site that details the ICD-10 code hierarchy, I wanted to see if I could convert the data into a format that would allow me to visualize the code counts into a D3.js example.  I figured it was good to exercise some XML knowledge outside of the complexity of the QRDA Category 1 XML.  Further,I wanted to learn a little more about the structure of the ICD-10 codes.

It is worth noting that the CMS ICD-10 XML is surprisingly easy to understand for the purposes of enumerating the full set of codes and the hierarchy.  The QRDA Category 1 XML… not so easy to understand.

What I did was to load the ICD-10 XML hierarchy into a simple Ruby program via REXML.  I created a aggregate count in a hash table of the second-level codes in the ICD-10 hierarchy by traversing the XML file.  I had to do this only at the second-level of the ICD-10 hierarchy because the sheer number of third-level ICD-10 codes broke the D3.js visualization examples.  To explain this a little more, the hierarchy of an example diabetes code down that the fourth level in ICD-10 follows:

E00-E89: Endocrine, nutritional and metabolic diseases
  |-> E08 Diabetes mellitus due to underlying condition
    |->E08.2 Diabetes mellitus due to underlying condition with kidney complications
      |->E08.22 Diabetes mellitus due to underlying condition with diabetic chronic kidney disease

So for the illustration of ICD-10 code counts, I stopped aggregating at just the second level of the hierarchy and count/aggregate codes from the third and forth levels.  Each tiny square in the illustration below represents the counts of just the second level of the ICD-10 space of roughly 68,000 total codes.

Once I had the counts of individual ICD-10 codes aggregated at second-level of the ICD-10 hierarchy, I exported a JSON file that could work with the D3.js example that I picked.  Below is a thumbnail (admittedly... illegible) of the ICD-10 code counts transformed with the D3.js treemap example.

Visualization of second level ICD-10 code counts

If you want to try and download a higher resolution image of the ICD-10 codes and actually read more of the details, click here.  HEADS UP… it is gianormous.

With the illustration, starting from left-to-right and then top-to-bottom, the sections in the ICD-10 data set that coincide with the colors in the illustration as follows.  The only confusing item is the last "chapter" from ICD-10 is the gray box in the bottom left "Factors influencing health status and contact with health services".  I think the D3.js code had to try and fit that section into the illustration.

• A00-B99: Certain infectious and parasitic diseases
• C00-D49: Neoplasms
• D50-D89: Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism
• E00-E89: Endocrine, nutritional and metabolic diseases
• F01-F99: Mental, Behavioral and Neurodevelopmental disorders
• G00-G99: Diseases of the nervous system
• H00-H59: Diseases of the eye and adnexa
• H60-H95: Diseases of the ear and mastoid process
• I00-I99: Diseases of the circulatory system
• J00-J99: Diseases of the respiratory system
• K00-K95: Diseases of the digestive system
• L00-L99: Diseases of the skin and subcutaneous tissue
• M00-M99: Diseases of the musculoskeletal system and connective tissue
• N00-N99: Diseases of the genitourinary system
• O00-O9A: Pregnancy, childbirth and the puerperium
• P00-P96: Certain conditions originating in the perinatal period
• Q00-Q99: Congenital malformations, deformations and chromosomal abnormalities
• R00-R99: Symptoms, signs and abnormal clinical/laboratory findings, not elsewhere classified
• S00-T88: Injury, poisoning and certain other consequences of external causes
• V00-Y99: External causes of morbidity
• Z00-Z99: Factors influencing health status and contact with health services

If you are interested, you can access the JSON file with the second level code counts from my GitHub repository that I setup.

Further, you could use this JSON with several other data hierarchy examples off the D3.js site if you are interested.  They use the same JSON format for representing the data, and you should be able to just drop the JSON that I created into that HTML if you tweak the name of the file in the examples and set your width and height of the demo to about one thousand times greater than what is provided since the about of data is so large.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

popHealth v2.4 Design for Meaningful Use Stage 2

As some of you know, I have been leading the open source popHealth project at MITRE for the past 4 years.  The MITRE popHealth team is currently working for the Veterans Health Administration (VHA).  Our work is focused on using popHealth as the software service allow the VHA to meet Meaningful Use Stage 2 certification for Clinical Quality Measures (CQMs).

Since we hadn't been working on popHealth in any substantial level in most of 2013, so we identified some problems for ongoing operations and maintenance (O&M) of the popHealth software when we started to put more and more engineers back on the project.  The concerns that the popHealth team had was the software that manages the User Interface (UI) presentation layer in the browser was becoming too complex and convoluted.  This had the potential for introducing difficulty when a non-MITRE contractor will eventually manage the popHealth software in an "O&M" mode for the VHA.  This will happen when our team completes the development task and moves off of this project.

What the popHealth team opted to do was to clean up the JavaScript and HTML so that there was an Application Programming Interface (API) that could provide JavaScript Object Notation (JSON) for the popHealth data needed for the UI directly to the browser.  This is a cleaner software design that also allows user to forgo the existing popHealth UI if they need to provide just the raw CQM data.

In the process of refactoring the popHealth web application infrastructure, the team opted to introduce some tweaks to the popHealth presentation layer but still maintain the existing interaction model.  What we decided to do was only slightly evolve the existing popHealth branding and look and feel, so that is is more maintainable from an engineers perspective, but does not introduce a radically different interaction model require re-training for all our existing users.

The popHealth v1.4 user interface that was aligned with the Meaningful Use Stage 2 program is below:

popHealth v1.4 Dashboard Design for Meaningful Use Stage 1

In the process of refactoring the popHealth web application infrastructure, the popHealth design team opted to introduce some tweaks to the presentation layer but still maintains the established L&F and interaction model.  The latest updated designs for popHealth v2.4 follow:

popHealth v2.4 Dashboard Design for Meaningful Use Stage 2

Changes in the evolved popHealth v2.4 design include:

• Eliminated confusing links for the "parameters" of each CQM on the right; now, there is just a link to access the patients.
• Reduced the number of colors and focused on a more muted and simplified color pallet 
• Eliminated the single horizontal bar visualization and replaced it with two stacked bars for the fractions
• Identified the exception and exclusion population with muted gray bars on the stacked bar charts
• Enhanced the performance rate fraction with a visual radial circle around the fraction value

If anyone has opinions on the latest designs, the team welcomes any and all constructive feedback.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

Designing Logos for a Portfolio of Healthcare Projects

Over the past three years, I have been involved with numerous open source healthcare projects, ranging from large reference implementations of Clinical Quality Measures, certification and testing tools adopted by the federal government, a small research project assessing CQM complexity/cost/tractability, this blog, and other ideas that failed to launch.

What I have been endorsing throughout the course of the past three years has been to establish a common "brand" that could be applied to various open source healthcare projects, but still provide a unique look and feel for each individual project.  

The design approach that organically evolved was to develop logos followed the following rules:

• Use only circles
• The circles may come in different sizes and arrangements
• Limiting the logos to a pallet of only two colors
• Color opacity could be changed

You can see the portfolio of these project logos below:

Portfolio of logos for open source healthcare projects I have worked on

Since I am a "left brain" engineer at heart, the various arrangements of the circles has been an easier task than color selection.  I have often struggled with finding the magic combination of colors for anything from these logos to our interior design of our hours.

The best resource that I have found been able to identify is ColorLovers.com.  For those not familiar with the site, it is an evolving resource for palettes and colors that users can share, rank and comment on.

When I had to select to colors, I went to the filter for "most loved" colors of all time, and tried to identify pallets where instead of 5 colors, there were 2 colors used in the pallet.  Hopefully this is a helpful trick.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.

Remaining "Agile" with a Medium-Sized Team

I am hoping to start leading a new (really old... but soon-to-be revived) open source health IT project very soon.  The good news is that it's appearing that the project will be well-supported, and that we will be able to staff the project team up to a very good size.  I'm also planning on continuing to embrace an agile project management process using scrum.

One of the key benefits to having a successful agile process has been the shared situational awareness of the members of the project which allows the team to self-organize.   If I am needed less on the small but blocking minute-to-minute decision making, there's a higher probability of exceeding the expected outcomes of our sponsor while simultaneously growing new leaders for the future.  Additionally, I am happy to perform less tactical work and focus more on strategy.  

With the right project team (talented... responsible... multi-domain... passionate) it is remarkably easy to be successful on any project; from a project that is technology-heavy software development activity, to one that is primary focused on steering or influencing federal policy.

After originally being a software engineer, have been leading software projects for about 7 years, and my largest projects have been right around 5-8 Full Time Equivalent (FTE).  I am now facing the prospect of leading a 14 FTE project... this is a first for me.  

One concern of leading a project of this size is the increasing number of individual communication links that would be needed for everyone on the team to know what everyone else on the team is working on to maintain shared situational awareness on the project.  

To illustrate this, you can see how the communication links will continue to increase non-linearly with more and more staff.  L = number of communication links, T = number of members of the team.

Additionally about half of the 14 Full Time Equivalent (FTE) project team are co-located in an agile bullpen room, while the other half are geographically distributed across the East Coast... from Massachusetts, New Jersey, Virginia, all the way down to Florida.  The good news with this distributed team is that the entire roster will be based in Eastern Standard Time.  At least we will not be depriving individuals of sleep with meetings that span numerous time zones!

My thoughts of dealing with this if/when this new project starts, and try the Scum Alliance model of hosting a "Scrum of Scrums".  My current plan is to have co-located team maintain their agile "bullpen" and have one scrum, and have the geographically-distributed team scrum either after or before them.  The two team would have a size of approximately 7 FTE each.  They should both be able to maintain a cross-domain roster of engineers, healthcare informaticists, clinicians, policy leaders, and Clinical Quality Measure SMEs.  

To follow the "Scrum of Scrums" process, I plan to identify a scrum master from each team, and after the daily scrums, the scrum masters, product owner and stakeholders will host a second scrum.  Again following the Scrum Alliance variant on the "Scrum of Scrums" meeting, the agenda for that meeting will deviate a little by addressing the following questions:

Because the "Scrum of Scrums" meetings may not be daily and because one person is there representing his or her entire team, these three questions need to be rephrased a bit to be "team-focused" and they suggest it's beneficial to add a fourth question, identifying if the two teams might be unintentionally stepping on each others' toes:

1. What has each team done since yesterday?
2. What will each team do today?
3. Are there any impediments to the team's plan today?
4. Are you about to put something in the other team’s way?

It looks good and makes sense.  I hope to report out on how this "Scrum of Scrums" works.  

Fingers crossed...

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.