The Kamira project has listened to my evangelizing the use of Kiviat charts as the visualization technique used with CQMs. In addition to being fairly useful for understanding the CQM complexity results, I have found that the Kiviat visualizations also have a "sexiness" factor that I have been leveraging to continue the research. See the Kamira Cyclomatic Complexity dashboard design below:
 |
| Kamira complexity dashboard design |
What we did with the Kiviat visualization for the complexity results is leverage the five attributes that could always be used for parts of the CQM algorithmic logic:
Because some of the MU Stage 2 CQM logical sections far exceeded CMU's threshold of 50 for "Untestable, Extreme Risk" we found that there were some CQMs that had logical sections that were "off the scale" when it came to Cyclomatic Complexity. The worst violator is "NQF 0038: Childhood Immunizations" which is actually part of the core set of MU Stage 2 for Eligible Professional CQMs.
Since we couldn't use a linear scale for those CQMs, so we capped the scale at 60, and switched the presentation of the results around by providing a red background around the Cyclomatic Complexity value. Normally, there isn't a background and the results are just black text. The idea here was to try and call out that there was something fundamentally wrong with complexity values that are so high.
See the 6 most complex CQMs for MU Stage 2 Eligible Professionals (EP) below:
From a software engineer's perspective, if you have to manually implement these CQMs in software code, you are going to be very hard pressed to test and validate that any system implementing this logic accurately. Ie. you have a challenging job ahead of you to test that any implementation of these 6 CQMs are in fact accurate. This also means that there's a higher probability of having a lurking software bug in your implementation.
Also, for healthcare providers who need to understand these CQMs in order to improve the quality of care that they are providing to their patients, they will also have a challenge. Providers needing to maintain a mental model of the CQMs and understand what actions should be taken for their patients will likely face challenges.
Ideally, the complexity of the CQMs should never even get this high. What I would want to see is some consideration by measure stewards on Cyclomatic Complexity during the development process. Another consideration for the policy leaders would be refusal to accept any CQM submitted that had a Cyclomatic Complexity value that exceeded a threshold.
The Kamira project's work has initially focused on the Meaningful Use Stage 2 Eligible Professional CQMs. The reason that these were initially selected is because we were easily able to instrument the JavaScript code that is implemented in the popHealth and Cypress project's quality measure engine. However, the work can be expanded to include the MU Stage 2 Eligible Hospital (EH) CQMs when those are implemented in popHealth and Cypress, currently planned for late April 2013. Unfortunately, I expect that the EH CQMs will actually be more complex than the EP CQMs.
Additionally, I have plans to expand the work into financial analysis and feasibility of implementing CQMs, but for now the lowest hanging fruit was the complexity analysis.
The full visualization of the Meaningful Use Stage 2 CQM Complexity Data is available here, and the detailed results in a JSON file is available here.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.
- Initial Patient Population (IPP)
- Denominator
- Numerator
- Exclusion
- Exception
- 1-10: Very Simple, Low Risk (green)
- 11-20: Nominal, Moderate Risk (yellow)
- 21-50: Complex, High Risk (orange)
- >50: Untestable, Extreme Risk (red)
Because some of the MU Stage 2 CQM logical sections far exceeded CMU's threshold of 50 for "Untestable, Extreme Risk" we found that there were some CQMs that had logical sections that were "off the scale" when it came to Cyclomatic Complexity. The worst violator is "NQF 0038: Childhood Immunizations" which is actually part of the core set of MU Stage 2 for Eligible Professional CQMs.
Since we couldn't use a linear scale for those CQMs, so we capped the scale at 60, and switched the presentation of the results around by providing a red background around the Cyclomatic Complexity value. Normally, there isn't a background and the results are just black text. The idea here was to try and call out that there was something fundamentally wrong with complexity values that are so high.
See the 6 most complex CQMs for MU Stage 2 Eligible Professionals (EP) below:
 |
| 6 Most Complex CQMs for MU Stage 2 Eligible Professionals |
Also, for healthcare providers who need to understand these CQMs in order to improve the quality of care that they are providing to their patients, they will also have a challenge. Providers needing to maintain a mental model of the CQMs and understand what actions should be taken for their patients will likely face challenges.
Ideally, the complexity of the CQMs should never even get this high. What I would want to see is some consideration by measure stewards on Cyclomatic Complexity during the development process. Another consideration for the policy leaders would be refusal to accept any CQM submitted that had a Cyclomatic Complexity value that exceeded a threshold.
The Kamira project's work has initially focused on the Meaningful Use Stage 2 Eligible Professional CQMs. The reason that these were initially selected is because we were easily able to instrument the JavaScript code that is implemented in the popHealth and Cypress project's quality measure engine. However, the work can be expanded to include the MU Stage 2 Eligible Hospital (EH) CQMs when those are implemented in popHealth and Cypress, currently planned for late April 2013. Unfortunately, I expect that the EH CQMs will actually be more complex than the EP CQMs.
Additionally, I have plans to expand the work into financial analysis and feasibility of implementing CQMs, but for now the lowest hanging fruit was the complexity analysis.
The full visualization of the Meaningful Use Stage 2 CQM Complexity Data is available here, and the detailed results in a JSON file is available here.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. © Rob McCready, 2013.
No comments:
Post a Comment