Test Maturity and the Obamacare Website


From the kick-off it must be very clear that no member of Celtic Testing Experts or the Company itself was involved with the testing of the Obamacare Website or the assessment of the issues that are to be discussed in this article. Both the issues and its origins as well as that of the CMMI (Capability and Maturity Model Integrated – for Development) and TMMI (Testing Maturity Model Integrated) can be found on the Web by searches using the supplied URLs. It must also be mentioned that this is not an in-depth assessment but merely based on the findings of individuals who investigated and interviewed those involved.

When systems fail, it is natural to focus on Testing to determine why the defects weren’t identified during the Testing cycles. Therefor the blame usually gets placed solidly on Testing. Testing is there to find defects in the system but Development needs the output of reviews and Testing to find defects in the process (SDLC) used to produce the System. Contractors to Government should have a high CMMI and TMMI Level proving that they not only have processes of high maturity but continuously operate in a mature environment.

The CMMI and TMMI are required by many Department Of Defense and U.S. Government contracts, since it will give the Government the peace at heart that they are dealing with a mature Company operating with best practices and reliable and proven oversight.

‘Maturity’ is based on how we as humans mature. Whatever we do we first PLAN what we want to do. Then we DO what we planned. Then we MEASURE the outcome. Lastly we ACT on our measurements. The CMMI and TMMI address maturity levels (There are five levels but it is necessary to be on Level 3 for handling complicated and high cost projects). Each Maturity Level has a number of Core Process Areas and sub-Process areas (With defined Best Practices) on which the Company will be appraised.

The following are the Core Process Areas for Levels 2 & 3 of the CMMI (Development):

Maturity Level 2 – Repeatable

Configuration Management, Measurement and Analysis, Project Monitoring and Control, Project Planning, Process and Product Quality Assurance, Requirements Management, Supplier Agreement Management.

Maturity Level 3 – Defined

Decision Analysis and Resolution, Integrated Project Management, Organizational Process Definition, Organizational Process Focus, Organizational Training, Product Integration, Requirements Development, Risk Management, Technical Solution, Validation, Verification

Looking at the Core Process Areas above and the Development Issues identified in Appendix ‘A’, one can immediately experience the lack of maturity in the development processes with obvious failures in each of the Core Process Areas. Project Management, Risk Management, Requirements Management Verification and Validation and Technical Solution are the areas of most failures.

The Senior Vice President of CGI Federal, the contractor who designed the site, called Healthcare.gov “one of the most complicated systems ever conceived”. At Celtic Testing Experts everyone will tell you that End-to-End Testing cannot be left till five days before the System is supposed to go live. More so when it is “the most complicated systems ever” built for Government and a high hit rate is expected. In this case Performance Testing was of utmost importance.

Speaking of Testing, the following are the Core Process areas for Levels 2 & 3 of the TMMI:

Level 2 – Managed

Test Policy and Strategy, Test Planning, Test Monitoring and Control, Test Design and Execution, Test Environment.

Level 3 – Defined

Test Organization, Test Training Program, Test Life Cycle and Integration, Non-functional Testing, Peer Reviews.

As is with the CMMI each of the TMMI Core Processes mentioned above are subdivided into sub-processes each containing continuously updated Best Practices. This is how as a Company IT matures and also how we mature. As little as measurements alone in our life can assure us maturity so little can Testing alone assure quality Software for a Company.

Again looking at the TMMI Levels 2 & 3 Core Processes above and the Testing issues in Appendix ‘A’, we can immediately see why the immature Testing processes begged for failure. Test Planning and Test Strategy, two of the most important aspects of Testing were greatly neglected with an Exit Criteria maybe still not achieved. Business Analysts completely misjudged the number of applications expected and in the short time before going live Performance Testing couldn’t be done properly on the expected numbers. Limited Performance Testing showed the System couldn’t handle up to 5000 applications, let alone the 250,000 received. Still the System went live. With all the defects still logged at going live, Regression Testing was also way off schedule.

But, all of this was known. So why did the System go live with all its shortcomings, defects and unproved requirements? Maybe for the same reason Businesses go live with Systems. They can make money whilst the Customers and IT identify defects. The Government, politically, needed to have something in place as promised rather than having a defect free system available. Now way after the System went live and the volume pressure subsided, attention is given to all the known defects and shortcomings. The System will work correctly – eventually.

Will this attitude ever change? I don’ know because there is too much money around. Rework on defects cost huge amounts to Business every month. Studies have shown that one and a half to four times the cost of developing a System (Normally two times) is spent extra on rework. The same study also determined that more than 90% of projects costing $95 million or more never went into production.

Maybe it is time for our IT everywhere to mature – Celtic Testing Experts can help.

Appendix A – Issue lists derived from Articles on Obamacare Development and Testing

Development Issues

  • Security Requirements not well defined or expertly reviewed
  • Rushed Software Development through SDLC. Security not well integrated
  • Hardware not well planned – last minute changes to accommodate system needs
  • Website was expected to draw around 60,000 simultaneous users but instead drew many more, around 250,000
  • The administration knew the website wasn’t ready, why did they roll it out anyway?
  • The senior vice president of CGI Federal, the contractor who designed the site, called Healthcare.gov one of the most complicated systems ever conceived.
  • The basic architecture of the site, built by federal contractors overseen by the Department of Health and Human Services, was flawed in design, poorly tested and ultimately not functional.
  • “You need there to be good people on the inside to make good contracting decisions and good people on the outside to do the work,”
  • Even on the back end of the site, data was garbled and, in some cases, unusable
  • The nightly reports that insurance companies receive from the federal government on new enrollees in the health plans have been riddled with errors, including syntax mistakes, and transposed or duplicate data, according to industry veterans
  • In other cases, insurers received multiple enrollments and cancelations from the same person, but since the documents lacked timestamps, it has been impossible to know which form is the most recent.
  • “We are seeing and hearing that enrollment files going to carriers are incomplete, there are errors,”
  • In three weeks or so when they start receiving these in mass volume, tens of thousands per day, it doesn’t matter if there’s a 1 percent error rate. Insurers don’t have resources to go through them and clean them up.”
  • Others (Developers) rewrote computer code over and over to meet what they considered last-minute requests for changes from the government or other contractors.
  • As questions mount over the website’s failure, insider interviews and a review of technical specifications by The Associated Press found a mind-numbingly complex system put together by harried programmers who pushed out a final product that congressional investigators said was tested by the government and not private developers with more expertise.
  • They complained openly to each other about what they considered tight and unrealistic deadlines. One was nearly brought to tears over the stress of finishing on time, one developer said. Website builders saw red flags for months

Testing Issues

  • Not Testing for Security requirements
  • ObamaCare website was only equipped to handle 1,100 users a day before it was launched
  • Insufficient testing process contributed to an error-prone rollout.
  • Last-minute nature of the testing.
  • There hadn’t been an end-to-end test, five days before the October 1 launch
  • HealthCare.gov was unable to consistently handle 500 users at once in the testing, and tests failed with 2,000 users over a three-day period,
  • Key contractors said the government spent weeks instead of the months needed on testing the site.
  • Some key testing of the system did not take place until the week before launch

 

There had been no tests to determine whether a consumer could complete the process from beginning to end: create an account, determine eligibility for federal subsidies and sign up for a health insurance plan, according to two sources familiar with the project.

The exchange system failed hundreds of tests administered by employees of the Centers for Medicare and Medicaid Services. These were the simplest sort of “beta” tests, the kind of thing I used to do all the time in my IT career. You plug in some simple manufactured data and see if the system can swallow it and regurgitate on command.

The tests were postponed until late September—an early warning sign of disaster that was kept hidden from the American people. When they finally occurred, they were a complete failure. The system “ground to a halt,” “froze,” and “crashed” under the lightest and most rudimentary data entry by a tiny pool of CMS testers.

“It was unequivocally clear from testing this wasn’t ready,” was the conclusion forwarded to HHS Secretary Kathleen Sebelius… who serenely ignored these desperate warnings and launched anyway. And worse, the head of CMS said the site’s issues “did not show up in testing” during testimony on Tuesday.

Appendix B – Partial articles and references on Obamacare Website Development and Testing

Computer Security Expert: Obamacare Website Security ‘Much Worse Off’ Than Before

The Obamacare website is even less secure than it was in November, David Kennedy, head of computer security consulting firm TrustedSec LLC, told Fox News Sunday.

Kennedy testified before Congress Thursday that the site was “100 percent” insecure and personal information for consumers at healthcare.gov was at risk, Reuters reports:

“What we learned was that they had rushed through what we call the software development life cycle where they actually build the application,” Kennedy said. ”So when you do that, security doesn’t really get integrated into it. And what happened with the rocky launch in October is they slapped a bunch of servers in trying to fix the website just to keep it up and running so that people could actually go and use it. The problem is they still didn’t imbed any security into it.”

(http://freebeacon.com/computer-security-expert-obamacare-website-security-much-worse-off-than-before/)

ObamaCare website could only handle 1,100 users a day before launch, docs show

The problem-plagued ObamaCare website was only equipped to handle 1,100 users a day before it was launched, documents released by the House Oversight and Reform Committee reveal.

The Obama administration has repeatedly insisted that the website’s repeated crashes were due to unexpectedly high traffic. U.S. Chief Technology Officer Todd Park said on Oct. 6 that the website was expected to draw around 60,000 simultaneous users but instead drew many more, around 250,000.

(http://www.foxnews.com/politics/2013/11/07/obamacare-website-could-only-handle-1100-users-day-before-launch-docs-show/)

Not enough pre-launch testing of Obamacare website, contractors testify.

Contractors who helped to build the troubled sign-up website for Obamacare said Thursday that an insufficient testing process contributed to an error-prone rollout.

They put blame on the Obama administration – specifically the health agency that had the key role of “systems integrator” for the Healthcare.gov website – for the last-minute nature of the testing.

(http://www.csmonitor.com/USA/Politics/2013/1024/Not-enough-pre-launch-testing-of-Obamacare-website-contractors-testify-video)

Forbes reported: We’ve known for at least nine months that Obamacare’s website could turn out to be a “third-world experience.” In recent weeks, we’ve learned that there hadn’t been an end-to-end test of whether Americans could enroll as late as September 26, five days before the October 1 launch. So the obvious question has been: if the administration knew the website wasn’t ready, why did they roll it out anyway? The report comes from Amy Goldstein and Juliet Eilperin of the Washington Post. Goldstein and Eilperin interviewed “more than two dozen current and former administration officials and outsiders who worked alongside them” to get to the bottom of why the rollout of Obamacare’s insurance exchanges has been so problematic.

(http://www.forbes.com/sites/theapothecary/2013/11/03/on-sept-5-test-of-obamacares-website-cms-staffers-secretly-rooted-for-it-to-fail/)

Days before launch, Obamacare website failed to handle even 500 users.

(Reuters) – In the last days before the botched October 1 launch of President Barack Obama’s healthcare website, the team in charge was seeing alarming results from performance tests, according to internal emails released by Republican lawmakers investigating the rollout.

HealthCare.gov was unable to consistently handle 500 users at once in the testing, and tests failed with 2,000 users over a three-day period, according to a series of emails between members of the information technology team at the Centers for Medicare and Medicaid Services, or CMS.

(http://www.reuters.com/article/2013/11/22/us-usa-healthcare-website-idUSBRE9AL03K20131122)

Obamacare website contractors say government spent just two weeks testing site

In a House committee filled with frustrated lawmakers of both parties, key contractors said the government spent weeks instead of the months needed on testing the site. The senior vice president of CGI Federal, the contractor who designed the site, called Healthcare.gov one of the most complicated systems ever conceived.

http://www.nydailynews.com/news/politics/contractors-obamacare-site-tested-2-weeks-article-1.1496119#ixzz2r43QtLCV)

No One Tested Healthcare.gov Until It Was Too Late

Some key testing of the system did not take place until the week before launch, according to [one] person [close to the project]. As late as Sept. 26, there had been no tests to determine whether a consumer could complete the process from beginning to end: create an account, determine eligibility for federal subsidies and sign up for a health insurance plan, according to two sources familiar with the project.

Traffic Didn’t Crash the Obamacare Site Alone. Bad Coding Did Too

The basic architecture of the site, built by federal contractors overseen by the Department of Health and Human Services, was flawed in design, poorly tested and ultimately not functional. “You need there to be good people on the inside to make good contracting decisions and good people on the outside to do the work,” explained Clay Johnson, a Democratic technology consultant who recently worked as a White House fellow. “Right now, it’s the blind leading the blind.”

Even on the back end of the site, data was garbled and, in some cases, unusable. The nightly reports  that insurance companies receive from the federal government on new enrollees in the health plans have been riddled with errors, including syntax mistakes, and transposed or duplicate data, according to industry veterans.  In other cases, insurers received multiple enrollments and cancelations from the same person, but since the documents lacked timestamps, it has been impossible to know which form is the most recent. Companies have resorted to contacting enrollees directly to get answers, a solution possible only because so few have been able to sign up.  ”We are seeing and hearing that enrollment files going to carriers are incomplete, there are errors,” said Dan Schuyler, a director of exchange technology at Leavitt Partners, a firm that consulted with several states in setting up their websites. “In three weeks or so when they start receiving these in mass volume, tens of thousands per day, it doesn’t matter if there’s a 1 percent error rate. Insurers don’t have resources to go through them and clean them up.”
Read more: Traffic Didn’t Crash the Obamacare Site Alone. Bad Coding Did Too. | TIME.com http://swampland.time.com/2013/10/24/traffic-didnt-crash-the-obamacare-site-alone-bad-coding-did-too/#ixzz2r484vDIe

Obamacare Website Programmers Complained About Unrealistic Deadlines

Crammed into conference rooms with pizza for dinner, some programmers building the Obama administration’s showcase health insurance website were growing increasingly stressed. Some worked past 10 p.m., energy drinks in hand. Others rewrote computer code over and over to meet what they considered last-minute requests for changes from the government or other contractors.

As questions mount over the website’s failure, insider interviews and a review of technical specifications by The Associated Press found a mind-numbingly complex system put together by harried programmers who pushed out a final product that congressional investigators said was tested by the government and not private developers with more expertise.

Project developers who spoke to the AP on condition of anonymity — because they feared they would otherwise be fired — said they raised doubts among themselves whether the website could be ready in time. They complained openly to each other about what they considered tight and unrealistic deadlines. One was nearly brought to tears over the stress of finishing on time, one developer said. Website builders saw red flags for months.

(http://www.huffingtonpost.com/2013/10/22/obamacare-website-programmers_n_4141411.html)

 

Leave a Reply