Intelligence Testing Whitepaper
TestStream is unlike any other technology or testing approach available
Today’s modern healthcare systems are large, integrated electronic systems that incorporate thousands of cross-functional actions in unique environments. They require extensive testing and validation. This is to ensure they are operating as the manufacturer intended and providing accurate and error-free data processing within the healthcare provider’s organization.
The FDA has mandated that electronic blood banks undergo quality assurance testing and validation after every change to these systems. The FDA and AABB consider these systems 510(k) medical devices and issue updated testing and validation guidelines annually. However, outside of blood banks, the same crucial testing and attention to changes is just as important.
To test these highly complex systems, healthcare organizations typically employ teams of trained IT professionals armed with automated software tools and some quantity of manual test scripts. All of these testing tools require individual scripts to be written, loaded and maintained continuously to operate correctly. These scripts are single-patient, dedicated functionality focused, end-to-end “patient journeys” that are unique to the system, and version, and very specific in nature. These scripts test a minute fraction of the testable system and, as such, do not address the large-volume, en mass testing required to effectively validate all the variables and conditions incurred in a “day in the life” of these systems. These highly integrated systems are simply too large and complex to effectively maintain manually.
Traditionally, the largest part of the hospital’s testing overhead expense has been to train and retain a talented resource pool to test the system. The problem with this approach is once a single change is made, those individual test scripts now cease to function. Subsequent workarounds to the test script itself is required to get a conditional “Pass”. The major drawback to this approach is the scripts themselves cannot self-adapt to changes that inevitably get introduced into the system regularly. Scripts must be manually debugged with a subsequent trial and error process in an attempt to get a script pass each and every time they are executed.
Take this exercise and multiply it times hundreds of thousands of times across every facility with every variable… and you start to understand the extreme scope and challenges associated with validation and why it’s so difficult. These systems are in a constant state of flux and manual approaches can provide only at best single digit percentage test coverage. Today’s modern systems are simply too large and complex with too many changes being introduced daily to effectively test by traditional methods. Clinical errors continue to manifest and proliferate over time that poses a very real risk to patients — and these become even more critical as these systems evolve in complexity and integration capabilities.
Taking test scripts and their headaches and limitations aside for the moment, we must address first the quality state of the system under test. With a faulty system build configuration, system testing is a complete waste of time. Unknown errors introduce themselves in the build configuration, causing once passed test scripts to fail. Critical risks to patient safety are introduced through routine use and maintenance updates and creep in over time and lurk in the build configurations. Unless you have absolute up-to-the-minute system build intelligence to uncover these system flaws and correct them upfront, they will continue to flourish undetected and create endless havoc for test teams.
The endless cycle of script-based testing has proven ineffective, costly and dangerous to patient safety. It’s clear a different approach is necessary to successfully test and validate both the operational and quality assurance requirements for the LIS/HIS.
TestStream is unlike any other technology or testing approach available. TestStream is not a test tool, but instead a proprietary, broad-spectrum, intelligence-based learning engine that intimately understands the system under test and applies this system knowledge to test the system just like a human would — through the front end graphical user interface (GUI). TestStream understands your unique build configuration through its own data collection and analysis processes. Once the system is learned and understood shortly after installation, TestStream utilizes our own proprietary visual recognition technology which has the uncanny ability to “see” the screen and understand what it’s “seeing”. Because TestStream is smart enough to test the system just like a human being, it can register the patient, order the medication, and schedule the surgery. By moving the mouse and typing the keyboard, TestStream has the ability to test your entire order catalog across every department and facility as it understands your own build infrastructure and site specific configurations, options, customizations and variables.
TestStream’s artificial intelligence approach to clinical testing incorporates these three major process elements:
Not just testing and reporting the issues found, TestStream’s HealthScore system categorizes the problems, allowing for prioritization of critical issues. This unique display shows the health of the system at any point in time. This is possible through TestStream’s risk assessment system as it truly understands the environment it is testing.