The advent of computing has undoubtedly proven a point – machine is mankinds greatest invention. And as time passes, these machines are becoming smarter by the day with their ability to learn business and personal tasks, convert human physiology into scientific algorithms and formulae, and identify objects by their features. Remember the super computer IBM Deep Blues chess victory over Garry Kasparov? Another well-known example is the popular social networking platforms ability to automatically recognize photos, and tag relevant people from your friend list.
Over the past five years, machine learning (ML) has made remarkable progress, especially in the text, imaging, speech recognition, and natural language processing (NLP) areas. Other applied areas include probability and statistics, fuzzy logic and decision trees. Practical application of ML has resulted in number of business and personal innovations from data mining and business intelligence, to telecommunications and gaming, driverless cars and even robotic surgery. Recognizing this potential to drive innovation and business results, large corporate giants are investing in the domain, through acquisition of various ML startups. The DeepMind, API and Moodstocks AI deals are just a few examples.
What is the Role of QA & Validation?
Since 2013, the life sciences and medical sectors are also witnessing ML startup activity. Mankinds greatest invention is gearing up to make a positive impact on the quality of human life. Lets explore a few instances and examine the need and importance of Quality Assurance and validation for ML in life sciences and healthcare industries.
Today, ML provides indispensable algorithms, systems, and tools for lab diagnosis, image and cycle management, intelligent clinical data analysis, and computer aided medical procedures such as organ recognition and tissue characterization. Data of genomics research are irresistible both in their amount and their complication. Microarrays, structural genomics and proteomics methods are generating data with multiple dimensions representing the complex connections that occur among various molecular elements of the cell. By making the data available to the ML systems in an accessible, refined and usable form, ML system can help bioinformatics make sure the genomics research to advantage human health is improved.
Further, as ML technology advances, new solutions and ecosystems such as augmented reality and the Internet of Things (IoT), are making inroads into the healthcare and life sciences space. In fact, ML and the IoT are inseparable. Take sensors for instance. When fitted into human bodies, these sensors, working as part of a larger healthcare IoT ecosystem, collect real-time vital body parameters such as sugar levels and heart beats, and transmit the medical data to healthcare servers, which in turn, alert medical practitioners on their mobile devices. In some cases, medical technologists have taken machine intelligence to an altogether new level enabling the machines to learn from patients medical history, and even suggest immediate medication, before the practitioner intervenes.
While this use case could truly be a life saver, it could also put lives at risk, if the machine malfunctions, or does not learn correctly. Plus, security, authentication, data integrity, confidentiality, availability and privacy issues too, must be strictly addressed. That brings me to an important point – its not just enough to validate functionality of medical devices in isolation, its equally, and perhaps more important, to continuously assure machine learning algorithms, processes and systems the entire medical IoT ecosystem, as a whole.
While data quality and audit trail validation can be done in batches, to assure the suitability of performance measures, in todays times of Agile development and delivery, functional and other testing must happen in real time, as events unfold. Also, as ML involves use of complex algorithms, white box testing is more critical than functional testing. Unlike typical computer system validations processes adopted by LS, validation of machine learning systems is different not just in complexity, but also in the execution process. Given the highly regulated nature of the LS industry, regulators too, will need to continuously catch up with evolving technology, and release guidance and standards regularly. Currently, the 510(K) standard addresses submission and clearance of apps that makes use of ML, and act as medical devices. The Health Insurance Portability and Accountability Act (HIPAA) lays down guidelines for Protected Health Information (PHI). Other relevant regulations include ISO 27001, ISO 27018 and EU Model Clauses compliance. While these are good starts, ML is not well articulated in FDA guidance documents. And until regulators come up with well-defined guidelines, companies will need to turn to the experience and judgment of their quality assurance and testing practitioners and auditors, to evolve validation strategies for ML systems. In my next post, I will outline the process, steps and deliverables for effective ML validation. Stay tuned.