In my previous post, I highlighted how mankinds greatest invention the machine, with its learning ability, is making a positive impact on human life, by driving innovations in the highly regulated, life sciences sector. The post stressed the need for assurance and validation of machine learning (ML) systems. I also emphasized the need for self-regulation, until regulators come up with detailed guidance on ML applications.
In this post, I will share a broad framework and a step-by-step approach for ML validation. But before getting down to specifics, let me explain the difference between supervised and unsupervised ML. In supervised learning, machines are trained using output datasets much like how we, as kids, learned to recognize animals, tress, and objects from pictorial charts. Unsupervised learning, instead of using sample output datasets (similar to the pictorial charts for kids), classifies the data into classes, and lets the algorithm differentiate objects correctly. This is similar to visiting a new country, exploring new places and areas of interest, with research building on your current level of familiarity, and determining the best transport options, routes and paths that can save time and money.
Given the algorithmic need for understanding behavior designs and patterns, scope definition for both these ML types can be difficult, involving multiple iterations and varying degrees of complexity, and sometimes even resulting in significant scope creep. Such dynamic requirements specifications are best suited for Agile methodologies. In turn, the Agile way of working demands unprecedented levels of team collaboration, with each team member having skills and ability to code, build, qualify infrastructure, and also act as a consultant, whenever required.
A typical ML process begins with a problem statement a formal understanding of the problem, and the need for machine learning to solve the problem. The proposed system is also categorized as per GAMP-5 a regulation and standard for validation of automated pharmaceutical production systems.
Next comes the data preparation phase, involving definition of speech, vision and image data sets, along with sample inputs (the kid pictorial charts for machines). This phase also involves development of deep learning algorithms and machine learning mechanisms (in our example, the algorithmic intelligence to identify and differentiate between edible and non-edible).
The third phase involves spot checks for evaluating algorithms, through test harnesses. Although there are a number of ways to conduct test harness in machine learning, train the test and cross validation are two widely used harness methods. Train the test involves usage of scripts to evaluate algorithms for learning correctness. Cross validation is a sampling method, to determine algorithm performance through reliable estimates. This phase requires more white box testers.
Finally, the learn and enhance phase involves evaluating the desired output, and fine tuning the algorithms for enhanced accuracy, greater efficiency, and improved performance. The deliverables of the process include code review and installation qualifications for the algorithms, evidence sampling for batches, performance qualification protocols, scripts, summary reports, and a security and privacy compliance checklist, in line with HIPAA title 2 on administrative simplification.
Despite being all about algorithms and deep programs, programming is not all, when it comes to ML systems. With most ML systems being IoT and cloud based, its also necessary to qualify the infrastructure before hosting ML systems on the cloud. Aspects such as single sign on for private clouds, application security of the IoT ecosystem, hardware and software specifications and performance, IQ scripts, and production standard operating procedures, are equally important. The infrastructure qualification plan and summary report are important deliverables of this activity. To guard the ML systems against the risk of hardware wear out and deterioration, due to extensive use, peripherals too, especially storage systems, must be regularly tested and subjected to troubleshooting.
When selecting infrastructure for ML systems, companies must be choosy, and ensure that infrastructure partners will not just support uniqueness of every ML solution, but also be able to provide the required levels of scalability and elasticity. Infrastructure partners must also qualify the infrastructure, especially when its used for life sciences applications. When working with multiple infrastructure partners, the responsibility assignment matrix, also known as the RACI matrix, is a good framework to drive clarity and reduce risk of regulatory noncompliance.
On the workforce front, with ML requiring skillful and analytical white box testers to conduct test harness and debugging, the data scientist, deep programmers, and validation consultant roles are becoming popular and prominent. While external hiring is always an option, its also good to have an internal, train the trainer strategy in place.
When embarking on your ML journey, dont take the big bang approach, as any large investment poses high risk to business. Start small and make steady progress. Create prototypes, demonstrate with successful pilots, and then scale up. To capitalize on the ML revolution, think out of the box, from all perspectives. Only then will you be able to mitigate risks and maximize returns from ML investments. Remember, machine smartness depends on business intelligence – and that includes the combined intelligence of your management and workforce, working in unison, to make your machines smarter. And smart machines, with equally smart processes and solutions to drive them, will not just take you closer to customers, but also ahead of competition.