With the advances in artificial intelligence (AI) and machine learning (ML), we see before us a huge opportunity to deploy these technologies in everyday use. From fully autonomous customer interactions to plant failure prediction, the ML domain is virtually endless. However, given an almost limitless supply of options how do teams specify and focus on which areas to start their ML journey. Any project needs a scope, a timeline, and a budget, in order to establish whether it is worth doing. Yes, we may consider using agile methods for implementation, but we still need a definition of ‘done’, and before we begin, a definition of ‘ready’.
Identifying the goal; defining the outcome
As with any project, when determining where to start an ML implementation the focus should be to ascertain what we are trying to achieve. What is the objective of this ML project? Multiple times, we see that in the initial phase, the goals are quite vague. For example, improve customer satisfaction, predict engine failures, and forecast disease outbreaks. These are fine for a starting point. We can then apply a fairly systematic approach to refining the ML specifications.
Let us take the case of ‘improving customer satisfaction’; the approach however applies to all scenarios.
To begin, we must answer the fundamental question: what does customer satisfaction mean for our business and how do we measure it today? If we are already at a satisfaction index of 99%, then ‘improving’ it may be very difficult and time consuming, and our efforts will only result in a small incremental value. This brings us to a pertinent question about our goal: is it realistic, achievable, and desirable? How does the cost-benefit analysis look like? Aiming for a 100% on customer satisfaction is a fantastic goal, but if the starting point is 99%, the effort and time invested in closing that gap of 1% may be better spent elsewhere.
Having decided on an output that is worth the investment, the team can consider how it will be measured. Will a binary answer suffice, for instance, ‘yes’ – the customer is satisfied, or ‘no’ – he/she isn’t, or will this be a classification outcome ranging from dissatisfied to neutral, to satisfied, to extremely satisfied?
A further consideration when defining the model scope is to consider any particular constraints. In our customer satisfaction example, we may want to develop a predictive model that predicts which associate can handle a specific type of call the best. However, we would have to allow for the fact that one associate can only handle one call at a time, and only, so many per day. So, the availability of an associate becomes a constraint on future outcomes.
Building the model
With the outcome and goal in mind, we can start to evaluate data to drive this prediction. General data assessment rules of velocity, veracity, and volume will apply. If your result is expected in real time, you may have to consider alternative sources, if some are batch oriented, highly static in nature, or even only available externally.
During production, the trained model will be making predictions based on the actual data rather than the data used to design it. A further step is to consider how the model will be validated while in full operation. If we take the example of a service desk, where satisfaction may be measured through the ability of the agent to handle a call efficiently, at what stage and how we will actually be able to see whether the satisfaction is as we predicted, and what tools, software, and infrastructure will be required to achieve it.
We must also be mindful of the risk of bias entering into the model and think through the strategy, policies, and techniques that need to be applied to counter it. This process becomes especially significant when dealing with data that can be influenced by gender, ethnicity, age, orientation, and/or other potentially discriminating factors. Furthermore, when using data to train and develop models, adherence to GDPR regulations will also have to be considered and reviewed.
Last but not the least, what will we do with the result? In our example, we predict that the satisfaction of this service desk interaction is going to be either positive or negative. Who will use the result and how? The activities required to leverage the result, which may range from integration into a desktop application to feeding into another ML model, must also be considered and planned.
A quick recap: steps to consider
In conclusion, when planning to embark on an ML project, the following is a sample checklist:
- What is the outcome and how will we measure it?
- Under what constraints will this model operate?
- What data do we need to predict the outcome?
- Where is the source of that data?
- How to manage the risk of bias?
- How to ensure GDPR compliance?
- How to validate the model in production?
- How will the users utilize the outcome?
One final challenge
Over the years, teams have developed multiple estimation models for software development. Development teams traditionally use bottom-up approaches, T-shirt sizing approaches, user stories, Planning Poker models, and so on, however, there are currently no clear guidelines as to how long it takes to generate and train an ML model to the desired levels of accuracy. Benchmarks and baselines are few and far between, which means, there is a significant element of risk and guesswork at this stage. Furthermore, the access and availability of infrastructure to train and even run these models must be factored into the total costs of the equation, especially if the problem requires the development of very deep learning with very large data sets.
While there is a lot of buzz around AI and ML today, the success of a transformation project will largely depend on how well the predicted outcome is leveraged and how quickly results can be realized. This approach will most likely help to get it right, the first time around.