Phase II: Exploration (4-6 weeks)
At this stage, the objective is to build a Proof-of-Concept model that will be installed as an API. Once we have trained a baseline model that is performing the task, we can estimate the goal performance of our production-ready solution.
Once again, smart matching of business goals with ML metrics prove to be beneficial. When testing a recommendation system for an e-commerce site, if there were no advanced recommendations before, a trained baseline model can be implemented at a very early stage, practically when it's performing the task.
On the other hand, this may become an extremely difficult problem if we are working on improving an already effective recommendation system.
At the end of the exploration phase, the team should be able to estimate what performance can be achieved with any of the metrics planned during the discovery phase.
Phase III: Development (+3 months)
It's time for the bespoke software development team to work iteratively until they reach a production-ready solution. As there are fewer uncertainties with each step of the project, at this stage estimation is getting more precise.
When training an algorithm, we can react to each output of our experiments as we watch the computer program write another computer program.
An algorithm learns very fast, we make a test on a set of data, we apply the metric, and see if the result has reached our goal or, if not, how far away we are. If we wanted our recommendation module to predict a viewer watching the whole movie with 70% accuracy, and our model is still at 55%, we need to readjust the algorithm and run it once again until we reach the goal.
If the result is not improving, sometimes the engineers have to apply a different model or change the method, or adjust the data. We do it until we reach the goal measured with the metric.
In this phase, the team is working in sprints, deciding after each iteration what to do next. The outcomes of each sprint can be predicted efficiently, however planning more than one sprint ahead is a mistake, especially in Machine Learning, where you are often sailing on uncharted water.
Phase IV: Improvement (indefinite)
When we have already deployed a solution to the production environment, business decision makers are often tempted to end the project in order to cut costs. In Machine Learning technology this is often a mistake. Usually, the data, such as user preferences and trends, changes over time. That is why an AI model needs to be constantly monitored and reviewed to protect it from erosion and degradation.
Machine Learning projects need time to achieve satisfying results. Even if you are lucky and your algorithms beat the benchmarks immediately, chances are it's just one strike, and your program will get completely lost using a different dataset.
That is why the improvement phase is perpetual. It can be done efficiently and does not need as many resources as the previous two phases, nevertheless, it has to be done. Continuous monitoring will not only protect the model from degradation, but improve it over time.
Machine Learning brings uncertainty to the project. That is why pays off to get the best and most experienced engineering team involved. Defining business goals and metrics, sketching architecture, and planning technical requirements at the earliest stage will determine the failure or success of your venture.