Top 5 Challenges for Machine Learning Projects

Mateusz Opala

Aug 9, 2018 • 9 min read

Although scientists, engineers, and business mavens agree we might have finally entered the golden age of artificial intelligence you have to be ready to face much more challenges than you think when planning a machine learning project. Entrepreneurs, designers, and managers tend to overestimate the present capabilities of machine learning. They expect the algorithms to learn quickly and deliver precise predictions to complex queries. They expect wizardry.

Commercial use of machine learning, especially deep learning methods, is relatively new. They require vast sets of properly organized and prepared data to provide accurate answers to the questions we want to ask them. A business working on a practical machine learning application needs to invest time, resources, and take substantial risks.

A typical artificial neural network has millions of parameters; some can have hundreds of millions. A training set usually consists of tens of thousands of records. While a network is capable of remembering the training set and giving answers with 100 percent accuracy, it may prove completely useless when given new data. It is just one of the limits to current deep learning algorithms. Read on top 5 challenges for machine learning projects.

The black box problem

The early stages of machine learning belonged to relatively simple, shallow methods. For example, a decision tree algorithm acted strictly according to the rules its supervisors taught it: "if something is oval and green, there's a probability P it's a cucumber." These models weren't very good at identifying a cucumber in a picture, but at least everyone knew how they work.

Deep Learning algorithms are different. They build a hierarchical representation of data - layers that allow them to create their own understanding. After analyzing large sets of data, neural networks can learn how to recognize cucumbers with astounding accuracy. The problem is that their supervisors - the machine learning engineers or data scientists - don't know exactly how they do it. The problem is called a black box.

Artificial Intelligence supervisors understand the input (the data that the algorithm analyses) and the output (the decision it makes). While the engineers are able to understand how a single prediction was made, it is very difficult to understand how the whole model works.

Some AI researchers, agree with Google's Ali Rahimi, who claims that machine learning has recently become a new form of "alchemy", and the entire field has become a black box. It is a significant obstacle in the development of other AI applications like medicine, driverless cars, or automatic assessment of credit rating. What if an algorithm’s diagnosis is wrong? How will a car manufacturer explain the behavior of the autopilot when a fatal accident happens? How will a bank answer a customer’s complaint?

The black box is an issue in machine learning for in-app recommendation services. It turns out that web application users feel more comfortable when they know more or less how the automatic suggestions work. That is why many big data companies, like Netflix, reveal some of their trade secrets.

The research shows artificial intelligence usually causes fear and other negative emotions in people. People are afraid of an object looking and behaving "almost like a human." The phenomenon is called "uncanny valley".

We accept machines that act like machines, but not the ones that do the human stuff, like talking, smiling, singing or painting. Of course, this may change with time, as new generations grow up in a digital environment, where they interact with robots and algorithms.

Talent deficit

Although many people are attracted to the machine learning industry, there are still very few specialists that can develop this technology and that is currently one of common machine learning problems. A good data scientist who understands machine learning hardly ever has sufficient knowledge of software engineering.

The problem is drastic. The Chinese tech giant Tencent estimated at the end of 2017 that there were just about 300,000 researchers and practitioners dealing with AI worldwide. Element AI, nn independent company, estimates that "fewer than 10,000 people have the skills necessary to tackle serious artificial intelligence research". Machine learning engineers and data scientists are top priority recruits for the most prominent players such as Google, Amazon, Microsoft, or Facebook.

It makes salaries in artificial intelligence field skyrocket, but also makes the average quality of specialists available on the market plummet. I wrote about general tech brain drain before. With machine learning, the problem seems to be much worse.

According to NYT in the US, people with just a few years of experience in artificial intelligence projects earned in up to $500,000 per year in 2017, while the best will get as much as NBA superstars. In a court filing in 2016, Google revealed that one of the leaders of its self-driving-car division earned $120 million in incentives before he left for Google's competitor - Uber.

uber google machine learning-746025-edited-801931-edited

Data is not free at all in machine learning projects

As I mentioned above, to train a machine learning model, you need big sets of data. It may seem that it's not a problem anymore, since everyone can afford to store and process petabytes of information. While storage may be cheap, it requires time to collect a sufficient amount of data. Moreover, buying ready sets of data is expensive.

There are problems in machine learning of a different nature. Preparing data for algorithm training is a complicated process. Here's an interesting post on how it is done. You need to know what issue you want your algorithm to solve, because you will need to plan classification, clustering, regression, and ranking ahead.

You need to establish data collection mechanisms and consistent formatting. Then you have to reduce data with attribute sampling, record sampling, or aggregating. You need to decompose the data and rescale it. It is a complex task that requires skilled engineers and time. So even if you have infinite disk space, the process is expensive.

If you plan to use personal data, you will probably face additional challenges. People around the world are more and more aware of the importance of protecting their privacy. They may be unwilling to share them with you or issue a formal complaint if when they realize you did it, even if you obtained all they gave you their consent.

Personal data and big data activities have also become more difficult, risky and costly with the introduction of new regulations protecting personal data, such as the famous European General Data Protection Regulation.

The machine learning technology is very young

Once again, from the outside, it looks like a fairytale. The biggest tech corporations are spending money on open source frameworks for everyone. The Alphabet Inc. (former Google) offers TensorFlow, while Microsoft cooperates with Facebook developing Open Neural Network Exchange (ONNX). These systems are powered by data provided by business and individual users all around the world.

However, the central problem of machine learning is that all these environments are very young. The first version of TensorFlow was released in February 2017, while PyTorch, another popular library, came out in October 2017. Web application frameworks are much, much older - Ruby on Rails is 14 years old, and the Python-based Django is 13 years old.

On one hand young technology uses the most contemporary solutions, on the other, it may not be production-ready, or be borderline production ready.

You need time to achieve any satisfying results and planning is difficult

Traditional enterprise software development is pretty straightforward. You have your business goals, functionalities, choose technology to build it, and assume it will take some months to release a working version. In machine learning development has more layers. The engineers are writing a program that will generate a program, which will learn to perform the actions you planned when setting your business goals. Just adding these one or two levels makes everything much more complicated.

The challenge is that machine learning takes much more time. You have to gather and prepare data, then train the algorithm. There are much more uncertainties. That is why, while in traditional website or application development an experienced team can estimate the time quite precisely, a machine learning project used for example to provide product recommendations can take much less or much more time than expected. Why? Because even the best machine learning engineers don't know how the deep learning networks will behave when analyzing different sets of data. It also means that the machine learning engineers and data scientists cannot guarantee that the training process of a model can be replicated.

Understand the limits of contemporary machine learning technology

It's very likely machine learning will soon reach the point when it's a common technology but the main machine learning problems are yet to be solved. Nevertheless, engaging in an AI project is a high risk, high reward enterprise. You need to be patient, plan carefully, respect the challenges machine learning technology brings, and find people who truly understand machine learning and are not trying to sell you an empty promise.

Related topics

More posts by this author

Mateusz Opala

New call-to-action

Read more on our Blog


Check out the knowledge base collected and distilled by experienced professionals.