Do We Over-Index on Machine Learning for Business Applications?

Machine learning (ML) is one of those buzzwords that both big companies and startups love to use as a statement of credibility. Ie. “We use machine learning to harness X to power Y and impact Z.” The main problem with this approach though is that not every company needs to use ML in their business depending on the size and sector the company operates in. An analogy to use to explain the impactfulness of ML is let’s say you need a vehicle for transportation services. Do you want to buy a large semi-truck or a smart car, or even a bicycle? The answer ultimately depends on what you are using the vehicle for. If you are planning on moving large amounts of goods from point A to point B, you may want to use a semi-truck. But if you are planning on using a vehicle for a simple daily commute, maybe a smaller car or bike is sufficient. The same is true for applying ML to different scenarios whether it be in understanding market segments, reducing cloud data storage, or optimizing retail storefronts.

Below are some of my main takeaways for balancing ML with human decision-making in business applications.

Importance of Sample Size and Scope

In understanding any applications for ML in business settings, the most important question — as with any other data science problem — is how much sample has been collected for the problem. This depends on a few factors:

• What is the scope of the problem? Are we looking at specific types of products? How many consumers purchase or use these products?
• How was data collected? Was it using a survey or logging events within an app to understand user behavior or through payment info.? Were there any mistakes in data collection? If so, how much of the data collected is not valid?
• What is our objective in analysis? Are we trying to optimize a product for a market? What dimensions do we want to break this problem down by (ie. demographic info. segments such as age, gender, etc.)?

Ultimately, these factors will all impact the necessary sample size in order to perform analysis. It is completely possible that it may be impossible to collect enough sample depending on the question being asked. For example, if I want to conduct a cluster analysis on what types of cars with a cost > $1M that people buy, I will likely have a small sample size since the number of individuals who can afford a $1M+ car is limited to the upper echelons in socioeconomic status. Conversely, if I asked the same question about cars with a value > $30K, I would have a much larger sample size to include middle class cars that are more readily affordable, but there may be bias I did not account for. Ultimately, sample size is the number one decision-maker in terms of being able to use data to solve your problem vs. using human intuition.

Machine Learning for Modeling Services or Product

Depending on the industry and type of problem being solved, there is a way for the problem to be framed such that a model can be constructed to help identify a solution. For example, let’s say that you have been asked by a large clothing manufacturer to predict the most popular types of t-shirts being sold next year to each distributor or retailer since each company has a slightly different brand logo which can cost more or less money to print. There are many different ways to model this problem assuming you have enough historical data for training, test, and potentially validation sets (for boosting/cross-validation).

• Optimize for the which t-shirt types are likely to be bought by companies. This can be solved by using multiclass classification or CART modelling. We would still keep seasonality in mind as a variable. In this approach, the goal is to identify which types of t-shirts are more likely to be sold and so the response variable is categorical in nature. When we test the model, we compare if the actual t-shirt type and predicted t-shirt type are the same or different to measure accuracy. This method of modelling does not implicitly take cost into account and so it is up to us to remove shirt types that may be niche and hard to manufacture efficiently.
• Optimize for the t-shirt type with the highest profit margin. This can be solved by using a simple linear regression model potentially augmented using random forest or boosting. Again, we keep seasonality in mind as a variable. This approach is arguably better since we can consolidate any revenue and cost metrics per row into a single value. Since this approach uses linear regression, we would likely need to be careful of bias which can cause overfitting so we should look at VIF scores before we test for the predicted response value of profit margin and use RMSE to compute error.

In this particular scenario, since the time to run these types of models would take no more than a day or two, we could possibly compare the accuracy levels of both approaches and identify which is better.

Ultimately, this is just an example of how ML can be used in business decisions. There are other possibilities such as in performing competitive analysis before product development or even extending the approach in the example above to understanding consumer behavior as a retail front.

Holistic Decision-Making

As mentioned above when discussing the importance of sample size, there are many situations where there may not be enough sample to use ML-based techniques for business intelligence.

In these situations, such as comparing contracts for purchase orders or when a new category of product is created, it is important to use human intuition in problem solving.

The main problem though is when we choose to use human intuition even when we have data in front of us to use. This is a common problem in some industries such as supply chain management where operations teams will likely have enough data that is being collected for inventory management, bill of lading, etc., but the tools being used are old, outdated, legacy systems which makes analyzing data a challenge.

Balancing Strategic and Data-Driven Decisions

There is a fine balance between using human intuition and relying on models for decision-making in business. ***It is important to note that even ML is not a precise science and that tuning models and measuring sensitivity to perturbation definitely require human intuition and potentially some brute forcing.

As a general rule, it is always good to use human intuition to form a hypothesis about a problem or to make decisions with significantly large scope, and then rely on data science fundamentals such as A/B testing and building complex ML models in order to optimize or modify the decision being made. There are, of course, exceptions to this rule, but it works as a general framework for problem-solving in business applications.

Should We “Over-Index” on ML?

My personal opinion as an industrial engineer-turned-product manager is that the answer should be yes. Granted, I don’t have a PhD in AI or ML, or have worked for an extended period of time as a data scientist, so feel free to take my opinion with a grain of salt, but my reasoning is that as a general rule in business, we have been approaching different systems such as supply chain management, market research, etc. using traditional models, which are great starting points, but there is definitely room to improve.

My opinion is that if you are part of a R&D-based company or a firm looking to go through a digital transformation, having a few data scientists that help with ML modeling can be incredibly impactful to understanding more about a market or portfolio. But if you are working on a startup/for a small business, the net benefit of machine learning is a lot lower since the cost of having a data scientist on your team is incredibly high and the returns are not as impactful in the near term. Again, these are generalizations, so keep in mind there are exceptions in both of these cases.