00:00:03 Probabilistic forecasts: introduction and overview.
00:01:34 Uncertainty and accuracy in probabilistic forecasting.
00:02:25 Probabilistic forecasts: managing supply chain asymmetries.
00:04:33 Elusive boundaries and unlikely events in probabilistic forecasts.
00:07:43 Mathematical models’ role in probabilistic forecasting.
00:09:15 Evaluating accuracy of a probabilistic forecast.
00:11:14 Drawbacks of classic forecasts over probabilistic ones.
00:13:07 Industries’ reliance on classic forecasts and Excel’s limits.
00:15:23 Probabilistic forecasting’s best applications.
00:18:43 Industries where probabilistic forecasting isn’t needed.
00:20:03 Probabilistic forecasting: adoption and reasons.
00:22:34 Future outlook of probabilistic forecasting.
00:24:27 Future methods: omitting explicit computation of probabilities.
00:25:37 Zooming in on relevant futures and demands.
00:26:14 Uncertainties in product demand timeline.
00:27:03 ‘What-if’ scenarios: impact of price adjustments.
00:27:56 Importance of selective exploration.
Summary
In this discussion with Kieran Chandler, Joannes Vermorel, founder of Lokad, elucidates the nature and benefits of probabilistic forecasting. Unlike deterministic forecasts, probabilistic forecasts are considered to offer a range of outcomes, each with a certain probability. This approach is viewed as a better way to address supply chain asymmetries, such as the different implications of demand overestimation and underestimation. It does not limit itself to averages but is assessing a continuum of potential scenarios. Although complex, advancements in computational power and deep learning are making probabilistic forecasting more accessible. Vermorel is predicting a future where forecasting embraces uncertainty and integrates a multitude of variables, promising a more detailed and realistic depiction of possible futures.
Extended Summary
In this episode of Lokad TV, Kieran Chandler initiates a dialogue with Joannes Vermorel, the founder of Lokad, discussing probabilistic forecasting, its advantages, implementation, and utilization in businesses.
Vermorel elucidates that probabilistic forecasts denote a type of predictions where future knowledge remains imperfect. Unlike deterministic forecasts which predict a single definitive outcome, probabilistic forecasts outline a spectrum of potential outcomes, each associated with a particular probability. The idea is to embrace the uncertainty inherent in future occurrences. This method may not guarantee absolute precision, but it potentially provides a better chance of relevance for decision-making.
According to Vermorel, the primary advantage of probabilistic forecasts over traditional forecasts lies in the capability to handle asymmetries in supply chains. He highlights that overestimation and underestimation of demand could lead to asymmetric outcomes. For example, in the aerospace sector, overestimating demand could yield a surplus screw costing 50 euros, while underestimating could result in grounding an aircraft, incurring hundreds of thousands in delay costs.
Vermorel underscores that traditional forecasting methods often aim for an average outcome. Still, he points out that in supply chains, costs are usually driven more by extreme events. He further illustrates the problem with examples from the aerospace and food retail industries where excessive inventory can result in discards and financial loss.
Discussing extremes, Vermorel makes clear that there are no hard boundaries, but rather a continuum of infrequent events. For a typical product, there could be a 5% chance of witnessing twice the daily demand, a 1% chance of observing four times the daily demand, and a minuscule likelihood of observing ten times the daily demand. Probabilistic forecasting does not limit itself to averages but investigates a range of possible outcomes.
While Vermorel recognizes the challenge of assessing infinite future possibilities, he argues that modern computational resources permit a large range of risks to be incorporated. Extreme events like a shipment sinking are unlikely, but delays at customs or other logistical issues could be considered, as they may have similar effects on supply.
Vermorel proceeds to unravel the intriguing nature of probabilistic forecasting. He introduces the need for suitable metrics to evaluate the accuracy of probabilistic forecasts, ideally assigning more weight to events with higher probabilities assigned by the model.
Drawing a parallel to a hypothetical prediction about Italy winning the World Cup, Vermorel demonstrates that a model’s precision is reflected in how closely its assigned probabilities align with actual events. He compares probabilistic forecasting with traditional forecasting, stating that while the former may not be inherently more accurate, it presents richer information by considering a wider range of potential outcomes.
Vermorel continues by explaining that probabilistic forecasts can be “collapsed” into classic forecasts by taking an average. However, this process omits valuable information about extreme or “tail” events - those with surprisingly high or low demand. These events are often more financially impactful in supply chain contexts, where deviations from the average can lead to expensive outcomes like stockouts or inventory write-offs.
Despite these advantages, Vermorel acknowledges that many industries still employ classic forecasting techniques, often utilizing Excel. He clarifies that this is because of Excel’s accessibility and convenience in creating simple forecasts. Transitioning towards probabilistic forecasting would require abandoning Excel due to the complexity and computational intensity of considering a large number of potential futures.
Vermorel points out that industries marked by high uncertainty, such as fashion, aerospace maintenance, e-commerce, and store-level retail, are ideally suited for probabilistic forecasting. These industries grapple with unpredictability, from capricious fashion trends to sporadic needs for specific aircraft parts, and from the long tail of e-commerce sales to fluctuating store-level sales in large markets.
Vermorel identifies situations where probabilistic forecasting may
be less suitable, such as industries or instances where future outcomes can be accurately predicted. For example, cement production or certain automotive production lines, where long-term contracts provide clear visibility into future requirements. Here, traditional forecasting methods are sufficient. The real value of probabilistic forecasting, Vermorel notes, emerges in situations with substantial uncertainty, where future outcomes cannot be precisely anticipated.
The conversation then shifts towards why probabilistic forecasting is increasingly popular, despite not being a novel concept. Vermorel identifies two main factors: the decreased cost of processing power and the emergence of statistical methods like deep learning. A decade ago, the computational resources needed for probabilistic calculations were exorbitantly expensive. With costs decreasing, these methods have become more accessible. Also, advancements in deep learning, a subfield of AI propelled by probabilistic modeling, have further boosted the rise of probabilistic forecasting.
Discussing the future of probabilistic forecasting, Vermorel asserts confidently that there is no retreat to classical methods. Probabilistic forecasting offers more insights into the future, making it counterproductive to return to methods offering lesser information. However, he does concede the complexities involved, especially when forecasting scenarios involve multiple factors or products. The scenarios to explore expand exponentially with each added item, rendering explicit probability calculations almost impossible. This, Vermorel believes, will drive future methods towards calculations that do not strive to express all probabilities, an approach already employed by deep learning.
The discussion concludes with Vermorel painting a picture of how future forecasting might accommodate uncertainty and integrate complex variables. By contemplating all potential futures, including variations in product demand, supply timeframes, and price adjustments, the possibilities become virtually limitless. However, Vermorel stresses that the goal should not be to examine each potential future individually, but to employ mathematical techniques that enable the exploration of many possible scenarios without necessarily listing them. While this approach poses numerous challenges, it also heralds new opportunities for exploration in the forecasting realm.
Full Transcript
Kieran Chandler: Today on Lokad TV, we’re going to be discussing exactly what probabilistic forecasts are, why they can be beneficial, and also how they can be implemented into businesses to improve the way that they operate. So Joannes, a subject of interest at the moment, we have so many sports fans and bookmakers trying to work out who’s exactly going to win the World Cup. Perhaps a good place to start is: What are probabilistic forecasts?
Joannes Vermorel: Probabilistic forecasts represent a class of forecasts where you have imperfect information about the future. You have a sense of the probable futures, the futures that have a chance of happening, versus the futures that do not have a chance of happening. Typically, when people think of forecasts, they consider them as final, like, “This team is going to win”. But the point is that you don’t know for sure; it’s just a certain probability that this team is going to win. A more accurate forecast is to actually have this shortlist of teams that are very strong and have collectively a very high probability of winning. It’s not as satisfying as knowing the winner, but nobody can ever know that due to the uncertainty at stake. Probabilistic forecasting is about making a statement about the future that involves probabilities. It embraces the very notion that you don’t know everything about the future and you don’t pretend to know.
Kieran Chandler: So what’s the main benefit of this compared to more traditional forecasting techniques?
Joannes Vermorel: The primary benefit is that probabilistic forecasts give you an angle to approach all the asymmetries that you have in supply chain. What I mean by asymmetries is the fact that the cost of overestimating or underestimating demand is not symmetrical. For example, let’s take aerospace. If you overestimate your demand, you might have a screw in stock that you never use. But if you underestimate your need for screws, you can have an aircraft grounded just because it’s one screw short, and that could cost you hundreds of thousands in rerouting passengers and delays. The problem with classic forecasts is that you’re aiming for the average. But in supply chain, it’s not necessarily the average that you want to secure. Your costs are typically much more driven by the extremes. If you have too much inventory in food retail, you might have to discard it entirely, losing all the investment.
Kieran Chandler: So if we talk about these extremes, they’re basically boundaries, aren’t they? So how can we establish where these boundaries are?
Boundaries can be elusive; it’s a matter of probability. For instance, if usually you have in a store a demand of, let’s say, five units a day for a given product, then you might have a 5% chance of observing ten units being demanded on any given day, a 1% chance of having 20 units being asked for, and an almost zero percent chance of having like 50 units being asked for on any given day. So, there is no final boundary, it’s a continuum of events that become rarer and rarer, and you can assess the probabilities of that. However, in this continuum, are you saying you’re predicting every single possibility? Surely, you have to draw the line somewhere. You can’t know exactly what’s going to happen tomorrow. For example, if you’re having products delivered to you, there is a probability that the ship they’re coming on might sink. Can you really use every single possible future?
Joannes Vermorel: There is a limit to what we can assess due to computational resources. Yes, we have computers with lots of memory and processing power, but we must restrict the number of features we assess to a finite number. However, computers have tremendous computing resources. So, even if the number of futures they can assess is finite, it can still be extremely large. For example, for a product that usually sells only a few units a day, you can still affordably assess the probability of selling one thousand units even if it’s a slim chance of one in a million. Likewise, for the risk of a ship sinking, it’s maybe one chance out of a million, but a computer can perform billions of computations per second.
While we might not consider the risk of a ship sinking, we can assess the risk of a ship being detained indefinitely at customs. That can happen, and it can cause a three-month delay because of problems with the customs process. Such a delay would be nearly equivalent to the ship sinking, as far as your shipment is concerned. For instance, if you’re expecting swim suits, the season will be over by the time you receive them. It will be winter, and your product would be useless.
Kieran Chandler: Yes, a ship at the bottom of the ocean is indeed an extreme example. Let’s talk about the mathematical models you mentioned. How do we know which one is best to use? From what I understand, there are different forecasts and different forecasting techniques depending on the mathematical model we use.
Joannes Vermorel: First, you need a mathematical model that generates probability distributions, which is very different from models you might use in Excel. When people think of forecasting, they typically imagine some kind of moving average. They consider what the demand was last week or last year, average the relevant time period, and that gives them a forecast. It’s not a bad method, but it produces a single point estimate.
When you want to move towards the probabilistic world, you need something that generates a distribution of probabilities. You have a variety of mathematical models available to you. The most famous one is the Poisson model, or if you want to be really fancy, you can have a look at negative binomial models. These are different classes of parametric models, but you can also have nonparametric models.
Kieran Chandler: I understand that the use of a more sophisticated mathematical model can generate probabilities that can help predict demand. However, this doesn’t seem to be the end of the process. No matter what happens, your model can always say “I told you so”. If it predicts 10 units of demand and we observe 10 units, the model is right. If we observe 100 units, the model still says there was a probability for this to happen. So how do we know if a model is good or not?
Joannes Vermorel: You’re correct. That’s why we need better metrics, metrics suitable for probabilistic forecasts. If your model assigns a high probability to an event that actually occurs, then your model is performing well. For instance, if I predict that Italy has an 80% chance of winning the World Cup and they don’t win, the model was inaccurate. However, if I say Italy has a 5% chance and they don’t win, then the model was reasonably accurate. These metrics measure how much weight, in terms of probability, you’re putting towards things that are actually happening.
Kieran Chandler: It’s interesting you mention accuracy. How does the accuracy of a probabilistic forecast compare to a traditional forecast? They seem to be measuring very different things.
Joannes Vermorel: Indeed, they do. A probabilistic forecast is not, by design, more accurate than a classic forecast. However, a distribution of probabilities can be collapsed to a classic forecast by taking the average. The problem with this is you lose all information about the tails – the events where demand could be surprisingly high or low. You can measure the accuracy of a probabilistic forecast with a traditional metric like mean absolute percentage error, but it doesn’t really make sense. The goal is to capture more information about surprising events. You want your forecast to be accurate where it really matters financially. In supply chain management, this isn’t always the average situation.
Kieran Chandler: So, in essence, the benefit of probabilistic forecasting is that it allows you to view a wider picture and produce richer forecasts?
Joannes Vermorel: Yes, exactly. It gives you more dimensions, more depth to understand the future.
Kieran Chandler: But despite this, many in the industry still use traditional forecasting techniques. Why are people still content with using these methods?
Joannes Vermorel: I wouldn’t necessarily say they’re happy to use these techniques. The reality is most supply chains still rely heavily on tools like Excel, which aren’t conveniently designed to produce probabilistic forecasts.
Kieran Chandler: Forecasts, I mean it’s possible to produce some poetic forecast but it’s nowhere near as convenient. Producing a classic forecast is just about building some kind of moving average recipe and then you’re good. However, when you want to move towards the probabilistic world, you have to give up on Excel. Not only do you have to stop generating the forecast in Excel, but you also have to stop making the decisions in Excel. Why is this?
Joannes Vermorel: Your decision will be an exploration of all the possible futures. You’re going to assess all possible decisions and reflect those decisions against all possible futures to evaluate the economic outcome for each single decision. This way, you can directly select the best decision based on all possible outcomes. Suddenly, you see, you have a large number of features to consider and a large number of decisions to evaluate against even larger possible futures. It becomes computationally much more intensive and fundamentally incompatible with Excel.
Kieran Chandler: So, if I understand correctly, the reason people are not doing that is primarily because they lack the necessary tools. They move towards Excel not because they prefer it, but because ERP failed at delivering the kind of sophisticated risk analysis they need for making the right decisions for their supply chain. So, if we talk about those industries, which industries does probabilistic forecasting work best in? Where are you seeing the best results for a probabilistic forecast?
Joannes Vermorel: Probabilistic forecasts truly shine when there is uncertainty. For example, if you want to produce electricity consumption forecasts at a national scale, you can do it with a high degree of precision. You can have like a 0.5% accurate forecast if you want to forecast the electricity consumption of France per one-hour time slot, probably up to 48 hours in advance. This is a situation where you almost know the future perfectly. Same thing if you want to forecast how much traffic you will get on the roads, you can have very accurate forecasts as it’s highly predictable. But if you move towards domains where uncertainty is greater, that’s where probabilistic forecasts become more valuable.
Kieran Chandler: Can you give some examples of those domains?
Joannes Vermorel: Absolutely. Industries like fashion, where the trends are highly erratic, are good examples. Fashion has a lot of irreducible uncertainty. Aerospace and maintenance in general also have a great deal of uncertainty, not because the aircraft are uncertain, but because you have many parts that are rarely needed. You don’t know when you’re going to need a part, and you have so many spare parts and aircraft that it’s not like selling bottles of milk in an open market where you’re selling hundreds of units every single day. It’s much more erratic.
Ecommerce in general is another example. The long tail of products is actually super long and most of your sales come from products that have intermittent, erratic sales. And let’s not forget everything that happens at the point of sale and at the store level. Even if you look at what is happening in a store, even in a hypermarket that can have up to one hundred thousand references, you only have, in Europe for example, something like 2,000 products where you’re going to sell five units or more every single day. All the rest of the products are going to sell less than five units a day. So, it’s small numbers, and the erratic nature is large. Probabilistic forecasts shine here because they give you insight into the risks that you have for the inventory decisions that you make.
Kieran Chandler: So, bottom line, probabilistic forecasts shine in areas where there is a high level of uncertainty?
Joannes Vermorel: That’s correct.
Kieran Chandler: Whatever you have a lot of uncertainty, and you need to optimize your decisions against factoring all the ways that you have on one end of the spectrum surprisingly low demand, and the other end of the spectrum surprisingly high demand. Okay, so we’ve spoken a lot about the benefits of probabilistic forecasting. We’ve spoken about where it works well. But how about those industries where it’s not quite so appropriate to use it? Are there industries where actually classic forecasting is fine?
Joannes Vermorel: Yes, for example, if you are producing cement and you have clients that give you a backlog of orders for the next three years, then you don’t need forecasts. If you know the future, it can also happen for some production lines in the automotive industry. When you know that 12 months ahead, you know exactly what you’re going to produce because it’s a large car maker that gives you a roadmap that is very precise and can only divert from max five percent. If there is no residual uncertainty left about your plans and it’s just a matter of pure execution, then indeed, probabilistic forecasting is not going to help you. Probabilistic forecasting is only going to help you if there is some kind of erraticity. If you cannot know the future perfectly, if you already have your roadmap that is frozen for the next 12 months, then basically, you do not care about probabilistic forecasting.
Kieran Chandler: Okay, and why is it that companies are starting to use probabilistic forecasts now? I mean, it’s not an especially new technology, right? So why is it now the time that they’re starting to see it used in the industry a bit more commonly?
Joannes Vermorel: There are probably several reasons. First, it’s a lot more computationally intensive so you end up with statistical models that consume anywhere from 100 to 1,000 times more computational power. The good news is processing power has never been so cheap so it’s rarely the bottleneck. But still, it means that a decade ago, most of these probability calculations were dramatically expensive. It’s very different to be able to run your supply chain on a 2,000 euro budget a month for processing power, or two million euro a month for processing power. It makes quite a difference in practice. That’s what three orders of magnitude mean in terms of cost. So, clearly, the fact that processing power is much cheaper has helped a lot to make these methods a lot more practical. The second thing is that there is an entire class of statistical methods known as deep learning, which is where this artificial intelligence buzzword comes from. It’s all about deep learning and deep learning is all about probabilistic forecasting underneath. You might not care or understand the technicalities, you might just enjoy the fact that you have a piece of software that does voice recognition for you, but it’s actually driven by probabilistic calculations under the hood. First, we had more processing power, then we had mathematical theories like deep learning that came out on top in terms of AI benchmarks. For example, when AI managed to out-compete players like the world champion in Go, it was a probabilistic method that was used, not a combinatorial one.
Kieran Chandler: Okay, so it sounds like probabilistic forecasting is very much a thing of the now, but what about the future? I mean, how do you see the next steps for probabilistic forecasting? Can you see it lasting very long, or how do you see that?
Joannes Vermorel: Yes, I mean, I think that the cat is out of the box and it will not come back. We probably will not revert back to classical forecasting again. Once you have a probabilistic forecast, you know a lot more about the future, so it would be very strange to revert back to an approach that was fundamentally giving you much less insight.
Kieran Chandler: Less information about the future now, even if you say we want to explore all possible futures. In practice, we don’t explore all possibilities. For example, we can say I have a probability of selling zero, one unit, two units, three units of this product and I can make a similar analysis for another product. But what about the joint probability for those two products together?
Joannes Vermorel: Indeed. Suddenly, I have to estimate maybe a hundred scenarios to consider all the demand for my product A. I have to assess a hundred scenarios for all the demand for product B. But what about looking at all the scenarios for product A and B together? That’s like ten thousand scenarios to look at. And if I add a third product with a hundred scenarios, that would be a million scenarios to look at. The situation becomes rapidly more complicated if you want to express all the probabilities explicitly. I think what we will see more and more in the future are methods that don’t even try to express those probabilities. You don’t even try to compute all the possibilities for all the things that can happen. You have methods that actually do those calculations without explicitly stating the probabilities. This is what deep learning and AI techniques are about. They compute probabilities, but not by expressing everything as a probability. The big bonus is that you can explore scenarios about the future that are extremely complicated, and way beyond the capacity of any reasonable computer, or even a fleet of computers.
You can still explore all those features with smart mathematical tricks. The essence of deep learning is that you won’t explore the future randomly. You want to zoom in on the futures that are most likely relevant for the forecast. So, you want to focus on the areas that are relatively dense, where there’s a higher chance of being a future of interest, instead of randomly trying to explore everything.
This approach will unlock tons of scenarios. For example, one of the things that we will try to explore probably this year will be to explore not only all the possible demand levels for products but also look at all the possible horizons in time. You want a forecast for the demand that can start at any point of time and can end at any point of time, both randomly.
This is a way, for example, to reflect a scenario where you have a shipment coming through a ship and you have uncertainty. You don’t know exactly when the product will stop being available for sale in your store, online or offline, and you want to take this uncertainty into account when you make your plans for inventory.
You need to take into account the fact that you have uncertainty about when the goods will be received and when the demand will actually start and end. If you want to look even further, it would be very interesting to start exploring what-if scenarios.
As part of our long-term roadmap, we even plan to start exploring what are all the possible futures if you consider all the pricing adjustments that you can make on your products. You want to see what all the possible futures for demand are if you leave your prices as they are and what if you start exploring all the possibilities for all the price adjustments that you can make on top.
When you start thinking of all those possible futures, the numbers become extremely large. The trick is that you do not want to try to iterate individually over all those futures. You want to have some kind of mathematical techniques that let you explore a lot of them without trying to enumerate them.
Kieran Chandler: Well, it sounds like there’s so many possibilities. I’m glad it’s left to computers because otherwise, my brain’s gonna probably explode. But we’re going to have to leave it for today. Thanks for taking the time out to tell us all about probabilistic forecasting. It’s been really interesting. Thank you. That’s all for today’s episode. We’ll be back again next week, but until then, make sure you’re subscribed to our videos and we’ll see you again soon. Bye for now.
