Top 10 lies of forecasting vendors
Prediction is very difficult, especially about the future. Niels Bohr
Lokad has been in the forecasting business for more than half a decade now, and while we took an immense pride in sticking to what we believe to be the truth, we did witness countless times competitors getting away with unabashed lies. Hence, for the record and for the fun of it, let’s list the top 10 lies of forecasting vendors.
1. Our forecasts are accurate
2. We will forecast your promotions
We have already written extensively about promotion forecasts. In a nutshell, getting promotion forecasts right is insanely hard. In particular, it takes a massive data qualification effort to have anything exploitable for statistics. The problem is even harder for ecommerce where the visibility given to any product can go from nothing to front page placement in minutes – and back to nothing in minutes as well. Claiming that a forecasting technology is good enough to forecast promotions - during the course of a vendor selection process – only proves that the vendor is dangerously little experienced with actual promotion forecasts. More seasoned or more honest vendors would argue there is not a single chance that such a promotion forecasting benchmark does not end-up in a major garbage-in garbage-out exercise.
3. We are experienced with your vertical
Vendors are so experienced that they don’t hesitate to propose forecasting methods that are guaranteed to fail such as daily or weekly median forecasts in case of (e)commerce. At Lokad, it took us a few years to realize that median forecasts were irremediably dysfunctional as far inventory optimization was concerned. We started to realize the depth of the problem the day we started to observe the actual impact of our forecasts on the business of our clients. It’s wasn’t pretty but it proved very enlightening. If the forecasting vendor does not emphasize that your reorder points and reorder quantities are the only relevant forecasts for inventory optimization, the vendor is either deceptive or clueless.
4. Our technology is intuitive and efficient
Statistical forecasting is so intensely counter-intuitive that there is no way to have both something intuitive AND efficient. It’s either intuitive and completely incorrect, or unsettling and possibly correct. It’s just the way the human brain works. We are not geared for a correct perception of randomness. We see patterns everywhere when it’s just statistical noise. What makes statistical forecasts efficient against human “expert” forecasts is precisely the fact that while the forecasting models are pretty dumb by human intelligence standards, when correctly designed, those models are not biased.
5. You will get X% less stock and Y% less stock-outs
Enterprise software vendors absolutely love case studies claiming ground-breaking results. One private joke at Lokad is that, in order to catch up with competitor claims, we should also start claiming that our software cures cancer too. The reality is that the immense majority of those case studies are utter garbage. Numbers are made up, testimonials are made up and observed improvements, if any, are vastly unrelated to the forecasting solution. The customer is happy because the case study gives them some good press and drive their own competition nuts. Combo: by claiming very loudly that the super-expensive enterprise solution XYZ was the best thing that ever happened to your business, you drive a competitor in doing the same outrageously expensive mistake.
6. You will have the freedom to tune your forecasts
And the bullet that you will shoot in your own foot comes free of charge; and you should have a look at the incredible health care plan that you can buy from us too. Giving the opportunity to the client to have full control on its forecasts is not technically a lie, but it remains an extremely deceptive move when considering the consequences. It’s a case of moral hazard. In case of success, the technology of the vendor gets praised. In case of failure, the incompetent staff of the client gets blamed. Worse, if the first attempt fails, then there is no alternative but to purchase some extra training sessions from the vendor. Obviously, there is some serious money to be made in prolonging the problem.
7. We use state-of-art technologies
As far statistical learning is concerned state-of-the-art technologies are found in voice recognition, face recognition, spam filtering and all the other ultra-horizontal machine learning problems where companies like Google, Microsoft and Apple are investing hundreds of millions. It might be humbling to say, but demand forecasting is a somewhat niche business that does not justify hundreds of millions of R&D investments. As a result, even the most technologically aggressive companies like us, are about a decade behind what could be really considered as state-of-art in machine learning. However, the reality is that most forecasting vendors are half-a-century behind state of the art: ARIMA, Holt-Winters and Box-Jenkins models were already there back in 1970.
8. A forecasting benchmark cannot lie
Statistics are certainly the most refined way to lie. Lies can be made up from numbers in all sort of ways. At Lokad, we have faced several benchmarks where returning zeroes as forecasts for all products would have ensured an outstanding victory in the benchmarking process. Yet, setting all inventory levels at zero does not really strike as a wise business option. Whenever a vendor claims X percents of error, you should really ask yourself how many dollars of errors are you going to get. Minimizing percents of error is a very different thing than measuring the dollars of error. Believing that the first drives the latter is vastly mistaken – but vendors won’t hesitate to claim the exact opposite because percents of error are much safer grounds to decline any responsibility in the subsequent mess.
9. Our software is the best
The vendor website looks like it was designed in the late nineties; it’s not possible to try the software online; there are no screenshots, no public pricing, no public documentation and no API either, but, trust us, it’s really good software. As Riley said, When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck. Software is one of those businesses where there is absolutely no barrier to selling everything online. Unless your software is deeply dysfunctional, you have zero incentive in keeping it hidden. Decent vendors don’t argue they are the best, they just let clients try and see for themselves.
10. We manage your inventory AND your forecasts
Yes, sure. And some chess masters happen to be soccer champions too. Software is all about focus, and what it takes, as a company, to be good at designing, say, an ERP or a WMS, is about the exact opposite of what it takes to be good at designing a forecasting software. ERP companies make poor forecasting tools, simply because forecasting is just one add-on among dozens if not hundreds. Despite whatever vendors might say, forecasting cannot be the No1 priority of an ERP vendor; it’s a second-class citizen by design. Respectively, if you have a capable team of data scientists, you can’t keep them driven to design the hundreds of screens it takes to have a full-fledged ERP.
Reader Comments (1)
It’s refreshing to see such honesty in a vendor written article - thanks for posting it. Of course this sort of over-promising is not limited in any way to forecasting or even analytic tools. Buyers beware ! Andrew Gibson, Crabtree Analytcs 4 years ago | Andrew Gibson
