Introduction: False promises?
as a consultant and manager in the data sphere, I’ve sat through my fair share of slide deck presentation. On both sides. And any slide deck worth its salt promises something, often about efficiency or productivity. You have probably heard something like this:
- This tool with make your data scientists 40% more productive!
- You will spend 30% less time fixing bugs by doing this. You could basically implement a 6-hour work day and still come out on top!
- With our solution, you could code up two projects in the time it took you previously to do only one. This halves the amount of time to production!
Sometimes the promises do not work simply because the proposed product is bad. But why does this never seem to actually work even with good products? You could end up switching to a product that you genuinely love, but still not really see the promised improvement. Why? Are the numbers you got presented lies?
My background as a PhD in mathematics has probably scarred me for life in more ways than one. One of the deepest scars is my need to understand precisely what numbers represent. And the numbers you find in the statements above are all indicating one thing, while telling a completely different story if you stop and think.
While lying certainly happens, what is a much more common practice is being misleading. This sort of marketing assumes that you don’t think critically when presented with numbers. Let’s think critically together and see what we come up with.
Lies, lies, and marketing
So what is the issue with the productivity statements?
The main issue is that they claim to optimize a certain aspect of the process, while (indirectly) promising global productivity gains.
Let’s go through a simple example to understand what this means.
Say you’re a big player in AI and have recently launched a product that is great at helping data scientists with model parameter selection. Cool! Initial surveys show that it has given data scientists a 20% increase in productivity for model parameter selection. You initially present this as:
Our tool has improved the productivity of model parameter selection for data scientists by 20%.
Happy with this impressive result, you send your statement of to marketing and they come back only with minor adjustments:
Our tool has improved model parameter selection, making data scientists 20% more productive.
You shrug and wonder for a second what these people in marketing are really paid to do when they only shuffled around a few words. In reality, they have now shifted your statement from something that is moderately impressive to something that is insanely impressive.
Why? The adjustment from marketing makes it seem like the product makes data scientists 20% more productive in general. But your survey only really talked about productivity during the time the data scientists are selecting model parameters. What’s the real difference?
A data scientist does loads of things, including prototyping, stakeholder management, coordination meetings, etc. While machine learning is often at the front and center of how they would describe themselves, many data scientists only spend around 40% of their time on typical data science tasks. A big chunk of these 40% is debugging data quality issues, pipeline management, and data validation. Model parameter selection may only take up 10% of their 40% time doing data science tasks. Multiplying shows us that this is only 4% of their total time.
If the data scientist adopted a tool to make the model parameter selection 20% more productive, that would only make a difference of barely 1% of their total time. You would not notice this during a work week. In fact, with the added complexity of learning a new tool in the beginning, you might even see a decrease in productivity in the beginning.
The best part? Look at the statement carefully:
Our tool has improved model parameter selection, making data scientists 20% more productive.
It certainly seems to argue that data scientists would be 20% more productive overall, but this is just one interpretation. If pressed, marketing would make the connection between the start and end of the sentence, and say that it is implied that the productivity increase is only for model parameter selection.
So you effectively get to say one thing, while falling back on another if the misleading statement is discovered. The pay for marketing comes from shuffling around the right words!
A better way? Focus on cognitive load rather than productivity
What does the story I just told really tell you? If you have many different tasks that are complex (like a data scientist does), then aiming for productivity gains is really not pushing the needle that much.
Don’t get me wrong. If you have an easy opportunity to become 20% more productive with one of your tasks, go for it! But don’t expect it to result in more than a percent or two difference in total productivity.
What can we do instead when we have many different tasks that are complex? We can use cognitive load as a metric, and try to reduce that instead.
Say that a competing company developed their own tool for model parameter selection. Instead of trying to speed up the process, their tool had the sole purpose of reducing the cognitive load of the data scientist. So the process of model selection would take the same amount of time, but the data scientist would feel energized and ready for another challenge after selecting model parameters.
Most people, myself included, cannot work 8 hours a day and be at the top of our game at all times. Some days I feel like I have 6 effective hours in me. Other days it is more like 2 effective hours. If one process does not require that much cognitive load, then I can work longer effectively. This often results in the same total productivity of a few percent, but with the added benefit of improved morale.
So next time someone presents a “40% increase in productivity“, ask them the following:
- How much of the total work time does this productivity increase affect?
- How much cognitive load does this take away or introduce?