The Pareto principle, also known as the 80-20 rule, asserts that 80% of consequences come from 20% of causes, rendering the remainder way less impactful.<\/p>\n
Those working with data may have heard a different rendition of the 80-20 rule: A data scientist spends 80% of their time at work cleaning up messy data as opposed to doing actual analysis or generating insights. Imagine a 30-minute drive expanded to two-and-a-half hours by traffic jams, and you\u2019ll get the picture.<\/p>\n<\/p>\n
While most data scientists spend more than 20% of their time at work on actual analysis, they still have to waste countless hours turning a trove of messy data into a tidy dataset ready for analysis. This process can include removing duplicate data, making sure all entries are formatted correctly and doing other preparatory work.<\/p>\n
On average, this workflow stage takes up about 45% of the total time, a recent Anaconda survey<\/a> found. An earlier poll<\/a> by CrowdFlower put the estimate at 60%, and many other surveys<\/a> cite figures in this range.<\/p>\n None of this is to say data preparation is not important. \u201cGarbage in, garbage out\u201d is a well-known rule in computer science circles, and it applies to data science, too. In the best-case scenario, the script will just return an error, warning that it cannot calculate the average spending per client, because the entry for customer #1527 is formatted as text, not as a numeral. In the worst case, the company will act on insights that have little to do with reality.<\/p>\n The real question to ask here is whether re-formatting the data for customer #1527 is really the best way to use the time of a well-paid expert. The average data scientist is paid between $95,000 and $120,000 per year, according to various estimates. Having the employee on such pay focus on mind-numbing, non-expert tasks is a waste both of their time and the company\u2019s money. Besides, real-world data has a lifespan, and if a dataset for a time-sensitive project takes too long to collect and process, it can be outdated before any analysis is done.<\/p>\n What\u2019s more, companies\u2019 quests for data often include wasting the time of non-data-focused personnel, with employees asked to help fetch or produce data instead of working on their regular responsibilities. More than half of the data being collected by companies is often not used at all<\/a>, suggesting that the time of everyone involved in the collection has been wasted to produce nothing but operational delay and the associated losses.<\/p>\n The data that has been collected, on the other hand, is often only used by a designated data science team that is too overworked to go through everything that is available.<\/p>\n The issues outlined here all play into the fact that save for the data pioneers like Google and Facebook, companies are still wrapping their heads<\/a> around how to re-imagine themselves for the data-driven era. Data is pulled into huge databases and data scientists are left with a lot of cleaning to do, while others, whose time was wasted on helping fetch the data, do not benefit from it too often.<\/p>\n The truth is, we are still early when it comes to data transformation. The success of tech giants that put data at the core of their business models set off a spark that is only starting to take off. And even though the results are mixed for now<\/a>, this is a sign that companies have yet to master thinking with data.<\/p>\nAll for data, and data for all<\/h2>\n