Everyone talks about how great Big Data is, and how it will revolutionize our world. They also talk about Big Data Risks in terms of security and privacy. I know I often discuss them in my tutorials and talks (most recently at Better Software East on Privacy, Security, and Trust in the Mobile Age), but that may be the least of our worries when it comes to Big Data Risks. The real danger of Big Data that I was able to glean from a recent keynote by Malcolm Gladwell at the BlackLine User Conference #InTheBlack15 was that acquiring masses of data does not really help us to make better decisions.


Big Data Risks - Confidence is Dangerous

Malcolm Gladwell on Big Data Risks — Confidence is Dangerous

Overinflated Confidence Due to Big Data Risks, Can Lead to Big Mistakes

Malcolm Gladwell on Big Data Risks — Overinflated Confidence Due to Information, Can Lead to Big Mistakes


After a certain point, just like everything else, there is the law of diminishing returns. In other words, if we have perfect information or let’s say, all the relevant information we can get our hands on, the decisions that we make are marginally better than the decisions we’d make if we had 1/3 of the information. As Malcolm Gladwell said, we gain all this information and what happens? We become more confident in the decisions that we make, but in fact, the decisions are not any better, or marginally better. He went on to cite many examples, from Civil War history as well as recent history, where the decision makers had been very confident in their assessment of the situation and their decision, but that perhaps so much XBOSoft Knowledge Center - Blogintelligence actually led to the wrong decision due to overconfidence. He also noted that those people that make it to high levels in government and industry, usually exude high levels of confidence, and are thereby able to lead their followers and constituents to believe in them. Yet it is those people in high ranking positions that make the biggest mistakes. For instance, you wouldn’t expect a custodian at McDonald’s (this was one of my first jobs:))  to make a big, far-reaching mistake. On the other hand, Alan Greenspan could make huge mistakes based on his position. I guess you had to be there to get the impact and feel the moment when he said: “What we need is less confidence in our leaders and more humility.”

Thus comes the worry about Big Data. We think that all of this information is going to be analyzed and ‘ban-analyzed’ (my own invented term) to help us make better decisions and glean insights. However, m
y position is that as we gain more intelligence through ‘big data’, we should not become overconfident. Mistakes based on overconfidence can be big ones, 1929, 2007, etc. Even if you think you have the answer based on all the information that you have so diligently collected, be open to other possibilities.

As software engineers, what decisions are we making that could be gravely wrong?