why BIG DATA is BIG?

The last few months have been an interesting expedition for me. Having been in the midst of a lot of intelligence is humbling…at the same time very enlightening. The fact that the company has been more about building ideas (west coast) than profiting from them (east coast) has been a calming balm for jaded nerves. The fact that I am part of that very ‘profiting’ juggernaut is a perspective that has proven to be invaluable.

There is a palpable excitement in the valley that is hard not to spot every-time each of my 5 trips has taken me there this year. And it all centers around this monster theme of ‘data’…or as it is unpretentiously referred to – BIG DATA (a term I think I can give credit to my friend Roger Magoulas at O’Reilly…and Roger should claim it as well).

Every BIG idea, BIG business model, BIG pitch, BIG money, BIG solution is driven by BIG DATA.

The fact that I believe that the DATA REVOLUTION is THE biggest economic engine since the INDUSTRIAL REVOLUTION might explain my not-so-visible enthusiasm for it.

And what exactly do I see in it:

1. Data in the world is exploding…has been exponentially increasing the last few years and is now mutating at an uncontrollable pace (see graphic - too much data)

2. It will continue to gather pace. As sensors that feel, monitor, track and record data continue to explode (GPS enabled cellphone, computer, email device, health monitors, energy meters, credit/debit cards, browser, social networks, security cameras, satellites, tracking chips in dogs…)…so will the data they generate.

3. Almost EVERY single socio-economic entity (companies, governments, individuals) has the same exploding phenomenon and doesn’t really know (yet) what to do with it all.

4. Technology is finally democratizing the access to and processing of this massive amount of data. Or as I like to call it, it is finally time to put the Information back in Information Technology. Simple, scalable and cheap hardware software platforms are being rapidly created, deployed and improved upon.

5. And is debunking the myth that not all data is useful (data trash is history…finally). Never again should an IT manager be allowed to ask the question “can I delete historical data’ much less “archive this data to tape”.

6. A new breed of talent is being bred into the workforce with an understanding of what can be done with this explosive combo of data and data technology, creating the sexiest new jobs on the planet. I refer to them as data quants (others call them data scientists) and they are destined to become the vikings of the data kingdom.

7. And finally all this is bringing together the potent mix of problems needing answers (and we have no dearth of problems), data that can lock onto the solutions, technologies that hold the keys to that data and talent that can bring it all to life. QED

And (I hope) you can see why I am jumping in my plane seat as I write this…(thank you gogo wireless…although you could have picked better than a pornstar alias as the brand-name for the in-air wireless service)