I like big data and I cannot lie,
You IT Pros can’t deny
When a vendor walks in with an itty-bitty cost
And Big Data for your boss
You get sprung! Get out the checkbook
‘Cause you know you’re on the hook
…
So your data’s a tall mountain, 1’s and 0’s in a big fountain
But big data’s not worth it, son, unless you got real information
Big data don’t want none unless you got integration, hon!
Okay, I’m no Sir Mix-A-Lot. And I won’t be quitting my day job as Adeptia’s CTO to pursue a rap career anytime soon.
But I did want to comment on all this hype surrounding big data. There’s no doubt that big data is powerful, especially if it can be connected and made into actionable information. You can use big data to enhance business practices and grow sales…but not without big data integration.
So before you whip out the checkbook for a big data project, I want you to stop and consider these three questions:
1. What Technology Do You Need?
It’s more than just Hadoop. Of course you need to be able to put all of that data somewhere and process it, but that’s not the whole picture.
What many people don’t think of is the integration side of the story. From what sources will the data be pulled? How do we aggregate the data? The connection of data sources allows businesses to pull insights, but big data can easily become unmanageable if there is no structured approach to analyzing it — it’s just flat and meaningless. A fountain of 1’s and 0’s.
There are services that assist with integrating disparate data sources, but they differ in overall approach. Some solutions offer tools that allow IT to make connections more efficient when integrating data sets. Alternatively, other offerings place integration capabilities in the hands of business users.
2. How Will You Integrate Big Data?
We know that big data comes from multiple sources and locations, but how do we make it meaningful? If you’re an online retailer looking to analyze user behavior in order to better drive sales, you’ll most likely pull data from multiple sources, such as the website log or the e-commerce application log.
Then you’ll need to aggregate the various sources to develop a pool of big data and use it to discern what your online visitors are interested in and adjust your sales and marketing accordingly.
Traditionally, creating these connections between multiple data sources required buying integration software platforms and finding the right individuals to use them. Boiling an ocean of integration software vendors can be time-consuming; it’s essentially swimming through an alphabet soup of acronyms (ETL, EDW, RDBMS, REST, API, and JSON)…and you still may end up hating the integration software you chose. My advice here is to choose a vendor with the easiest-to-use platform who can address the broadest spectrum of integration use cases.
3. Who Will Do The Integration?
Based on the highly technical skills (our Director of Solutions Engineering calls it, “godlike coding powers”) required to use most data integration platforms, the IT department was usually tasked with performing integration. But IT is busier than ever these days — and integration tasks always seem to fall to the bottom of their to-do list.
Let’s think for a second — does this model make sense? IT isn’t analyzing the data after all – they’re just establishing connections between data sources to efficiently aggregate data. Business users are the ones analyzing the data to develop insights, and this analysis can only be done when the data is pulled together. If business users have to wait around for IT to get through their daily to-do list, they could be waiting a long time.
There is a rise in the “self-service” trend, where users take matters into their own hands (just look at cloud adoption and “shadow IT”). Self-service integration is also going to take hold, and soon data scientists and analysts will be breaking the IT bottleneck to make their own connections when and where they need them. Smart CIOs would do well to get ahead of this trend and choose an integration software platform that enables business users to do integration, while allowing IT to still maintain control and governance.
Unlike “Baby Got Back,” big data isn’t a one-hit-wonder. But to make it a consistent top performer, you need to be able to efficiently connect the big data points. So let me leave you with the remixed words of Sir Mix-a-Lot to make my point: “Big Data Don’t Want None Unless You Got Integration, Hon.”