Big Data #6: Before Giving Your Big Data Project to a Contractor …
Big Data people are the rock stars of the corporate IT. Their salaries and benefits are high in the sky and still climbing, the supply still not able to satisfy the demand. The technology is complex, the infrastructure requirements are high and the tools for the trade change almost everyweek by undergoing frequent updates.
Even if you bring the people and the tools, the data is not ready for complex analytics. In the best case scenario, the data is kept in well-maintained databases in well-maintained corporate datacenters. In the normal case scenario (covering more than 95% of the companies), the data is scattered in a lot of databases, running on different systems with unknown relationships, created by unknown people aeons ago. I leave the worst case scenario to your imagination.
While this is the case today, I cannot say that the situation is not workable and all the databases and the infrastructure should be put in shape right away. If the data is already there, working no-matter-what and if there is the cross-platform language SQL, an IT Professional can get his hands dirty and play with the already available data. Before preparing a request for proposal and choosing a contractor, why wouldn’t an IT department see what it can do with the data?
RELATED: Big Data #7: Preparing for the Big Data Workflows
Before lining up the contractors for their proposals, bids, project plans, I suggest the company to analyze what it can do with the data. Big Data is about seeking answers to the questions that weren’t asked before. The answers are about, but not limited to, understanding your customer behaviors, changes in behavior and the impacts of emerged technologies. For example, if you are working for Moleskine, you would be asking questions about how people use notebooks, the impact of Palm and the impact of the Galaxy Note series on the consumer behavior and the market dynamics. Put this scenario to your company: ask a simple question.
RELATED: Geek Ethicist: Skype Therapy
After you ask the question, examine what you have at hand to answer the question. How many years of data do you have? What are your sales trends? What is the situation of your data in the databases? Do you have tools that more or less answer the same questions? You may have 10 years of data, scattered in – say – Microsoft SQL and Sybase databases and a web application that analyzes visitor/member data to make recommendations. In that case, you already have an infrastructure that is working. Before handing your data to a contractor, play with this data. It may show you that it is highly possible that you don’t need a Big Data contractor: maybe just a database update/upgrade/consolidation is what you need. Then, perhaps your Big Data project will reshape to forming in-house Big Data team. A couple of days’ play with your data may completely change your view on the project.
At this point, you already have a good, working knowledge of what you have at hand, what your requirements are and what you are expecting from a contractor. You can now engage the business-side (non-IT) people in your organization and define expectations in more detail. Once the ideas are ripe, you can work with the Big Data contractor in several ways. Just to count a few:
- Upgrade/update/consolidate databases and only deploy Big Data tools,
- Keep existing infrastructure and have the contractor only develop models,
- Upgrade/update/consolidate databases with IT staff and have the contractor only deliver training,
- Build in-house Big Data applications, outsource models.
RELATED: What Will Be the Biggest Hosting Trends for 2015?
As your company walks the Big Data way, your data assets will reshape: existing data sets will be cleaner, new data sets will be introduced and external data sets may be incorporated. With these advancements and the experience your company gains down the road will make you find and deploy the right Big Data solution for your company.
References:
- Featured Image: venturebeat.com
