What is BigData? It is the question that will arise in the minds when someone wants to learn Hadoop and other distributed processing tools. <\/p>\n\n\n\n
There is no end to learn about any technology as technology is growing along with you each day.<\/p>\n\n\n\n
Over the last decade, there has been an exponential increase in data in every sector. Estimation says it is 2.5 exabytes\/day <\/p><\/div><\/div>\n\n\n\n
Companies like Facebook, Twitter, Google are already generating Petabytes of data every day.<\/p>\n\n\n\n
“Big data are nothing but a collection of large and complex data sets that are difficult to process using traditional database management tools or traditional data processing applications”<\/p><\/blockquote>\n\n\n\n
When the complexity increases, the efficiency of the traditional tool will decrease. <\/p>\n\n\n\n
Various sources for data(BigData)<\/h2>\n\n\n\n
Stock exchange 1 TB, Smartphones, Youtube upload contributes 48 hrs of data\/min, social networks like Twitter, FB(more than 10 TB data daily), 30 million network sensors across the globe generating data throughout the day, Instagram, etc<\/p>\n\n\n\n
There are a few types in which we can categorize these rapidly growing data.<\/p>\n\n\n\n
Types of Data<\/strong><\/h2>\n\n\n\n
- Structured<\/strong> \u2013 The structured data are data that have a proper schema like RDBMS(with structures).<\/li>
- Semistructured<\/strong> \u2013 Semi-structured data has its own structure but it doesn\u2019t have the detailed schematic way of storing data. This includes XML, JSON.<\/li>
- Unstructured<\/strong> \u2013 These are data that don\u2019t have a structure but only data in it. It includes weblogs, anti-virus logs, etc.<\/li><\/ul>\n\n\n\n
Big data is a broad term for large volume or complex data sets that are difficult to process using simple data-management tools and applications.<\/p>\n\n\n\n
Attributes that describe Bigdata<\/strong><\/h2>\n\n\n\n
- Variety <\/strong>\u2013 This is nothing but the different types of complex data that is getting generated like sensors, stock exchange updates, social network data, etc.<\/li>
- Velocity <\/strong>\u2013 This is the pace in which the data are getting dumped and retrieved for processing.<\/li>
- Volume <\/strong>\u2013 Volume represents the amount of data that is stored over a period of time.<\/li><\/ul>\n\n\n\n
So, handling this much data is a definite problem. Isn\u2019t it? \u201cBig data\u201d is a problem statement and one of the solutions for handling this is with the help of a framework called \u201cHadoop\u201d.<\/p>\n\n\n\n
Hadoop <\/strong>is a framework that helps to solve this problem statement with its framework components. When you think about handling BigData there are two main things that will come in the mind.<\/p>\n\n\n\n
- Storage<\/li>
- Processing<\/li><\/ol>\n\n\n\n
- HDFS(Hadoop distributed file system) takes care of storage.<\/li>
- MapReduce(MR) takes care of processing.<\/li><\/ul>\n\n\n\n
<\/p>\n\n\n\n