Get started for Cloudera Hadoop

Cloudera provides its distribution of hadoop, people are getting more attracted towards cloudera rather then original Apache hadoop. Cloudera provides almost all the sub packages which apache hadoop provides, Cloudera applys lots of patches to apache hadoop before providing it as its own distribution.

Cloudera also provides some extra subproject (which apache hadoop does not provides) like cloudera hue , Oozie, Sqoop, Flume etc

To install Cloudera:

Standalone mode: http://patodirahul.blogspot.com/2011/03/running-cloudera-in-standalone-mode.html

Pseudo distributed mode: http://patodirahul.blogspot.com/2011/03/running-cloudera-in-pseudo-distributed.html

Distributed mode: http://patodirahul.blogspot.com/2011/03/running-cloudera-in-distributed-mode.html

Start yourself with Apache Hadoop

What is Hadoop: Hadoop is a framework written in Java for running applications on large clusters of commodity hardware and incorporates features similar to those of the Google File System and of MapReduce.

Why Hadoop: MapReduce is Google's secret weapon: A way of breaking complicated problems apart, and spreading them across many computers. Hadoop is an open source implementation of MapReduce, and its own filesystem HDFS(Hadoop distributed file system). 

Hadoop has defeated Super Computer in tera sort: Hadoop clusters sorted 1 terabyte of data in 209 seconds, which beat the previous record of 297 seconds in the annual general purpose (daytona) terabyte sort benchmark. The sort benchmark, which was created in 1998 by Jim Gray, specifies the input data (10 billion 100 byte records), which must be completely sorted and written to disk. This is the first time that either a Java or an open source program has won.

To understand hadoop: http://patodirahul.blogspot.com/2011/03/understanding-what-is-hadoop.html

To install hadoop:

In standalone mode: http://patodirahul.blogspot.com/2011/03/hadoop-in-standalone-mode.html

In psedo distributed mode: http://patodirahul.blogspot.com/2011/03/hadoop-in-pseudo-distributed-mode.html

In distributed mode: http://patodirahul.blogspot.com/2011/03/hadoop-in-distributed-mode.html

HSearch

HSearch is an open source, NoSQL Search Engine. Use it when you need real-time search on your Big Data. This project's goal is to index over 100 billion records atop your commodity hardware cluster. HSearch is an open source, distributed, multi-format, structured and unstructured content search engine built on HBase platform. As the complete index is stored in HBase table, it inherits the HBase storable capabilities.

HSearch features include:

• Multi-XML formats
• Record and document level search access control
• Continuous index updation
• Parallel indexing using multi-machines
• Embeddable inside application
• A REST-ful Web service gateway that supports XML
• Auto sharding
• Auto replication

Java development 2.0: Big data analysis with Hadoop MapReduce

Pentaho Makes Hadoop Easy

More and more enterprises are turning to Hadoop to reduce costs and improve their ability to extract actionable business insight from the vast amount of data being collected throughout the enterprise. However, Hadoop presents challenges to users – steep technical learning curve, a lack of qualified technical staff and the lack of availability of development and deployment applications for performing data integration and business intelligence with Hadoop.

Pentaho breaks down these barriers and makes it easy to use Hadoop

• Pentaho lowers on-ramp for Hadoop users
  Lowers complexity and learning curve for Big Data analytics
• Pentaho offers full BI Suite
  Data to dashboards (ETL, OLAP, reporting, dashboards, mining) functionality with seamless Hadoop integration
• Rapidly integrate Hadoop into existing data architectures
  Easily integrates data between Hadoop and databases, data warehouses and other enterprise data stores
• Agile BI and modern platform, deployed on-premise or on-demand
  Pentaho solution drives rapid deployments and brings scalability via 100% Java platform
• Commitment to open source
  Frees up $$ for more servers, CPUs

The IBM Distribution of Apache Hadoop

http://www.alphaworks.ibm.com/tech/idah

Distributed data processing with Hadoop, Part 2: Going further

http://www.ibm.com/developerworks/linux/library/l-hadoop-2/index.html

Distributed data processing with Hadoop, Part 1: Getting started

http://www.ibm.com/developerworks/linux/library/l-hadoop-1/

Tools for Moving from an SQL Database to Hadoop

http://java.dzone.com/articles/tools-moving-sql-database

Sqoop

Sqoop is a tool designed to help users of large data import existing relational databases into their Hadoop clusters. Once present in HDFS, these datasets can be combined with other HDFS-resident data using MapReduce. Sqoop is based on JDBC, enabling it to talk to a wide variety of database platforms. Imports are performed in parallel using MapReduce.

Hive/HiveJDBCInterface

Contents

1. Integration with Pentaho
2. Integration with SQuirrel SQL Client

Starting to learn Hive

Hive

Hive is a data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and analysis of large datasets data stored in Hadoop files. It provides a mechanism to put structure on this data and it also provides a simple query language called Hive QL which is based on SQL and which enables users familiar with SQL to query this data. At the same time, this language also allows traditional map/reduce programmers to be able to plug in their custom mappers and reducers to do more sophisticated analysis which may not be supported by the built-in capabilities of the language.