Commencement of Data Processing in MapReduce Architecture
In the world of big data processing, Hadoop's MapReduce framework plays a significant role. This article will delve into the steps involved in the initialization of a MapReduce job in Hadoop.
The process begins with a client submitting a job, which includes the necessary executable code (JAR with Mapper and Reducer logic), input data location, and the output path. The Job Tracker, upon accepting the job, splits it into multiple tasks corresponding to input splits and assigns these tasks to Task Trackers or nodes based primarily on data locality to minimize network transfer.
Key steps in this initialization phase include job submission, input data splitting, strategic task assignment with data locality in mind, task container launching, and intermediate and final output setup on HDFS.
Before tasks begin execution, the Application Master sets up the output path by calling on the OutputCommitter. For small jobs, tasks can run in the same JVM as the Application Master, a concept known as an Uber Task. An Uberised job is one where all tasks run in the same JVM as the Application Master, saving overhead for such jobs.
The number of Reducer tasks can be set using the property and can be configured in code with . Each reducer gets its own task object.
Conditions for Uberization include having fewer than 10 mappers, only 1 reducer, and an input size smaller than one HDFS block. If these conditions are met, the MapReduce job employs the Uber task behavior, which can be controlled using configuration properties like , , , and .
The Application Master (AM) used in MapReduce is called MRAppMaster and controls the entire job, keeping track of progress and assigning tasks to different nodes. The AM also sets up temporary directories for each task to write its intermediate output, ensuring that partial or corrupted output is prevented in case of task failure. Once a task finishes successfully, temp output is safely committed to the final directory.
In summary, the MapReduce job initialization in Hadoop follows a pipeline starting from job submission, input data splitting, strategic task assignment with data locality in mind, task container launching, and ends with intermediate and final output setup on HDFS. This is optimized for distributed execution efficiency and fault tolerance.
In the process of MapReduce job initialization, the client submits a job that includes the necessary technology like the JAR file containing the Mapper and Reducer logic, data location, and output path. This job, upon acceptance by the Job Tracker, is combined with data-and-cloud-computing technology like HDFS for efficient storage and processing, and a trie data structure may be utilized for its scalable search and insert operations.
