Cloudera Enterprise 6.3.x | Other versions

Managing YARN (MRv2) and MapReduce (MRv1)

  Note: This page contains references to CDH 5 components or features that have been removed from CDH 6. These references are only applicable if you are managing a CDH 5 cluster with Cloudera Manager 6. For more information, see Deprecated Items.

CDH supports two versions of the MapReduce computation framework: MRv1 and MRv2, which are implemented by the MapReduce (MRv1) and YARN (MRv2) services. YARN is backwards-compatible with MapReduce. (All jobs that run against MapReduce also run in a YARN cluster).

The MapReduce v2 (MRv2) or YARN architecture splits the two primary responsibilities of the JobTracker — resource management and job scheduling/monitoring — into separate daemons: a global ResourceManager and per-application ApplicationMasters. With YARN, the ResourceManager and per-host NodeManagers form the data-computation framework. The ResourceManager service effectively replaces the functions of the JobTracker, and NodeManagers run on worker hosts instead of TaskTracker daemons. The per-application ApplicationMaster is, in effect, a framework-specific library and negotiates resources from the ResourceManager and works with the NodeManagers to run and monitor the tasks. For details of this architecture, see Apache Hadoop NextGen MapReduce (YARN).

Defaults and Recommendations

  • In a Cloudera Manager deployment of a CDH cluster, the YARN service is the default MapReduce computation framework.In CDH 5, the MapReduce 1 service has been deprecated. However, the MapReduce service is fully supported for backward compatibility through the CDH 5 lifecycle.
  • For production uses, Cloudera recommends that only one MapReduce framework should be running at any given time. If development needs or other use case requires switching between MapReduce and YARN, both services can be configured at the same time, but only one should be running (to fully optimize the hardware resources available).

Migrating from MapReduce to YARN

Cloudera Manager provides a wizard described in Importing MapReduce Configurations to YARN to easily migrate MapReduce configurations to YARN. The wizard performs all the steps (Switching Between MapReduce and YARN Services, Updating Services Dependent on MapReduce, and Configuring Alternatives Priority for Services Dependent on MapReduce) on this page.

The Activity Monitor role collects information about activities run by the MapReduce service. If MapReduce is not being used and the reporting data is no longer required, then the Activity Monitor role and database can be removed:
  1. Do one of the following:
    • Select Clusters > Cloudera Management Service.
    • On the Home > Status tab, in Cloudera Management Service table, click the Cloudera Management Service link.
  2. Click the Instances tab.
  3. Select checkbox for Activity Monitor, select Actions for Selected > Stop, and click Stop to confirm.
  4. Select checkbox for Activity Monitor, select Actions for Selected > Delete, and click Delete to confirm.
  5. Manage the Activity Monitor database. The example below is for a MySQL backend database:
    1. Verify the Activity Monitor database: 
      mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| amon               |
+--------------------+
    2. Back up the database: 
      mysqldump -uroot -pcloudera amon > /safe_backup_directory/amon.sql
      Drop the database: 
      mysql> drop database amon;

Once you have migrated to YARN and deleted the MapReduce service, you can remove local data from each TaskTracker host. The mapred.local.dir parameter is a directory on the local filesystem of each TaskTracker that contains temporary data for MapReduce. Once the service is stopped, you can remove this directory to free disk space on each host.

For detailed information on migrating from MapReduce to YARN, see Migrating from MapReduce 1 (MRv1) to MapReduce 2 (MRv2).

Continue reading:

Switching Between MapReduce and YARN Services

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

MapReduce and YARN use separate sets of configuration files. No files are removed or altered when you change to a different framework. To change from YARN to MapReduce (or vice versa):
  1. (Optional) Configure the new MapReduce or YARN service.
  2. Update dependent services to use the chosen framework.
  3. Configure the alternatives priority.
  4. Redeploy the Oozie ShareLib.
  5. Redeploy the client configuration.
  6. Start the framework service to switch to.
  7. (Optional) Stop the unused framework service to free up the resources it uses.

Updating Services Dependent on MapReduce

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

When you change the MapReduce framework, the dependent services that must be updated to use the new framework are:
  • Hive
  • Sqoop 2
  • Oozie
To update a service:
  1. Go to the service.
  2. Click the Configuration tab.
  3. Select Scope > service name(Service Wide).
  4. Select Scope > All.
  5. Locate the MapReduce Service property and select the YARN or MapReduce service.
  6. Enter a Reason for change, and then click Save Changes to commit the changes.
  7. Select Actions > Restart.
The Hue service is automatically reconfigured to use the same framework as Oozie and Hive. This cannot be changed. To update the Hue service:
  1. Go to the Hue service.
  2. Select Actions > Restart.

Configuring Alternatives Priority for Services Dependent on MapReduce

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

The alternatives priority property determines which service—MapReduce or YARN—is used by clients to run MapReduce jobs. The service with a higher value of the property is used. The MapReduce service alternatives priority is set to 91 and the YARN service is set to 92.

To configure the alternatives priority:
  1. Go to the MapReduce or YARN service.
  2. Click the Configuration tab.
  3. Select Scope > Gateway Default Group.
  4. Select Category > All.
  5. Type Alternatives in Search box.
  6. In the Alternatives Priority property, set the priority value.
  7. Enter a Reason for change, and then click Save Changes to commit the changes.
  8. Redeploy the client configuration.

Configuring MapReduce To Read/Write With Amazon Web Services

These are the steps required to configure MapReduce to read and write with AWS.

  1. Save your AWS access key in a .jceks file in HDFS. 
    hadoop credential create fs.s3a.access.key -provider \
    jceks://hdfs/<hdfs directory>/<file name>.jceks -value <AWS access key id>
  2. Put the AWS secret in the same .jceks file created in previous step. 
    hadoop credential create fs.s3a.secret.key -provider \
    jceks://hdfs/<hdfs directory>/<file name>.jceks -value <AWS secret access key>
  3. Set your hadoop.security.credential.provider.path to the path of the .jceks file in the job configuration so that the MapReduce framework loads AWS credentials from the .jceks file in HDFS. The following example shows a Teragen MapReduce job that writes to an S3 bucket. 
    hadoop jar <path to the Hadoop MapReduce example jar file> teragen \
     -Dhadoop.security.credential.provider.path= \
     jceks://hdfs/<hdfs directory>/<file name>.jceks \
     100 s3a://<bucket name>/teragen1

You can specify the variables <hdfs directory>, <file name>, <AWS access key id>, and <AWS secret access key>. <hdfs directory> is the HDFS directory where you store the .jceks file. <file name> is the name of the .jceks file in HDFS.

To configure Oozie to submit S3 MapReduce jobs, see Configuring Oozie to Enable MapReduce Jobs To Read/Write from Amazon S3.

Page generated August 29, 2019.