Cloudera Enterprise 6.3.x | Other versions

Backing Up Cloudera Manager

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

The following steps create a complete backup of Cloudera Manager:

  1. Collect Information for Backing Up Cloudera Manager
  2. Back Up Cloudera Manager Agent
  3. Back Up the Cloudera Management Service
  4. Back Up Cloudera Navigator Data
  5. Stop Cloudera Manager Server & Cloudera Management Service
  6. Back Up the Cloudera Manager Databases
  7. Back Up Cloudera Manager Server
  8. Start Cloudera Manager Server & Cloudera Management Service

Collect Information for Backing Up Cloudera Manager

  2. Collect database information by running the following command: 
    cat /etc/cloudera-scm-server/db.properties
    For example: 
    ...
com.cloudera.cmf.db.type=...
com.cloudera.cmf.db.host=database_hostname:database_port
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=SOME_PASSWORD
  3. Collect information (host name, port number, database name, user name and password) for the following databases.
    • Reports Manager
    • Navigator Audit Server
    • Navigator Metadata Server
    • Activity Monitor

    You can find the database information by using the Cloudera Manager Admin Console. Go to Clusters > Cloudera Management Service > Configuration and select the Database category. You may need to contact your database administrator to obtain the passwords.

  4. Find the host where the Service Monitor, Host Monitor and Event Server roles are running. Go to Clusters > Cloudera Manager Management Service > Instances and note which hosts are running these roles.
  5. Identify the location of the Cloudera Navigator Metadata Server storage directory:
    1. Go to Clusters > Cloudera Management Service > Instances.
    2. Click the Configuration tab.
    3. Select Scope > Navigator Metadata Server.
    4. The Navigator Metadata Server Storage Dir property stores the location of the directory.
  6. Ensure that Navigator Metadata Server Java heap is large enough to complete the upgrade. You can estimate the amount of heap needed from the number of elements and relations stored in the Solr storage directory.
    1. Go to Clusters > Cloudera Management Service > Instances.
    2. In the list of instances, click Navigator Metadata Server.
    3. Select Log Files > Role Log File.
    4. Search the log file for solr core nav_elements and note the number of element documents.
    5. Search the log file for solr core nav_relations and note the number of relation documents.
    6. Multiply the total number of documents by 200 bytes per document and add to it a baseline of 2 GB: 
      ((num_nav_elements + num_nav_relations) * 200 bytes) + 2 GB
      For example, if you had 68813088 elements and 78813930 relations, the recommended Java heap size is ~30 GB: 
      ((68813088 + 78813930) * 200) + 2 GB = 29525403600 bytes = ~29.5 GB + 2 GB = ~ 31.5 GB
    7. Set the heap value in the Java Heap Size of Navigator Metadata Server in Bytes property in Clusters > Cloudera Management Service > Configuration.

Back Up Cloudera Manager Agent

  Note: Commands are provided below to backup various files and directories used by Cloudera Manager Agents. If you have configured custom paths for any of these, substitute those paths in the commands. The commands also provide destination paths to store the backups, defined by the environment variable CM_BACKUP_DIR, which is used in all the backup commands. You may change these destination paths in the command as needed for your deployment.
The tar commands in the steps below may return the following message. It is safe to ignore this message: 
tar: Removing leading `/' from member names

Backup up the following Cloudera Manager agent files on all hosts:

  • Create a top level backup directory. 
    export CM_BACKUP_DIR="`date +%F`-CM"
echo $CM_BACKUP_DIR
mkdir -p $CM_BACKUP_DIR
  • Back up the Agent directory and the runtime state. 
    sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-agent.tar --exclude=*.sock /etc/cloudera-scm-agent /etc/default/cloudera-scm-agent /var/run/cloudera-scm-agent /var/lib/cloudera-scm-agent
  • Back up the existing repository directory.
    RHEL / CentOS
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.d
    SLES
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/zypp/repos.d
    Debian / Ubuntu
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/apt/sources.list.d

Back Up the Cloudera Management Service

  Note: Commands are provided below to backup various files and directories used by Cloudera Manager Agents. If you have configured custom paths for any of these, substitute those paths in the commands. The commands also provide destination paths to store the backups. You may change these destination paths in the command as needed for your deployment.
  1. On the host where the Service Monitor role is configured to run, backup the following directory: 
    sudo cp -rp /var/lib/cloudera-service-monitor /var/lib/cloudera-service-monitor-`date +%F`-CM
  2. On the host where the Host Monitor role is configured to run, backup the following directory: 
    sudo cp -rp /var/lib/cloudera-host-monitor /var/lib/cloudera-host-monitor-`date +%F`-CM
  3. On the host where the Event Server role is configured to run, back up the following directory: 
    sudo cp -rp /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-eventserver-`date +%F`-CM

Back Up Cloudera Navigator Data

  1.   Important: Upgrading from Cloudera Manager 5.9 (Navigator 2.8) and earlier can take a significant amount of time, depending on the size of the Navigator Metadata storage directory. When the Cloudera Manager upgrade process completes and Cloudera Navigator services restart, the Solr indexing upgrade automatically begins. No other actions can be performed until Solr indexing completes (a progress message displays during this process). It can take as long as two days to upgrade a storage directory with 60 GB. To help mitigate this extended upgrade step, make sure to clear out all unnecessary metadata using purge, check the size of the storage directory, and consider rerunning purge with tighter conditions to further reduce the size of the storage directory.
  2. Make sure a purge task has run recently to clear stale and deleted entities.
    • You can see when the last purge tasks were run in the Cloudera Navigator console (From the Cloudera Manager Admin console, go to Clusters > Cloudera Navigator. Select Administration > Purge Settings.)
    • If a purge hasn't run recently, run it by editing the Purge schedule on the same page.
    • Set the purge process options to clear out as much of the backlog of data as you can tolerate for your upgraded system. See Managing Metadata Storage with Purge.
  3. Stop the Navigator Metadata Server.
    1. Go to Clusters > Cloudera Management Service > Instances.
    2. Select Navigator Metadata Server.
    3. Click Actions for Selected > Stop.
  4. Back up the Cloudera Navigator Solr storage directory. 
    sudo cp -rp /var/lib/cloudera-scm-navigator /var/lib/cloudera-scm-navigator-`date +%F`-CM
  5. If you are using an Oracle database for audit, in SQL*Plus, ensure that the following additional privileges are set: 
      GRANT EXECUTE ON sys.dbms_crypto TO nav;
  GRANT CREATE VIEW TO nav;
    where nav is the user of the Navigator Audit Server database.

Stop Cloudera Manager Server & Cloudera Management Service

  1. Stop the Cloudera Management Service.
    1. Log in to the Cloudera Manager Admin Console.
    2. Select Clusters > Cloudera Management Service.
    3. Select Actions > Stop.
  3. Stop the Cloudera Manager Server.
    RHEL 7, SLES 12, Debian 8, Ubuntu 16.04 and higher
    sudo systemctl stop cloudera-scm-server
    RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04 or 14.04
    sudo service cloudera-scm-server stop

Back Up the Cloudera Manager Databases

  1. Back up the Cloudera Manager server database – Run the following command. (The command displayed below depends on the database you selected in the form at the top of this page. Replace placeholders with the actual values returned from the db.properties file):
    MySQL
    mysqldump --databases database_name --host=database_hostname --port=database_port -u user_name -p > $HOME/database_name-backup-`date +%F`-CM.sql
      Note: If the db.properties file does not contain a port number, omit the port number parameter from the above command.
    PostgreSQL/Embedded
    pg_dump -h database_hostname -U user_name -W -p database_port database_name > $HOME/database_name-backup-`date +%F`-CM.sql
    Oracle
    Work with your database administrator to ensure databases are properly backed up.

    For more information about backing up databases, see Backing Up Databases.

  2. Back up All other Cloudera Manager databases - Use the database information that you collected in a previous step. You may need to contact your database administrator to obtain the passwords.
    These databases can include the following:
    • Reports Manager
    • Navigator Audit Server
    • Navigator Metadata Server
    • Activity Monitor (Only used for MapReduce 1 monitoring).

    Run the following commands to back up the databases. (The command displayed below depends on the database you selected in the form at the top of this page. Replace placeholders with the actual values.):

    MySQL
    mysqldump --databases database_name --host=database_hostname --port=database_port -u database_username -p > $HOME/database_name-backup-`date +%F`-CM.sql
    PostgreSQL/Embedded
    pg_dump -h database_hostname -U database_username -W -p database_port database_name > $HOME/database_name-backup-`date +%F`-CM.sql
    Oracle
    Work with your database administrator to ensure databases are properly backed up.

Back Up Cloudera Manager Server

  Note: Commands are provided below to backup various files and directories used by Cloudera Manager Agents. If you have configured custom paths for any of these, substitute those paths in the commands. The commands also provide destination paths to store the backups, defined by the environment variable CM_BACKUP_DIR, which is used in all the backup commands. You may change these destination paths in the command as needed for your deployment.
The tar commands in the steps below may return the following message. It is safe to ignore this message: 
tar: Removing leading `/' from member names
  1. Log in to the Cloudera Manager Server host. 
    ssh my_cloudera_manager_server_host
  2. Create a top-level backup directory. 
    export CM_BACKUP_DIR="`date +%F`-CM"
echo $CM_BACKUP_DIR
mkdir -p $CM_BACKUP_DIR
  3. Back up the Cloudera Manager Server directories: 
    sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-server.tar /etc/cloudera-scm-server /etc/default/cloudera-scm-server
  4. Back up the existing repository directory.
    RHEL / CentOS
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.d
    SLES
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/zypp/repos.d
    Debian / Ubuntu
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/apt/sources.list.d

Start Cloudera Manager Server & Cloudera Management Service

  2. Start the Cloudera Manager Server.
    RHEL 7, SLES 12, Debian 8, Ubuntu 16.04 and higher
    sudo systemctl start cloudera-scm-server
    If the Cloudera Manager Server starts without errors, no response displays.
    RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04 or 14.04
    sudo service cloudera-scm-server start
    You should see the following: 
    Starting cloudera-scm-server: [ OK ]
  3. Start the Cloudera Management Service.
    1. Log in to the Cloudera Manager Admin Console.
    2. Select Clusters > Cloudera Management Service.
    3. Select Actions > Start.
Page generated August 29, 2019.