When a purge job runs, any running Navigator tasks—extractions, policy application, or other background tasks—are stopped so that the purge can run immediately. When the purge task completes, the tasks that were stopped are restarted from the beginning. The interruption for the purge task may delay collecting new audits and metadata but does not affect what content is collected.

A purge task won't stop another purge task: if a purge task if already running when a new purge task is triggered, the new purge task will be ignored. For example, if the scheduled metadata purge is running, an administrator won't be able to kick off a purge task for deleting managed metadata properties; if a scheduled metadata purge is still running when the next scheduled purge starts, the new task is ignored.

Based on the purge behavior, consider the following recommendations to make sure that Navigator purge tasks don't conflict with extraction tasks:

• Purge Timing. Because purge tasks stop extraction and other Navigator jobs, it's important to schedule the purge task when it will cause the least disruption: users and processes will be unable to use Cloudera Navigator until the purge process completes.
• First Time Purge Runs. For new installations of Navigator or installations starting with a fresh storage directory, schedule purge jobs after you've seen the HDFS and Hive HMS extractions shift from bulk extraction to incremental extraction. The first extractions of metadata from HDFS run against every HDFS directory in the cluster and can run for many hours. Subsequent extractions run incrementally, only extracting metadata for new and changed entities. If the initial extractions are never allowed to complete, the shorter, faster incremental extractions don't begin. The same is true for HMS extractions.

  To avoid a purge task from stopping the initial bulk extraction, avoid scheduling purge tasks until after extraction has shifted to the incremental mode.

Use the Cloudera Navigator console to configure a schedule for a regular weekly purge of deleted and stale metadata from the Navigator Metadata Server and its associated database.

Required Role: Metadata Administrator (or Full Administrator)

Here is an example of a revised schedule:

  Note: If no schedules appear after you configure the purge process, your environment may not have automated purge enabled. To check if the automated purge is enabled, add /debug to the Navigator URL (http://fqdn-a.example.com:7187/debug), click Configuration, and check the value of nav.purge.enabled. If the property is set to false, you can enabled it in the debug page for this Navigator session; change the property in Cloudera Manager by adding nav.purge.enabled=true to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties.

Purge processes look for metadata that is associated with deleted files and tables and with operation executions that are older than the configured threshold date.

Hive Metadata

• Hive operations
  • That don't produce output
  • That have all operation executions that were executed earlier than the threshold date
• Hive operation executions
  • Associated with Hive operations that don't produce output
  • That were executed earlier than the threshold date
• Hive sub-operations
  • Associated with Hive operations that were purged
• All relations associated with the purged entities

Impala Metadata

• Impala operations
  • That don't produce output
  • That have all operation executions that were executed earlier than the threshold date
• Impala operation executions
  • Associated with Impala operations that don't produce output
  • That were executed earlier than the threshold date
• Impala sub-operations
  • Associated with Impala operations that were purged
• All relations associated with the purged entities

Sqoop Metadata

• Sqoop import and export operations
  • That have all operation executions that were executed earlier than the threshold date and no existing downstream entities
• Sqoop operation executions
  • that were executed earlier than the threshold date
• All relations associated with the purged entities

YARN Metadata

• YARN operations
  • That have all operation executions that were executed earlier than the threshold date
• YARN operation executions
  • That were executed earlier than the threshold date
• All relations associated with the purged entities

Pig Metadata

• Pig operations
  • That have all operation executions that were executed earlier than the threshold date and don't apply to tables connected to existing HDFS files
• Pig operation executions
  • That were executed earlier than the threshold date
• Pig tables
  • That were created by an operation execution executed earlier than the threshold date and also not connected to an existing HDFS file
• Pig fields
  • Fields in purged tables
• All relations associated with the purged entities

HDFS Metadata

• HDFS directories
  • That have been deleted longer than the configured threshold
  • AND don't have a logical-physical relation with another entity (such as a Hive table)
  • AND don't have children (sub directories or files) that aren't ready to be purged
  • AND aren't an endpoint in a data flow relation
• HDFS files
  • Deleted file metadata are purged only when the containing directory is purged
• All relations that have both endpoints associated with purged entities