Cloudera Enterprise 6.3.x | Other versions

Navigator Metadata Server Tuning

This page can help you tune your Navigator Metadata Server instance for peak performance. See also Setting up Navigator Role Instances on Different Hosts.

Continue reading:

Memory Sizing Considerations for Navigator Metadata Server

Unlike Navigator Audit Server for which configured Java heap size is rarely an issue, Navigator Metadata Server encompasses two processes that have a direct impact on memory, specifically:
  • Extracting metadata from the cluster and creating relationships among the metadata entities (facilitating lineage)
  • Querying to find entities

Navigator Metadata Server uses Solr to index, store, and query metadata. Indexing occurs during the extraction process, with the resulting Solr documents—data structure used by Solr for index and search—stored in the specified Navigator Metadata Server Storage Dir. Because the metadata is indexed, querying is fast and efficient. However, Solr indexing runs in-process with Navigator Metadata Server, so the amount of memory configured for the Java heap is critical.

There is a direct correlation between the number of Solr documents contained in the index and the size of the Java heap required by Navigator Metadata Server, so this setting may need to be changed over time as the number of Solr documents making up the index increases. To calculate optimal Java heap setting for your system, see Estimating Optimal Java Heap Size Using Solr Document Counts.

Estimating Optimal Java Heap Size Using Solr Document Counts

Each time it starts up, Navigator Metadata Server counts and logs the number of Solr documents in the datadir (the Navigator Metadata Server storage directory), as shown below: 
2016-11-11 09:24:58,013 INFO com.cloudera.nav.server.NavServerUtil:
  Found 68813088 documents in solr core nav_elements
2016-11-11 09:24:58,705 INFO com.cloudera.nav.server.NavServerUtil:
  Found 78813930 documents in solr core nav_relations

These counts can be used to estimate optimal Java heap size for the server, as detailed below. If your normal setup provides less than 8 GB for the Navigator Metadata Server heap, consider increasing the heap before performing an upgrade. See Setting the Navigator Metadata Server Java Heap Size for details about using Cloudera Manager Admin Console to modify this setting when needed.

  Note: If your system has been running without interruption for awhile, the data in the log may be stale. Restart the Navigator Metadata Server when it is convenient and then look at the log to find the current number of documents to use in the calculation.
To estimate the Java heap size:
  1. Open the log file for Navigator Metadata Server. By default, logs are located in /var/log/cloudera-scm-navigator.
  2. Find the number of documents in solr core nav_elements line in the log.
  3. Find the number of documents in solr core nav_relations line in the log.
      Warning: If the number of relations is more than 2 times the number of elements, you may have encountered the problem described by TSB-259 Upgrade needed for Cloudera Navigator due to high risk of failure through exhausting data directory resources. Follow the instructions for mitigating the problem rather than simply changing your Java heap.
  4. Multiply the total number of documents by 200 bytes per document and add to a baseline of 2 GB: 
    (num_nav_elements * 200 bytes) + 2 GB
    For example, using the log shown above, the recommended Java heap size is ~14 GiB: 
    (68813088 * 200) + 2 GB
13762617600 bytes = ~12.8 GiB + 2 GB (~1.8 GiB) = ~ 14–15 GiB
  Important: If you are upgrading to a Cloudera Manager version 5.10.0 or later, you'll need considerably more Java heap to perform the upgrade. Upgrading to the 5.10 release includes the unusual performance of a schema change in Solr. This operation requires additional Java heap for the duration of the upgrade process. To calculate the recommended Java heap for the upgrade, multiply the total number of documents by 200 bytes per document and add to a baseline of 2 GB: 
((num_nav_elements + num_nav_relations) * 200 bytes) + 2 GB
For example, using the log shown above, the recommended Java heap size is ~31 GiB: 
((68813088 + 78813930) * 200) + 2 GB
29525403600 bytes = ~29.5 GiB + 2 GB (~1.8 GiB) = ~ 31.3 GiB

Purging the Navigator Metadata Server of Deleted and Stale Metadata

In addition to adjusting the Java heap size, administrators can also clear the Navigator Metadata Server of stale and deleted metadata prior to an upgrade or whenever system performance seems slow. Purging stale and deleted metadata also helps speed up display of lineage diagrams. Purge fully removes metadata that has been deleted.

Cloudera Navigator console (Administration tab) provides a fully configurable Purge Settings page. See Managing Metadata Storage with Purge for details. You can also invoke purge capability using the Cloudera Navigator APIs. See Using the Purge APIs for Metadata Maintenance Tasks for details.

Page generated August 29, 2019.