CDH includes a FUSE (Filesystem in Userspace) interface into HDFS. The hadoop-hdfs-fuse package enables you to use your HDFS cluster as if it were a traditional filesystem on Linux. Proceed as follows.

  Note: FUSE does not currently support file append operations.

Before you start: You must have a working HDFS cluster and know the hostname and port that your NameNode exposes. If you use parcels to install CDH, you do not need to install the FUSE packages.

To install hadoop-hdfs-fuses On Red Hat-compatible systems:

sudo yum install hadoop-hdfs-fuse

To install hadoop-hdfs-fuse on Ubuntu systems:

sudo apt-get install hadoop-hdfs-fuse

To install hadoop-hdfs-fuse on SLES systems:

sudo zypper install hadoop-hdfs-fuse

You now have everything you need to begin mounting HDFS on Linux.

To set up and test your mount point in a non-HA installation:

mkdir -p <mount_point>
hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>

where namenode_port is the NameNode's RPC port, dfs.namenode.servicerpc-address.

To set up and test your mount point in an HA installation:

mkdir -p <mount_point>
hadoop-fuse-dfs dfs://<nameservice_id> <mount_point>

where nameservice_id is the value of fs.defaultFS. In this case the port defined for dfs.namenode.rpc-address.[nameservice ID].[name node ID] is used automatically. See Enabling HDFS HA for more information about these properties.

You can now run operations as if they are on your mount point. Press Ctrl+C to end the fuse-dfs program, and umount the partition if it is still mounted.

  Note:

To find its configuration directory, hadoop-fuse-dfs uses the HADOOP_CONF_DIR configured at the time the mount command is invoked.

To clean up your test:

umount <mount_point>

You can now add a permanent HDFS mount which persists through reboots.

To add a system mount:

1. Open /etc/fstab and add lines to the bottom similar to these:

   hadoop-fuse-dfs#dfs://<name_node_hostname>:<namenode_port> <mount_point> fuse allow_other,usetrash,rw 2 0

   For example:

   hadoop-fuse-dfs#dfs://localhost:8020 /mnt/hdfs fuse allow_other,usetrash,rw 2 0

     Note:

   In an HA deployment, use the HDFS nameservice instead of the NameNode URI; that is, use the value of dfs.nameservices in hdfs-site.xml.

2. Test to make sure everything is working properly:

   mount <mount_point>

Your system is now configured to allow you to use the ls command and use that mount point as if it were a normal system disk.

For more information, see the help for hadoop-fuse-dfs:

hadoop-fuse-dfs --help