Hadoop Administration : Part 2 Volumes

Volumes:

MapR provides volumes as a way to organize data and manage cluster performance. A volume is a logical unit that allows you to apply policies to a set of files, directories, and sub-volumes. A well-structured volume hierarchy is an essential aspect of your cluster's performance. As your cluster grows, keeping your volume hierarchy efficient maximizes your data's availability. Without a volume structure in place, your cluster's performance will be negatively affected.

Creating a Volume

When creating a volume, the only required parameters are the volume type (Standard,Local or Remote) and the volume name. You can set the ownership, permissions, quotas, and other parameters at the time of volume creation, or use the Volume Properties dialog to set them later.By default, the root user and the volume creator have full control permissions on the volume

Creating New Volume:

Ex: New Volume for user divakar(dannap001c)

If you see any permission issue, create Volume from datanode cli user as mapr and change the owner and permissions accordingly

maprcli volume create -name user.dannap001c -path /user/dannap001c -topology /data/user -quota 50G

hadoop fs -chown dannap001c /user/dannap001c

hadoop fs -chmod 750 /user/dannap001c

Volumes In MapR MCS:

.

Volume in HDFS:
hadoop fs -ls /user/dannap001c
drwx------   - dannap001c  CHQCDC_Users                    10 2014-08-28 15:41 /user/dannap001c

To mount/Unmount a volume using the MapR Control System:
1) In the Navigation pane, expand the MapR-FS group and click the Volumes view.
2) Select the checkbox beside the name of  each volume you wish to mount.
    Click the Mount/Unmount button

When Should I Use Volumes?

You can use volumes to enforce disk usage limits, set replication levels, establish ownership and accountability, and measure the cost generated by different projects or departments.Create a volume for each user, department, or project.You can mount volumes under other volumes to build a structure that reflects the needs of your organization.The volume structure defines how data is distributed across the nodes in your cluster.Create multiple small volumes with shallow paths at the top of your cluster's volume hierarchy to spread the load of access requests across the nodes.

Volume Mount:

Mounts one or more specified volumes. Permissions required: mnt, fc, or a

Ex: Mount the volume "test-volume" at the path "/test":

maprcli volume mount -name test-volume -path /test

Some Question & Answers:

1) I'm trying to understand how I can figure out how "fill" a volume is in cases where no quota has been defined for the volume; that is, the -quota and -advisoryquota parameters are set to 0.In such cases, what drives the volume running out of space?

There are probably three answers to this depending on what angle you take to this question.

1) You can see [1] how much space each volume uses on the MapR Control System (MCS). If you like, you can also accumulate the space for multiple volumes by defining an accounting entity for all of the volumes of interest.

2) A volume can run out of space when the topology it lives in becomes full or when the cluster itself fills up. Filling up the topology will cause an alarm and the volume will live partially outside specified topology. Filling up the cluster is more serious and will prevent writes. You can fill up a cluster the obvious way, but it can (does!) also happen when you have a cluster that is 80% full and you have a rack power failure that takes out >20% of your available space. If you don't fix that within the default maintenance window your cluster will naturally fill up with replicas.

3) A volume can also fill up its available space for meta-data. There is no hard and fast limit for meta-data as part of a volume, but since the first 64K of each file is kept in-line inside directories, you may be consuming more of this than you might expect.

2) I want to create a volume(MyVolume) but it can not mount the volume. it says this error: Failed to mount volume MyVolume, Permission denied its permission is root and admin.how can i create volume that comes to folder my cluster?

You need to be root to mount the volume. Or please check in permissions whether the logged in user has mount permissions or not. Grant mount permissions and you could mount it.

You should not provide the mount location as an existing folder. The parent folder path should be passed on to the mount location.

3) Is it possible to assign certain nodes to a volume?
I have some data nodes (older machines) that I want to use for long-term storage.
I would like to be able to direct older files to those nodes. What is the best way to do that?

Yes. You can absolutely do this