Hadoop Administration - Cluster Management

Basic knowledge on Slots

ü  Each node contains -> 24 CPUS = 12 Cores(Physical)

ü  Total nodes (Masters & Slaves) = 108

ü  Number of Nodes Excluding Masters (CLDB & Jt) = 100

ü  Total Cores=100 nodes *12 Cores = 1200 Cores

ü  Total number of Cores after enabling Hyper Threading = 1200*2 = 2400 Cores

ü  Each Node will take 2 cores one for OS and 2^nd one for File System,Warden and TT à 100 nodes will consume 200 cores

ü  Total Number of remaining Cores = 2400 -200 = 2200 Cores

ü  Roughly Each Core will handle 1.5 slots.

ü  2200 Cores * 1.5 = 3300 Slots ( Map Task & Reduce Tasks)

   Example 2:
   Total nodes = 13 data nodes
    Each data node = 22 Physical ( 44 CPUs, after hyper thread enabled)
    Total CPUs = 13*44 = 572 cores ( 1 CPU = 1 Core)
     Each data node will consume 2 cores one for OS and another one for Hadoop Service.
     13*2 = 26
    Remaining CPUs = 572-26=546
    Total number of slots = 546*1.5 (  Roughly Each Core will handle 1.5 slots)
    Total slots = 819 slots for Mappers and Reducers    


Disk Size on each node:

Master Node:
Memory : 94.4 GB
Disk Space =883 GB/3 disks 
CPU =32
Network I/O =34.6 KB/Sec
Disk I/O = 144KB/Sec

Slave Node:
 Memory =126 GB
Disk Space = 30.9 TB /12 disks (2.5 TB/disk)
CPU=24



Chuck Size: