当前位置:网站首页>Important components of Apache Hadoop

Important components of Apache Hadoop

2020-11-09 22:19:38 chain_xx_wdm

Hadoop = HDFS( distributed file system ) + MapReduce( Distributed computing framework ) + Yarn( Resource coordination framework ) + Common modular

HDFS

Hadoop Distribute File System A highly reliable , High throughput distributed file system
" Divide and rule "

image.png

  • NameNode(nn): Storing files Metadata . Like the file name , File directory structure , File attribute ( Generation time , replications , File permissions ), And the list of blocks for each file and the DataNode etc.
  • SecondaryNameNode(2nn): auxiliary NameNode Work better , be used for monitor HDFS state Auxiliary background program for , Every once in a while obtain HDFS Metadata snapshot
  • DataNode(dn): On the local file system Storage File block data , And block data check

NN, 2NN, DN These are both role names , It's also the name of the process , It refers to the computer node name

MapReduce

MapReduce Calculation = Map Stage + Reduce Stage
Map The stage is branch The stage of , Parallel processing of input data
Reduce The stage is close The stage of , Yes Map Stage results are summarized

image.png

Yarn

 The framework of job scheduling and cluster resource management 

image.png

  • ResourceManager(rm): Handle client requests , start-up / monitor ApplicationMaster, monitor NodeManager, Resource allocation and scheduling
  • NodeManager(nm): Single node Resource management on , Processing comes from ResourceManager The order of , Processing comes from ApplicationMaster The order of
  • ApplicationMaster(am): Data segmentation , For applications Apply for resources and allocate Give internal tasks , Mission monitoring And Fault tolerance
  • Container: The abstraction of the task running environment , Encapsulates the CPU, Memory and other multi-dimensional resources and environment variables , Start command, etc Information related to task operation

ResourceManager It's the boss , NodeManager It's little brother , ApplicationMaster It's the computing task force

image.png

版权声明
本文为[chain_xx_wdm]所创,转载请带上原文链接,感谢