当前位置:网站首页>Hadoop learning (3) - Yarn

Hadoop learning (3) - Yarn

2020-11-09 13:13:42 It lost schoolboy

YARN yes hadoop Resource scheduling platform of .

1. YARN Basic architecture

YARN It consists of the following components .Resource ManagerNode ManagerApplication Master and Container form .

  1. Resource Manager RM Is a global Explorer . Responsible for resource management and task scheduling of the whole system . The core component consists of two . Scheduler Scheduler  and Application task manager ASMApplications Master

Scheduler Responsible for allocating resources to applications .ASM Responsible for starting 、 monitor AMApplication Master

  1. Node Manager:
    1. Periodic direction RM Report node resources and Container Running state .
    2. receive AM The order of . start-up 、 stop it Container.
  2. Application Master: Each task submitted by the user contains a AM.AM The main functions of the :
    1. Segmentation data .
    2. For applications to RM Application resources (container
    3. And NM signal communication , start-up 、 Stop task
    4. Monitor the running status of tasks . Provide fault tolerance mechanism .

 

  1. Container:

Container It's the abstraction of resources ( Memory and CPU).

2.  Job submission process

The user to YARN After submitting the task ,YARN Running the application in two phases . start-up AM and AM Create application Request resources for the task Monitor the task execution process until the task is completed .

 

establish AM

1)  Client to YARN Submit the application , And ask for RM Allocate resources

2) RM To start a Container, And run on it AM.

AM Create application

3) AM towards RM Medium ASM register .

4) AM towards RM Apply for and access resources .

5) AM After obtaining resources , Will correspond to the resource NM Establishing a connection .

6)  Start the task , Corresponding Container Will be in AM Registered at .

7)  Each task is directed to AM Periodic reporting on performance .

 

8) AM Run back to RM Log off and close yourself

3.  Need to improve 】 Resource scheduler

HADOOP There are three job schedulers for . First in, first out scheduler FIFO, Capacity scheduler (capacity scheduler) And fair scheduler (fair scheduler). The default is capacity scheduler .

 

(1) FIFO : Queue save job, First come, first go .

(2) Capacity Scheduler:

Support multiple queues , Each queue adopts a FIFO scheduling strategy .

(3) Fair Scheduler

版权声明
本文为[It lost schoolboy]所创,转载请带上原文链接,感谢